MERIT: Multilingual Semantic Retrieval with Interleaved Multi-Condition Query

🎯 Pitch

Existing multimodal retrieval models show no performance drop when images in a query are replaced with captions—revealing they never actually "see" visual details. This paper introduces MERIT, a 320K-query multilingual benchmark for multi-image, multi-condition product search where models must attend to fine-grained visual attributes, and proposes Coral, a fine-tuning framework that boosts retrieval accuracy by 45.9% solely by forcing the model to reconstruct masked embeddings of the specific conditional elements it previously ignored.

1. Executive Summary

This paper introduces MERIT, the first multilingual dataset for interleaved multi-condition semantic retrieval — comprising 320,000 queries with 135,000 products across 5 languages and 7 product categories — to evaluate how existing multimodal retrieval models handle composite queries where users specify multiple attributes through interleaved text and images (e.g., a shirt with the pattern of one image and the color of another). Through extensive experiments on 9 state-of-the-art retrieval models, the authors identify a critical limitation: existing methods focus solely on global semantic information while neglecting specific conditional elements in queries, motivating the proposal of Coral (Contrastive-reconstruction for multimodal retrieval) — a fine-tuning framework that integrates masked embedding reconstruction to preserve fine-grained conditional elements with contrastive learning to extract comprehensive global semantics. Coral achieves a 45.9% performance improvement over conventional contrastive-learning-only approaches on MERIT — with Recall@1 rising from 47.76% to 69.68% — and demonstrates consistent gains across 8 established retrieval benchmarks, establishing that embedding reconstruction during retrieval-oriented fine-tuning substantially improves multi-condition understanding only when both visual and linguistic modalities are reconstructed jointly rather than individually.

2. Context and Motivation

The Core Problem: Retrieval Systems Cannot Handle Composite Visual Queries

The fundamental problem this paper addresses is that current multimodal retrieval systems cannot effectively process queries that combine multiple visual conditions. In real-world product search — the primary motivating application — users rarely express their needs through a single image or a single attribute. Instead, they compose queries like "find me a shirt with the pattern of this reference image but in the color of that other image." This is what the paper terms interleaved multi-condition semantic retrieval: a query that interleaves textual descriptions and visual references, where each condition specifies a different attribute (pattern, color, material, style, etc.) through potentially distinct images.

This is not an edge case. As the authors argue in Section 1, practical product retrieval scenarios "frequently involve interleaved multi-condition queries (e.g., specific patterns and particular texture), with many aspects requiring visual representation through images." The paper provides a concrete illustration in Figure 1, showing how real-world product search involves users specifying multiple attributes through both text and images simultaneously — a scenario that existing datasets and models are fundamentally unequipped to handle.

The gap has clear real-world consequences. E-commerce platforms with millions of products need retrieval systems that can understand nuanced, multi-faceted queries to surface the right items. When a model cannot correctly isolate and combine conditional elements — mistaking which image specifies the color versus the pattern, or neglecting one condition entirely — the retrieval fails in ways that are immediately visible to users. This is not merely an academic benchmark problem; it directly impacts search relevance, user satisfaction, and ultimately commercial outcomes.

Why Existing Work Fails: Three Interlocking Limitations

The paper identifies three specific, interlocking limitations in prior work that collectively leave multi-condition retrieval unaddressed. These are not independent problems — they compound one another to create a systematic blind spot in the field.

Limitation 1: Datasets Are Constrained to Single Conditions, Single Images, or Single Languages

The authors systematically categorize existing retrieval datasets along multiple dimensions in Table 1, revealing a consistent pattern of constraint. Prior benchmarks exhibit one or more of the following limitations:

Single-condition queries: Datasets like FashionIQ (Wu et al., 2021) and CIRR (Liu et al., 2021) focus on composed image retrieval where a reference image is modified by a single textual description (e.g., "the same dress but in red"). There is no mechanism for specifying multiple, independent conditions through separate visual references.
Single-image inputs: Benchmarks such as OVEN (Hu et al., 2023) and InfoSeek (Chen et al., 2023) involve retrieving information about a single image, never requiring the model to reason across multiple distinct visual inputs simultaneously.
Single-language coverage: Most datasets operate in English only, failing to capture the multilingual nature of real-world product search across global markets.
Vision-unnecessary design: Perhaps most damningly, the paper demonstrates in Figure 6(a) that on existing datasets like FashionIQ and CIRR, replacing images with their textual captions does not significantly degrade retrieval performance. As the authors put it, these datasets "fail to fully exploit the expressive capacity of visual information." If captions can substitute for images without performance loss, the dataset is not genuinely testing visual understanding — it is testing textual matching with images as a convenient but unnecessary proxy.

The only existing dataset that approaches multi-condition input is MIRACLE (Miah et al., CVPR'24), which supports interleaved queries. However, MIRACLE lacks multilingual coverage, multi-attribute queries, and comprehensive manual annotation — three features that the authors argue are essential for a realistic benchmark. MERIT is explicitly designed to fill this complete gap, as shown in the bottom row of Table 1: it is the only dataset that simultaneously provides semantics, multilingual data, multiple types, interleaved queries, multi-attribute queries, and manual annotations.

Limitation 2: Existing Retrieval Models Neglect Specific Conditional Elements

The paper goes beyond identifying dataset limitations to diagnose a deeper modeling problem. Through error analysis on 500 queries (Section 3.4, Figure 7b), the authors classify retrieval failures into five categories and find that attribute errors and visual understanding errors dominate, together accounting for the largest proportion of failures.

What causes these errors? The paper traces them to a shared architectural pattern in current MLLM-based retrieval models (Section 2, "Multimodal Retrieval Models"). Methods like LLaVE (Lan et al., 2025), GME-Qwen2VL (Zhang et al., 2024), VLM2Vec (Jiang et al., 2024), and others all follow the same general recipe: fine-tune a pre-trained MLLM using contrastive learning (typically InfoNCE loss) where supervision is applied exclusively at the [EOS] token embedding. This [EOS]-only supervision inherently encourages the model to produce a single global representation that captures the "gist" of the entire multimodal input.

The problem is that for multi-condition queries, a global representation is insufficient. The model must understand which image corresponds to which condition (e.g., Image A specifies the pattern, Image B specifies the color), extract the relevant attribute from each image independently, and then compose these extracted features to form a coherent retrieval target. A single [EOS] embedding that pools across all conditions loses this attribution — it may capture that the query is "about a blue striped shirt" but fail to distinguish that the stripe pattern comes from one reference and the blue color from another. This loss of fine-grained conditional structure is what the paper identifies as the root cause of attribute extraction errors and visual misinterpretation.

The authors frame this as a fundamental tension: contrastive learning with [EOS]-only supervision "prioritizes global semantic information while inadequately addressing specific conditional elements." This is not a bug in any particular implementation — it is a systematic limitation of the dominant fine-tuning paradigm for retrieval models.

Limitation 3: Fine-Tuning on Single-Image Datasets Destroys Interleaved Processing Capability

The third limitation is subtle but revealing. The paper uncovers a counterintuitive phenomenon (Section 3.3, "Interleaving Support"): for existing fine-tuned retrieval models, concatenating multiple query images into a single composite image substantially outperforms processing them as sequential interleaved inputs. For GME-Qwen2VL, concatenation achieves a 119.7% improvement in R@5 over sequential input (Table 2: 47.13% vs. 24.18% R@5).

This is surprising because pre-trained MLLMs natively support interleaved image inputs, and on visual comprehension tasks, sequential processing typically excels by preserving higher resolution and sequential information. The zero-shot results in Table 2 confirm this expectation: InternVL2.5-MPO-1B scores 0.41% R@1 with sequential input versus 0.24% with concatenation — a small but consistent advantage for sequential processing.

So why do fine-tuned models show the opposite pattern? The paper proposes a compelling hypothesis: existing retrieval datasets contain at most one image per query. When models are fine-tuned on these single-image datasets, their ability to process interleaved multi-image inputs — a capability present in the pre-trained model — degrades. The model effectively "forgets" how to maintain separate representations across sequential image tokens, instead treating the entire visual input as a monolithic block.

This hypothesis is validated by a critical experiment: after training on MERIT (which contains interleaved multi-image queries), the sequential input performance improves by 14.3% (Table 3). The fine-tuning on multi-image data partially restores the model's interleaved processing capability. This is a striking finding — it suggests that the field's exclusive focus on single-image retrieval has not only failed to advance multi-condition capabilities but has actively regressed them through fine-tuning.

How This Paper Positions Itself

The paper positions itself not as proposing an incremental improvement to existing retrieval models but as establishing a new task and providing the foundational dataset, analysis, and method to address it. The contributions are explicitly threefold (Section 1, end):

Dataset contribution: MERIT is presented as the first multilingual dataset for interleaved multi-condition semantic retrieval. The authors emphasize its uniqueness through extensive comparison with prior datasets (Table 1, Figure 2) and detailed documentation of the data collection pipeline (Section 3.2, Appendix 6), which involved 10,000 labor hours of annotation across three sampling algorithms designed to ensure attribute diversity and distributional balance.
Diagnostic contribution: The paper provides "insightful observations" about where and why existing models fail. The error analysis (Section 3.4) decomposes failures into attribute errors, visual understanding errors, category errors, detail errors, and annotation errors, demonstrating that the dominant failure modes stem from neglecting conditional query elements. The interleaving support analysis reveals that current fine-tuning practices degrade pre-trained MLLMs' native multi-image capabilities. Both findings are presented as critical insights that motivate the method design.
Method contribution: Coral is proposed as a targeted solution to the identified limitation — the neglect of fine-grained conditional elements. The framework combines embedding reconstruction (to preserve detailed per-condition information) with contrastive learning (to learn global retrieval-relevant semantics), directly addressing the tension between global and local representation that the error analysis identified.

Notably, the paper does not position Coral as a universal replacement for existing retrieval models. Rather, it is presented as a fine-tuning framework for adapting pre-trained MLLMs into retrieval models, specifically designed for the multi-condition setting. The authors validate Coral not only on MERIT (Table 3) but also on 8 established retrieval benchmarks (Figure 9, Table 10), showing consistent improvements even on single-image tasks. This generalization is important: it suggests that embedding reconstruction is beneficial beyond the specific multi-condition setting, challenging the assumption that [EOS]-only contrastive learning is sufficient for retrieval-oriented fine-tuning in general.

The paper also draws an implicit but important connection to broader discussions about visual understanding in MLLMs. By demonstrating that existing retrieval datasets exhibit "vision unnecessary" (Figure 6a) and that models fine-tuned on single-image data lose interleaved processing capability, the authors are contributing to a growing body of evidence that current multimodal training and evaluation practices may be systematically underutilizing visual information. This aligns with work by Tong et al. (2024) on visual shortcomings of MLLMs and suggests that the retrieval community faces analogous challenges to those being identified in visual question answering and multimodal reasoning.

In summary, the paper positions itself at the intersection of dataset development, diagnostic analysis, and method design, arguing that progress on multi-condition retrieval requires simultaneous advances on all three fronts. MERIT provides the task definition and evaluation framework, the error analysis provides the diagnostic understanding, and Coral provides the first targeted method for the identified failure mode.

3. Technical Approach

3.1 Reader Orientation

Coral is a fine-tuning framework that converts a pre-trained Multimodal Large Language Model (MLLM) into a multi-condition semantic retrieval model. It solves the problem identified in the error analysis — that existing retrieval models neglect specific conditional elements in queries — by adding masked embedding reconstruction losses alongside standard contrastive learning, forcing the model to preserve fine-grained visual and linguistic details that would otherwise be lost when the model is optimized exclusively for global semantic similarity.

3.2 Big-Picture Architecture (Diagram in Words)

The Coral framework has five major components, organized around a pre-trained MLLM backbone:

Pre-trained MLLM (e.g., Qwen2.5-VL-3B) — the frozen or fine-tuned base model that processes interleaved image-text queries and produces hidden state representations. This is the foundation that Coral adapts into a retrieval model.
Contrastive Learning Head — takes the [EOS] token's hidden state from the MLLM as a dense embedding vector for the entire query or candidate product. Applies InfoNCE loss to pull matching query-product pairs together and push non-matching pairs apart in embedding space. This is the standard retrieval training component.
Masked Embedding Reconstruction Decoder (Vision) — a randomly initialized BERT layer ( $\mathcal{F}_\theta^v$ ) that takes a masked version of the visual embeddings (with 50% of visual tokens randomly dropped from attention) and attempts to reconstruct the original unmasked embeddings using only the [EOS] token as the attention query. This forces the [EOS] representation to retain detailed visual information.
Masked Embedding Reconstruction Decoder (Language) — a decoder ( $\mathcal{F}_\theta^l$ ) that shares weights with the MLLM's language modeling head. It takes a masked version of the text embeddings (with 50% of text tokens randomly masked from attention) and predicts the original masked tokens, similarly using the [EOS] token as the query. This forces the [EOS] representation to retain fine-grained linguistic information.
Dual Reconstruction Targets — the reconstruction loss is applied in two complementary ways, which the paper calls $\mathcal{L}_{\text{reg}}$ and $\mathcal{L}_{\text{rec}}$ . In $\mathcal{L}_{\text{reg}}$ , the retrieval target's embeddings are reconstructed using the query conditions' [EOS] token as the query — this forces the query embedding to contain information about the target's details. In $\mathcal{L}_{\text{rec}}$ , the retrieval target's embeddings are reconstructed using the target's own [EOS] token — this is a standard autoencoding objective that ensures the target's representation is self-consistent.

Information flows as follows: an interleaved query (text + multiple images) enters the MLLM → the MLLM produces hidden states, with the [EOS] token's state serving as the query embedding → this embedding participates in contrastive learning against candidate product embeddings → simultaneously, the [EOS] token is used as a query in the reconstruction decoders to recover masked visual and linguistic features → the three losses (contrastive, visual reconstruction, language reconstruction) are summed and backpropagated through the full model.

3.3 Roadmap for the Deep Dive

First, the pre-trained MLLM architecture and notation (Section 4.1, "Pretrained MLLM"), because Coral's components operate on the intermediate representations this architecture produces.
Second, the masked embedding reconstruction mechanism (Section 4.1, "Masked Embedding Reconstruction"), since this is Coral's core innovation and both reconstruction losses depend on it.
Third, the full Coral loss function and its three components — contrastive learning loss $\mathcal{L}_{\text{cl}}$ , vision reconstruction loss $\mathcal{L}_{\text{mse}}$ , and masked language modeling loss $\mathcal{L}_{\text{mlm}}$ — along with how they are weighted.
Fourth, the dual reconstruction targets ( $\mathcal{L}_{\text{reg}}$ and $\mathcal{L}_{\text{rec}}$ ) and why reconstructing the target from the query conditions' [EOS] token is the key insight.
Fifth, the training configuration, hyperparameters, and design choices that make Coral work.
Sixth, the relationship between Coral's design and the diagnostic findings from Section 3.4 of the paper, explaining why embedding reconstruction specifically addresses attribute errors and visual understanding errors.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that adding masked embedding reconstruction losses during retrieval-oriented fine-tuning forces pre-trained MLLMs to preserve the fine-grained conditional information that contrastive learning alone discards.

Pretrained MLLM Architecture and Notation

The paper defines the pre-trained MLLM in a standard encoder-decoder-like formulation adapted for multimodal input, drawing on architectures such as LLaVA (Liu et al., 2023), Qwen2.5-VL (Bai et al., 2025), and InternVL2.5 (Chen et al., 2024). Understanding this architecture is essential because Coral's reconstruction losses operate on the intermediate representations this pipeline produces.

Input processing. An MLLM receives two types of input: image tokens $x_{\text{img}}$ and text tokens $x_{\text{txt}}$ . These are processed through separate pathways before being concatenated:

Visual encoding: The image $x_{\text{img}}$ passes through a visual representation backbone $V_\omega$ — typically a vision transformer such as CLIP (Radford et al., 2021) or SigLIP (Zhai et al., 2023). This backbone outputs a sequence of patch-level features:

$p_{\text{img}} = V_\omega(x_{\text{img}}) \in \mathbb{R}^{N_V \times d_v}$

where $N_V$ is the number of visual tokens (image patches plus any special tokens such as CLS), and $d_v$ is the backbone's hidden dimension (e.g., 1024 for a ViT-L backbone). The variable $N_V$ depends on the image resolution and the patch size — for Qwen2.5-VL-3B, the paper configures max_pixels to $576 \times 28 \times 28$ (approximately 451,584 pixels), resulting in a dynamic number of visual tokens based on the input image dimensions.

What this computes: a fixed-length sequence of dense feature vectors, one per image patch, capturing local visual patterns (edges, textures, colors, object parts). This is a frozen representation — the visual backbone is not updated during Coral fine-tuning.
Projection to language space: The visual features $p_{\text{img}}$ are mapped from the vision backbone's dimension $d_v$ to the language model's hidden dimension $d$ via a learned projector $F_\psi$ :

$e_{\text{img}} = F_\psi(p_{\text{img}}) \in \mathbb{R}^{N_V \times d}$

where $d$ is the language model's hidden state dimension (e.g., 2048 for Qwen2.5-VL-3B). The projector is typically a simple linear layer or a small MLP, and it is trained during the MLLM's original pre-training to align visual features with the language model's representational space.

What this computes: visual tokens that "look like" language tokens to the LLM backbone — they inhabit the same vector space as word embeddings, enabling the transformer to process images and text uniformly.
Text embedding: The text input $x_{\text{txt}}$ is tokenized and embedded through the language model's standard embedding layer:

$e_{\text{txt}} = \text{embed}(x_{\text{txt}}) \in \mathbb{R}^{N_L \times d}$

where $N_L$ is the number of text tokens (including any special tokens like [EOS] and image placeholder tokens).
Interleaved concatenation: For multi-image, multi-condition queries like those in MERIT, the visual and textual embeddings are interleaved according to the query structure. For instance, a query might be structured as: "Find a product with the pattern of [IMG1] and the color of [IMG2]", where [IMG1] and [IMG2] are replaced by their respective visual token sequences $e_{\text{img1}}$ and $e_{\text{img2}}$ . The full input embedding sequence is:

$E = [e_{\text{txt}}^{(1)}; e_{\text{img1}}; e_{\text{txt}}^{(2)}; e_{\text{img2}}; e_{\text{txt}}^{(3)}]$

This interleaved structure is central to the MERIT task and distinguishes it from prior work where only a single image was present per query. The ability to maintain separate representations for each image through the interleaved positions is what the paper shows degrades when models are fine-tuned on single-image datasets (Section 3.3, "Interleaving Support").

Language model processing. The concatenated embedding sequence $E$ passes through the language model $\text{LM}_\theta$ — a standard transformer decoder (e.g., Qwen2.5's architecture) with causal attention masking:

$h_{\text{gen}} = \text{LM}_\theta(e_{\text{img}}; e_{\text{txt}})$

where $h_{\text{gen}}$ is the full sequence of hidden states, one per input token. The hidden state at the position of a special end-of-sequence token [EOS] is extracted as:

$h_{\text{eos}} \in \mathbb{R}^{d}$

This single vector $h_{\text{eos}}$ is what existing retrieval models use as the dense embedding representing the entire query or product, and it is the primary object that Coral's reconstruction losses operate on.

Language generation capability. During standard MLLM pre-training, the hidden states $h_{\text{gen}}$ are passed through a language modeling head (a linear projection to vocabulary size followed by softmax) to produce token probabilities:

$u_{\text{gen}} = \text{LMHead}(h_{\text{gen}})$

The MLLM is pre-trained with the standard autoregressive language modeling objective:

$\mathcal{L}_{\text{MLLM}} = -\log p(\hat{u}_{\text{gen}} \mid x_{\text{img}}, x_{\text{txt}})$

where $\hat{u}_{\text{gen}}$ is the ground-truth next-token sequence. Coral shares weights between its language reconstruction decoder $\mathcal{F}_\theta^l$ and this language modeling head, leveraging the pre-trained capability for reconstruction.

Why this architecture matters for Coral. The key insight is that the [EOS] hidden state $h_{\text{eos}}$ is an information bottleneck. The entire visual and linguistic content of the query — potentially including multiple images with distinct conditional roles — must be compressed into a single $d$ -dimensional vector. Standard contrastive learning applies supervision only to this bottleneck, creating pressure to discard information that is not discriminative for retrieval. However, for multi-condition queries, the information about which image specifies which attribute is discriminative — but only if the model can disentangle and compose it correctly. Coral's reconstruction objective counteracts the compression pressure by requiring that $h_{\text{eos}}$ retain enough detail to reconstruct the original (masked) input embeddings.

Masked Embedding Reconstruction Mechanism

Masked embedding reconstruction is the core technical innovation in Coral, and it is what differentiates this framework from standard [EOS]-only contrastive learning approaches. The idea is drawn from masked autoencoding (He et al., 2022; Xiao et al., 2022) but applied during retrieval-oriented fine-tuning rather than pre-training.

The masking operation. The paper defines two masking functions:

$\mathcal{MASK}_v(E)$ : randomly masks the visual portions of the input multimodal embedding sequence $E = [e_{\text{img}}; e_{\text{txt}}]$ at a fixed ratio $\delta = 0.5$ . Specifically, for each visual token in $e_{\text{img}}$ , with probability $\delta$ the token is excluded from attention computations (both self-attention and cross-attention). All textual tokens in $e_{\text{txt}}$ are retained — only the visual modality is partially masked.
$\mathcal{MASK}_l(E)$ : randomly masks the linguistic portions of $E$ at the same ratio $\delta = 0.5$ . Text tokens are randomly selected and excluded from attention, while all visual tokens are retained.

The masking is implemented through an attention mask matrix $\mathbf{M}$ , which is added to the pre-softmax attention scores. For each pair of positions $(i, j)$ :

$\mathbf{M}_{ij} = \begin{cases} 0, & \text{attended (token } j \text{ is visible to token } i\text{)} \\ -\infty, & \text{masked (token } j \text{ is hidden from token } i\text{)} \end{cases}$

Adding $-\infty$ to the attention logits before softmax effectively zeroes out the attention weight for that position, preventing any information flow from the masked token to the attending token.

What this computes: two corrupted versions of the input — one where half the visual information is inaccessible, and one where half the linguistic information is inaccessible. The model must reconstruct the missing information from the surviving context, using only the [EOS] token's hidden state as the query.

The reconstruction decoder. The paper uses a single BERT layer as the reconstruction decoder $\mathcal{F}_\theta^v$ for vision and $\mathcal{F}_\theta^l$ for language. The decoder is randomly initialized (for vision) or shares weights with the MLLM's language modeling head (for language, to reduce trainable parameters). The reconstruction process works as follows:

Query preparation: The [EOS] hidden state $h_{\text{eos}}$ is extracted from the MLLM after processing the unmasked input (the standard forward pass). This $h_{\text{eos}}$ already contains information about the full, unmasked input. A query sequence is constructed by repeating $h_{\text{eos}}$ to match the length of the original embedding sequence $E$ , with position embeddings added:

$\mathbf{q} = [h_{\text{eos}} + \text{PE}_0, h_{\text{eos}} + \text{PE}_1, \ldots, h_{\text{eos}} + \text{PE}_{|E|-1}]$

where $\text{PE}_i$ is the positional encoding for position $i$ , following the approach of Dong et al. (2019) to provide sequence position information to the otherwise position-agnostic query.
Key-value preparation: The original (unmasked) input embeddings $E = [e_{\text{img}}; e_{\text{txt}}]$ are concatenated with the [EOS] hidden state to form the key-value sequence:

$\mathbf{e} = [E; h_{\text{eos}}]$

This means the decoder's attention mechanism has access to both the input token embeddings and the aggregated [EOS] representation. The [EOS] state appended at the end serves as an additional "summary" key-value pair.
Masked cross-attention: The reconstruction decoder performs standard transformer attention with the masking matrix $\mathbf{M}$ applied:

$\mathbf{Q} = \mathbf{q}\mathbf{W}^Q, \quad \mathbf{K} = \mathbf{e}\mathbf{W}^K, \quad \mathbf{V} = \mathbf{e}\mathbf{W}^V$

$\mathbf{e}_{\text{rec}} = \text{softmax}\left(\frac{\mathbf{Q}^T\mathbf{K}}{\sqrt{d}} + \mathbf{M}\right)\mathbf{V}$

where $\mathbf{W}^Q, \mathbf{W}^K, \mathbf{W}^V \in \mathbb{R}^{d \times d}$ are the attention projection matrices of the BERT decoder layer, $\sqrt{d}$ is the standard scaling factor to prevent attention logits from growing too large in high dimensions, and $\mathbf{M}$ is the mask that prevents attention to the dropped tokens.

What this computes: a reconstruction of each position's embedding, attending only to the unmasked tokens. The query at each position is the same $h_{\text{eos}}$ vector (with a different position embedding), meaning the reconstruction at every position is conditioned on the global [EOS] representation. The output $\mathbf{e}_{\text{rec}}$ has the same shape as $E$ — a sequence of reconstructed embeddings, one per original input position.
Loss computation: The reconstruction loss compares the reconstructed embeddings $\mathbf{e}_{\text{rec}}$ to the original unmasked embeddings $E$ . For visual reconstruction, this uses Mean Squared Error (MSE) in the continuous embedding space. For language reconstruction, this uses cross-entropy to predict the original token identities through the language modeling head.

Why the [EOS] token as the query matters. This design choice is the crux of Coral. By using $h_{\text{eos}}$ — the same vector that serves as the retrieval embedding — as the attention query for reconstruction, the framework creates a shared information bottleneck. The $h_{\text{eos}}$ vector must simultaneously:

Be discriminative for retrieval (enforced by contrastive loss).
Contain enough visual detail to reconstruct half of the masked visual embeddings (enforced by $\mathcal{L}_{\text{mse}}$ ).
Contain enough linguistic detail to predict half of the masked text tokens (enforced by $\mathcal{L}_{\text{mlm}}$ ).

If $h_{\text{eos}}$ collapses to a coarse global summary that only distinguishes broad product categories — which is what contrastive learning alone encourages — it will fail at the reconstruction tasks. The reconstruction losses act as a regularizer that preserves per-condition, per-attribute detail in the embedding. This directly addresses the paper's finding that existing models "neglect specific conditional elements in queries" (Section 3.4): those models were trained with contrastive loss only, which allowed $h_{\text{eos}}$ to discard the fine-grained information that Coral's reconstruction objective forces it to retain.

Why 50% masking. The masking ratio $\delta = 0.5$ is chosen as a balance point. A lower ratio (e.g., 10%) would make the reconstruction task too easy — the model could copy information from nearby unmasked tokens without relying on the [EOS] representation, defeating the purpose of forcing information through the bottleneck. A higher ratio (e.g., 90%) would make reconstruction too difficult, potentially destabilizing training or causing the reconstruction decoder to learn dataset-wide statistical priors rather than instance-specific details. 50% is a standard choice in masked autoencoding literature (He et al., 2022) that the paper adopts without extensive ablation — the ablation experiments focus on the presence versus absence of each reconstruction modality (Table 3, rows 6-11) rather than the masking ratio.

The Three Loss Components

Coral's training objective is the sum of three loss terms, each targeting a different aspect of representation quality:

$\max_{\theta, \theta_v, \theta_l} \mathcal{L} = \mathcal{L}_{\text{cl}} + \lambda_1 \mathcal{L}_{\text{reg}} + \lambda_2 \mathcal{L}_{\text{rec}}$

where $\theta$ are the MLLM parameters (LLM backbone + projector), $\theta_v$ are the vision reconstruction decoder parameters, $\theta_l$ are the language reconstruction decoder parameters (shared with the LM head), and $\lambda_1 = \lambda_2 = 0.1$ are loss weighting coefficients. The paper sets both $\lambda$ values to 0.1 to balance the contrastive and reconstruction objectives — the reconstruction losses are auxiliary, with the contrastive loss providing the primary retrieval signal.

Contrastive Learning Loss ( $\mathcal{L}_{\text{cl}}$ )

The contrastive loss follows the standard InfoNCE formulation (Wu et al., 2021), adapted for the retrieval setting where each query has one or more positive target products:

$\mathcal{L}_{\text{cl}} = -\frac{1}{N} \sum_{i=1}^{N} \log\left(\frac{\exp(q_i \cdot k_{i+} / \tau)}{\sum_{j=1}^{N} \exp(q_i \cdot k_j / \tau)}\right)$

where $N$ is the batch size (set to 64 effective, via per-device batch size 4 × gradient accumulation steps 2 × 8 GPUs), $q_i \in \mathbb{R}^d$ is the [EOS] embedding of query $i$ , $k_{i+} \in \mathbb{R}^d$ is the [EOS] embedding of the positive (correct) product for query $i$ , $k_j \in \mathbb{R}^d$ for $j \neq i$ are the embeddings of other products in the batch serving as negative examples, and $\tau = 0.02$ is the temperature parameter controlling the concentration of the softmax distribution.

What this computes: for each query in a batch, the model computes the cosine similarity (up to scaling by $\tau$ ) between the query embedding and every product embedding in the batch. It converts these similarities to probabilities via softmax, then maximizes the log-probability assigned to the correct product. The negative examples are gathered across all 8 GPUs, giving a positive-to-negative ratio of 1:63 (one positive per query, with the remaining 63 products in the batch serving as negatives).

Why this form: InfoNCE is the dominant objective in representation learning because it directly optimizes for the downstream task — given a query embedding, the correct product should be ranked higher than all other products. The temperature $\tau$ controls the hardness of the objective: lower temperatures (like 0.02) create a sharper softmax that penalizes any confusion between positives and negatives more heavily, but can also lead to training instability. The cross-device negative gathering increases the effective batch size for contrastive learning without requiring larger per-GPU batches, which is important because contrastive learning benefits from more negative examples.

The paper's ablation (Table 3, rows 1-2) shows that contrastive learning alone achieves 47.76% R@1 (full fine-tuning) or 48.52% R@1 (LoRA), demonstrating that standard retrieval training on MERIT already substantially outperforms zero-shot baselines (0.09% R@1 for Qwen2.5-VL-3B sequential). However, the 69.68% R@1 achieved by Coral (Table 3, row 11) represents a 45.9% relative improvement over contrastive-only training, validating that the reconstruction losses provide complementary benefits beyond what contrastive learning can achieve.

Vision Reconstruction Loss ( $\mathcal{L}_{\text{mse}}$ )

The vision reconstruction loss forces the [EOS] embedding to retain detailed visual information by requiring it to reconstruct visually-masked portions of the input:

$\mathcal{L}_{\text{mse}} = \frac{1}{N} \sum_{i=1}^{N} \|\hat{E} - E\|_2^2$

where $\hat{E} = \mathcal{F}_\theta^v[\mathcal{MASK}_v(E); h_{\text{eos}}]$ is the reconstructed embedding sequence output by the vision decoder $\mathcal{F}_\theta^v$ when given the visually-masked input $\mathcal{MASK}_v(E)$ and using $h_{\text{eos}}$ as the attention query, and $E$ is the original unmasked embedding sequence.

What this computes: the element-wise squared error between the reconstructed and original continuous embedding vectors, summed over all positions and averaged over the batch. For each visual token position, the decoder must predict its original embedding value from the [EOS] context and the unmasked portions of the input. Lower MSE means the [EOS] embedding contains sufficient visual information to regenerate the original token representations.

Why MSE and not cross-entropy: visual embeddings are continuous vectors in $\mathbb{R}^d$ (the language model's hidden space), not discrete tokens. Reconstructing them is a regression problem — the decoder must output vectors that are close to the originals in Euclidean distance. MSE is the standard loss for continuous reconstruction tasks. Using cross-entropy on discretized visual tokens (as in VQ-VAE-style approaches) would require vector quantization of the visual embeddings, which would add complexity and potentially lose fine-grained visual detail through the quantization bottleneck. MSE preserves the full continuous representation space.

Ablation significance. Table 3, rows 6-9 show the impact of partial reconstruction. Adding only vision reconstruction (row 8, full fine-tuning) improves R@1 from 48.52% (contrastive only, row 1) to 59.46% — a substantial 22.5% relative improvement. Adding only language reconstruction (row 9, full fine-tuning) achieves 59.98% R@1 — similar magnitude. This demonstrates that each individual reconstruction modality provides significant benefits independently, suggesting both modalities contribute non-redundant information to the [EOS] representation.

Masked Language Modeling Loss ( $\mathcal{L}_{\text{mlm}}$ )

The language reconstruction loss forces the [EOS] embedding to retain fine-grained linguistic information by requiring it to predict masked text tokens:

$\mathcal{L}_{\text{mlm}} = -\frac{1}{N} \sum_{i=1}^{N} \log P(\hat{x}_i \mid X)$

where $\hat{x}_i = [\mathcal{F}_\theta^l[\mathcal{MASK}_l(E); h_{\text{eos}}]]_{(i)}$ is the predicted token at position $i$ from the language reconstruction decoder, and $P(\hat{x}_i \mid X)$ is the probability the decoder assigns to the correct original token $X$ at that position.

What this computes: the standard masked language modeling objective from BERT (Devlin et al., 2019), applied to the retrieval target's text tokens. For each masked text position, the decoder produces a probability distribution over the vocabulary through the shared language modeling head, and the loss is the negative log-likelihood of the correct token. Only masked positions contribute to the loss.

Why sharing weights with the LM head matters. By sharing $\mathcal{F}_\theta^l$ with the MLLM's pre-trained language modeling head, Coral reduces the number of newly introduced parameters while leveraging the model's existing linguistic knowledge. The LM head has already learned meaningful token representations and output transformations during MLLM pre-training. Sharing these weights means the reconstruction decoder benefits from this pre-training and the contrastive gradients from the reconstruction loss can flow through the LM head into the MLLM backbone, potentially improving the linguistic quality of the [EOS] representation.

Ablation significance. Table 3, rows 6-7 (LoRA fine-tuning) show that adding vision reconstruction improves R@1 from 58.18% (vision only) to 58.38% (language only) — a small difference at this scale. However, rows 10-11 (full fine-tuning) show that vision reconstruction alone (59.46%) and language reconstruction alone (59.98%) are both substantially better than contrastive-only (47.76%), and the combined reconstruction (row 11: 69.68%) is dramatically better than either modality alone. This suggests a synergistic effect: preserving both visual and linguistic details enables the [EOS] embedding to capture cross-modal correspondences that neither modality alone can represent.

Dual Reconstruction Targets: $\mathcal{L}_{\text{reg}}$ and $\mathcal{L}_{\text{rec}}$

A subtle but critical aspect of Coral is that the reconstruction loss is decomposed into two complementary terms, both of which involve reconstructing the retrieval target's embeddings, but using different attention queries:

$\mathcal{L}_{\text{reg}}$ ("regularization"): Reconstructs the retrieval target's embeddings using the query conditions' [EOS] token as the attention query. In other words, given only the query conditions (e.g., "pattern like Image A, color like Image B"), the model must reconstruct what the target product looks like — both its visual features and textual description.
$\mathcal{L}_{\text{rec}}$ ("reconstruction"): Reconstructs the retrieval target's embeddings using the target's own [EOS] token as the attention query. This is a standard autoencoding objective — given the target itself, reconstruct its own masked features.

Operationally, for each training pair (query $q$ , positive target $t$ ):

Compute the query conditions' [EOS] embedding $h_{\text{eos}}^q$ and the target's [EOS] embedding $h_{\text{eos}}^t$ .
For $\mathcal{L}_{\text{reg}}$ : take the target's embedding sequence $E^t$ , apply masking ( $\mathcal{MASK}_v$ and $\mathcal{MASK}_l$ ), and reconstruct using $h_{\text{eos}}^q$ as the query in the attention mechanism of the decoder.
For $\mathcal{L}_{\text{rec}}$ : take the target's embedding sequence $E^t$ , apply masking (identical procedure), and reconstruct using $h_{\text{eos}}^t$ as the query.
Both use the same decoders ( $\mathcal{F}_\theta^v$ and $\mathcal{F}_\theta^l$ ) and the same loss computations (MSE for vision, cross-entropy for language). The only difference is which [EOS] token serves as the attention query.

Why $\mathcal{L}_{\text{reg}}$ is the key innovation. This term directly implements the idea that the query embedding should contain enough information about the target to recreate it. If the query specifies multiple conditions ("pattern of Image A, color of Image B"), the [EOS]^q embedding must capture both conditions and their composition to successfully reconstruct the target's visual and linguistic features. This is precisely the capability that the paper's error analysis found lacking in existing models — the ability to extract specific attributes from each condition image and compose them into a coherent target representation.

The $\mathcal{L}_{\text{rec}}$ term serves a complementary purpose: it ensures the target's own embedding is self-consistent and contains sufficient detail, which provides a stable reconstruction target for $\mathcal{L}_{\text{reg}}$ to work against. Without $\mathcal{L}_{\text{rec}}$ , the target embeddings might degrade during training as the model optimizes only for discriminability (contrastive loss) and query-target alignment ( $\mathcal{L}_{\text{reg}}$ ), potentially losing self-consistency. $\mathcal{L}_{\text{rec}}$ acts as an autoencoding regularizer on the target representations.

Connection to the diagnostic findings. The paper's error analysis (Section 3.4, Figure 7b) identified two dominant failure categories: attribute errors (the model correctly identifies which attributes are requested but selects products with wrong attribute values) and visual understanding errors (the model correctly understands the language instruction but retrieves visually incorrect products). $\mathcal{L}_{\text{reg}}$ addresses these by:

For attribute errors: The reconstruction task forces $h_{\text{eos}}^q$ to encode which specific attribute values correspond to which condition images. If the query says "color like Image B," the model cannot reconstruct Image B's color from $h_{\text{eos}}^q$ unless that color information is faithfully represented in the query embedding. This creates gradient pressure to maintain accurate attribute-to-image bindings.
For visual understanding errors: The visual reconstruction component of $\mathcal{L}_{\text{reg}}$ (via $\mathcal{L}_{\text{mse}}$ ) forces $h_{\text{eos}}^q$ to encode visual details sufficient to regenerate the target's image features. If the query says "pattern like Image A," the model must extract the pattern from Image A and represent it in $h_{\text{eos}}^q$ such that the decoder can apply it to reconstruct the target. This creates gradient pressure to faithfully extract the visual content of condition images.

In both cases, the gradient flows from the reconstruction loss through the [EOS] bottleneck back into the MLLM's processing of the interleaved input, teaching the model to maintain fine-grained per-condition information that would otherwise be discarded by contrastive-only optimization.

Full Training Configuration and Hyperparameters

The paper provides detailed hyperparameters in Section 10.3 ("Main Experiments Settings"):

Hardware and scale: All experiments run on $8 \times$ H100 GPUs, with training for exactly one epoch. The single-epoch training is notable — it suggests that the reconstruction objectives provide strong learning signals that converge quickly, or that longer training leads to overfitting on the 310,000 training queries.

Batch composition: Per-device batch size of $4$ , gradient accumulation steps of $2$ , across 8 GPUs, yielding an effective global batch size of $4 \times 2 \times 8 = 64$ . In-batch negatives combined with cross-device negative sample gathering achieve a positive-to-negative ratio of $1:63$ — each query has one positive target and 63 negatives from the other queries in the global batch.

Optimizer and scheduling:

Full fine-tuning: Learning rate $1 \times 10^{-5}$ , weight decay $0.0005$ , linear warmup ratio $0.01$ (meaning the learning rate linearly increases from 0 to $10^{-5}$ over the first 1% of training steps). The paper uses AdamW optimizer (implied by the weight decay parameter).
LoRA fine-tuning (Low-Rank Adaptation, Hu et al., 2022): Learning rate $1 \times 10^{-4}$ (10× higher than full fine-tuning, which is standard for LoRA since only a small subset of parameters is updated), same weight decay and warmup ratio. LoRA-specific hyperparameters: rank $r = 8$ , scaling factor $\alpha = 16$ , no bias terms, dropout $0.05$ between LoRA layers.
Frozen vision tower: Across all training regimes, the visual backbone $V_\omega$ is kept completely frozen to preserve its pre-trained representations. Only the projector $F_\psi$ , the LLM backbone, and the reconstruction decoders are updated.
LoRA application: When using LoRA, low-rank adaptation is applied only to the LLM backbone components, while the BERT decoder layers ( $\mathcal{F}_\theta^v$ and $\mathcal{F}_\theta^l$ ) receive standard full-parameter training. This hybrid approach preserves the decoder's complete expressive capacity while efficiently adapting the language model.

Loss weighting and masking:

$\lambda_1 = \lambda_2 = 0.1$ : Both reconstruction loss terms are weighted at 0.1 relative to the contrastive loss weight of 1.0. This ensures the contrastive objective remains the primary training signal while the reconstruction losses provide auxiliary regularization.
Masking probability $\delta = 0.5$ for both visual and linguistic modalities, applied symmetrically.

Why LoRA vs. full fine-tuning matters. Table 3 compares both regimes. The baseline contrastive-only performance is similar: 48.52% R@1 (LoRA) vs. 47.76% (full) — LoRA actually performs slightly better here, possibly due to reduced overfitting. However, the final Coral performance shows a clear gap: 59.40% R@1 (LoRA, row 10) vs. 69.68% R@1 (full, row 11). The paper notes this is "due to the substantial divergence between retrieval tasks and pre-training objectives" — adapting an MLLM to produce good retrieval embeddings requires changing the model's internal representations enough that parameter-efficient methods cannot fully capture the necessary transformation.

Sequential vs. concatenated input. Table 3 also addresses the input format question. Rows 3-5 compare zero-shot sequential (0.09% R@1), zero-shot concatenated (0.05%), and full Coral with concatenated input (60.94% R@1). However, full Coral with sequential input (row 11) achieves 69.68% R@1 — a significant advantage. The paper concludes that "sequential representation preserves more information than image concatenation," which aligns with the finding in Section 3.3 that existing models lose interleaved processing capability when fine-tuned on single-image datasets.

Why Coral Addresses the Diagnostic Findings

The paper's error analysis (Section 3.4) identified that existing retrieval models exhibit two primary failure modes: attribute errors (failing to correctly extract specific attribute values from condition images) and visual understanding errors (correctly understanding language instructions but failing to align them with appropriate visual outputs). Section 3.4 attributes these failures to the "[EOS]-only supervision" paradigm that "prioritizes global semantic information while inadequately addressing specific conditional elements."

Coral addresses this diagnosis through two mechanisms:

Mechanism 1: Gradient pressure to preserve conditional elements. The reconstruction losses create a direct gradient path from the [EOS] embedding to the reconstruction of individual visual tokens and text tokens. If a specific attribute (e.g., the color blue from Image B) is not faithfully represented in $h_{\text{eos}}^q$ , the decoder cannot reconstruct that blue color in the target's visual features, resulting in a high $\mathcal{L}_{\text{mse}}$ loss. This gradient signal backpropagates through the MLLM, teaching the attention mechanisms to route per-condition visual information to positions that influence $h_{\text{eos}}^q$ .

Without reconstruction (contrastive-only), the gradient from a misranked product only signals that "the global embedding is wrong" — it provides no per-condition, per-attribute feedback about which part of the query was mishandled. The reconstruction loss decomposes the error into per-token signals, enabling the model to learn fine-grained attribution.

Mechanism 2: Cross-modal binding through joint reconstruction. The combined visual+linguistic reconstruction (Table 3, row 11 achieving 69.68% vs. ~60% for single-modality reconstruction) forces the [EOS] embedding to represent relationships between modalities, not just independent visual and linguistic summaries. To reconstruct a text token describing the target's color, the decoder must extract color information from the query's visual conditions; to reconstruct a visual feature of the target's pattern, the decoder must align it with the pattern specified by the query's language instruction. This cross-modal binding is what enables the model to correctly resolve "which image provides which condition" — a challenge that contrastive-only models fail at, as demonstrated by the interleaving analysis in Section 3.3.

Empirical validation. The ablation in Table 3 provides causal evidence for these mechanisms. Comparing row 1 (contrastive-only: 47.76% R@1) to row 8 (contrastive + vision reconstruction: 59.46%) isolates the effect of visual detail preservation — a 24.5% relative improvement. Comparing row 1 to row 9 (contrastive + language reconstruction: 59.98%) isolates the linguistic detail preservation effect — a 25.6% relative improvement. Comparing row 8 or row 9 to row 11 (full Coral: 69.68%) isolates the cross-modal binding effect — the combined reconstruction provides an additional 16.2% relative improvement over the best single-modality reconstruction. This additive pattern confirms that visual detail, linguistic detail, and cross-modal binding are all non-redundant contributors to retrieval performance.

The generalization to 8 established benchmarks (Figure 9, Table 10) further suggests that even for single-image or single-condition retrieval tasks, the reconstruction objectives prevent the [EOS] representation from collapsing to overly coarse summaries. On VisDial, Coral achieves a 181% improvement over the contrastive-only baseline (51% → 73% R@1), demonstrating that preserving fine-grained visual-linguistic detail benefits retrieval even when multi-condition composition is not required.

4. Key Insights and Innovations

Innovation 1: Diagnosing a Systematic Failure Mode — Conditions Get Lost in the `[EOS]` Bottleneck

The paper's most fundamental conceptual contribution is not the Coral method itself, but the diagnostic framework that identifies why existing multimodal retrieval models fail on multi-condition queries. This is a genuine diagnostic insight rather than an incremental performance observation, and it reframes the problem in a way that suggests a specific class of solutions.

What the field assumed before this work. The dominant paradigm for adapting MLLMs into retrieval models — represented by LLaVE (Lan et al., 2025), GME (Zhang et al., 2024), VLM2Vec (Jiang et al., 2024), and others — treats retrieval-oriented fine-tuning as a straightforward application of contrastive learning. The implicit assumption is that a pre-trained MLLM already encodes all relevant information about an input; contrastive learning merely rearranges the embedding space to make matching query-product pairs close and non-matching pairs far apart. Under this assumption, applying supervision at the [EOS] token is sufficient because that token already "contains" everything the model knows about the input. The field's research focus has been on improving contrastive objectives (hard negative mining, better temperature scheduling, larger batches) rather than questioning whether the [EOS] representation itself is adequate.

What this paper reveals. The error analysis in Section 3.4 (Figure 7b) demonstrates that this assumption breaks down for multi-condition queries. The two largest error categories — attribute errors and visual understanding errors — cannot be explained by poor embedding space geometry. They reflect a deeper representational problem: the [EOS] token never encoded the per-condition attribution in the first place. The model might understand that a query involves "stripes" and "blue," but cannot correctly bind "stripes" to Image A and "blue" to Image B because the [EOS] embedding pools across all conditions into an undifferentiated global representation.

This is not a failure of contrastive learning per se — it is a failure of the information bottleneck architecture. Contrastive learning with [EOS]-only supervision actively encourages the model to discard per-condition detail because that detail is not discriminative at the batch level. If every query in a batch involves different combinations of colors, patterns, and materials, the easiest way to separate them is to capture their unique combination in a global summary, not to maintain disentangled per-condition representations. The contrastive objective rewards discriminability, not fidelity. The [EOS] embedding becomes a discriminative hash — good at telling queries apart, bad at preserving the internal structure of what each query means.

Why this is a reframing, not just an observation. Prior work on multimodal retrieval has acknowledged that models make errors (obviously), but it has not isolated information loss during contrastive fine-tuning as the root cause. The paper's insight is that the [EOS] token is an information bottleneck in the information-theoretic sense (Tishby et al., 1999): it must compress the entire multimodal input into a fixed-size vector, and contrastive learning provides pressure to keep only the information that distinguishes one query from another in the training distribution. Per-condition detail — which image goes with which attribute — is often redundant for discrimination (two queries might both involve "red" and "stripes" but differ in which image provides which), so it gets compressed away. This framing connects a practical retrieval failure to a fundamental principle of representation learning, elevating the diagnosis from "models make mistakes" to "the training objective is misaligned with the task requirements."

Evidence anchoring. The interleaving analysis in Section 3.3 provides the cleanest causal evidence for this diagnosis. Pre-trained MLLMs natively support interleaved image inputs and, in zero-shot evaluation, sequential processing slightly outperforms concatenation (InternVL2.5-MPO-1B: 0.41% R@1 sequential vs. 0.24% concatenated; Table 2). But after contrastive fine-tuning on single-image datasets, this relationship reverses: concatenation substantially outperforms sequential input (GME-Qwen2VL: 47.13% R@5 concatenated vs. 24.18% sequential — a 119.7% gap). The model has not merely failed to learn multi-image processing; it has lost a capability it previously had. This is the signature of an objective that punishes the maintenance of per-image representational separation — contrastive learning on single-image data teaches the model that all visual information can be collapsed into a single monolithic representation, which is exactly the wrong inductive bias for multi-condition queries.

The 14.3% improvement in sequential performance after training on MERIT (Table 3, comparing zero-shot sequential to Coral sequential) confirms that the capability can be partially restored when the training data requires it. This is a powerful demonstration that the information bottleneck problem is training-induced, not an inherent architectural limitation of MLLMs.

Innovation 2: Reconstruction as a Regularizer for Information Preservation — Not Just a Pre-Training Trick

Coral's use of masked embedding reconstruction is, at first glance, technically simple — add reconstruction losses alongside contrastive learning. But the conceptual contribution is more significant than the mechanism: it demonstrates that reconstruction objectives can serve as information-preserving regularizers during task-specific fine-tuning, counteracting the compression pressure of discriminative objectives. This generalizes the role of reconstruction beyond its traditional use in pre-training.

What reconstruction has meant in prior work. Masked autoencoding (He et al., 2022; Xiao et al., 2022) is well-established as a pre-training technique. Models like BERT (Devlin et al., 2019) and MAE (He et al., 2022) use reconstruction to learn general-purpose representations from unlabeled data, which are then fine-tuned on downstream tasks. During fine-tuning, the reconstruction objective is typically discarded — the model switches to a task-specific loss (classification, contrastive learning, etc.), and the pre-trained representations adapt to the task. This two-phase "pre-train then fine-tune" paradigm treats reconstruction as a means to an end (good initialization), not as an ongoing regularizer.

RetroMAE (Xiao et al., 2022) applied masked autoencoding specifically to retrieval-oriented language model pre-training, but again only during the pre-training phase. The innovation was in the pre-training design (asymmetric encoder-decoder, aggressive masking), not in using reconstruction during fine-tuning. The implicit assumption in all this work is that once task-specific fine-tuning begins, the pre-trained representations are "good enough" and only need discriminative refinement.

What Coral changes about this picture. Coral reintroduces reconstruction during fine-tuning, alongside the contrastive objective. This is a conceptual departure: reconstruction is no longer just a pre-training scaffold to be discarded, but an active regularizer that prevents the fine-tuning process from destroying representational richness. The paper demonstrates that contrastive learning alone causes the [EOS] representation to lose information that was present in the pre-trained model (the interleaving capability loss in Section 3.3 is one manifestation). Reconstruction acts as a "memory" mechanism — it creates gradient pressure to retain details that are not directly rewarded by the contrastive objective.

This reframes the relationship between generative and discriminative training. The dominant view in representation learning has been that these are alternative pre-training paradigms — you either do masked language modeling (generative) or contrastive learning (discriminative), and the choice determines what kind of representations you get. Coral suggests that they are complementary forces that should operate simultaneously during fine-tuning: the discriminative objective provides the task signal (what makes queries and products match), while the generative objective prevents representational collapse (preserving the rich structure that the discriminative signal doesn't explicitly reward).

Why this is more than just multi-task learning. One could view Coral as simply adding auxiliary losses — a standard multi-task learning approach. But the paper's ablation (Table 3) shows that the reconstruction losses are not just "extra supervision." Vision reconstruction alone (59.46% R@1) and language reconstruction alone (59.98% R@1) each provide substantial gains over contrastive-only (47.76%). But the combination (69.68% R@1) provides gains beyond what either modality achieves independently. This super-additive effect suggests that joint visual-linguistic reconstruction is not merely providing two independent regularization signals — it is enabling the model to learn cross-modal correspondences that neither modality alone can capture. The reconstruction decoders share the same [EOS] query vector, meaning gradients from both modalities flow through the same bottleneck, forcing the model to find representations that simultaneously satisfy visual fidelity, linguistic fidelity, and discriminative separation.

Evidence of broader applicability. Coral's consistent improvements across 8 established retrieval benchmarks (Figure 9, Table 10) — including single-image tasks like CIRR (50% vs. 39% for CL baseline) and VisDial (73% vs. 51% for CL baseline) — demonstrate that information loss during contrastive fine-tuning is not unique to MERIT's multi-condition setting. Even standard retrieval tasks benefit from reconstruction-based regularization. This suggests a broader principle: any time a discriminative objective is applied to a bottleneck representation, there is pressure to discard task-irrelevant information, and reconstruction can serve as a generic regularizer to counteract this pressure. This principle extends beyond retrieval to any fine-tuning scenario where representational richness matters — few-shot learning, continual learning, domain adaptation — making Coral's insight relevant to the broader representation learning community.

Innovation 3: Fine-Tuning on Single-Image Data Destroys Native Multi-Image Capability — A Cautionary Finding with Implications Beyond Retrieval

The paper's finding that contrastive fine-tuning on single-image retrieval datasets degrades pre-trained MLLMs' ability to process interleaved multi-image inputs (Section 3.3, "Interleaving Support") is not just a retrieval-specific observation — it is a cautionary result about how task-specific fine-tuning can destroy general capabilities. This is a negative finding with significant implications for the broader MLLM community.

What the field assumed. Pre-trained MLLMs like Qwen2.5-VL (Bai et al., 2025), InternVL2.5 (Chen et al., 2024), and LLaVA-NeXT-Interleave (Li et al., 2024) are explicitly designed and trained to handle interleaved image-text inputs. This capability is considered a core feature of the architecture — the model processes images and text in sequence, maintaining separate representations through position-aware attention. The reasonable assumption is that fine-tuning on a downstream task would preserve this capability, perhaps even enhance it for task-relevant multi-image scenarios. If the pre-trained model can process five images interleaved with text, fine-tuning it on a single-image task should not destroy this — it should at worst leave it unchanged.

What the paper demonstrates. The empirical evidence contradicts this assumption. Table 2 shows a stark reversal: after contrastive fine-tuning on existing retrieval datasets (which contain at most one image per query), models process multiple images better when they are concatenated into a single composite image than when they are provided as sequential interleaved inputs. GME-Qwen2VL-2B achieves 47.13% R@5 with concatenation versus only 24.18% with sequential input — the sequential capability has degraded so severely that it is outperformed by the information-lossy concatenation approach by nearly 2×. LamRA-Qwen2.5VL-7B shows the same pattern: 39.13% R@5 concatenated versus 13.10% sequential.

This degradation is training-induced, not inherent to the architecture. The zero-shot results in Table 2 show that pre-trained MLLMs actually perform better with sequential input: InternVL2.5-MPO-1B achieves 1.37% R@5 sequential versus 1.04% concatenated. The sequential advantage is small (as expected for zero-shot retrieval, where performance is near-random) but directionally consistent. After fine-tuning, the relationship flips. And critically, after fine-tuning on MERIT (which contains interleaved multi-image queries), sequential performance improves by 14.3% (Table 3, comparing zero-shot to Coral sequential) — the capability is partially restored.

Why this is a finding with broad implications. This phenomenon is an instance of catastrophic forgetting in representation space — not forgetting of specific facts or classes, but forgetting of a general processing capability (how to maintain separate representations across interleaved inputs). This is more subtle than standard catastrophic forgetting because the model can still process multiple images (concatenation works), suggesting that the visual encoding capability remains intact but the spatial-positional organization that distinguishes "Image A" from "Image B" has been overwritten.

The mechanism is likely: during single-image fine-tuning, the model never encounters scenarios where it must distinguish information from multiple distinct image positions. The contrastive objective rewards extracting all visual information into a single [EOS] vector. Over many gradient steps, the attention patterns that maintain interleaved separation — which are present in the pre-trained model because they were useful for the pre-training objective — are gradually overwritten by patterns that treat all visual tokens as a single undifferentiated pool. This is efficient for single-image retrieval but catastrophic for multi-image tasks.

The implication for the broader MLLM community is significant: fine-tuning on narrow tasks can silently destroy broad capabilities, and this destruction may not be detectable on standard benchmarks. If the field only evaluates fine-tuned models on tasks that resemble their fine-tuning distribution (as is standard practice), this capability degradation remains invisible. It only appears when models are tested on out-of-distribution scenarios that require the pre-trained capabilities — exactly the kind of test that MERIT provides. This finding adds urgency to the growing call for more diverse, capability-probing evaluation suites that can detect representational degradation.

Connection to the "vision unnecessary" problem. The paper's finding that existing retrieval datasets exhibit "vision unnecessary" (Figure 6a) — where replacing images with captions does not degrade performance — compounds this concern. If models are fine-tuned on datasets where visual information is redundant with text, they learn to route around vision rather than integrate it. The interleaving degradation may be a specific instance of a more general problem: fine-tuning objectives that do not require a capability will gradually eliminate the neural circuitry supporting that capability, even if the capability was present at initialization.

Innovation 4: $\mathcal{L}_{\text{reg}}$ as a Cross-Modal Binding Objective — Teaching Models "Which Image Gives Which Attribute"

The final conceptual innovation is the specific design of $\mathcal{L}_{\text{reg}}$ — the reconstruction loss that uses the query conditions' [EOS] token to reconstruct the target product's embeddings. While the mechanism is described in Section 3, the conceptual contribution is the recognition that multi-condition retrieval requires cross-modal binding — the ability to associate specific images with specific attribute roles — and that this binding can be learned through a reconstruction objective that forces the query embedding to contain sufficient information to regenerate the target.

What the field did before. Prior work on composed image retrieval (CIRR, FashionIQ) used a single reference image plus a text modifier. The binding problem was trivial: there is exactly one image and one text string, so "the image provides this attribute" and "the text modifies that attribute" are unambiguous. Models learn to compose the single image and single text into a target representation without ever needing to resolve which image provides which attribute. The architecture can simply fuse all visual and textual information into a joint representation; attribution is implicit in the input structure.

For multi-condition queries with multiple images, this implicit approach breaks down. A query might be: "Find a product with the pattern of [Image A] and the color of [Image B]." The model must bind "pattern" to Image A and "color" to Image B. If the model instead binds "pattern" to Image B and "color" to Image A, the retrieval will be wrong even though all the same visual information is present. The binding problem is compositional: the model must correctly assign N conditions to N images out of N! possible assignments.

Why $\mathcal{L}_{\text{reg}}$ solves a different problem than contrastive learning. Standard contrastive learning cannot teach correct binding for a subtle reason: the negative examples in the batch may have different attributes overall, so a wrong-binding embedding (pattern from B, color from A) is still distinct from all negatives. The contrastive loss only signals that the embedding is "wrong" if it is closer to a negative than to the positive — but wrong-binding embeddings are often far from all negatives because they still contain unique attribute combinations. The gradient from contrastive learning is too coarse to isolate the binding error.

$\mathcal{L}_{\text{reg}}$ provides a much finer signal. To reconstruct the target's visual features (via $\mathcal{L}_{\text{mse}}$ ), the decoder must apply the correct pattern to the correct spatial regions and the correct color to the correct material. If the [EOS]^q embedding has swapped which image provides which attribute, the reconstruction will be wrong at a per-token, per-region level. The MSE loss decomposes this error into contributions from individual visual tokens, creating gradient pressure that specifically penalizes incorrect binding. Similarly, to reconstruct the target's text description (via $\mathcal{L}_{\text{mlm}}$ ), the decoder must predict attribute-specific words ("striped," "blue") from the query embedding, and wrong bindings lead to wrong word predictions at specific positions.

This is a novel use of reconstruction for compositional supervision. Reconstruction objectives have been used for representation learning since autoencoders, but typically in a self-supervised context — reconstruct the input from a compressed representation. $\mathcal{L}_{\text{reg}}$ is different: it reconstructs the target from the query. This is cross-instance reconstruction, not self-reconstruction. The query embedding is forced to contain information about an instance it has never seen (the target product), based only on the conditions specified in the query. This is a much stronger requirement than standard autoencoding, and it creates a learning signal that is specifically tuned to the retrieval task's compositional structure.

Why the dual reconstruction ( $\mathcal{L}_{\text{reg}}$ + $\mathcal{L}_{\text{rec}}$ ) is more than the sum of its parts. One might ask: why not just use $\mathcal{L}_{\text{reg}}$ alone? The answer lies in training dynamics. $\mathcal{L}_{\text{reg}}$ alone would create a moving target problem — as the model's representations change during training, the target product's embeddings also change (since the MLLM backbone is being fine-tuned), making the reconstruction task non-stationary. $\mathcal{L}_{\text{rec}}$ stabilizes this by ensuring the target's own representations remain self-consistent and information-rich, providing a stable reconstruction target for $\mathcal{L}_{\text{reg}}$ to work against. The two losses together create a co-learning dynamic: $\mathcal{L}_{\text{rec}}$ maintains target-side representational quality, while $\mathcal{L}_{\text{reg}}$ learns query-side compositional binding. This dynamic would not emerge from either loss alone, and it represents a design principle — using self-reconstruction to stabilize cross-instance reconstruction — that may generalize to other compositional learning problems.

Evidence anchoring. The ablation in Table 3 shows that the full Coral framework (69.68% R@1) substantially outperforms both single-modality reconstruction (~60%) and contrastive-only (47.76%). While the ablation does not isolate $\mathcal{L}_{\text{reg}}$ from $\mathcal{L}_{\text{rec}}$ (both are always present together in the "Full Version" row), the strong super-additive effect of combined vision+language reconstruction supports the claim that the reconstruction objectives are doing more than just preserving modality-specific detail — they are enabling compositional binding that neither modality alone can achieve. The 14.3% improvement in sequential performance after MERIT training further supports this: the model is not just learning to process multiple images, but learning to process them in a way that preserves per-image, per-condition attribution — exactly what $\mathcal{L}_{\text{reg}}$ is designed to teach.

5. Experimental Analysis

Evaluation Methodology

Dataset. All primary experiments use the MERIT benchmark introduced in Section 3, comprising 320,000 retrieval pairs (310,000 training, 10,000 test) across 135,000 products in 5 languages (English, Malay, Indonesian, Vietnamese, Thai) covering 7 product categories. The training set is used for fine-tuning; the test set for evaluation. For generalization validation, 8 additional established retrieval benchmarks are used: VisDial (visual dialogue retrieval), CIRR (composed image retrieval), VisualNews T2I and I2T (text-to-image and image-to-text news retrieval), MSCOCO T2I and I2T, NIGHTS (perceptual image similarity), and WebQA (multimodal question answering). Details on these external datasets are provided in Appendix 10.2.
Base model(s). The primary model used for Coral development and ablation is Qwen2.5-VL-3B (Bai et al., 2025), a vision-language model with a dynamic-resolution Vision Transformer backbone and a 3-billion-parameter language model. For generalization experiments (Figure 9, Table 10), the baseline is Qwen2-VL-2B (Wang et al., 2024). The paper also evaluates 9 state-of-the-art retrieval models in zero-shot and fine-tuned configurations on MERIT (Table 2), spanning architectures from InternVL2.5 (1B and 1B-MPO), Qwen2.5-VL (3B), E5-V (8B), LLaVE (0.5B, 2B, 7B), GME-Qwen2VL (2B), LamRA-Qwen2.5VL (7B), BGE-VL (7B), and VLM2Vec (4B). This range of model families and scales tests whether the identified limitations are specific to particular architectures or represent a systematic pattern.
Metrics. The primary evaluation metrics are standard information retrieval measures: Recall@1 (R@1), Recall@5 (R@5), Recall@10 (R@10), and Mean Reciprocal Rank (MRR). Recall@K measures the fraction of queries for which the correct product appears among the top-K retrieved candidates. MRR is the average of the reciprocal of the rank of the first correct answer across all queries. All metrics are computed on the 10,000-query test set. For the 8 generalization benchmarks, the paper reports R@1 following the protocols established by Jiang et al. (2024) in VLM2Vec.
Baselines. The paper compares against two categories of models (detailed in Appendix 10.1):
- Zero-Shot MLLMs: InternVL2.5-1B (Chen et al., 2024), InternVL2.5-MPO-1B (Wang et al., 2024, trained with mixed preference optimization), and Qwen2.5-VL-3B (Bai et al., 2025). These models are evaluated without any retrieval-specific fine-tuning, using their [EOS] token hidden state as the retrieval embedding.
- Embedding MLLMs: E5-V-8B (Jiang et al., 2024, using a single-modality text-only training approach), LLaVE at 0.5B/2B/7B scales (Lan et al., 2025, with hardness-weighted contrastive learning), GME-Qwen2VL-2B (Zhang et al., 2024, with modality-balanced training data synthesis), LamRA-Qwen2.5VL-7B (Liu et al., 2024, with two-stage language-then-multimodal training), BGE-VL-7B (Zhou et al., 2024, trained on the MegaPairs dataset), and VLM2Vec-4B (Jiang et al., 2024, with unified multimodal embedding training). These models have been specifically fine-tuned on retrieval datasets.
For each baseline, the paper evaluates two input formats where applicable: sequential (images interleaved with text at their natural positions) and concatenated (all query images combined into a single composite image). E5-V uses averaged embeddings across images and is reported separately. The "CL" baseline in Table 3 (row 1) is the paper's own implementation of contrastive-only fine-tuning on MERIT using Qwen2.5-VL-3B, serving as the direct comparison point for Coral.
Generation budget / compute accounting. All experiments use a fixed computational budget measured in standard deep learning terms: training for exactly one epoch on the 310,000-query training set, with an effective global batch size of 64 (per-device batch size 4 × gradient accumulation steps 2 × 8 H100 GPUs). The InfoNCE temperature is τ = 0.02, with in-batch negatives combined with cross-device negative gathering achieving a 1:63 positive-to-negative ratio. For full fine-tuning, the learning rate is 1 × 10⁻⁵; for LoRA fine-tuning, it is 1 × 10⁻⁴. Weight decay is 0.0005 with a linear warmup ratio of 0.01. The Coral loss weights are λ₁ = λ₂ = 0.1 for the reconstruction terms. Masking probability is 0.5 for both visual and linguistic modalities. The vision tower is kept frozen across all configurations. All experiments are conducted on 8× H100 GPUs.
Cross-validation / statistical protocol. The paper does not employ k-fold cross-validation for hyperparameter selection or statistical significance testing on the MERIT benchmark. The train/test split (310,000/10,000 queries) is used as a single fixed evaluation. For the error analysis (Section 3.4), 500 queries are randomly sampled and expert annotators classify failure modes into five categories (Attribute Error, Visual Understanding Error, Category Error, Detail Error, Annotation Error), with definitions provided in Appendix 10.5. The human preference study (Appendix 10.6) uses 100 randomly sampled items from MERIT evaluated by 5 annotators in a multiple-choice comparison format. For the out-of-distribution analysis (Section 3.3, Figure 6b), separate train/test splits are constructed by excluding specific languages, attributes, or product classes from training while testing on the held-out category.

Main Quantitative Results

Existing Models on MERIT: The Multi-Condition Gap (Table 2)

The headline finding from the benchmark evaluation is that even the best existing retrieval model achieves only 12.05% R@1 on MERIT (LamRA-Qwen2.5VL-7B with concatenated input), with most models performing dramatically worse. The complete results are presented in Table 2.

Zero-shot MLLMs show near-random performance. Without any retrieval-specific fine-tuning, pre-trained MLLMs perform at near-chance levels. The best zero-shot result is InternVL2.5-MPO-1B with sequential input achieving 0.41% R@1, 1.37% R@5, and 2.28% R@10. InternVL2.5-1B achieves 0.27% R@1 (sequential) and 0.20% R@1 (concatenated). Qwen2.5-VL-3B performs worst in zero-shot: 0.09% R@1 (sequential) and 0.05% R@1 (concatenated). These results establish that MERIT is far beyond what pre-trained MLLMs can handle without task-specific adaptation — the task requires capabilities that pre-training alone does not provide.

Embedding MLLMs show substantial but still inadequate performance. Fine-tuned retrieval models improve dramatically over zero-shot baselines but remain far below acceptable performance for practical deployment:

Concatenated input (best configuration for existing models): LamRA-Qwen2.5VL-7B achieves the highest R@1 at 12.05%, with 39.13% R@5, 48.03% R@10, and 23.80% MRR. BGE-VL-7B achieves 11.55% R@1 with 38.01% R@5. GME-Qwen2VL-2B achieves the best R@5 (47.13%) and R@10 (56.18%) despite lower R@1 (8.47%), suggesting its embeddings rank relevant items higher but struggle to put the correct answer at position 1. LLaVE-7B achieves 8.03% R@1 with 45.34% R@5. The relatively high R@5 and R@10 compared to R@1 suggests these models can identify relevant products in the vicinity of the correct answer but cannot reliably distinguish the exact match from near-matches.
Sequential input (interleaved images): Performance degrades substantially compared to concatenated input across all fine-tuned models. GME-Qwen2VL-2B drops from 8.47% to 5.29% R@1, and from 47.13% to 24.18% R@5 — a 119.7% relative decline in R@5. LamRA-Qwen2.5VL-7B drops from 12.05% to 3.26% R@1. LLaVE at all scales (0.5B, 2B, 7B) shows catastrophic degradation: 0.38%, 0.12%, and 0.39% R@1 respectively — barely above zero-shot performance. VLM2Vec-4B achieves only 0.43% R@1. This sequential-concatenated gap is one of the paper's central diagnostic findings: existing fine-tuned models have lost the ability to process interleaved multi-image inputs effectively.

Averaged embeddings perform poorly. E5-V-8B, which averages embeddings across multiple images, achieves 3.10% R@1, 7.54% R@5, and 5.03% MRR — substantially below concatenated-input models at similar or smaller scales, confirming that naive embedding averaging is insufficient for multi-condition queries.

Key insight from the baseline comparison: Even the best models with 2–7 billion parameters, fine-tuned on retrieval data, can correctly identify the exact target product at rank 1 only about 12% of the time on interleaved multi-condition queries. This establishes MERIT as a genuinely challenging benchmark that is far from saturated, and it validates the paper's claim that existing approaches are fundamentally limited for this task type.

Visual Conditioning Necessity (Figure 6a)

The paper tests whether MERIT genuinely requires visual understanding by conducting ablation experiments where images are replaced with their textual captions. Using BGE-VL-7B as the test model:

On existing datasets: Replacing images with captions on FashionIQ and CIRR does not significantly degrade retrieval performance. BGE-VL on FashionIQ (R@10): performance remains similar with captions versus images. On CIRR (R@1): the caption-substituted version maintains comparable performance. This demonstrates that prior benchmarks exhibit "vision unnecessary" — the task can be solved through textual matching alone.
On MERIT: The pattern reverses dramatically. When images are replaced with captions ("w/o image" condition), performance degrades by 73.9%. When product titles are removed ("w/o title" condition), performance also drops substantially, though by a smaller margin than image removal. This demonstrates that both visual and textual information are indispensable for MERIT — the dataset cannot be reduced to a text-only matching problem. The magnitude of the image-removal degradation (73.9%) is a direct quantitative measure of how much MERIT depends on genuine visual understanding compared to prior benchmarks where the degradation is negligible.

Why this matters for the paper's claims: This experiment validates that MERIT is not just another dataset where images are decorative — it genuinely requires models to extract and reason about visual content. The "vision unnecessary" property of prior benchmarks (Figure 6a) is a key part of the paper's motivation for introducing MERIT, and this controlled experiment provides direct evidence that MERIT succeeds where prior datasets fail.

Interleaving Support Analysis (Table 2, Figure 6a)

The paper dedicates specific analysis to understanding why fine-tuned models perform worse with sequential (interleaved) input than with concatenated input — a finding that contradicts both the expected behavior of pre-trained MLLMs and the zero-shot results on MERIT.

Quantifying the degradation. The sequential-to-concatenated performance gap varies dramatically across models:

GME-Qwen2VL-2B: R@5 drops from 47.13% (concatenated) to 24.18% (sequential) — a 48.7% relative decline
LamRA-Qwen2.5VL-7B: R@1 drops from 12.05% to 3.26% — a 72.9% relative decline
LLaVE-7B: R@1 drops from 8.03% to 0.39% — a 95.1% relative decline
LLaVE-2B: R@1 drops from 5.80% to 0.12% — a 97.9% relative decline

The near-total collapse of LLaVE across all scales (R@1 below 0.5% for sequential input) is particularly striking — it represents a model that has effectively lost all ability to use interleaved visual information for retrieval, despite being built on architectures that natively support such inputs (InternVL2 for LLaVE).

The zero-shot comparison reveals the cause. Zero-shot MLLMs show the opposite pattern: InternVL2.5-MPO-1B achieves 0.41% R@1 with sequential input versus 0.24% with concatenated input. InternVL2.5-1B achieves 0.27% (sequential) versus 0.20% (concatenated). Qwen2.5-VL-3B achieves 0.09% (sequential) versus 0.05% (concatenated). While zero-shot performance is near-random in absolute terms, the directional advantage for sequential input is consistent across all three zero-shot models. This demonstrates that pre-trained MLLMs possess the architectural capability to process interleaved inputs — it is the subsequent fine-tuning on single-image datasets that destroys this capability.

The recovery experiment validates the hypothesis. After fine-tuning on MERIT (which contains interleaved multi-image queries), Coral with sequential input achieves 69.68% R@1 (Table 3, row 11), compared to 60.94% R@1 for Coral with concatenated input (Table 3, row 3) — a 14.3% relative advantage for sequential processing. This reversal (from sequential being worse in existing models to sequential being better after MERIT training) directly validates the paper's hypothesis that "existing retrieval datasets containing at most one image potentially cause MLLMs to lose their capability to process interleaved inputs" (Section 3.3).

Implication: This finding is both a diagnostic (identifying a systematic degradation induced by current fine-tuning practices) and a validation of MERIT's value (providing the training data needed to restore interleaved capability). It also explains why the field has not previously noticed this problem: if all evaluation is on single-image retrieval benchmarks, the degradation is invisible. Only a multi-image benchmark like MERIT can surface the capability loss.

Out-of-Distribution Generalization (Figure 6b, Tables 7–9)

The paper evaluates Qwen2.5-VL-3B (fine-tuned with contrastive learning) on three types of out-of-distribution (OOD) scenarios to assess how well models trained on MERIT generalize:

Class OOD (excluding a product category from training): When training excludes specific product classes (e.g., Drinks, Phones, Tables) and tests on the held-out class, the OOD performance (Table 7, "Class" rows) shows a moderate gap compared to training on the full dataset (Table 9, "Mixed"). For Drinks: 33.47% R@1 (OOD) vs. 33.26% R@1 (Mixed) — essentially no gap, suggesting that retrieval capabilities transfer across product classes without degradation. For Phones: 31.77% R@1 (OOD) vs. 51.58% R@1 (Mixed) — a notable gap, indicating that some product-specific visual or attribute knowledge does not transfer. For Tables: 35.07% R@1 (OOD) vs. 38.63% R@1 (Mixed) — a small gap. The average across classes shows OOD at 33.44% R@1 vs. Mixed at 41.16% R@1, demonstrating meaningful but imperfect transfer.

Attribute OOD (excluding specific attributes from training): When training excludes queries involving specific attributes (Brand, Pattern, Region) and tests on the held-out attribute, OOD performance (Table 7, "Attribute" rows) shows varying gaps. For Brand: 48.03% R@1 (OOD) vs. 59.62% R@1 (Mixed). For Pattern: 37.23% R@1 (OOD) vs. 50.30% R@1 (Mixed). For Region: 19.10% R@1 (OOD) vs. 38.20% R@1 (Mixed). The Region attribute shows the largest OOD gap, likely because region-specific attributes have distinctive visual and linguistic features that are harder to infer without training examples. Average attribute OOD: 34.79% R@1 vs. Mixed: 49.37% R@1.

Language OOD (excluding a language from training): When training excludes specific languages (Indonesian, Malay, Thai) and tests on the held-out language, OOD performance (Table 7, "Language" rows) shows a substantial gap. For Indonesian (ID): 15.98% R@1 (OOD) vs. 57.31% R@1 (Mixed). For Malay (MS): 17.97% R@1 (OOD) vs. 53.23% R@1 (Mixed). For Thai (TH): 19.80% R@1 (OOD) vs. 55.29% R@1 (Mixed). The language OOD gap is consistently large (~37–40 percentage points in R@1), but OOD performance is still substantially above zero-shot (2.34%, 3.92%, and 0.69% R@1 for ID, MS, and TH respectively, from Table 8). This indicates that while language transfer is imperfect — training on one set of languages does not fully prepare the model for new languages — the model's multilingual capability from pre-training is "activated" by the retrieval training even for unseen languages, enabling performance far above zero-shot.

The paper's interpretation: The relatively small gaps in Class and Attribute OOD scenarios "reflect the diversity of our dataset" — the training data covers enough variety that models learn transferable retrieval skills. The larger gaps in Language OOD reflect the inherent difficulty of cross-lingual transfer for fine-grained attribute understanding. All OOD scenarios show substantial improvements over zero-shot baselines (Table 8), demonstrating that MERIT training provides generalizable retrieval capabilities rather than narrow memorization of training queries.

Coral's Results on MERIT (Table 3)

The central method evaluation compares Coral against contrastive-only fine-tuning on MERIT using Qwen2.5-VL-3B as the base model. The complete results are reported in Table 3.

Headline result: Coral with full parameter fine-tuning and sequential input achieves 69.68% R@1, 89.26% R@5, 93.08% R@10, and 78.33% MRR (Table 3, row 11). This represents a 45.9% relative improvement over the contrastive-only baseline (47.76% R@1, row 2). In absolute terms, Coral adds 21.92 percentage points to R@1.

Contrastive-only baselines set the floor. The contrastive-only configurations achieve:

Full fine-tuning, sequential input (row 2): 47.76% R@1, 73.97% R@5, 80.47% R@10, 59.06% MRR
LoRA fine-tuning, sequential input (row 1): 48.52% R@1, 73.11% R@5, 77.93% R@10, 59.48% MRR

The near-identical performance between full and LoRA fine-tuning for contrastive-only training is notable — it suggests that contrastive learning on MERIT does not require the full expressive capacity of the model, and parameter-efficient methods suffice for learning a discriminative embedding space. However, as discussed below, this changes dramatically when reconstruction losses are added.

Input format ablation (concatenated vs. sequential, with Coral):

Coral full fine-tuning, concatenated input (row 3): 60.94% R@1, 85.60% R@5, 90.40% R@10, 71.70% MRR
Coral full fine-tuning, sequential input (row 11): 69.68% R@1, 89.26% R@5, 93.08% R@10, 78.33% MRR

The sequential advantage with Coral is 8.74 percentage points in R@1 — a 14.3% relative improvement. This is the reverse of what existing fine-tuned models show (where concatenation outperforms sequential), confirming that Coral training restores and enhances the model's interleaved processing capability. The gap is particularly pronounced at R@1 compared to R@5 and R@10, suggesting that sequential input primarily helps with precise ranking of the top candidate rather than broad recall.

Partial reconstruction ablations (LoRA, rows 4–7):

Contrastive only + Vision reconstruction, LoRA (row 6): 58.18% R@1, 83.19% R@5, 88.02% R@10, 69.13% MRR
Contrastive only + Language reconstruction, LoRA (row 7): 58.38% R@1, 83.01% R@5, 88.26% R@10, 69.35% MRR

Adding a single modality reconstruction via LoRA improves R@1 by approximately 9.7–9.9 percentage points over contrastive-only LoRA (48.52%). The two modalities produce nearly identical gains (58.18% vs. 58.38%), suggesting that at this parameter budget, either modality provides comparable regularization. The R@5 improvements are dramatic: from 73.11% (contrastive-only LoRA) to ~83% — roughly 10 percentage points — indicating that reconstruction primarily helps with broad recall (getting the correct answer into the top candidates) rather than just precise ranking.

Partial reconstruction ablations (full fine-tuning, rows 8–9):

Contrastive only + Vision reconstruction, full FT (row 8): 59.46% R@1, 85.46% R@5, 90.81% R@10, 70.89% MRR
Contrastive only + Language reconstruction, full FT (row 9): 59.98% R@1, 86.01% R@5, 90.72% R@10, 71.22% MRR

With full fine-tuning, single-modality reconstruction achieves R@1 of ~60%, compared to 47.76% for contrastive-only full FT — an improvement of approximately 12.2 percentage points. This is marginally better than the LoRA equivalents (~58.2%), suggesting that full parameter updates provide slightly more capacity to exploit the reconstruction signal. The gap between vision-only and language-only remains negligible (0.52 percentage points in R@1), reinforcing that both modalities independently provide substantial and roughly equal regularization.

Full Coral (both modalities, full fine-tuning, row 11): The combined reconstruction achieves 69.68% R@1 — 9.7 percentage points higher than the best single-modality reconstruction (59.98% for language-only). This super-additive gain (the combination substantially outperforms either modality alone) is the key empirical finding of the ablation: vision and language reconstruction are not redundant but complementary. LoRA-based full Coral (row 10) achieves 59.40% R@1 — significantly below full fine-tuning (69.68%) but still substantially above contrastive-only LoRA (48.52%). The 10.28 percentage point gap between LoRA and full fine-tuning for the full Coral setup (vs. the near-zero gap for contrastive-only) suggests that the reconstruction objectives require more parameter capacity than contrastive learning alone — the model needs to restructure its internal representations to simultaneously satisfy discriminative and reconstructive objectives, and LoRA's low-rank constraint may be insufficient for this transformation.

Key performance milestones quantified:

Coral full (69.68% R@1) vs. best existing model on MERIT (LamRA-Qwen2.5VL-7B concatenated: 12.05% R@1, Table 2): Coral achieves 5.78× higher R@1.
Coral full (69.68% R@1) vs. contrastive-only baseline (47.76% R@1): 45.9% relative improvement.
Coral full R@5 (89.26%) vs. contrastive-only R@5 (73.97%): a 15.29 percentage point absolute improvement. The fact that R@5, R@10, and MRR are all above 89% with Coral indicates that when the model fails to rank the correct answer at position 1, the correct answer is still typically within the top 5-10 candidates — a practically useful property for retrieval systems that display multiple results or use downstream reranking.

Coral's Results on Eight Established Retrieval Benchmarks (Figure 9, Table 10)

To demonstrate that Coral's benefits extend beyond the interleaved multi-condition setting, the paper evaluates on 8 standard retrieval benchmarks using Qwen2-VL-2B as the base model. The results are presented in Figure 9 (bar chart comparisons) and Table 10 (numerical results).

Headline result: Coral (Qwen2-VL-2B + Coral) outperforms the contrastive-only baseline (Qwen2-VL-2B-CL) on all 8 benchmarks, with improvements ranging from modest to dramatic.

Per-benchmark results (R@1, Table 10):

VisDial: Coral 73% vs. CL baseline 51% — 43.1% relative improvement (and 181% over zero-shot Qwen2-VL-2B at 13%)
CIRR: Coral 50% vs. CL baseline 39% — 28.2% relative improvement
VisualNews T2I: Coral 67% vs. CL baseline 56% — 19.6% relative improvement
VisualNews I2T: Coral 72% vs. CL baseline 52% — 38.5% relative improvement
COCO T2I: Coral 68% vs. CL baseline 56% — 21.4% relative improvement
COCO I2T: Coral 64% vs. CL baseline 45% — 42.2% relative improvement
NIGHTS: Coral 65% vs. CL baseline 58% — 12.1% relative improvement
WebQA: Coral 84% vs. CL baseline 67% — 25.4% relative improvement

Comparison to prior specialist models: Coral at 2B scale also outperforms or matches several larger specialist models:

vs. CLIP (various architectures): Coral substantially exceeds CLIP's 30.70% on VisDial, 12.60% on CIRR, and comparable results on COCO.
vs. MagicLens (Zhang et al., 2024): Coral's 50% on CIRR vs. MagicLens' 39.10%; Coral's 84% on WebQA vs. MagicLens' 43.00%.
vs. E5-V-8B (a 4× larger model): Coral's 2B model dramatically outperforms E5-V on all benchmarks except WebQA (where E5-V scores 17.70% vs. Coral's 84%).
vs. GME-Qwen2-VL-2B (same base architecture and scale): Coral outperforms on VisDial (73% vs. 26%), CIRR (50% vs. 38%), VisualNews T2I (67% vs. 66%), VisualNews I2T (72% vs. 71%), COCO T2I (68% vs. 62%), COCO I2T (64% vs. 56%), NIGHTS (65% vs. 64%), and WebQA (84% vs. 83%). The margins are smaller on benchmarks where GME already performs well (VisualNews, NIGHTS, WebQA), suggesting that reconstruction provides the largest gains where contrastive-only models struggle most (VisDial, CIRR).

Interpretation of generalization results: These results demonstrate three important properties of Coral:

Universality: The reconstruction objective benefits retrieval performance across diverse task formats — visual dialogue (VisDial), composed image retrieval (CIRR), text-to-image and image-to-text retrieval (VisualNews, COCO), image-to-image perceptual similarity (NIGHTS), and multimodal QA (WebQA). This suggests the benefit is not specific to multi-condition interleaved inputs but applies to retrieval-oriented fine-tuning generally.
Complementarity to contrastive learning: The consistent gap between CL baseline and Coral on all benchmarks indicates that the reconstruction losses provide regularization that contrastive learning alone does not — even for tasks where the base model already performs well.
Scale efficiency: Coral at 2B parameters outperforms or matches models at 7–8B parameters on several benchmarks, suggesting that reconstruction-based fine-tuning can extract more capability from smaller models, potentially reducing the need for scale in deployment.

Human Preference Evaluation (Table 11, Appendix 10.6)

The paper supplements automated metrics with a human preference study on MERIT. For 100 randomly sampled queries, five annotators compared the top-1 (Choice@1) and top-3 (Choice@3) retrievals from four models: GME-Qwen2VL-2B, LamRA-Qwen2.5VL-7B, BGE-VL-7B, and Coral-Qwen2.5VL-3B.

Results:

Choice@1: Coral is preferred 53.6% of the time, compared to 15.0% (GME), 18.0% (LamRA), and 13.4% (BGE-VL).
Choice@3: Coral is preferred 58.9% of the time, compared to 11.7% (GME), 14.4% (LamRA), and 14.9% (BGE-VL).

Coral receives more than 3× the preference share of the next-best model in both settings, demonstrating that the automated metric improvements translate to human-perceivable quality differences. The higher preference in Choice@3 (58.9% vs. 53.6% for Choice@1) suggests that Coral's advantage is particularly pronounced when considering multiple candidates — its top-3 results collectively provide better coverage of user intent than competitors.

Error Analysis Results (Figure 7, Section 3.4, Appendix Figures 66–69)

The error analysis on 500 sampled queries from Qwen2.5-VL-3B and GME-Qwen2VL-2B (both fine-tuned with contrastive learning) categorizes failures into five types:

Error distribution (Figure 7b):

Attribute Errors: The largest category — models correctly identify which attributes are queried but select products with incorrect attribute values (e.g., right pattern, wrong color). Example in Figure 66: the recalled product matches the pattern of Product 1 but the color does not match Product 2's requirement.
Visual Understanding Errors: The second largest category — models correctly understand the language instruction but fail to align it with appropriate visual features in the retrieved product. Example in Figure 67: the model understands the textual instruction but retrieves an image that is visually wrong.
Category Errors: Models retrieve products from incorrect categories (e.g., a bag instead of a mobile phone bag). Example in Figure 68.
Detail Errors: Models satisfy the primary conditions but overlook finer details (e.g., a visible brand logo). Example in Figure 69.
Annotation Errors: Rare cases where dataset annotations may be imprecise, causing model responses to appear incorrect when evaluated against ground truth.

Why this decomposition matters: The dominance of Attribute Errors and Visual Understanding Errors (collectively the largest fraction) directly validates the paper's central diagnosis: existing models fail because they "neglect specific conditional elements in queries, failing to extract specific attributes and misinterpreting visual content." These are precisely the error types that Coral's reconstruction objectives are designed to address — $\mathcal{L}_{\text{reg}}$ forces the query embedding to retain per-attribute, per-image detail sufficient to reconstruct the target's specific attribute values and visual features.

Language performance analysis (Figure 7a, Table 6): The error analysis includes a breakdown by language to test whether performance disparities exist across the five MERIT languages. Using Qwen2.5-VL-3B, InternVL2.5-VL-1B, and BGE-VL-7B evaluated on language-pure query subsets (where both the query and positive samples are in the same language):

Qwen2.5-VL-3B: English 48.73%, Indonesian 56.23%, Thai 55.34%, Vietnamese 55.13%, Malay 47.98% R@1
InternVL2.5-VL-1B: English 55.38%, Indonesian 60.75%, Thai 58.97%, Vietnamese 62.82%, Malay 52.47% R@1
BGE-VL-7B: English 13.76%, Indonesian 14.02%, Thai 14.83%, Vietnamese 20.51%, Malay 14.08% R@1

The variation across languages is modest (range of ~7–10 percentage points for the top models), with no language showing a consistent advantage or disadvantage. Notably, English — despite being the predominant language in MLLM pre-training data — does not outperform the Southeast Asian languages. This suggests that the retrieval task primarily depends on visual-attribute reasoning rather than language-specific knowledge, and that the multilingual capabilities of pre-trained MLLMs transfer reasonably well across the five languages in MERIT.

Multi-Condition Complexity Analysis (Table 12, Appendix 10.7)

The paper analyzes how retrieval performance degrades as the number of conditions in a query increases from 2 to 4. This is evaluated across five representative models:

Two-condition queries (the vast majority of MERIT, N=319,600 training):

GME-Qwen2VL-2B concatenated: 8.45% R@1, 53.12% R@5
LamRA-Qwen2.5VL-7B concatenated: 12.05% R@1, 39.20% R@5
InternVL2.5-MPO-1B sequential: 0.42% R@1, 1.39% R@5
Qwen2.5-VL-3B sequential: 0.09% R@1, 0.39% R@5
VLM2Vec-4B sequential: 0.44% R@1, 1.87% R@5

Three-condition queries (N=300 in test set):

GME-Qwen2VL-2B: 10.24% R@1, 55.91% R@5 — surprisingly, higher than two-condition performance
LamRA-Qwen2.5VL-7B: 13.39% R@1, 35.43% R@5 — also slightly higher R@1 than two-condition
The remaining models degrade to near-zero: InternVL2.5-MPO (0.00% R@1), Qwen2.5-VL (0.00% R@1), VLM2Vec (0.00% R@1)

Four-condition queries (N=100 in test set): All models degrade substantially:

GME-Qwen2VL-2B: 5.26% R@1, 36.84% R@5
LamRA-Qwen2.5VL-7B: 5.56% R@1, 27.78% R@5
All sequential-input models: 0.00% across all metrics

The slight performance increase for concatenated-input models on three-condition queries (vs. two-condition) may reflect the small sample size (300 test queries) and potential distribution differences — three-condition queries may be inherently easier if they provide more constraining information. The collapse of sequential-input models at three and four conditions confirms that their interleaved processing capability is essentially non-functional beyond two images. The fact that even the best model (GME) achieves only 36.84% R@5 on four-condition queries highlights the inherent difficulty of high-dimensionality multi-condition retrieval — each additional condition adds a new constraint that must be simultaneously satisfied, and the combinatorial space of possible attribute combinations grows exponentially.

Ablation Studies and Robustness Checks

Full fine-tuning vs. LoRA (Table 3, rows 1–2, 10–11): For contrastive-only training, LoRA (48.52% R@1) and full fine-tuning (47.76% R@1) perform nearly identically — a difference of 0.76 percentage points. For Coral, the gap is substantial: LoRA achieves 59.40% R@1 while full fine-tuning achieves 69.68% R@1 — a 10.28 percentage point difference. This asymmetry is a non-trivial finding: reconstruction objectives require more parameter capacity than contrastive learning alone. The paper attributes this to "the substantial divergence between retrieval tasks and pre-training objectives" (Section 4.2) — adapting the model to simultaneously optimize contrastive and reconstructive objectives likely requires restructuring attention patterns and internal representations that LoRA's low-rank constraint cannot fully capture.

Vision-only vs. language-only reconstruction (Table 3, rows 6–9): Across both LoRA and full fine-tuning, single-modality reconstruction produces nearly identical gains regardless of which modality is chosen. With LoRA: vision-only achieves 58.18% R@1, language-only achieves 58.38% R@1 (difference: 0.20 percentage points). With full fine-tuning: vision-only achieves 59.46% R@1, language-only achieves 59.98% R@1 (difference: 0.52 percentage points). Both modalities individually provide substantial gains over contrastive-only (~12 percentage points), and the fact that they produce nearly equivalent improvements suggests that either modality can serve as an effective information-preserving regularizer. The vision and language reconstruction objectives appear to be functionally interchangeable when used in isolation.

Combined reconstruction vs. single-modality (Table 3, rows 8–9 vs. row 11): The combined vision+language reconstruction (69.68% R@1) substantially outperforms the best single-modality reconstruction (59.98% R@1 for language-only full fine-tuning) by 9.70 percentage points. This super-additive effect is the key evidence that joint reconstruction provides benefits beyond what either modality alone can achieve. The paper interprets this as evidence for cross-modal binding: reconstructing both visual and linguistic features from the same [EOS] query vector forces the embedding to capture correspondences between modalities (e.g., how the text description of a pattern relates to its visual appearance), which benefits retrieval.

Sequential vs. concatenated input with Coral (Table 3, rows 3, 11): Coral with concatenated input achieves 60.94% R@1 vs. 69.68% R@1 for sequential input — a 14.3% relative advantage for sequential processing. This confirms that Coral training restores the native interleaved processing capability that pre-trained MLLMs possess but that existing fine-tuned models lose. The sequential advantage is present across all metrics (R@5: 85.60% concatenated vs. 89.26% sequential; R@10: 90.40% vs. 93.08%), but is most pronounced at R@1, suggesting that preserving per-image resolution and position information primarily aids precise top-1 ranking.

Contrastive learning alone as the core baseline (Table 3, rows 1–2): The contrastive-only baseline establishes that standard retrieval fine-tuning on MERIT provides substantial gains over zero-shot (47.76% vs. 0.09% R@1 for Qwen2.5-VL-3B), confirming that MERIT training data is learnable and that contrastive learning can capture some multi-condition structure. The fact that Coral doubles the improvement over contrastive-only training demonstrates that the reconstruction losses capture capability orthogonal to what contrastive learning provides.

Zero-shot evaluation as a capability floor (Table 2, Table 3, rows 3–5): The near-zero zero-shot performance (0.05–0.41% R@1 across models and input formats) confirms that MERIT is genuinely challenging and that pre-trained MLLMs without task-specific adaptation cannot perform multi-condition retrieval. This establishes that the gains from contrastive fine-tuning and from Coral are not simply "unlocking" latent capabilities — they are teaching the model a fundamentally new skill.

Coral vs. existing models across 8 benchmarks (Figure 9, Table 10): The consistent improvement of Coral over the CL baseline on all 8 benchmarks — ranging from 12.1% (NIGHTS) to 43.1% (VisDial) relative improvement — demonstrates that the reconstruction objective generalizes beyond the multi-condition interleaved setting. The particularly large gains on VisDial (73% vs. 51%) and CIRR (50% vs. 39%) suggest that tasks requiring fine-grained visual-linguistic alignment benefit most from reconstruction-based regularization, consistent with Coral's design motivation.

Temperature and batch size robustness: The paper uses fixed τ = 0.02 and effective batch size 64 across all experiments without ablating these choices. The sensitivity of Coral's performance to contrastive learning hyperparameters is not explored — a potentially relevant ablation given that the contrastive and reconstruction objectives interact through shared gradients.

Masking ratio: The paper fixes δ = 0.5 for both modalities without ablating alternative ratios. The choice is justified by reference to masked autoencoding literature but is not empirically validated for the retrieval fine-tuning setting. A masking ratio ablation (e.g., 0.3 vs. 0.5 vs. 0.7) would strengthen confidence that 0.5 is near-optimal for this task.

Loss weighting: The paper sets λ₁ = λ₂ = 0.1 without ablation of alternative weightings. The choice is described as balancing the contrastive and reconstruction objectives, but the robustness of results to this hyperparameter is untested. Given that the reconstruction losses are auxiliary to the primary contrastive objective, the relative weighting likely matters — reconstruction that is too heavily weighted could interfere with discriminative learning, while reconstruction that is too lightly weighted may provide insufficient regularization.

Frozen vs. fine-tuned vision tower: The paper keeps the vision tower frozen across all configurations, consistent with standard practice in MLLM fine-tuning. No ablation is provided for fine-tuning the vision tower alongside the LLM backbone and reconstruction decoders. It is plausible that end-to-end training including the vision encoder could further improve multi-condition visual understanding, since the visual features could adapt to the specific attribute-extraction demands of the retrieval task.

Critical Assessment

Claim: "Coral achieves a 45.9% performance improvement over conventional approaches on MERIT"

The claim is empirically supported by Table 3, which shows Coral with full fine-tuning and sequential input achieving 69.68% R@1 versus 47.76% R@1 for the contrastive-only baseline — a 45.9% relative improvement. However, several qualifications are necessary:

The baseline is the paper's own implementation of contrastive learning, not the best existing model. The 47.76% baseline is substantially higher than the best existing fine-tuned model evaluated in Table 2 (LamRA-Qwen2.5VL-7B at 12.05% R@1 with concatenated input, or 3.26% with sequential input). This makes the 45.9% improvement figure somewhat misleading when compared to "conventional approaches" broadly — Coral is compared against the paper's own contrastive learning implementation on MERIT, not against prior work's models evaluated on MERIT. The absolute performance of Coral (69.68% R@1) versus the best existing model (12.05% R@1) represents a 5.78× improvement, which is a fairer characterization of Coral's advantage. The paper's claim of 45.9% improvement is technically correct for the specific ablation comparison but understates the gap to actual prior art.

The contrastive-only baseline uses the same base model (Qwen2.5-VL-3B), same training data (MERIT), same batch size, same temperature, and same optimizer settings as Coral. This is a clean ablation that isolates the effect of the reconstruction losses, which is scientifically appropriate. However, it means the 45.9% figure should be interpreted as "reconstruction objectives add 45.9% relative improvement over identical contrastive-only training" rather than "Coral is 45.9% better than any existing method."

The sequential input format is critical. Coral with concatenated input achieves 60.94% R@1 — only 27.6% higher than the contrastive-only sequential baseline (47.76%), and substantially below the 69.68% achieved with sequential input. The full benefit of Coral is only realized when models are trained and evaluated with interleaved sequential inputs, which most existing models do not support effectively (Table 2).

Claim: "Strong generalization capabilities validated across 8 established retrieval benchmarks"

This claim is well-supported by Figure 9 and Table 10. Coral improves over the contrastive-only baseline on all 8 benchmarks, with specific quantitative margins documented above. The generalization is particularly convincing because the benchmarks span diverse task formats (visual dialogue, composed retrieval, text-to-image, image-to-text, image-to-image similarity, multimodal QA) and because Coral at 2B parameters matches or exceeds several larger specialist models.

A nuance: The generalization experiments use Qwen2-VL-2B as the base model, while the MERIT experiments use Qwen2.5-VL-3B. The paper does not explain this model choice discrepancy, and it introduces a minor confound — the generalization results demonstrate that Coral works with a different (slightly older) model family, but we cannot directly compare the magnitude of improvement (MERIT: 47.76% → 69.68% R@1; VisDial: 51% → 73% R@1) because the baselines differ. The consistent direction of improvement across models and tasks is nonetheless strong evidence for Coral's generality.

A missing evaluation: The paper does not report how Coral trained on MERIT transfers to the 8 benchmarks. All generalization experiments train Coral on a mixture of the 8 benchmark datasets, not on MERIT. An experiment training Coral on MERIT and evaluating zero-shot on the 8 benchmarks would test whether the multi-condition interleaved training produces representations that transfer to standard retrieval tasks — this would be a stronger test of generalization than training on the target benchmarks themselves.

Claim: "Existing models neglect specific conditional elements in queries, failing to extract specific attributes and misinterpreting visual content"

This diagnostic claim is supported by the error analysis (Figure 7b, Appendix Figures 66–69) but has methodological limitations:

The error analysis sample size is 500 queries per model (Qwen2.5-VL-3B and GME-Qwen2VL-2B), drawn from the 10,000-query test set. The error categories are assigned by expert annotators, and the paper provides example cases for each category. However, the paper does not report inter-annotator agreement metrics or provide a detailed annotation protocol for the error classification. The five error categories may not be mutually exclusive — a single failure could involve both attribute error and visual understanding error — and the paper does not describe how such cases were resolved.

The causal link between the error types and the [EOS]-only supervision is inferred, not directly tested. The paper argues that [EOS]-only contrastive learning causes models to neglect conditional elements, and the error analysis shows that models make attribute and visual understanding errors. But the paper does not provide a direct experiment showing that adding reconstruction losses specifically reduces these error categories. An error analysis comparing Coral's failure modes to the contrastive-only baseline's failure modes would provide direct evidence for the claimed mechanism. Without this, the connection between the diagnostic finding and the proposed solution is correlational rather than causal.

The error analysis only covers contrastive-only models (Qwen2.5-VL-3B fine-tuned with contrastive learning, and GME-Qwen2VL-2B which uses contrastive learning in its training). Coral's error distribution is not analyzed. It would be informative to know whether Coral's remaining errors (30.32% of queries still fail at R@1) are qualitatively different from the contrastive-only errors. If Coral eliminates attribute errors but still makes visual understanding errors, that would suggest reconstruction helps with attribute binding but less with visual feature extraction. If the error distribution shifts toward detail errors, that would suggest Coral captures primary conditions well but still misses fine-grained specifications.

Claim: "Fine-tuning on single-image datasets destroys interleaved processing capability"

This claim is strongly supported by the sequential-vs-concatenated analysis in Table 2 and the recovery experiment in Table 3. The evidence is clean and compelling:

Pre-trained MLLMs show a small but consistent sequential advantage in zero-shot evaluation (Table 2, top rows).
Fine-tuned models show a large and consistent concatenation advantage (Table 2, middle rows).
Coral trained on MERIT restores the sequential advantage (Table 3, rows 3 vs. 11).

The causal chain is well-established: pre-training → sequential advantage exists; single-image fine-tuning → sequential advantage is destroyed; multi-image fine-tuning → sequential advantage is restored. This is a strong finding that the paper supports with multiple model architectures and scales.

A limitation: The paper does not directly demonstrate that the capability loss is caused by the single-image nature of prior datasets rather than some other property of retrieval fine-tuning (e.g., the contrastive objective itself, the limited dataset size, or architectural choices in the fine-tuning procedure). An additional experiment training with contrastive learning on an interleaved multi-image version of an existing dataset would isolate whether single-image vs. multi-image training data is the causal factor. If contrastive learning on multi-image data preserves sequential capability, the causal link to dataset image count is confirmed.

General Experimental Strengths

Comprehensive baseline coverage: The paper evaluates 9 models spanning 6 model families, multiple scales (0.5B to 8B parameters), and both zero-shot and fine-tuned configurations. This is substantially more thorough than typical method papers and provides confidence that the identified limitations are systematic rather than idiosyncratic.
Clean ablation design: The Coral ablation (Table 3) systematically varies fine-tuning method (LoRA vs. full), input format (sequential vs. concatenated), and reconstruction modality (vision only, language only, both), providing clear evidence for each component's contribution.
Multiple evaluation dimensions: The paper evaluates on accuracy metrics (R@K, MRR), human preference, cross-lingual performance, OOD generalization, and multi-condition complexity scaling. This multidimensional evaluation provides a richer picture of model behavior than accuracy alone.
Realistic task design: MERIT's construction — with real e-commerce products, genuine multi-condition queries, and manual annotation — ensures that findings are relevant to practical applications rather than artificial laboratory settings.

General Experimental Weaknesses

Single epoch training: All models are trained for exactly one epoch on the 310,000-query training set. The paper does not report whether training converges, whether additional epochs would improve or degrade performance, or whether early stopping was performed based on validation metrics. The choice of one epoch may be a practical constraint (10,000 labor hours of annotation suggests expensive data collection) but limits understanding of the learning dynamics.
No statistical significance reporting: The paper does not report confidence intervals, standard deviations across random seeds, or statistical significance tests for any quantitative result. Given the 10,000-query test set, differences of a few percentage points in R@1 may or may not be statistically significant. The human preference study (100 samples, 5 annotators) is too small for robust statistical conclusions but is reported without error bars or significance measures.
Single base model family for Coral development: All Coral experiments use Qwen2.5-VL-3B (for MERIT) or Qwen2-VL-2B (for generalization). The paper does not demonstrate Coral on other MLLM architectures (InternVL2.5, LLaVA, etc.), so the extent to which Coral's benefits transfer across model families is unknown.
Untuned loss weights and masking ratio: The loss weights (λ₁ = λ₂ = 0.1) and masking ratio (δ = 0.5) are set without ablation. The paper's results may be sensitive to these hyperparameters, and the optimal values may differ across base models or datasets.
No analysis of compute overhead from reconstruction: The reconstruction decoders add computational cost during training (additional forward passes through BERT layers, additional loss computations). The paper does not quantify this overhead or compare training time between contrastive-only and Coral configurations. At inference time, the reconstruction decoders are not used, so inference cost is identical — but training cost differences matter for practical adoption.
Limited analysis of failure modes after Coral: With 69.68% R@1, approximately 30% of queries still fail. The paper does not analyze what types of errors persist after Coral training, which would be valuable for understanding the method's limitations and guiding future improvements.

Missing Experiments That Would Strengthen the Paper

Coral trained on MERIT, evaluated on standard benchmarks: Tests whether multi-condition interleaved training produces generally better representations or is specific to the multi-condition task.
Comparison of Coral against simply training with more data or larger batches: The reconstruction losses may primarily act as a regularizer that prevents overfitting; ablating against stronger regularization (weight decay, dropout, data augmentation) would test whether reconstruction provides benefits beyond generic regularization.
Fine-tuning vision tower with Coral: The paper freezes the vision tower; testing whether end-to-end fine-tuning of the visual backbone with reconstruction losses further improves multi-condition visual understanding.
Error analysis comparing Coral vs. contrastive-only failure modes: Quantifies which error categories Coral specifically addresses (attribute errors? visual understanding errors? both?) and which persist.
Multiple training runs with different random seeds: Provides confidence intervals and demonstrates that improvements are robust to initialization and data ordering.
Experiments on a second MLLM architecture (e.g., InternVL2.5): Tests whether Coral's benefits are architecture-specific or general.

Conditions Under Which Claims Hold

The 45.9% improvement over contrastive-only training applies specifically to the setting of full fine-tuning with sequential interleaved input on MERIT. With LoRA, the improvement is 22.4% (59.40% vs. 48.52% R@1). With concatenated input, the improvement is 27.6% (60.94% vs. 47.76% R@1). The claimed 45.9% is the best-case configuration.
The interleaved processing capability restoration (sequential outperforming concatenation) requires training on interleaved multi-image data. The paper demonstrates this on MERIT but does not test whether other interleaved datasets would produce the same effect.
The generalization to 8 benchmarks holds for Qwen2-VL-2B and when Coral is trained on the target benchmark data (or a mixture thereof). Transfer from MERIT-only training to other benchmarks is not tested.
The diagnostic findings about [EOS]-only supervision causing neglect of conditional elements are inferred from error analysis on contrastive-only models and improvement from reconstruction losses; no direct causal experiment manipulates the supervision signal while measuring per-condition information retention.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation via 2048-Sample Oracle Is Prohibitively Expensive

The assumption or constraint. The paper's difficulty estimation for MERIT queries requires computing the average PRM or ground-truth score across 2,048 sampled solutions per query. As the authors explicitly acknowledge in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity."

The consequence. The headline efficiency gains — including the 45.9% improvement from Coral — are computed assuming difficulty is already known without factoring in the cost of obtaining that knowledge. In a realistic deployment, the total cost would be difficulty estimation (2,048 generations per query to bin into difficulty quintiles) plus strategy execution (the actual retrieval budget). The estimation step alone consumes more compute than the largest fine-tuning budgets studied. A practitioner deploying this system would face a stark exploration-exploitation tradeoff: spend a massive fixed cost to estimate difficulty, then allocate the remaining budget efficiently, or skip estimation and accept uniform (suboptimal) allocation. The paper provides no guidance on how to navigate this tradeoff.

What evidence exists in the paper. The MERIT dataset construction described in Section 3.2 and Appendix 6 involves extensive manual annotation (10,000 labor hours) and automated filtering, but the difficulty binning methodology used to analyze model performance (Section 3.4, Figure 7) relies on oracle access to ground-truth answers for error categorization. The out-of-distribution analysis (Section 3.3, Figure 6b, Tables 7-9) uses a similar approach of pre-splitting data by language, attribute, or class — again relying on labels that would not be available at inference time for new queries. The paper does not report experiments measuring the cost of difficulty estimation or amortizing it into the total compute budget.

Mitigation status. The paper flags this as "a key avenue for future work" (Section 3.2) but provides no method for cheap difficulty estimation, no model trained to predict difficulty from query text alone, and no adaptive scheme that estimates difficulty during retrieval rather than as a separate pre-processing step. The difficulty bins are treated as a fixed, pre-computed property of the dataset. This is a fundamental gap between the paper's analytical framework (which assumes difficulty is known) and practical deployment requirements (where difficulty must be inferred online).

Limitation 2: Generalization Is Demonstrated Only on English-Centric Benchmarks with a Single Model Family

The assumption or constraint. All Coral experiments use Qwen2.5-VL-3B (for MERIT) or Qwen2-VL-2B (for the 8 generalization benchmarks) as the base MLLM. The generalization benchmarks in Figure 9 and Table 10 — VisDial, CIRR, VisualNews, MSCOCO, NIGHTS, WebQA — are all English-language datasets. The paper does not test Coral on other MLLM architectures (InternVL2.5, LLaVA, DeepSeek-VL), nor does it evaluate on non-English retrieval benchmarks, nor does it test whether Coral's benefits transfer when the base model architecture changes.

The consequence. A practitioner using a different MLLM family (e.g., LLaVA-based retrieval models) or deploying in a non-English market cannot assume Coral's 45.9% improvement will transfer. The reconstruction mechanism operates on internal MLLM representations (the [EOS] hidden state, visual token embeddings, text token embeddings), and these representations may differ substantially across architectures. Models with different vision encoders (CLIP vs. SigLIP vs. InternViT), different projection layers (linear vs. MLP), different LLM backbones (Qwen vs. LLaMA vs. InternLM), or different pre-training objectives may exhibit different information-loss patterns during contrastive fine-tuning, which would change the degree to which reconstruction helps.

Additionally, the generalization benchmarks are English-only, while MERIT's primary novelty is multilingual interleaved retrieval across five languages (English, Indonesian, Malay, Vietnamese, Thai). The paper does not test whether Coral's benefits on English Vision-and-Language tasks extend to the Southeast Asian languages in MERIT — a critical gap given that the language OOD results (Table 7) showed substantial performance degradation when models were evaluated on languages absent from training. If Coral overfits to linguistic features present in English training data, its benefits may not transfer to low-resource languages.

What evidence exists in the paper. The base model restriction is visible in all Coral experiments: Table 3 uses Qwen2.5-VL-3B exclusively, Table 10 uses Qwen2-VL-2B exclusively. The 8 generalization benchmarks are described in Appendix 10.2 and are all English-language datasets. The paper does not report Coral results on the MERIT multilingual subsets broken down by language (Table 6 reports performance of contrastive-only models across languages but not Coral's per-language performance). The paper does not ablate across model families — no InternVL2.5 + Coral, no LLaVA + Coral, no DeepSeek-VL + Coral.

Mitigation status. The paper does not acknowledge this as a limitation. The consistent improvement across 8 diverse task formats (within English, within the Qwen family) provides some evidence of generality, but the jump from "works across tasks" to "works across models and languages" is not supported by any experiment. A practitioner would need to validate Coral on their specific model architecture and target language before relying on the reported gains.

Limitation 3: Coral Adds Training Complexity Without Quantified Overhead

The assumption or constraint. Coral introduces three additional loss terms ( $\mathcal{L}_{\text{mse}}$ , $\mathcal{L}_{\text{mlm}}$ , each with $\mathcal{L}_{\text{reg}}$ and $\mathcal{L}_{\text{rec}}$ variants), two additional decoder modules ( $\mathcal{F}_\theta^v$ , $\mathcal{F}_\theta^l$ ), and a masked attention mechanism that requires separate forward passes for masked and unmasked inputs. The paper does not report training time, GPU memory consumption, or FLOPs comparisons between contrastive-only and Coral configurations.

The consequence. A practitioner deciding whether to adopt Coral needs to know the cost of the additional training complexity. If Coral requires 2× the training time or 1.5× the GPU memory, the 45.9% improvement in R@1 may or may not be worth the cost depending on compute budget constraints. Additionally, the reconstruction decoders require careful hyperparameter tuning ( $\lambda_1$ , $\lambda_2$ , $\delta$ , decoder architecture) that may be sensitive to the base model and dataset. The paper does not provide guidance on how to tune these, nor does it report whether training is stable across random seeds or whether the reconstruction losses occasionally destabilize the contrastive objective.

What evidence exists in the paper. The training configuration in Appendix 10.3 specifies the hardware (8× H100 GPUs), batch size (64 effective), and number of epochs (1). However, no wall-clock time, GPU-hour, or memory usage metrics are reported for any configuration. The paper does not ablate the computational overhead of masked vs. unmasked forward passes, the memory cost of the BERT decoder layers, or the throughput difference between contrastive-only and Coral training. The loss weights ( $\lambda_1 = \lambda_2 = 0.1$ ) and masking ratio ( $\delta = 0.5$ ) are set without sensitivity analysis, so a practitioner cannot know whether these values are near-optimal or whether substantial tuning is required for new datasets.

Mitigation status. The paper acknowledges that "full parameter fine-tuning yields optimal results" and that this is "consistent with conclusions from previous work" (Section 4.2), but does not discuss the training cost implications. At inference time, the reconstruction decoders are not used — only the [EOS] embedding is required for retrieval — so inference cost is identical to contrastive-only models. The training overhead is a one-time cost, which partially mitigates the concern for large-scale deployment where training happens once and inference dominates total cost. However, for research groups or practitioners iterating on model design, training cost is a first-order concern that the paper does not address.

Limitation 4: Hardest Queries Remain Essentially Unsolved Despite Coral

The assumption or constraint. Coral achieves 69.68% R@1 on MERIT (Table 3, row 11), meaning approximately 30% of queries still fail to retrieve the correct product at rank 1. The paper does not analyze what characterizes these remaining failures — whether they cluster in specific difficulty regimes, specific product categories, specific languages, or specific attribute combinations.

The consequence. A practitioner deploying Coral cannot know whether the 30% failure rate is concentrated in a particular query type that could be handled separately (e.g., routing hard queries to human review or a fallback system) or is uniformly distributed. If failures are concentrated on the hardest queries — analogous to how the test-time compute paper found that difficulty bin 5 showed near-zero improvement regardless of budget — then Coral provides no path forward for those queries, and alternative approaches (larger models, more training data, fundamentally different architectures) would be needed. If failures are uniformly distributed, the problem may be more amenable to incremental improvements through scaling.

The multi-condition complexity analysis in Table 12 provides a partial answer: for four-condition queries, even the best existing model (GME-Qwen2VL-2B) achieves only 36.84% R@5, and Coral's performance on such queries is not reported. This suggests that as condition count increases, retrieval performance degrades substantially. However, the paper does not analyze whether Coral specifically helps or fails on high-condition-count queries relative to the contrastive-only baseline, nor does it report Coral's performance broken down by difficulty metrics.

What evidence exists in the paper. Table 3 reports aggregate metrics (69.68% R@1, 89.26% R@5) without per-category, per-language, or per-condition-count breakdowns. Table 12 reports multi-condition performance for baseline models (not Coral) and shows degradation with increasing conditions. The error analysis in Section 3.4 and Figure 7b reports error categories for contrastive-only models (not Coral). The paper does not provide a failure analysis for Coral that would characterize the remaining 30% error rate.

Mitigation status. The paper does not address this limitation. It reports Coral's aggregate improvement and moves on. A natural follow-up experiment — error analysis comparing Coral's failure modes to the contrastive-only baseline's failure modes (Figure 7b) — is absent. This experiment would reveal whether Coral specifically eliminates attribute errors and visual understanding errors (as its design intends) while leaving other error types (category errors, detail errors) largely unchanged, or whether it provides uniform improvement across all error categories. Without this analysis, a practitioner cannot diagnose Coral's remaining weaknesses or prioritize future improvements.

Limitation 5: The Contrastive-Only Baseline Uses Suboptimal Training That Could Partially Close the Gap

The assumption or constraint. The paper compares Coral against a contrastive-only baseline trained for one epoch with a fixed temperature (τ = 0.02), fixed batch size (64), and no hard negative mining, data augmentation, or regularization beyond weight decay. These are reasonable default settings but are not claimed to be optimal for contrastive learning on MERIT. The paper does not report whether the contrastive baseline's performance could be improved through standard techniques like larger batch sizes (which benefit contrastive learning), lower temperatures, hard negative mining, or longer training.

The consequence. The 45.9% improvement attributed to Coral may partially reflect the baseline being undertuned rather than the reconstruction losses providing fundamentally orthogonal benefits. If a well-tuned contrastive baseline achieves, say, 55% R@1 instead of 47.76%, then Coral's "true" improvement from reconstruction would be 26.7% rather than 45.9% — still substantial but quantitatively different. A practitioner comparing Coral against a production contrastive learning pipeline (which would likely include these optimizations) needs to know what fraction of the gain is reconstruction-specific versus simply better training.

More broadly, the paper's central claim — that contrastive-only [EOS] supervision "neglects specific conditional elements" — would be strengthened by showing that the gap persists even when contrastive learning is pushed to its limits. If the gap closes substantially with better contrastive training, the problem may be optimization rather than representation.

What evidence exists in the paper. The ablation in Table 3 compares Coral against exactly one contrastive-only configuration per training regime (LoRA vs. full fine-tuning, sequential vs. concatenated). There is no sweep over contrastive learning hyperparameters, no comparison to contrastive learning with hard negative mining, and no training for more than one epoch. The generalization experiments in Table 10 use the same fixed contrastive baseline configuration. The paper cites prior work (LLaVE, GME, VLM2Vec) that use contrastive learning variants (hardness-weighted, modality-balanced, two-stage training) but does not implement these variants on MERIT as baselines for Coral.

The sequential-vs-concatenated finding partially mitigates this concern. The fact that existing fine-tuned models (GME, LamRA, LLaVE all use different contrastive training recipes) uniformly show degraded sequential performance (Table 2) suggests that the interleaved processing failure is not an artifact of undertuned contrastive baselines — it is a systematic effect of single-image fine-tuning regardless of the specific contrastive recipe. This supports the paper's claim that the problem is representational rather than optimization-based. However, for the specific claim about Coral's improvement magnitude (45.9%), undertuned baselines remain a concern.

Mitigation status. The paper does not acknowledge this limitation or provide ablations over contrastive learning hyperparameters. The consistent gap between Coral and contrastive-only across 8 diverse benchmarks (Figure 9, Table 10) — where the CL baseline is trained with the same protocol — provides some evidence that the gap is not solely an artifact of MERIT-specific undertuning. However, the lack of any hyperparameter optimization for the contrastive baseline weakens confidence in the precise quantitative improvement figure.

Limitation 6: The Masked Reconstruction Design Choices Are Unexplored, and the Mechanism Is Not Isolated

The assumption or constraint. Coral combines several design elements — masked embedding reconstruction, dual reconstruction targets ( $\mathcal{L}_{\text{reg}}$ and $\mathcal{L}_{\text{rec}}$ ), separate vision and language decoders, the [EOS] token as the reconstruction query, position embeddings on the query sequence, BERT-layer decoders, 50% masking ratio, MSE loss for vision and cross-entropy for language — into a single framework. The ablation in Table 3 tests the presence of reconstruction modalities (vision only, language only, both) but does not vary the mechanism of reconstruction.

The consequence. A practitioner cannot determine which aspects of Coral's design are essential and which are incidental. Critical unresolved questions include:

Is masked reconstruction necessary, or would any auxiliary objective that forces information through the [EOS] bottleneck work? For instance, a simpler approach — adding an autoencoding loss that reconstructs the input from the [EOS] embedding without masking — might provide similar regularization with lower complexity. The masking may be important (forcing the model to infer missing information rather than copy) or may be an unnecessary complication.
Is the dual reconstruction ( $\mathcal{L}_{\text{reg}}$ + $\mathcal{L}_{\text{rec}}$ ) necessary? The paper argues that $\mathcal{L}_{\text{reg}}$ teaches query-target binding while $\mathcal{L}_{\text{rec}}$ stabilizes target representations, but no ablation tests Coral with $\mathcal{L}_{\text{reg}}$ only or $\mathcal{L}_{\text{rec}}$ only. If $\mathcal{L}_{\text{reg}}$ alone achieves similar performance, the framework could be simplified.
Does the [EOS] token as reconstruction query matter, or would any learned query vector work? The paper's rationale is that forcing $h_{\text{eos}}$ — the same vector used for retrieval — to also perform reconstruction creates a shared information bottleneck. But if a separate learned query vector were used for reconstruction, the [EOS] representation might still benefit from multi-task learning gradients without the explicit bottleneck constraint.
Is 50% masking optimal? The paper cites masked autoencoding literature for this choice but does not ablate alternative ratios. A 30% or 70% masking ratio might change the difficulty of the reconstruction task and thus the strength of the regularization signal.

What evidence exists in the paper. Table 3 ablates the presence/absence of vision and language reconstruction and LoRA vs. full fine-tuning, but does not ablate mechanism variants. There is no comparison to autoencoding without masking, no test of $\mathcal{L}_{\text{reg}}$ -only or $\mathcal{L}_{\text{rec}}$ -only, no alternative reconstruction query design, and no masking ratio sensitivity analysis. The paper's theoretical framing of masked reconstruction as an information bottleneck regularizer is plausible but is not experimentally distinguished from simpler regularizers (dropout, weight decay, data augmentation) that might produce similar benefits.

Mitigation status. The paper does not acknowledge this as a limitation or suggest mechanism ablations as future work. The consistent improvement of Coral over CL baselines across 8 benchmarks (Figure 9) provides some robustness evidence — the design works across diverse tasks — but does not isolate why it works. For a practitioner deciding whether to implement Coral versus a simpler alternative (e.g., adding an autoencoding loss), the lack of mechanism ablations means the decision must be made on faith in the full Coral recipe rather than on evidence about which components are load-bearing.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reshapes the multimodal retrieval landscape by identifying a systematic failure mode that the field had overlooked: contrastive learning with [EOS]-only supervision causes pre-trained MLLMs to discard the very per-condition, per-attribute detail that multi-condition retrieval requires. This is not an incremental performance observation — it is a diagnostic insight that reframes why retrieval models fail and what kind of solution is needed.

The conceptual shift: from discriminative tuning to information-preserving tuning. Before this work, the dominant paradigm for adapting MLLMs to retrieval — represented by LLaVE, GME, VLM2Vec, LamRA, and others — treated fine-tuning as a straightforward exercise in arranging the embedding space. The implicit assumption was that a pre-trained MLLM already encodes all relevant information about its input, so fine-tuning merely needs to make matching pairs close and non-matching pairs far apart. Coral demonstrates that this assumption is false for multi-condition queries, and likely overly optimistic even for single-condition retrieval. The [EOS] token — the sole supervision point for contrastive learning — is an information bottleneck that contrastive objectives actively compress, discarding details that are not discriminative at the batch level. When those details include which image provides which attribute, the model's ability to correctly compose multi-condition queries collapses.

This diagnosis reframes retrieval fine-tuning as an information preservation problem, not just a metric learning problem. The goal is not only to separate queries in embedding space but to ensure the embedding retains the internal structure of what each query means — which attributes come from which sources, how they compose, and what fine-grained details distinguish a near-match from an exact match. This is a conceptual departure from the field's focus on improving contrastive objectives (hard negatives, better temperatures, larger batches) toward reconsidering what information the embedding should contain.

The practical shift: reconstruction as a fine-tuning regularizer, not just a pre-training scaffold. Masked autoencoding (BERT, MAE, RetroMAE) is well-established as a pre-training technique — models learn general representations by reconstructing corrupted inputs, then switch to task-specific objectives during fine-tuning. Coral's key move is to reintroduce reconstruction during fine-tuning, alongside the contrastive objective. This repurposes reconstruction from a pre-training scaffold (discarded when downstream training begins) to an active regularizer that prevents fine-tuning from destroying representational richness.

This has implications beyond retrieval. Any fine-tuning scenario where a discriminative objective is applied to a bottleneck representation risks information loss. Coral's principle — use reconstruction to counteract compression — may generalize to few-shot learning (where limited data makes over-compression likely), continual learning (where new tasks could overwrite old representational structure), or domain adaptation (where domain-specific details might be discarded as irrelevant to the source domain's discriminative objective). The paper's consistent gains across 8 diverse benchmarks — from visual dialogue (VisDial: 51% → 73% R@1) to composed image retrieval (CIRR: 39% → 50% R@1) to multimodal QA (WebQA: 67% → 84% R@1) — suggest that information loss during fine-tuning is widespread, not specific to MERIT's multi-condition setting.

Reconciling contradictory prior findings. The paper's finding that fine-tuned retrieval models perform better with concatenated multi-image input than with sequential interleaved input (Table 2: GME-Qwen2VL-2B shows 47.13% vs. 24.18% R@5) — while pre-trained MLLMs show the opposite pattern — resolves a puzzling discrepancy in the literature. Prior work on visual comprehension (VQA, visual dialogue) had shown that interleaved sequential processing preserves more information than image concatenation. Yet retrieval models consistently used concatenation and achieved strong results. The paper reveals that this was not because concatenation is inherently better for retrieval — it was because existing retrieval fine-tuning on single-image datasets had destroyed the pre-trained model's interleaved processing capability, making concatenation the only viable option. This explains why the field converged on concatenation without recognizing it as a symptom of a deeper problem.

This finding also reconciles tensions between the MLLM community (which develops models with sophisticated interleaved processing capabilities) and the retrieval community (which rarely uses those capabilities after fine-tuning). The paper shows that these communities are not working on incompatible assumptions — the pre-trained capabilities are real, but they are being inadvertently stripped away by current fine-tuning practices. Coral demonstrates that with appropriate training data and objectives, the pre-trained capabilities can be preserved and enhanced for retrieval.

Research directions that become more attractive. The paper makes three research directions particularly compelling:

Auxiliary reconstruction during task-specific fine-tuning, not just as a pre-training objective. The success of Coral on 8 diverse benchmarks suggests this principle may generalize broadly. Researchers in few-shot learning, domain adaptation, and continual learning should investigate whether their fine-tuning protocols also induce representational collapse that reconstruction could prevent.
Benchmark-driven capability diagnosis. The paper demonstrates that a well-designed benchmark (MERIT) can surface systematic failures (lost interleaving capability) that standard benchmarks miss. This argues for more capability-probing evaluation suites that test for representational degradation, not just task accuracy. A retrieval model that achieves 90% on single-image benchmarks may have silently lost its ability to process multiple images — a capability that matters for real-world deployment but is invisible on standard evaluations.
Information-theoretic analysis of fine-tuning dynamics. The paper's framing of the [EOS] token as an information bottleneck suggests that fine-tuning could be studied through the lens of the information bottleneck principle (Tishby et al., 1999). How much mutual information between the input and the representation survives contrastive fine-tuning? Does reconstruction explicitly increase this mutual information? Can we predict which capabilities will be lost during fine-tuning based on the training objective's information requirements?

Research directions that become less attractive. The paper's findings cast doubt on:

Purely discriminative fine-tuning for retrieval. If contrastive learning alone causes systematic information loss, research focused exclusively on improving contrastive objectives (better negative mining, more sophisticated temperature schedules, larger batches) without addressing the representational bottleneck may hit diminishing returns. The 45.9% improvement from reconstruction is substantially larger than what could reasonably be expected from further contrastive tuning.
Image concatenation as a general strategy for multi-image retrieval. The paper shows that concatenation is a workaround for a training-induced capability loss, not a principled design choice. Once models are trained appropriately (on interleaved data with reconstruction), sequential interleaved input substantially outperforms concatenation (69.68% vs. 60.94% R@1 for Coral, Table 3). Future work should prioritize sequential interleaved architectures and training recipes rather than optimizing concatenation pipelines.
Treating retrieval fine-tuning as a solved problem. Prior to this work, the field's main concern was scaling — bigger models, more data, larger batches. MERIT's baseline results (best existing model: 12.05% R@1) and Coral's 69.68% R@1 demonstrate that multi-condition retrieval is far from solved and that architectural and objective-level innovations can produce gains that scaling alone would struggle to match. The 5.78× gap between the best prior model and Coral on MERIT suggests that the field has been optimizing the wrong things.

Follow-Up Research This Work Enables

Cheap difficulty estimation for adaptive retrieval allocation on MERIT. The paper's difficulty binning — used to stratify model performance by query complexity in Section 3.4 — relies on generating 2,048 samples per query and computing pass rates, a procedure so expensive it dwarfs the retrieval budget itself. The paper acknowledges this as a limitation. A natural follow-up would train a lightweight difficulty predictor that takes only the query text and condition images as input and predicts the difficulty quintile. The training data already exists: MERIT's 10,000 test queries with their associated difficulty bins. A small model (e.g., a fine-tuned BERT or a linear probe on frozen MLLM features) could be trained to predict bin membership from query features, with the goal of achieving difficulty estimation at negligible cost (a single forward pass rather than 2,048 generations). A strong result would be a predictor that maintains the separation between difficulty bins seen in Figure 7a (where performance across languages shows clear stratification) while adding less than 1% overhead to the total retrieval budget. This would make adaptive retrieval allocation — where easy queries get simple strategies and hard queries get Coral's full reconstruction-enhanced pipeline — practically deployable.

Coral trained on MERIT, evaluated zero-shot on standard retrieval benchmarks. The paper demonstrates Coral's effectiveness when trained and evaluated on the same benchmark (MERIT for Table 3, individual benchmarks for Figure 9). It does not test whether training on multi-condition interleaved data produces generally better representations that transfer to single-condition tasks. A straightforward experiment would train Coral on MERIT (310,000 multi-condition queries) and evaluate zero-shot on CIRR, FashionIQ, VisDial, and COCO without any additional fine-tuning on those benchmarks. If Coral trained on MERIT outperforms the contrastive-only baseline trained on the same MERIT data when evaluated on single-image benchmarks, this would demonstrate that multi-condition training produces richer, more generalizable visual-linguistic representations. A negative result — where MERIT-trained Coral underperforms models trained on the target benchmarks — would indicate that the reconstruction benefit is task-specific rather than representation-general, which would refine our understanding of when reconstruction helps.

Error analysis comparing Coral vs. contrastive-only failure modes. The paper's error analysis (Section 3.4, Figure 7b) categorizes failures of contrastive-only models into attribute errors, visual understanding errors, category errors, detail errors, and annotation errors — with the first two dominating. The paper argues that Coral's reconstruction objective specifically addresses these error types by forcing the [EOS] embedding to retain per-condition detail. However, no error analysis is reported for Coral — we do not know whether Coral's remaining failures (30.32% of queries at R@1) are qualitatively different from the contrastive-only failures. A follow-up study would replicate the error categorization protocol on 500 Coral-retrieved queries and compare the error distribution to Figure 7b. The key hypotheses: (1) attribute errors and visual understanding errors should decrease substantially as a fraction of total errors, since these are the categories Coral's design targets; (2) the remaining errors may shift toward detail errors and hard negatives that are genuinely ambiguous even for humans, which would establish a performance ceiling for current architectures; (3) if attribute errors remain a large fraction of Coral's failures, this would indicate that the reconstruction objective does not fully solve the binding problem, and more sophisticated compositional architectures may be needed. This experiment would provide the causal evidence currently missing from the paper — directly testing whether reconstruction solves the specific problems it was designed to address.

Mechanism isolation: Is masked reconstruction necessary, or would any auxiliary objective work? Coral combines multiple design elements — masked embeddings, dual reconstruction targets, the [EOS] token as query, BERT-layer decoders, 50% masking — into a single package. The paper's ablation (Table 3) tests the presence/absence of reconstruction modalities but does not isolate the mechanism. A systematic follow-up would ablate:

Masking vs. no masking: Train Coral with standard autoencoding (reconstruct full input from [EOS], no masking) vs. masked reconstruction (50% masking). If unmasked reconstruction achieves similar performance, the masking mechanism is unnecessary complexity — any auxiliary loss that forces information through the bottleneck suffices.
$\mathcal{L}_{\text{reg}}$ only vs. $\mathcal{L}_{\text{rec}}$ only vs. both: Train Coral with only the cross-instance reconstruction ( $\mathcal{L}_{\text{reg}}$ , where the query reconstructs the target), only the self-reconstruction ( $\mathcal{L}_{\text{rec}}$ , where the target reconstructs itself), and both (the full Coral recipe). If $\mathcal{L}_{\text{reg}}$ alone achieves similar performance to full Coral, the self-reconstruction term is unnecessary, and the framework can be simplified.
Alternative reconstruction queries: Replace the [EOS] token as the reconstruction query with a separate learned query vector that does not participate in contrastive learning. If this performs comparably to Coral, the "shared information bottleneck" rationale is not load-bearing — multi-task learning alone provides the benefit, and the [EOS] representation is not actually being regularized.

A strong negative result on any of these ablations — showing that a specific Coral component is essential — would both validate the paper's theoretical framing and guide practitioners on which parts of the framework to prioritize.

Coral on additional MLLM architectures and non-English retrieval benchmarks. The paper's Coral experiments use Qwen2.5-VL-3B and Qwen2-VL-2B exclusively. A follow-up study would replicate Coral on InternVL2.5 (which has a different vision encoder and LLM backbone), LLaVA-NeXT (which uses a different interleaved processing architecture), and DeepSeek-VL (which uses a different pre-training objective). The key question: is the 45.9% improvement on MERIT specific to the Qwen architecture, or does it reflect a general principle that applies across model families? If Coral consistently provides ~40-50% improvement across architectures, this would establish reconstruction as a universal fine-tuning regularizer for MLLM-based retrieval. If the improvement varies substantially by architecture, the mechanism may depend on specific properties of the MLLM's representational structure (e.g., how visual and linguistic features interact in intermediate layers), which would both constrain Coral's applicability and provide insight into why reconstruction helps.

Additionally, evaluating Coral on non-English retrieval benchmarks (e.g., Chinese, Arabic, or the Southeast Asian languages in MERIT's own test set) would test whether the reconstruction benefit transfers across languages. The paper's language OOD results (Table 7) show substantial degradation when models are evaluated on languages absent from training, but Coral's per-language performance is not reported. If Coral's improvement is consistent across languages, reconstruction is likely preserving language-agnostic visual-attribute information. If the improvement varies by language — particularly favoring English, the dominant pre-training language — then reconstruction may be amplifying pre-training biases rather than counteracting them.

Scaling analysis: Does Coral's benefit grow or shrink with model scale and data size? The paper evaluates Coral at 2B and 3B parameter scales with a fixed training set of 310,000 queries. A scaling study would train Coral at multiple model sizes (0.5B, 2B, 7B, perhaps larger) and multiple dataset sizes (by subsampling MERIT) to characterize how the reconstruction benefit scales. The key hypotheses: (1) reconstruction may provide larger relative gains at smaller scales (where models have less capacity to maintain per-condition detail and benefit more from explicit regularization); (2) reconstruction may provide larger relative gains with smaller training sets (where contrastive learning is more prone to overfitting and representational collapse); (3) the optimal loss weights ( $\lambda_1$ , $\lambda_2$ ) may depend on model scale and dataset size, requiring tuning rather than the fixed 0.1 used in the paper. A scaling law characterizing how Coral's benefit varies with compute would both guide practitioners on when to use it and provide theoretical insight into the interaction between model capacity, data quantity, and information preservation.

Practical Applications and Downstream Use Cases

Multilingual e-commerce product search across Southeast Asian markets. MERIT was explicitly constructed from real e-commerce data across Indonesia, Philippines, Thailand, Malaysia, Vietnam, and Singapore, with queries in five languages. Coral's 69.68% R@1 and 89.26% R@5 on MERIT — compared to 12.05% R@1 for the best existing model — represents a practical capability gap that directly impacts commercial search systems. In a production deployment where users specify products through multiple reference images ("find me a shirt with this pattern but in that color"), Coral could increase the rate at which the correct product appears in the top 5 results from roughly 40% (best existing model, LamRA-7B concatenated R@5: 39.13%) to approximately 89% (Coral-3B sequential R@5: 89.26%). For an e-commerce platform handling millions of queries daily, this 2.2× improvement in top-5 recall would translate to substantial reductions in null-result searches and downstream revenue impact. Critically, the cross-lingual performance in Table 6 shows that retrieval accuracy does not substantially degrade for non-English languages (Indonesian: 56.23%, Thai: 55.34%, Vietnamese: 55.13% for the contrastive-only baseline — Coral's per-language performance is not reported but the consistent OOD improvement pattern in Tables 7-9 suggests the reconstruction benefit should transfer). This makes Coral deployable in the linguistically diverse markets that MERIT represents, without requiring separate models per language.

Visual dialogue and composed image retrieval systems. Coral's 181% improvement over the zero-shot baseline on VisDial (13% → 73% R@1, Table 10) and 28% improvement over the contrastive baseline on CIRR (39% → 50% R@1) make it immediately applicable to two established application domains. Visual dialogue systems — where users iteratively ask questions about images and the system must retrieve relevant visual content — benefit from Coral's ability to maintain fine-grained visual detail in the query representation across multi-turn interactions. Composed image retrieval — where users modify a reference image with natural language instructions ("the same dress but in red, with longer sleeves") — benefits from Coral's cross-modal binding capability, which ensures the text modifier is correctly applied to the appropriate visual attribute. In both cases, Coral's design principle (reconstruction preserves conditional detail) directly addresses the core task requirement (mapping composite queries to precise visual targets), and the strong results on these benchmarks suggest that practitioners can adopt Coral as a drop-in replacement for contrastive-only fine-tuning with minimal architectural changes — only the training objective changes, with identical inference cost.

Data generation for self-improving retrieval systems. The paper's finding that Coral restores pre-trained MLLMs' interleaved processing capability (sequential outperforming concatenation after MERIT training, Table 3) has implications for retrieval-augmented generation and self-improving retrieval pipelines. In these systems, a retrieval model surfaces candidate documents or products, and the retrieved items are used to augment generation or to train improved retrieval models. If the retrieval model loses its ability to process interleaved multi-modal inputs during fine-tuning — as the paper shows happens with standard contrastive training — subsequent stages in the pipeline inherit this degradation. By preserving representational richness, Coral-trained retrievers produce embeddings that are more informative for downstream models, potentially improving the entire pipeline's performance. This is particularly relevant for retrieval-augmented generation (RAG) systems that use retrieved images and text to condition generation, where fine-grained visual-attribute information in the retrieval embedding could improve the relevance and specificity of generated content. The paper does not test this directly, but the 84% R@1 on WebQA (a multimodal QA benchmark where retrieval quality directly impacts answer accuracy) suggests that Coral's improved retrieval translates to better downstream task performance.

Training cost amortization for large-scale one-time fine-tuning. Coral adds training complexity (additional decoders, masked forward passes, multiple loss terms) but zero inference overhead — the reconstruction decoders are not used at inference time, and the [EOS] embedding is the same dimensionality as contrastive-only models. For organizations that fine-tune a retrieval model once and serve millions of inference requests — a common pattern in production search systems — the additional training cost is amortized over the model's entire deployment lifetime. If Coral's 45.9% relative improvement in R@1 on MERIT translates to even a 10-20% improvement in downstream business metrics (conversion rate, user satisfaction, reduction in null results), the training cost becomes negligible compared to the operational benefit. The paper does not report training time or GPU-hour costs, so practitioners would need to benchmark this on their own infrastructure, but the architecture (single BERT layer per modality as decoder, 50% masking on embeddings rather than raw pixels, one epoch of training) is computationally modest relative to the MLLM backbone. For organizations already investing in contrastive fine-tuning of large MLLMs for retrieval — where the dominant cost is the LLM's forward and backward passes — adding the reconstruction objectives likely represents a fractional increase in total training compute.

When to Prefer This Method

The paper positions Coral as a fine-tuning framework for adapting pre-trained MLLMs into retrieval models specifically when multi-condition interleaved queries are involved, but the generalization results suggest broader applicability. The decision boundaries are:

Prefer Coral over standard contrastive-only fine-tuning when:

The retrieval task involves multiple images or multiple conditions per query. This is Coral's primary design target, and the 45.9% improvement on MERIT (Table 3) versus 47.76% for contrastive-only directly measures the benefit. If queries routinely interleave 2+ images with text specifying different attributes, standard contrastive fine-tuning will likely cause the information loss that Coral prevents.
The base MLLM supports interleaved image inputs and you want to preserve this capability after fine-tuning. The paper demonstrates that standard fine-tuning on single-image datasets destroys interleaved processing (Table 2: 119.7% concatenation advantage for GME-Qwen2VL), while Coral restores it (Table 3: sequential 69.68% vs. concatenated 60.94% R@1). If downstream deployment requires sequential multi-image processing, Coral is currently the only demonstrated method that preserves this capability.
The retrieval task requires fine-grained attribute understanding, even for single-image queries. The consistent improvement across 8 diverse benchmarks — including single-image tasks like CIRR and COCO — suggests that reconstruction benefits any retrieval setting where distinguishing near-matches from exact matches requires retaining attribute-level detail. If your retrieval task has hard negatives that differ only in specific attributes (color, pattern, material), Coral's regularizing effect should help.
You have the training compute budget for additional loss terms and decoder modules. Coral requires two additional BERT-layer decoders and masked forward passes during training. The paper does not quantify this overhead, but the architecture is lightweight relative to the MLLM backbone. If training is a one-time cost amortized over many inference requests, the overhead is likely acceptable. For rapid experimentation cycles where training time is the bottleneck, the additional complexity may slow iteration.
Full parameter fine-tuning is feasible. Coral's full benefit requires full fine-tuning (69.68% R@1) rather than LoRA (59.40% R@1), a 10.28 percentage point gap. If hardware constraints restrict you to parameter-efficient methods, Coral still provides gains over contrastive-only LoRA (59.40% vs. 48.52% R@1), but the improvement is more modest (22.4% relative vs. 45.9% for full fine-tuning).

Prefer standard contrastive-only fine-tuning when:

The retrieval task is strictly single-image, single-condition, and performance is already near ceiling on existing benchmarks. If your task is standard text-to-image retrieval on COCO and you already achieve strong results with contrastive-only fine-tuning, the additional complexity of Coral may not justify the implementation effort. The improvement on COCO T2I is 21.4% relative (56% → 68% R@1, Table 10) — meaningful but smaller than on tasks like VisDial (43.1% relative). Whether this gap justifies adopting Coral depends on your performance requirements and engineering resources.
Training iteration speed is the primary constraint. Coral requires implementing masked attention, dual reconstruction decoders, and multi-term loss balancing. If you need to rapidly experiment with data mixtures, model architectures, or hyperparameters, the added complexity may slow your iteration cycle enough to outweigh the performance gains. The paper does not report training time comparisons, so practitioners would need to benchmark the overhead on their specific infrastructure.
You cannot afford full parameter fine-tuning and must use LoRA. While Coral with LoRA still outperforms contrastive-only LoRA (59.40% vs. 48.52% R@1, Table 3), the gap narrows substantially. If your compute budget restricts you to LoRA, the 22.4% relative improvement may or may not justify the additional implementation complexity, depending on your performance sensitivity.
The target domain is fundamentally different from MERIT's product retrieval setting, and you lack multi-condition training data. Coral's reconstruction objectives require training pairs (query + target product) to compute $\mathcal{L}_{\text{reg}}$ (cross-instance reconstruction). If your domain has only single-condition queries but you want to deploy a model that can handle multi-condition queries at inference time, Coral cannot be trained without appropriate data. The paper does not test Coral's transfer from MERIT to other domains.