Scaling Language-Centric Omnimodal Representation Learning

ArXiv: 2510.11693

🎯 Pitch

Multimodal LLMs don't just generate text—their pretraining secretly encodes cross-modal alignment so powerful that text-only contrastive fine-tuning boosts image, audio, and video retrieval alike. This hidden alignment obeys a scaling law: better generative models hit higher representation ceilings, no huge multimodal contrastive data needed.

---

1. Executive Summary

This paper introduces a Language-Centric Omnimodal Embedding framework, termed LCO-EMB, that leverages the implicit cross-modal alignment established during MLLM generative pretraining—where the language decoder learns to exploit multimodal signals within a shared representation space—to achieve state-of-the-art multimodal representation quality using primarily text-only contrastive learning as a lightweight, post-hoc refinement stage. Through systematic analysis of anisotropy and kernel-level similarity on Qwen2.5-Omni-3B and Qwen2.5-VL models, the authors demonstrate that text-only contrastive fine-tuning with LoRA not only disperses collapsed textual representations but generalizes to improve isotropy in image, audio, and video embeddings, establishing that latent language-centric alignment is an inherent property of MLLMs. The paper further identifies a Generation-Representation Scaling Law (GRSL), formalized through a PAC-Bayesian generalization bound showing that an MLLM's generative quality determines an upper bound on its representational potential—a finding validated on the newly introduced SeaDoc benchmark for low-resource Southeast Asian visual document retrieval, where continual generative pretraining before contrastive learning improves retrieval performance. On the MIEB-Lite benchmark, LCO-EMB's multimodal variant trained on only ~370k triplets—approximately 21× less data than the prior state-of-the-art GME (~8M)—achieves 67.6% (7B VL) and 68.8% (7B Omni) average scores, outperforming all existing open-source and proprietary models and establishing that generative pretraining quality, not contrastive data scale, is the central driver of multimodal representation capability.

2. Context and Motivation

The Core Problem: Understanding Why MLLM-Based Embeddings Work

The central puzzle driving this paper is deceptively simple: multimodal large language model (MLLM)-based embedding approaches consistently outperform traditional CLIP-style models on complex representation tasks, but nobody knows why. The empirical observation is well-documented—recent models like E5-V, VLM2Vec, mmE5, and GME that fine-tune MLLMs with contrastive learning achieve state-of-the-art results on benchmarks like MIEB—but the underlying mechanism remains a black box. The paper frames this as a "critical research gap" (Section 1):

"However, the underlying reasons for the performance advantages of MLLM-based embedding approaches over traditional CLIP-based ones remain underexplored."

This is not merely a question of scientific curiosity. If the community doesn't understand why MLLM-based methods work, it cannot systematically improve them. Researchers are left to iterate through trial and error—trying different backbones, training recipes, and data mixtures—without a guiding principle. The paper argues that closing this explanatory gap is a prerequisite for principled progress in multimodal representation learning.

The problem is made more acute by an apparent paradox in the empirical landscape. MLLM-based approaches achieve compelling results with surprisingly little multimodal contrastive data. For instance, E5-V uses only ~276k text-only training pairs yet generalizes to image tasks. The paper's own LCO-EMB text-only variant (T) outperforms the proprietary Voyage Multimodal 3 on MIEB-Lite using the same ~276k text-only pairs—no images, no cross-modal pairs, just sentence-level entailment/contradiction triplets from NLI datasets. This makes no sense under the traditional CLIP paradigm, where cross-modal alignment is understood to emerge from massive-scale paired contrastive pretraining over hundreds of millions of image-text pairs. If contrastive learning against cross-modal pairs is what creates alignment, how can a model trained exclusively on text generalize to images, audio, and video?

The paper's central hypothesis—and what it sets out to empirically validate—is that generative pretraining, not contrastive learning, is where cross-modal alignment is actually established. In the MLLM architecture, the language decoder must learn to attend to and integrate multimodal signals (visual features from a vision encoder projected into the LLM's embedding space) to generate the correct textual output. This generative objective implicitly forces multimodal representations into a shared latent subspace within the decoder. Contrastive learning, in this view, does not create alignment from scratch—it merely refines and activates alignment that is already latent in the pretrained weights.

---

Why This Problem Matters: Practical and Theoretical Stakes

Practical Impact: The Economics of Multimodal Embedding Training

If the paper's hypothesis is correct—that generative pretraining quality, not contrastive data volume, is the primary driver of representation capability—the implications for how researchers and practitioners allocate compute budgets are profound. The current dominant paradigm, established by CLIP and its successors, treats contrastive learning as computationally intensive alignment training that must scale with data volume. CLIP-style models require hundreds of millions of image-text pairs, large batch sizes (often 32k+), and careful engineering to prevent training instability. This creates a barrier to entry: only organizations with access to massive multimodal datasets and substantial compute clusters can train competitive multimodal embedding models.

The paper's approach, by contrast, effectively decouples alignment from contrastive training. If alignment is already baked into the MLLM's pretrained weights, then contrastive learning can be lightweight—operating on small text-only datasets, using parameter-efficient methods like LoRA, and requiring only a fraction of the compute. The paper's experiments bear this out: text-only LoRA fine-tuning on ~276k NLI pairs takes approximately 4.7 GPU hours on H20 GPUs (Table 3), while a standard CLIP-style contrastive training run on 800K image-caption pairs takes ~453 GPU hours—a nearly 100× reduction in training cost while achieving superior performance.

For practitioners, this means:

• Lower data requirements: Text-only NLI datasets (SNLI, MNLI) are publicly available, well-studied, and require no multimodal collection pipelines. The paper's Scale-1M dataset is constructed from existing multilingual parallel corpora, again entirely text-based.
• Lower compute barriers: Training competitive multimodal embedding models becomes feasible on a single GPU-day rather than requiring distributed training clusters.
• Faster iteration: The lightweight nature of text-only LoRA fine-tuning enables rapid experimentation with different training configurations, backbones, and datasets.

Theoretical Significance: Reconceptualizing the Role of Contrastive Learning

Beyond the practical economics, the paper challenges a fundamental assumption in multimodal representation learning: that contrastive learning is the mechanism of cross-modal alignment. Under the dominant CLIP paradigm, a vision encoder and a text encoder are trained from scratch (or from unimodal pretrained initialization) with a contrastive objective that pulls matching image-text pairs together and pushes non-matching pairs apart. Alignment is the product of this training process.

The paper argues for a different view, which it formalizes in Section 3:

"CL can function as a lightweight, post-hoc refinement step for mapping pre-aligned generative embeddings into a similarity-matching space in MLLMs, which differs sharply from the computationally intensive CL required by CLIPs for alignment."

In this reconceptualization, contrastive learning serves three distinct functions that are more modest than the traditional view implies:

1. Dispersion (isotropy enhancement): Mitigating the representation degeneration problem where embeddings collapse into a narrow cone of the representation space (high anisotropy). This is a geometric correction, not semantic alignment.
2. Similarity projection: Mapping the pretrained generative representations—which are optimized for next-token prediction, not similarity matching—into a space where cosine similarity corresponds to semantic relatedness.
3. Discriminability sharpening: Enhancing the model's ability to distinguish between semantically similar but distinct inputs, which is exactly what NLI-style entailment/contradiction training provides.

This reframing has a corollary that the paper makes explicit: the quality ceiling is set by the pretrained generative model, not by the contrastive training process. No amount of contrastive fine-tuning can compensate for a weak generative backbone. This is the insight that the Generation-Representation Scaling Law (GRSL) formalizes.

Bridging Conflicting Observations

The paper's framework also helps reconcile seemingly contradictory observations in the literature. On one hand, CLIP-style models achieve remarkable zero-shot classification and linear probing performance, suggesting they learn robust visual representations. On the other hand, they plateau on tasks requiring deeper cross-modal comprehension—the paper explicitly cites "multilingual image retrieval, visual text representations, and tasks involving interleaved multimodal encodings" as areas where CLIP performance stagnates despite scaling (Section 1). The MIEB benchmark study by Xiao et al. (2025) provides systematic evidence for this plateau across 50 models and 130 tasks.

Why does this plateau exist? The paper's framework offers an explanation: CLIP-style models learn shallow cross-modal alignment through contrastive loss alone, optimizing for global image-text correspondence rather than fine-grained multimodal integration. MLLMs, by contrast, develop deep cross-modal alignment through the generative objective, where the decoder must attend to specific visual features (text in images, spatial relationships, compositional elements) to produce the correct token sequence. This deeper alignment manifests in precisely the task categories where CLIPs underperform—multilingual image retrieval (which requires the model to understand that the same visual concept can be described in different languages), visual STS (which requires fine-grained semantic judgment of text rendered as images), and document understanding (which requires reading and integrating textual content with visual layout).

---

Prior Approaches and Their Shortcomings

The paper situates itself against a rich landscape of prior work, identifying specific limitations that motivate the LCO-EMB approach.

CLIP-Style Models: The Dominant Paradigm Reaches Its Limits

The CLIP paradigm (Radford et al., 2021) and its derivatives—SigLIP (Zhai et al., 2023), EVA-CLIP (Sun et al., 2023), BLIP (Li et al., 2022)—constitute the dominant approach to multimodal representation learning. The recipe is well-known: joint training of separate vision and text encoders with a contrastive objective over massive paired datasets. Scaling efforts have focused on three axes: model size, dataset volume, and batch size.

The paper acknowledges CLIP's strengths: these models excel at linear probing and zero-shot classification, demonstrating that contrastive pretraining produces transferable visual features. However, the paper identifies specific failure modes that scaling alone does not address:

Failure 1: Plateauing on complex reasoning and interleaved tasks. The MIEB benchmark reveals that CLIP-style models underperform dramatically on tasks requiring reasoning over interleaved inputs—for instance, VQA-style tasks where a question refers to specific visual elements that must be localized and reasoned about. Traditional CLIP encodes images and text into separate global vectors; it has no mechanism for fine-grained cross-modal attention, which limits its capacity for compositional understanding.

Failure 2: Multilingual retrieval weakness. CLIP is predominantly trained on English image-text pairs. While some variants incorporate multilingual data, the contrastive objective only aligns images with the languages seen during training. An image aligned with the English caption "a dog playing in the park" has no implicit alignment with the Vietnamese equivalent "một con chó đang chơi trong công viên." MLLMs, by contrast, can learn that these linguistic expressions refer to the same visual concept through the decoder's multilingual generative pretraining.

Failure 3: Visual text blindness. CLIP-style vision encoders are not explicitly trained to read text in images. They may learn to associate visual patterns of text with their semantic content through co-occurrence statistics, but this is brittle. For document understanding tasks—where the model must extract structured information from text-heavy images—CLIP representations are fundamentally limited. The paper cites ColPali (Faysse et al., 2025) and Vidore (2024) as benchmarks that explicitly test this capability, and CLIP-style models consistently underperform.

Failure 4: Anisotropy as an architectural limitation. Section 2.1 documents that vanilla MLLM representations—like those from untuned language models (Ethayarajh, 2019; Gao et al., 2019)—suffer from anisotropy: hidden states collapse into a narrow region of the representation space, resulting in high expected cosine similarity between random inputs. This is not unique to MLLMs; it is a well-known property of self-supervised language models. However, CLIP-style models mitigate this through contrastive training's uniformity property (Wang and Isola, 2020), which explicitly pushes random pairs apart. The paper's innovation is showing that this uniformity can be achieved through text-only contrastive learning that generalizes across modalities—something that would be impossible if alignment were not already latent.

MLLM-Based Embedding Models: Promising but Unexplained

A wave of recent work has demonstrated that fine-tuning MLLMs with contrastive objectives produces superior multimodal embeddings. The paper references several key approaches:

• E5-V (Jiang et al., 2024): Fine-tunes an MLLM's language decoder with text-only contrastive learning on NLI data, achieving strong image-text retrieval performance. However, E5-V's paper does not provide a systematic investigation of why text-only training generalizes to vision. The authors treat it as an empirical finding without mechanistic explanation.

• VLM2Vec (Jiang et al., 2025): Trains vision-language models on massive multimodal embedding tasks using a combination of image-text pairs. While effective, this approach essentially replicates the CLIP paradigm using MLLM backbones—it does not exploit the latent alignment hypothesis.

• mmE5 (Chen et al., 2025) and GME (Zhang et al., 2024): Both leverage synthetic multimodal data at scale (2.1M and 8M pairs respectively) to fine-tune MLLMs for embedding tasks. GME, the prior state-of-the-art on MIEB, uses ~8M training pairs. While these methods achieve strong results, they are data-intensive and do not explicitly investigate the role of generative pretraining in their performance.

The paper argues that these approaches collectively demonstrate that MLLM-based embeddings work, but fail to answer why they work or how to optimize them systematically. This gap is the paper's primary target:

"This represents a critical research gap in understanding the limitations of CLIP-style models and the specific strengths MLLMs bring to these challenging scenarios." (Section 1)

Omnimodal Representation Learning: Piecemeal Approaches

The paper also contextualizes its work within the broader omnimodal (or "any-to-any") representation learning landscape:

• ImageBind (Girdhar et al., 2023): Uses the image modality as an anchor for contrastive alignment with all other modalities (text, audio, depth, thermal, IMU). While pioneering, this approach requires carefully curated cross-modal pairs for each modality pair and treats alignment as something that must be explicitly learned through contrastive training.

• One-PEACE (Wang et al., 2023): Explores a unified representation model across unlimited modalities but similarly relies on large-scale cross-modal contrastive pretraining.

The paper's position is that these modality-centric approaches—whether language-centric, vision-centric, or anchor-based—miss the key insight: a sufficiently powerful generative MLLM already contains cross-modal alignment in its pretrained representations. The task is not to create alignment but to activate and refine it.

Web-SSL and Vision-Centric Learning: A Complementary Perspective

The paper briefly references Web-SSL (Fan et al., 2025) and DINOv2 (Oquab et al., 2024) as examples of vision-centric representation learning that achieve strong results without language supervision. These approaches demonstrate that visual representations can capture OCR-level text understanding through pure self-supervised learning on images—challenging the assumption that text supervision is necessary for visual text understanding. The paper uses this as supporting evidence that alignment can emerge from objectives other than explicit cross-modal contrastive learning, reinforcing its central thesis.

---

How This Paper Positions Itself: From Explanatory Analysis to Method to Theory

The paper structures its contribution in three layers, each building on the previous:

Layer 1: Empirical diagnosis (Section 2). Before proposing any method, the paper conducts a systematic analysis of MLLM representations using two geometric tools:

• Anisotropy measurement (Section 2.1): Quantifying the degree of representation collapse before and after text-only contrastive learning, across text, image, audio, and video modalities.
• Kernel-level similarity (Section 2.2): Measuring the structural alignment between vision and language similarity kernels within the same MLLM, across layers and model sizes.

These analyses are not merely descriptive—they serve as a diagnostic to test the latent alignment hypothesis. If text-only contrastive learning improves isotropy for non-text modalities, that is strong evidence that the decoder's representation space is shared across modalities. If larger models exhibit stronger cross-modal kernel alignment, that supports the claim that generative capability scales with latent alignment quality.

Layer 2: Method grounded in insight (Sections 3–4). Building on the diagnostic findings, the paper proposes LCO-EMB as a framework that deliberately preserves the latent cross-modal alignment established during generative pretraining. The design choices—LoRA instead of full fine-tuning, text-only data as the primary training signal, minimal multimodal data for downstream calibration—are direct consequences of the diagnostic insight.

The paper explicitly contrasts this with the alternative: if one believes alignment must be created from scratch, one would use full fine-tuning with massive multimodal datasets (CLIP-style). If one believes alignment is already latent and just needs activation, one would use lightweight, parameter-efficient methods and minimal data—exactly what LCO-EMB does.

Layer 3: Theoretical formalization (Section 5). The paper's most ambitious contribution is the Generation-Representation Scaling Law (GRSL) and its PAC-Bayesian formalization. This moves beyond empirical observation to a theoretical claim: the generative quality of the MLLM prior sets a hard upper bound on the representational quality achievable through any amount of contrastive fine-tuning.

The bound (Theorem 1) decomposes the expected population contrastive risk into three terms:

1. The Generative Bottleneck (log N − I_P(X; Y)): The mutual information between input and target captured by the generative prior. A stronger generative model (higher I_P) directly lowers this bottleneck.
2. The Optimization Inefficiency (ϵ_P): The gap between the information-theoretic optimum and what finite-step contrastive fine-tuning actually achieves. The paper hypothesizes that a better prior makes the optimization landscape more favorable, reducing ϵ_P.
3. The PAC-Bayes Complexity Penalty: The KL divergence between the posterior (fine-tuned) and prior (pretrained) distributions, divided by sample size. This term justifies LoRA: by constraining the posterior to be close to the prior, the complexity penalty remains small.

This theoretical framing positions the paper as offering not just a method but a principled research agenda: to improve multimodal representations, invest in improving generative capabilities (via continued pretraining, better architectures, more data), not in scaling contrastive data. The SeaDoc experiment (Section 5.3) provides an initial validation of this agenda: continuing generative pretraining on OCR-intensive data before contrastive fine-tuning improves retrieval performance, but only when the generative training preserves the model's general alignment (by combining OCR data with general-domain captions to prevent catastrophic forgetting).

A Direct Contrast with the "Representation-Generation" Direction

The paper takes care to distinguish its GRSL from a related but inverse finding in the literature. Prior work (Cambrian-1, Tong et al., 2024; Yang et al., 2024) established a "Representation-Generation Scaling Law": the downstream generative performance of an MLLM scales with the representational quality of its vision encoder. This makes intuitive sense—a better visual feature extractor provides richer information for the language decoder to condition on.

The paper's GRSL is the inverse direction: the representational performance of an MLLM after contrastive refinement scales with its generative capability before refinement. This is less obvious and more consequential. It implies that the primary barrier to better multimodal embeddings is not the quantity of paired contrastive data but the quality of the generative model being refined. The paper explicitly names this reversal:

"In this work, we explore a fundamentally different concept: the 'Generation-Representation Scaling law' where the MLLM's representation abilities scale with its own generation capabilities." (Section 6)

This distinction is critical for understanding the paper's intellectual contribution. It is not merely proposing another MLLM-based embedding model—it is arguing for a causal arrow that runs from generative quality to representation quality, with contrastive learning playing a facilitative rather than causative role. If this claim holds, it reorients the entire multimodal representation learning research agenda toward generative pretraining as the primary lever, with contrastive refinement as a downstream optimization.

3. Technical Approach

3.1 Reader Orientation

This paper builds a framework—LCO-EMB—for producing high-quality multimodal embeddings (vectors representing images, text, audio, and video in a shared mathematical space) by starting from a pretrained multimodal large language model and applying lightweight contrastive fine-tuning that is primarily text-only. The core problem it solves is that prior methods either required massive multimodal paired datasets to achieve cross-modal alignment (CLIP-style) or used MLLM backbones without understanding why they worked, leading to inefficient training recipes. The shape of the solution is a diagnostic-then-prescribe approach: first empirically establish that MLLMs contain latent cross-modal alignment in their pretrained representations, then design a fine-tuning procedure that minimally perturbs the model to preserve that alignment while projecting representations into a similarity-matching subspace, and finally provide a theoretical framework showing that the generative quality of the backbone sets a hard upper bound on what contrastive refinement can achieve.

3.2 Big-Picture Architecture (Diagram in Words)

The LCO-EMB system has five major components:

1. Pretrained MLLM Backbone — a multimodal large language model (e.g., Qwen2.5-VL, Qwen2.5-Omni, LLaVA-Next) consisting of modality-specific encoders (vision, audio), projector layers that map modality representations into the language decoder's embedding space, and a language decoder (an LLM) that generates text conditioned on multimodal inputs. This backbone is the source of latent cross-modal alignment established during generative pretraining.

2. Language Decoder Isolation — the language decoder is extracted from the MLLM, separating it from the modality-specific encoders and projectors. Only the decoder receives trainable parameters; encoders and projectors remain frozen throughout.

3. LoRA Adapters — low-rank trainable matrices inserted into selected layers of the language decoder. These adapters learn representational refinements while the original pretrained weights stay fixed, preserving the decoder's generative knowledge and the latent cross-modal alignment embedded in its representations.

4. Language-Centric Contrastive Training — a contrastive learning objective applied to the LoRA-augmented language decoder using primarily text-only paired data (NLI entailment/contradiction triplets or multilingual parallel sentences). The decoder processes text inputs, produces embeddings from its last hidden state, and the contrastive loss pulls semantically similar texts together while pushing dissimilar ones apart.

5. Re-plugging and Multimodal Calibration (optional) — after text-only training, the LoRA weights are merged into the decoder, and the updated decoder is reinserted into the full MLLM architecture. For multimodal variants, a small amount of synthetic multimodal data (~94k pairs) further calibrates the embedding space for downstream task formats.

Information flows through the system in three stages. Stage one (diagnosis): Before any training, the decoder's representations are analyzed via anisotropy (expected cosine similarity between random embeddings) and kernel alignment (overlap in nearest-neighbor structures between modalities) to confirm latent alignment. Stage two (text-only refinement): The decoder undergoes contrastive fine-tuning exclusively on text pairs. Despite receiving no non-text data, the geometric improvements (reduced anisotropy) generalize to image, audio, and video embeddings because the decoder's latent space is shared across modalities. Stage three (multimodal calibration, if used): A small set of multimodal triplets tunes the projection into task-specific similarity formats. The final model can encode any modality supported by the backbone into a single shared embedding space suitable for retrieval, classification, clustering, and similarity matching.

3.3 Roadmap for the Deep Dive

• First, the formal diagnosis methodology: how anisotropy and kernel similarity are measured, what these metrics reveal about latent alignment, and why the quantification matters for establishing the paper's central empirical claim (Section 2 in the paper, but the mechanics belong here since they constitute the analytical apparatus).

• Second, the training objective and data construction: the contrastive loss formulation, the all-NLI and Scale-1M text-only datasets, the ~94k multimodal supplement, and how these datasets activate different representational capabilities.

• Third, the LoRA fine-tuning mechanism: what parameters are trained, on which layers, with what hyperparameters, and why LoRA (as opposed to full fine-tuning, linear probing, or CLIP-style training) is the appropriate choice given the latent alignment hypothesis.

• Fourth, the multimodal calibration stage: how the small set of synthetic multimodal triplets is constructed, what task formats it covers, and how it adjusts the embedding space without disrupting the text-only refinement.

• Fifth, the backbone architecture specifics across Qwen2.5-VL, Qwen2.5-Omni, and LLaVA-Next: how modality encoders, projectors, and the language decoder interact, and what it means to "isolate" the decoder for training.

• Sixth, the SeaDoc continual generative pretraining extension: how additional OCR-intensive generative training is applied before contrastive learning to test the Generation-Representation Scaling Law, including the data construction pipeline and training protocol.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper with a method grounded in its diagnostic findings. Its core idea is that MLLMs acquire cross-modal alignment during generative pretraining, and that this latent alignment can be activated through lightweight, primarily text-only contrastive fine-tuning that preserves—rather than overwrites—the pretrained representational structure. The method, LCO-EMB, is a direct implementation of this insight: use parameter-efficient fine-tuning (LoRA) on the language decoder alone, train with predominantly text-only data, and treat contrastive learning as a projection step rather than an alignment-creation step.

---

Diagnosis Methodology: Establishing Latent Cross-Modal Alignment

Before proposing any training method, the paper develops a diagnostic toolkit to test its central hypothesis: that MLLM representations exhibit latent cross-modal alignment before any contrastive fine-tuning. Two complementary geometric analyses are employed, each providing a different lens on the structure of the representation space.

Anisotropy Analysis: Measuring Representation Collapse

The first diagnostic tool quantifies anisotropy—the degree to which embeddings collapse into a narrow region of the representation space rather than spreading uniformly across the available dimensions. Language models trained with self-supervised objectives (like next-token prediction) are known to suffer from this "representation degeneration" problem (Ethayarajh, 2019; Gao et al., 2019): hidden states become concentrated in a cone with high expected cosine similarity between randomly chosen inputs. This is problematic for similarity-based tasks because it means all points are close to each other regardless of semantic content, making discrimination difficult.

The paper follows Ethayarajh (2019) and Xiao et al. (2023) in operationalizing anisotropy as the expected cosine similarity between pairs of embeddings drawn independently from the representation distribution:

$\text{Anisotropy} := \mathbb{E}_{\mathbf{h}_i, \mathbf{h}_j \sim \mathcal{D}} \left[ \cos(\theta_{ij}) \right] = \mathbb{E}_{\mathbf{h}_i, \mathbf{h}_j \sim \mathcal{D}} \left[ \frac{\mathbf{h}_i^\top \mathbf{h}_j}{\|\mathbf{h}_i\| \|\mathbf{h}_j\|} \right]$

where $\mathbf{h}_i$ and $\mathbf{h}_j$ are embedding vectors sampled independently and identically distributed from the empirical distribution $\mathcal{D}$ of the representation space, $\|\cdot\|$ denotes the Euclidean norm, and the expectation is taken over all pairs of embeddings in the sample.

What it computes: For a finite sample of $N$ embeddings $\{\mathbf{h}_1, \ldots, \mathbf{h}_N\}$, the empirical approximation averages the cosine similarity over all $\binom{N}{2}$ distinct pairs:

$\hat{\mathbb{E}}[\cos(\theta)] = \frac{2}{N(N-1)} \sum_{1 \leq i < j \leq N} \frac{\mathbf{h}_i^\top \mathbf{h}_j}{\|\mathbf{h}_i\| \|\mathbf{h}_j\|}$

A value close to 1 indicates severe anisotropy (all embeddings point in similar directions); a value near 0 indicates isotropy (embeddings are uniformly dispersed). The result is a single scalar per layer per modality that can be plotted to compare the geometry before and after contrastive fine-tuning.

Why this form: Cosine similarity is chosen because it is the metric used at inference time for retrieval and similarity matching tasks. If the representation space is to support these tasks, the cosine similarity between arbitrary embeddings should have low expected value (so that only semantically related pairs produce high similarities). The pairwise average is a non-parametric estimate of the concentration of the distribution and does not require fitting any model—it is a direct geometric measurement. Alternatives like the effective rank of the covariance matrix would capture dimensionality but not the concentration property that matters for discriminability.

The paper applies this analysis to Qwen2.5-Omni-3B, extracting embeddings from every layer of the language decoder for text, image, audio, and video inputs. Paired datasets are used so that the analysis compares corresponding semantic content across modalities: Pixmo Cap for image-text, AudioCaps for audio-text, and MSR-VTT for video-text. The key finding (Figure 1) is that text-only contrastive fine-tuning reduces anisotropy not only for text embeddings but also for image, audio, and video embeddings—an effect that would be impossible if the decoder's representation spaces for different modalities were geometrically separate. The generalization of isotropy improvement from text to non-text modalities is the paper's first piece of evidence for latent cross-modal alignment.

Kernel-Level Similarity: Measuring Structural Alignment Between Modality Representations

The second diagnostic tool examines whether different modalities induce structurally similar similarity spaces within the same MLLM. Two modalities are aligned not just if their embeddings live in the same vector space, but if the pattern of which inputs are similar to which other inputs is consistent across modalities. For example, if an image of a dog is embedded near images of other dogs and far from images of cats, and the caption "a brown dog playing fetch" is embedded near captions about dogs and far from captions about cats, then the similarity structures are aligned—the kernel induced by the vision encoder matches the kernel induced by the text encoder.

The paper adapts the mutual k-nearest neighbor (mutual kNN) metric from Huh et al. (2024), who used it to demonstrate that independently trained models (e.g., LLaMA and DINOv2) converge to similar similarity structures—the "Platonic representation hypothesis." The paper's innovation is to use this metric within a single MLLM to compare cross-modal kernels rather than across independent models.

For a given modality pair (e.g., vision and language), the procedure works as follows:

1. Take a batch of $b$ paired samples $\{(x_i, y_i)\}_{i=1}^b$ where $x_i$ is an input in one modality (e.g., an image) and $y_i$ is the semantically equivalent input in another modality (e.g., the image's caption). The assumption is that $x_i \triangleq y_i$—they share the same semantic content, serving as anchors for comparing the induced similarity structures.

2. Encode all samples through the MLLM's language decoder for both modalities, obtaining feature sets $\Phi = \{\phi_1, \ldots, \phi_b\}$ from modality $f$ (e.g., vision) and $\Psi = \{\psi_1, \ldots, \psi_b\}$ from modality $g$ (e.g., language), where $\phi_i = f(x_i)$ and $\psi_i = g(y_i)$. In practice, the paper uses the same model for both encodings (since the decoder processes multimodal inputs), so $f = g$ is the MLLM itself.

3. For each feature $\phi_i \in \Phi$, find its $k$ nearest neighbors within $\Phi$ (excluding itself), producing the neighbor set $\mathcal{S}(\phi_i)$. Similarly, for each $\psi_i \in \Psi$, find $\mathcal{S}(\psi_i)$ within $\Psi$. The nearest neighbors are determined by a distance function $d_{\text{knn}}$ (typically cosine distance).

4. Compute the kernel alignment score for each pair $(\phi_i, \psi_i)$ as the normalized intersection of their neighbor sets:

$m_{\text{NN}}(\phi_i, \psi_i) = \frac{1}{k} |\mathcal{S}(\phi_i) \cap \mathcal{S}(\psi_i)|$

where $|\cdot|$ denotes set cardinality, $k$ is the number of nearest neighbors considered, and the score ranges from 0 (no shared neighbors—completely different similarity structures) to 1 (all $k$ neighbors are identical—perfectly aligned similarity structures).

5. Average $m_{\text{NN}}$ across all $b$ pairs in the batch to obtain the overall kernel alignment metric for that layer.

What it computes: For each semantic anchor pair (e.g., an image and its caption), the metric asks: "Do the $k$ images most similar to this image correspond to the $k$ captions most similar to this caption, and vice versa?" The score is the fraction of matching neighbors, averaged across the batch, producing a value between 0 and 1 that quantifies how well the similarity structure of one modality's representation space aligns with the other's.

Why this form: The mutual kNN metric captures relational structure rather than absolute embedding positions. Two representation spaces could have the same pairwise distances but be rotated or translated relative to each other—cosine similarity would fail to recognize they encode the same relationships, but mutual kNN would report high alignment because neighbor rankings are invariant to rotations. This is critical for the paper's claim because it isolates structure from geometry: the claim is that the pattern of similarities is shared across modalities, not merely that embeddings live in the same space. If the paper used Procrustes alignment or CKA (Centered Kernel Alignment), it would confound structural alignment with geometric alignment (rotation/translation). Mutual kNN is purely structural.

The paper applies this analysis to Qwen2.5-VL-Instruct at 3B and 7B scales, computing layer-wise vision-language kernel alignment before and after text-only contrastive fine-tuning (Figure 2). Two findings emerge. First, kernel alignment improves after text-only contrastive fine-tuning across all layers, consistent with the anisotropy results: optimizing text representations generalizes to improve cross-modal structural alignment. Second, the 7B variant exhibits consistently higher kernel alignment than the 3B variant, both before and after fine-tuning (the paper attributes this to the expanded parameter space of the larger model yielding better latent cross-modal binding during pretraining). This second finding is the first hint of the Generation-Representation Scaling Law: larger models with stronger generative capabilities appear to learn more deeply aligned multimodal representations during pretraining, giving them a higher-quality starting point for contrastive refinement.

What the Diagnosis Establishes (and What It Doesn't)

Together, the anisotropy and kernel alignment analyses establish two empirical facts that motivate the method design:

1. Cross-modal alignment is latent in pretrained MLLMs. Text-only contrastive fine-tuning improves geometric properties (isotropy, kernel alignment) for non-text modalities despite the absence of any multimodal supervision during training. This is only possible if the decoder's representation space is shared across modalities—improving the space's structure for text necessarily improves it for images, audio, and video because they inhabit the same latent subspace.

2. Alignment quality scales with model scale and, by extension, generative capability. The 7B variant exhibits stronger cross-modal kernel alignment than the 3B variant both before and after fine-tuning, suggesting that more capable generative models learn better implicit alignment during pretraining.

These findings motivate the central design principle of LCO-EMB: preserve the latent alignment. If alignment is already present, agressive fine-tuning (full parameter updates, large learning rates, massive multimodal datasets) risks disrupting it. The method should instead make minimal, targeted modifications that project the existing aligned representations into a similarity-matching format.

---

Training Objective and Data Construction

LCO-EMB uses contrastive learning as its training objective, but with a specific formulation and data strategy designed to activate latent alignment rather than create it from scratch.

Contrastive Loss Formulation

The paper uses the standard InfoNCE (Information Noise-Contrastive Estimation) loss with in-batch negatives, a workhorse of representation learning. For a batch of $n$ paired samples $\{(q_i, p_i)\}_{i=1}^n$ where $q_i$ is a query (e.g., a premise sentence) and $p_i$ is its positive pair (e.g., an entailment hypothesis), the loss for query $q_i$ is:

$\mathcal{L}_{\text{InfoNCE}}(q_i, \{p_j\}_{j=1}^n) = -\log \frac{\exp(\text{sim}(\mathbf{q}_i, \mathbf{p}_i) / \tau)}{\sum_{j=1}^n \exp(\text{sim}(\mathbf{q}_i, \mathbf{p}_j) / \tau)}$

where $\mathbf{q}_i = f(q_i)$ is the embedding of query $i$ produced by the language decoder with LoRA adapters, $\mathbf{p}_j = f(p_j)$ is the embedding of positive candidate $j$, $\text{sim}(\cdot, \cdot)$ is the cosine similarity function, and $\tau$ is a temperature hyperparameter controlling the concentration of the similarity distribution.

What it computes: For each query in the batch, the loss treats its corresponding positive as the single correct match and all other $(n-1)$ positives in the batch as negatives. The numerator measures the similarity between the query and its positive (scaled by temperature). The denominator sums the query's similarity to all candidates in the batch. The loss is the negative log of the ratio—it is minimized when the query-positive similarity dominates all query-negative similarities. Over the full batch, the symmetric loss (applied in both query-positive and positive-query directions) pushes matched pairs together and non-matched pairs apart in cosine space.

Why this form: InfoNCE is the standard loss for contrastive representation learning because it is a lower bound on mutual information (Oord et al., 2018) and has the uniformity property (Wang and Isola, 2020) that directly addresses the anisotropy problem identified in Section 2.1. The temperature $\tau$ controls the hardness of negatives: low temperature concentrates the softmax on the hardest negatives (those with highest similarity), encouraging the model to separate confusable pairs; high temperature spreads the penalty across all negatives, producing a smoother optimization landscape. The paper does not explicitly state its temperature value in the main text, but standard practice (and the codebase implications) suggest $\tau \approx 0.05$-$0.1$ for text embeddings, consistent with Sentence-BERT and E5 training recipes.

The in-batch negative formulation is crucial for efficiency: no separate negative mining is required because every other sample in the batch serves as a negative. This makes training feasible at batch size 768 (and up to 1052 for multimodal variants) without external negative sampling infrastructure.

Text-Only Dataset: all-NLI

The primary training dataset for the text-only variant of LCO-EMB is all-NLI, a combination of the Multi-Genre Natural Language Inference (MNLI; Williams et al., 2018) and Stanford Natural Language Inference (SNLI; Bowman et al., 2015) datasets. Each instance consists of a premise sentence and three hypothesis sentences labeled as entailment (the hypothesis logically follows from the premise), neutral (the hypothesis might be true but is not logically entailed), or contradiction (the hypothesis cannot be true given the premise).

The paper constructs ~276k triplets from all-NLI using the following protocol:

• Positive pair: (premise, entailment hypothesis). The entailment relationship ensures semantic equivalence—the two sentences describe the same situation in different words, which is exactly what contrastive learning needs for positive pairs.
• Hard negative: (premise, contradiction hypothesis). Using contradictions as negatives is a deliberate choice: they are semantically opposite to the premise but share vocabulary and topic (e.g., "The man is running" vs. "The man is sitting"), making them harder to distinguish from positives than random negatives would be. This hard negative mining is critical for training discriminative representations.

The paper does not use neutral hypotheses in the contrastive triplet—they provide ambiguous supervision (neither clearly similar nor clearly dissimilar) and are discarded.

Why NLI data for multimodal embedding training? This is the paper's most counterintuitive design choice and the one that most directly operationalizes the latent alignment hypothesis. NLI data is purely textual—there are no images, no audio, no cross-modal pairs. Under the CLIP paradigm, training on NLI would produce a text embedding model with no multimodal capability. But under the latent alignment hypothesis, the hypothesis is that:

1. The MLLM's language decoder already contains multimodal representations because it was trained to generate text conditioned on visual (and audio) inputs.
2. Training the decoder to produce better text embeddings—by sharpening its ability to distinguish semantically similar from semantically different texts—will improve the entire shared representation space, including the subspaces occupied by non-text modalities.
3. The NLI task specifically trains semantic similarity discrimination: entailment pairs are near-paraphrases, contradictions are semantically opposed. This is precisely the capability needed for downstream tasks like retrieval and similarity matching, just applied to a text-only signal.

The paper's ablation results (Table 2) confirm that all-NLI training excels at Visual STS and Document Understanding tasks, suggesting that NLI supervision sharpens textual similarity perception in ways that generalize to vision-text similarity.

Text-Only Dataset: Scale-1M

The second text-only dataset, Scale-1M, is a curated collection of approximately 1 million sentence pairs sampled from 20 million multilingual parallel corpora. The constituent datasets include Global Voice (Nguyen and Daumé III, 2019), MUSE (Sachidananda et al., 2021), News Commentary (Tiedemann, 2012), Tatoeba (Artetxe and Schwenk, 2019), Talks (Reimers and Gurevych, 2020), WikiMatrix (Schwenk et al., 2021), and "other Sentence Transformers sources" (Reimers and Gurevych, 2019).

The key properties of Scale-1M are:

• Multilingual coverage: The parallel corpora span many languages, providing cross-lingual alignment signal during contrastive training. This is important because the MLLM's generative pretraining may have been English-dominant; Scale-1M extends the text representation refinement to non-English languages, which should generalize to improved cross-lingual multimodal alignment (as measured by multilingual image retrieval tasks).
• Descriptive and diverse text: Unlike NLI data, which is constrained to simple declarative sentences, Scale-1M includes varied text types (news, conversation, encyclopedia entries) that more closely resemble the captions and queries encountered in multimodal tasks. The paper explicitly states that this dataset is designed to "simulate image captions—aiming to activate image representations without direct image supervision" (Section 4.1).
• No explicit entailment structure: Parallel sentences are semantic equivalents rather than entailment pairs. This provides a different training signal from NLI: the model learns that different surface forms expressing the same meaning (cross-lingual translation pairs) should be close in embedding space, which complements the entailment/contradiction discrimination learned from NLI.

The ablation in Table 2 shows that Scale-1M-trained variants lead on Linear Probing and Multilingual Image Retrieval tasks, consistent with the hypothesis that diverse descriptive text activates image-relevant representations.

Model Soup: Combining Complementary Checkpoints

The paper demonstrates (Table 2) that the all-NLI and Scale-1M fine-tuned variants have complementary strengths: all-NLI excels at similarity discrimination (Visual STS, Document Understanding), while Scale-1M excels at representation transfer (Linear Probing) and multilingual alignment (Multilingual Image Retrieval). To capture both, the paper applies model soup (Wortsman et al., 2022): averaging the LoRA weights of the two independently fine-tuned checkpoints.

The procedure is simple weight-space averaging:

$\theta_{\text{ensemble}} = \frac{1}{2} (\theta_{\text{all-NLI}} + \theta_{\text{Scale-1M}})$

where $\theta$ denotes the LoRA adapter parameters (not the full model weights—only the low-rank matrices are averaged). The averaged checkpoint is then merged into the base model.

Why weight averaging works here: Model soup is effective when the fine-tuned models share a common basin in the loss landscape—they have explored different regions of a single wide minimum. The paper's finding that averaging improves overall performance (achieving the best average across all tasks in Table 2) suggests that all-NLI and Scale-1M fine-tuning optimize complementary aspects of the same underlying representation structure without diverging from each other. This is consistent with the latent alignment hypothesis: both fine-tuning runs are projecting the same pretrained representations into similarity-matching spaces, just emphasizing different dimensions of that projection.

Multimodal Supplement: ~94k Synthetic Triplets

For the multimodal variants of LCO-EMB, the paper adds approximately 94,000 synthetic multimodal triplets to the all-NLI base (totaling ~370k triplets). This supplement is deliberately small—the paper explicitly notes it is "minimal" and treats it as a calibration step rather than primary alignment training.

The supplement covers four categories, described in Appendix A:

1. Visual Document (~23k triplets): Constructed from ColPali (Faysse et al., 2025) and Docmatix. The paper deliberately limits this to ~23k—"rather than performing exhaustive data exposure"—noting that "large-scale visual document data, when not balanced with text and other task datasets, can degrade overall task generalization." This is a key practical finding: too much domain-specific multimodal data can cause overfitting to document formats at the expense of general representation quality.

2. Retrieval and Compositionality (~3k triplets): A small set from MS-COCO, augmented with length-perturbation techniques from LA(SER)³ (Xiao et al., 2023) to improve robustness to varying input lengths. The paper notes that these augmentations "not only improves length robustness but also enhances the model's spatial perception and image-text compositional reasoning"—an unexpected benefit suggesting that length augmentation creates training examples that require more careful cross-modal attention.

3. Multilingual/Diverse Text (~several thousand from Scale-1M): A subsample of the Scale-1M dataset to maintain linguistic diversity.

4. General Synthetic Data (~60k triplets): Synthetic samples in "diverse formats" designed to maintain instruction-following and interleaved alignment capabilities. The paper notes this "benefits classification tasks, improving both probing and zero-shot performance."

The multimodal supplement is applied after or jointly with the text-only training—the paper does not specify the exact training order, but the batch size scaling (1052 for multimodal vs. 768 for text-only, proportional to the dataset size ratio) suggests joint training on the combined ~370k triplet dataset.

Why such a small multimodal supplement? This is the clearest operationalization of the paper's thesis. If cross-modal alignment is already latent in the pretrained weights, the multimodal supplement serves only to "calibrate the embedding space for downstream task formats" (Section 4.1)—a minor adjustment relative to the primary function of the text-only data, which is to project representations into a similarity-matching subspace. The fact that ~94k multimodal pairs (versus GME's ~8M) achieves state-of-the-art results is presented as evidence that the paper's diagnosis is correct: the heavy lifting is done by generative pretraining, not by contrastive multimodal training.

---

LoRA Fine-Tuning: The Mechanism for Minimal Perturbation

The choice of Low-Rank Adaptation (LoRA) (Hu et al., 2022) is not incidental—it is directly motivated by the latent alignment hypothesis and is essential to the method's success.

How LoRA Works in LCO-EMB

LoRA inserts trainable low-rank matrices into selected layers of the frozen pretrained model. For a weight matrix $W \in \mathbb{R}^{d \times k}$ in the original model, LoRA decomposes the update $\Delta W$ into two low-rank matrices:

$\Delta W = B A$

where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$, with the rank $r \ll \min(d, k)$. The forward pass becomes $h = W x + \Delta W x = W x + B A x$, where $x$ is the input to the layer and $h$ is the output. During training, $W$ is frozen and only $A$ and $B$ receive gradient updates. At inference, $B A$ can be merged into $W$ for zero additional latency: $W_{\text{merged}} = W + B A$.

The scaling factor $\alpha$ modulates the contribution of the LoRA update to the overall forward pass:

$h = W x + \frac{\alpha}{r} B A x$

where a larger $\alpha$ amplifies the influence of the learned adaptation, and a smaller $\alpha$ keeps the model closer to its pretrained behavior.

For LCO-EMB, the paper applies LoRA to the language decoder only—the modality encoders (vision, audio) and projector layers are kept completely frozen. The specific hyperparameters are:

• Text-only variants: rank $r = 64$, alpha $\alpha = 16$. This low alpha-to-rank ratio ($\alpha/r = 0.25$) means the LoRA adaptation contributes modestly to the forward pass, keeping the model close to its pretrained behavior.
• Multimodal variants: rank $r = 64$, alpha $\alpha = 128$. The significantly higher alpha-to-rank ratio ($\alpha/r = 2$) allows the multimodal calibration data to exert stronger influence on the final representations while still operating within the LoRA constraint of low-rank updates.

The paper sweeps various rank and alpha values on Qwen2.5-VL-7B (Appendix D, Table 4), revealing several patterns:

• Tasks aligned with training data are insensitive to LoRA hyperparameters. Visual STS performance is "invariant to LoRA hyperparameters" because NLI training directly optimizes textual similarity discrimination, and the generalization to visual STS is near-perfect.
• Tasks requiring generalization benefit from moderate alpha. Multilingual retrieval, document understanding, and linear probing achieve optimal performance at moderate $\alpha$ values (e.g., $r = 8, \alpha = 16$ and $r = 64, \alpha = 128$).
• Tasks far from the training distribution degrade with high alpha. Compositionality and vision-centric QA suffer "significant performance degradation" at larger $\alpha$, showing "the importance of the preservation of the base model's knowledge for generalization to OOD tasks."
• Extreme hyperparameters cause training collapse. At $r = 256, \alpha = 512$, the model experiences "unrecoverable loss spikes in training," suggesting that excessive LoRA capacity disrupts the pretrained representational structure catastrophically.

Why LoRA Over Alternatives?

The paper compares LoRA against three alternative training strategies on Qwen2.5-VL 3B and 7B (Table 3):

CLIP-style contrastive fine-tuning (multimodal): Training the full model with contrastive loss on 800K PixmoCaps image-caption pairs. This underperforms text-only LoRA by substantial margins (e.g., 53.38 vs. 66.28 average on 3B) while requiring ~100× more training time (~453 GPU hours vs. ~4.7). The paper attributes this to two factors: (1) the contrastive objective deviates from the model's generative pretraining objective, causing optimization conflict, and (2) full-parameter training disrupts the latent cross-modal alignment established during pretraining.

Linear projection (text-only): Adding a trainable linear layer after the frozen decoder output. This barely improves over the native embeddings (52.69 vs. 51.49 on 3B) because a single linear layer "does not effectively leverage pretrained cross-modal structure"—it can only rotate and scale the existing representations without reshaping their internal geometry.

Full fine-tuning (text-only): Updating all decoder parameters with the text-only contrastive objective. This achieves reasonable results (62.10 on 3B) but consistently underperforms LoRA (66.28). The paper attributes the gap to "an objective mismatch: contrastive loss deviates from the model's pretraining objective, and full fine-tuning consequently induces larger perturbations to the pretrained parameters, which are more likely to disrupt the established cross-modal alignment."

The LoRA advantage is thus two-fold, consistent with Biderman et al. (2024)'s characterization of LoRA as "learning less and forgetting less":

1. Preserving generative knowledge: The frozen pretrained weights maintain the model's original capabilities, meaning the re-plugged MLLM can still perform generation tasks (though this is not evaluated for embedding quality).
2. Preserving latent alignment: The low-rank constraint prevents the decoder's representation space from being substantially reshaped. The alignment that was established during generative pretraining—where visual, audio, and textual representations were integrated into a shared latent subspace—remains intact because the core weight matrices are unchanged. The LoRA adapters learn a projection of this aligned space into a similarity-matching format, rather than rewriting the alignment itself.

The paper's theoretical analysis (Theorem 1) provides a formal justification: the PAC-Bayes complexity term $(\text{KL}(Q\|P) + \log(1/\delta)) / 2n$ penalizes posteriors $Q$ that diverge far from the prior $P$. LoRA, by constraining the posterior to a low-rank subspace around the pretrained weights, keeps $\text{KL}(Q\|P)$ small, meaning the generalization bound remains tight. Full fine-tuning would allow $\text{KL}(Q\|P)$ to grow large, increasing the bound and risking overfitting or disruption of pretrained structure.

Additional Training Hyperparameters

The paper specifies the following training configuration (Section 4.1):

• Optimizer: AdamW with cosine learning rate schedule.
• Peak learning rate: $4 \times 10^{-4}$ for most variants; reduced to $3 \times 10^{-4}$ for Qwen2.5-Omni-7B multimodal variants "due to loss spike."
• Batch size: 768 for text-only variants. For multimodal variants, the batch size is scaled by the ratio of total to text-only dataset size: with ~370k total triplets (276k text-only + 94k multimodal), the batch size becomes 1052 ($1.37 \times 768$).
• Training epochs: 2 epochs for all variants. This is a remarkably short training schedule—the paper's thesis is that only a "lightweight refinement stage" is needed, and 2 epochs of LoRA training suffices.
• Training cost: ~4.7 GPU hours on H20 GPUs for text-only 3B variants, ~9.3 GPU hours for 7B variants. This is the "lightweight" contrastive learning the paper contrasts with CLIP-style training.

---

Multimodal Calibration Stage: Task-Format Alignment

While the paper's primary claim is that text-only training suffices for cross-modal representation quality, the multimodal variants add a calibration stage to align the embedding space with downstream task formats. This stage is described in less detail than the text-only training, but its design principles are clear.

What the Calibration Data Covers

The ~94k multimodal triplets are constructed to cover the specific task formats encountered in MIEB and other multimodal benchmarks:

• Visual Document (~23k): Triplets formatted as (query, positive document page, negative document page). These train the model to distinguish document pages by their content relevance, which is the format of the Vidore/ColPali document retrieval tasks. The deliberate limit to 23k is a practical finding: more document data caused overfitting to document formats and degraded performance on non-document tasks.

• Retrieval and Compositionality (~3k): Image-caption triplets from MS-COCO with length augmentation. The length perturbation (from LA(SER)³) creates training examples where the caption length varies substantially, forcing the model to attend to semantic content rather than surface-level length correlations. The paper's finding that this also improves spatial perception and compositional reasoning is unexpected and suggests that length augmentation creates a form of implicit data augmentation for fine-grained alignment.

• Multilingual Text (~several thousand): A subsample of Scale-1M to maintain cross-lingual alignment. This ensures the calibration stage does not erase the multilingual capabilities established during text-only training.

• Synthetic Data (~60k): The largest component, consisting of "diverse formats" that maintain "instruction-following and interleaved alignment capabilities." The paper ties this to VQA tasks under the reasoning-as-retrieval paradigm: the model must understand an instruction (the question) that references a visual input and retrieve the correct answer. Synthetic data in this format trains the model to handle interleaved multimodal encodings.

Integration with Text-Only Training

The multimodal data is combined with the all-NLI text data (276k triplets) to form a single training set of ~370k triplets. Training proceeds jointly with the scaled batch size (1052). The LoRA hyperparameters are adjusted for multimodal variants ($\alpha$ increased from 16 to 128, peak learning rate kept at $4 \times 10^{-4}$ except for Omni-7B where it is reduced to $3 \times 10^{-4}$).

Why joint training rather than sequential? The paper does not explicitly justify this choice, but two practical considerations likely apply: (1) joint training is simpler to implement and avoids the need for learning rate scheduling across stages, and (2) training on the combined dataset ensures the improvements from calibration do not come at the expense of the text-only gains—the gradient signal from all-NLI remains present throughout training, preventing catastrophic forgetting of the text-refinement benefits.

What "Calibration" Means in Practice

The paper frames multimodal data as calibrating the embedding space "for downstream task formats." This is a precise claim: the text-only data projects the pretrained representations into a similarity-matching subspace (where cosine similarity corresponds to semantic relatedness), but the projection may not be optimal for the specific pattern of queries and candidates encountered in tasks like document retrieval (query: English question, candidates: document page images) or visual QA (query: interleaved question+image, candidates: answer texts). The small multimodal supplement fine-tunes this projection to the task formats without altering the underlying representation structure.

Evidence for this interpretation comes from Figure 4 and Table 1: the multimodal variants consistently outperform text-only variants on most task categories, but the gap is largest on tasks with distinctive formats (e.g., Document Understanding: 73.0 vs. 62.2 for 3B VL, a +10.8 point gain; Zero-shot Classification: 74.1 vs. 68.5 for 7B Omni, a +5.6 point gain). The text-only variants already achieve strong performance, but the multimodal calibrate aligns the similarity space with the specific evaluation protocols.

---

Backbone Architecture and Decoder Isolation

LCO-EMB is evaluated on three MLLM backbones, all of which share a common architectural template: modality-specific encoders, projector layers, and a language decoder.

Shared Architecture Template

The standard MLLM architecture (Liu et al., 2023; Bai et al., 2025; Xu et al., 2025) consists of:

1. Modality-specific encoders: A vision encoder (typically a ViT variant) processes images and videos into a sequence of visual feature tokens. An audio encoder (in Qwen2.5-Omni) processes audio into audio feature tokens. These encoders are pretrained on modality-specific objectives and produce representations in their own dimensional spaces.

2. Projector layers: Linear or MLP layers that map modality-specific features into the embedding space of the language decoder. For a vision encoder producing features of dimension $d_v$ and a language decoder with embedding dimension $d_l$, the projector is a function $p: \mathbb{R}^{d_v} \to \mathbb{R}^{d_l}$ that aligns the dimensionalities. In practice, this is often a simple linear projection or a 2-layer MLP.

3. Language decoder: An autoregressive transformer (an LLM) that receives the projected multimodal features interleaved with text tokens and generates text output token by token. During generative pretraining, the decoder learns to attend to multimodal features to produce the correct next token, which is the mechanism by which cross-modal alignment is implicitly learned.

Specific Backbones Used

Qwen2.5-VL-Instruct (3B and 7B): A vision-language model supporting image and video inputs. The 3B variant has 36 transformer layers in the language decoder (more layers than the 7B's 28 layers—a noted architectural curiosity), while the 7B variant has wider layers. The model has undergone instruction tuning, giving it strong generative capabilities that the paper leverages for latent alignment.

Qwen2.5-Omni (3B and 7B): An omnimodal model supporting text, image, video, and audio inputs. The architecture adds an audio encoder and audio projector alongside the vision components. This enables the paper's omnimodal evaluation (audio and video retrieval tasks in addition to vision-language tasks).

LLaVA-Next (8B): A vision-language model used primarily in the scaling law analysis (Figure 6). It follows the same encoder-projector-decoder architecture.

Decoder Isolation Procedure

The central operation in LCO-EMB is isolating the language decoder for text-only training:

1. Extraction: The language decoder is separated from the full MLLM. In PyTorch terms, the decoder's parameters and forward pass are extracted into a standalone module.

2. LoRA injection: LoRA adapters are inserted into selected layers of the decoder. The paper does not specify exactly which layers—common practice is to apply LoRA to all attention weight matrices ($W_q, W_k, W_v, W_o$) and optionally to feed-forward layers. The key constraint is that the embedding layer (the first layer that converts token IDs to vectors) must remain frozen: modifying it would shift the entire embedding space, breaking the alignment with the projector outputs that map visual/audio features into the decoder's space.

3. Text-only contrastive training: The LoRA-augmented decoder processes text inputs (premises and hypotheses from NLI, or parallel sentence pairs from Scale-1M), produces embeddings from its final hidden state (the "last token hidden" representation), and the InfoNCE loss is applied. The decoer never sees multimodal inputs during this stage.

4. LoRA merging: After training, the LoRA matrices are merged into the decoder's weights: $W_{\text{merged}} = W + \frac{\alpha}{r}BA$. This produces a single set of weights with no additional inference cost.

5. Re-plugging: The updated decoder is inserted back into the full MLLM, reconnecting it with the frozen encoders and projectors. The full model can now process multimodal inputs, with the decoder producing refined embeddings for all modalities.

Why freeze the embedding layer? This is a critical design detail implied by the method description but not explicitly stated. The embedding layer maps discrete tokens to the continuous representation space. If LoRA adapters were applied to the embedding layer, the visual features projected by the (frozen) vision projector would no longer align with the (updated) embedding space—the projector maps to the pretrained embedding distribution, and changing that distribution would break the cross-modal alignment. By keeping the embedding layer frozen, the decoder's input space remains identical to what it was during generative pretraining, and the LoRA adapters operate only on the internal layers where representations are transformed.

What the Represents After Re-plugging

After re-plugging, the MLLM can encode any supported modality into a single embedding vector. For text, the embedding is the decoder's last-layer hidden state at the end-of-sequence token. For images, the visual features pass through the vision encoder, through the projector, into the decoder alongside a text prompt (e.g., "Summarize the above in one word:"), and the decoder's final hidden state serves as the image embedding. Similarly for audio and video: modality-specific encoder → projector → decoder → last hidden state.

The critical property is that all modalities are embedded into the same vector space—the decoder's output space—because the projector maps modality features into the decoder's input space, and the decoder processes all modalities through the same transformer stack. This is why text-only contrastive training can improve image embeddings: the decoder's transformations are shared across modalities, so refining the geometric properties of the output space for text automatically refines them for all modalities that share that output space.

---

The SeaDoc Experiment: Testing the Generation-Representation Scaling Law

Section 5.3 introduces a specific experiment designed to test a key prediction of the Generation-Representation Scaling Law (GRSL): if representational quality after contrastive refinement is bounded by generative quality before refinement, then improving an MLLM's generative capabilities on a specific domain should improve its post-contrastive representation quality on that domain. The SeaDoc benchmark and the associated continual generative pretraining experiment constitute a direct test of this prediction.

The SeaDoc Benchmark

SeaDoc is a cross-lingual visual document retrieval benchmark for low-resource Southeast Asian languages, constructed specifically for this paper. The task format is: given an English query, retrieve the corresponding page from a corpus of 5,055 document pages written in Thai, Vietnamese, Malay, or Lao. This is designed to be extremely challenging because it requires the model to:

1. Understand the English query.
2. Read and comprehend text in the target SEA language from visual document pages (OCR capability).
3. Match the semantic content across languages (cross-lingual alignment).
4. Handle document layout, figures, and formatting (visual document understanding).

The data construction pipeline (detailed in Appendix E) is:

1. Corpus curation: 5,055 pages from 29 book publications across four SEA languages, covering diverse subjects (economics, natural sciences, technology, history, politics, art, psychology, education, country reports).

2. Annotation with Gemini-2.5-Flash: Each page undergoes OCR, English translation, and English query generation. The query is designed to be answerable exclusively from a single page, with the prompt instructing the model to ensure no other page in the corpus would be a valid match.

3. Quality control (three-stage):

   • Stage 1 (page filtering): Qwen2.5-7B-Instruct filters out functional pages (title pages, author pages, tables of contents), reducing the dataset to 4,491 content pages.
   • Stage 2 (scoring): The same model scores each annotation for Quality (informativeness and relevance) and Groundedness (exclusivity to the page) on a 10-point scale. Only examples with Quality ≥ 9 and Groundedness = 10 are retained.
   • Stage 3 (human review): In-house linguists conduct final review to ensure annotation quality.

4. Final dataset: 1,001 high-quality English queries for retrieval over the 5,055-page corpus. The evaluation metric is nDCG@10.

Continual Generative Pretraining Protocol

To test whether improving generative capability enhances representation quality, the paper compares two training pipelines for Qwen2.5-VL-3B:

Baseline: Apply lightweight contrastive learning directly to the pretrained Qwen2.5-VL-3B using text-only NLI data (the standard LCO-EMB text-only procedure). Then evaluate on SeaDoc.

Continual generative pretraining: Before applying lightweight contrastive learning, perform an additional stage of supervised fine-tuning (SFT) on generative tasks to enhance the model's image-to-text generation capability in the target domain. This SFT stage uses a mixture of:

• OCR data in SEA languages: Derived from the training split of SeaDoc (4k seed images not overlapping with the 1,001 test queries). For each training image, five SFT tasks are constructed: (1) OCR the image, (2) OCR then generate a question, (3) provide English translation given OCR'd text, (4) provide English translation directly from the image, (5) answer the generated question. With 4k seed images × 5 tasks, this produces approximately 20k training examples. Note that the test queries are excluded from this training set.

• General-domain image captioning data (PixmoCaps, ~710k examples): Included to prevent catastrophic forgetting of general visual-semantic alignment. Training exclusively on OCR data in low-resource languages causes "significant capability collapse" (Figure 7), aligning with observations in recent multimodal reasoning research that continual SFT on narrow domains degrades generalization.

After SFT, the model undergoes the same lightweight contrastive learning procedure as the baseline. The comparison isolates the effect of improved generative quality on subsequent representation quality.

Resolution Settings

Given that text in visual documents can be small—requiring high resolution for accurate OCR—the paper evaluates under two resolution settings:

• Low resolution: Maximum 262,144 pixels (standard practice).
• High resolution: Maximum 2,621,440 pixels (10× larger).

Experimental Results (Figure 7)

The SeaDoc experiment reveals several findings that support GRSL:

1. OCR-only SFT at low resolution degrades representation quality. The model trained exclusively on SeaDoc OCR data at low resolution experiences "significant capability collapse" compared to the baseline (no SFT). This is attributed to overfitting: the SFT objective narrows the model's generative distribution to OCR-like outputs, disrupting the general cross-modal alignment that LCO-EMB relies on.

2. Higher resolution partially mitigates the collapse. Training on SeaDoc with high resolution reduces the degradation because "text in visual documents is typically small; training with higher resolution allows for better grounding of the generated output in the visual text of the source image, as opposed to overfitting to example-level visual cues." In other words, high-resolution training forces the model to actually read the text rather than memorize spurious visual patterns—better generative learning transfers to better representation quality.

3. Adding PixmoCaps captions to the SFT mixture boosts retrieval performance post-contrastive learning. The combination of OCR data + general-domain captions preserves the latent image-text alignment during generative SFT, enabling the subsequent contrastive fine-tuning to effectively exploit it. This is the key validation of GRSL: generative pretraining that improves domain-specific capability while preserving general alignment leads to improved representation quality after contrastive refinement.

Why does this validate GRSL? The continual SFT experiment directly manipulates generative capability before contrastive learning and measures the downstream effect on representation quality. The fact that OCR+Captions SFT (which improves generative OCR ability) leads to better retrieval performance than no SFT (baseline) demonstrates the causal arrow: better generation → better representation after contrastive refinement. The collapse from OCR-only SFT shows that this arrow requires the generative improvement to preserve (or enhance) the cross-modal alignment structure—narrow generative improvements that disrupt alignment are counterproductive.

The theoretical bound from Section 5.2 provides the interpretation: OCR+Captions SFT increases the mutual information $I_P(X; Y)$ between visual document inputs $X$ and their textual content $Y$, which tightens the generative bottleneck term $\log N - I_P(X; Y)$ in the PAC-Bayes bound. OCR-only SFT at low resolution may actually decrease $I_P(X; Y)$ by overfitting to spurious correlations, explaining the counterintuitive performance drop.

---

Summary of Design Choices and Their Justifications

• Text-only primary training data: Contrastive fine-tuning on NLI and multilingual parallel text—rather than multimodal pairs—is the central operationalization of the latent alignment hypothesis. The expectation is that text-only refinement of the decoder's representation space will generalize to non-text modalities because the space is shared. Empirically validated by cross-modal anisotropy reduction (Figure 1) and kernel alignment improvement (Figure 2).

• LoRA over full fine-tuning: LoRA preserves the pretrained generative knowledge and cross-modal alignment by making low-rank modifications to frozen weights. Full fine-tuning disrupts the alignment (Table 3), and CLIP-style multimodal training is both worse-performing and 100× more expensive. The PAC-Bayes bound (Theorem 1) formally justifies this: LoRA keeps $\text{KL}(Q\|P)$ small, tightening the generalization guarantee.

• Decoder isolation: Training only the language decoder (frozen encoders and projectors) ensures that the modality-specific components are unchanged. The decoder is where cross-modal representations are integrated (because it processes projected multimodal features), so this targets the shared subspace directly.

• Frozen embedding layer: Implied by the architecture description—the embedding layer must remain frozen to maintain alignment between the projector outputs (which map to the pretrained embedding space) and the decoder's input representation. Modifying the embedding layer would break the visual-text alignment.

• all-NLI for sharpening semantic discrimination: Entailment as positives, contradiction as hard negatives—this trains the model to distinguish semantically similar from semantically different content, which is the core capability needed for retrieval and similarity tasks.

• Scale-1M for multilingual and descriptive coverage: Multilingual parallel sentences provide cross-lingual alignment signal and descriptive text that simulates image captions, activating visual representations without image supervision.

• Model soup for complementary strengths: Weight averaging of independently fine-tuned LoRA checkpoints combines the distinct benefits of all-NLI (similarity discrimination) and Scale-1M (representation transfer, multilingual alignment) without additional training.

• Minimal multimodal calibration (~94k triplets): Serves as a task-format alignment step, not primary alignment training. The deliberate small scale (~21× less than GME's 8M) demonstrates that generative pretraining—not contrastive data volume—is the primary driver of representation quality.

• Joint training of text and multimodal data: Ensures the calibration benefits do not come at the cost of the text-only refinement gains by keeping the all-NLI gradient signal active throughout training.

• SeaDoc continual SFT with general-domain data preservation: Validates that generative improvements enhance representation quality only when they preserve cross-modal alignment. OCR-only SFT causes collapse; OCR+Captions SFT improves retrieval. This directly supports GRSL's causal claim.

4. Key Insights and Innovations

Innovation 1: Latent Cross-Modal Alignment as a Testable Empirical Property — and the Diagnostic Toolkit for Detecting It

Before this paper, the field's understanding of why MLLM-based embedding models outperform CLIP-style models was essentially nil. Multiple groups had demonstrated the fact of superiority—E5-V, VLM2Vec, mmE5, GME all showed strong results—but the mechanism was opaque. The dominant assumption, inherited from the CLIP era, was that cross-modal alignment requires explicit contrastive training against paired multimodal data. Under this assumption, an MLLM's advantage would have to come from better architecture, better initialization, or the multimodal fine-tuning data itself. The possibility that alignment might already be present in pretrained MLLM weights, independent of any contrastive training, was simply not on the table as a hypothesis subject to empirical test.

This paper's first fundamental innovation is to operationalize "latent cross-modal alignment" as an empirically measurable property using two complementary geometric diagnostics—anisotropy estimation and kernel-level similarity analysis—and then use these diagnostics to demonstrate that the property holds across modalities, model scales, and backbones. This is not a method contribution or a performance gain; it is a conceptual reframing supported by a diagnostic apparatus that the field previously lacked.

The diagnostic design itself is clever in ways that prior work on representational geometry did not exploit. Prior anisotropy studies (Ethayarajh, 2019; Gao et al., 2019; Xiao et al., 2023) treated language models as unimodal objects and measured whether text representations collapsed. The paper's innovation is to use text-only contrastive fine-tuning as an intervention and then measure whether the geometric improvement generalizes to non-text modalities. If text-only training disperses image embeddings, the simplest explanation is that text and image representations share a geometric structure—which is a positive test for latent alignment. The kernel similarity diagnostic (adapted from Huh et al., 2024's cross-model Platonism work) provides convergent evidence by measuring structural alignment (do image similarities and text similarities share the same pattern of neighbor relations?) rather than absolute positioning.

The significance of this innovation extends beyond the paper's own method. By providing a diagnostic toolkit, the paper enables future research to evaluate MLLMs for latent alignment as a property, independent of any specific training recipe. A new MLLM architecture can be assessed before any contrastive fine-tuning: does text-only training on this backbone generalize geometrically to vision? If yes, the backbone has latent alignment; if no, something about its pretraining or architecture failed to establish it. This converts "MLLM-based embeddings work better" from a mysterious empirical regularity into a testable, diagnosable property of model checkpoints.

The evidence for this innovation is Figure 1 (cross-modal anisotropy reduction) and Figure 2 (cross-modal kernel alignment improvement after text-only CL), discussed in detail in Section 2 of the paper and broken down mechanically in Section 3.4 of this analysis. The key empirical claim—that the effect is causal (text-only training causes geometric improvement in non-text modalities) and not merely correlational—is established by the before/after design.

---

Innovation 2: The Reconceptualization of Contrastive Learning from Alignment Mechanism to Lightweight Projection Operator

The dominant paradigm in multimodal representation learning since CLIP (Radford et al., 2021) has treated contrastive learning as the mechanism by which cross-modal alignment is created. Under this view, the contrastive objective—pulling matched image-text pairs together, pushing non-matched pairs apart—is what teaches a vision encoder and a text encoder to inhabit a shared representational space. The scale of contrastive training matters because alignment is assumed to be built from scratch: more image-text pairs, larger batches, and longer training all contribute to better alignment.

This paper fundamentally challenges that view. Its central reconceptualization—articulated explicitly in Section 3 and operationalized throughout LCO-EMB—is that in the MLLM setting, contrastive learning serves as a lightweight projection operator that maps pre-existing, generatively-aligned representations into a similarity-matching subspace, rather than as an alignment-creation mechanism. The alignment itself is a product of generative pretraining, where the language decoder learned to integrate multimodal signals to produce correct textual outputs. Contrastive learning merely activates and refines what is already there.

This is a fundamental shift, not an incremental refinement, because it inverts the causal arrow assumed by prior work. Under the CLIP paradigm, more contrastive training → better alignment → better representations. Under this paper's paradigm, better generative pretraining → better latent alignment → better representations after any minimal contrastive refinement. The quality ceiling is set by the generative backbone, not by the contrastive training budget.

The practical consequence—validated extensively in Table 3 and the overall MIEB results—is that contrastive training can be radically downsized: ~276k text-only pairs, 2 epochs of LoRA training, ~4.7 GPU hours on H20 GPUs, versus ~453 hours for CLIP-style multimodal training on 800K image-caption pairs. The 100× compute reduction is not merely an efficiency gain; it is evidence for the reconceptualization. If alignment had to be created from scratch, text-only training on NLI data could not possibly produce competitive multimodal embeddings. The fact that it does, and that it outperforms CLIP-style training, is a direct empirical refutation of the alignment-via-contrastive-learning assumption for MLLM backbones.

This innovation also explains why prior MLLM-based approaches (E5-V, GME, mmE5) achieved good results despite using different training recipes: they were all, in effect, projecting pre-existing alignment into similarity spaces. But because they lacked this reconceptualization, they did not optimize for the right thing—preserving latent alignment during training—leading to inefficiencies (overuse of multimodal data, potential disruption through full fine-tuning) that LCO-EMB avoids.

The theoretical formalization in Theorem 1 (Section 5.2) elevates this from an empirical observation to a principled claim: the generative bottleneck term (log N − I_P(X; Y)) in the PAC-Bayes bound directly encodes that contrastive performance is capped by the mutual information captured during generative pretraining. This is not a loose analogy; it is a mathematical statement that the contrastive objective can only recover structure that the generative prior already encodes.

---

Innovation 3: The Generation-Representation Scaling Law (GRSL) as a Causal Relationship with a Formal Upper Bound

Prior work on scaling laws in multimodal models has focused primarily on the relationship between pretraining compute and downstream generative performance (Hoffmann et al., 2022, applied to MLLMs), or on how vision encoder quality affects MLLM generation (Cambrian-1; Yang et al., 2024—the "Representation-Generation" direction). The paper's GRSL is a fundamentally different claim about the inverse causal direction: an MLLM's generative capability determines an upper bound on its representational capability after contrastive refinement.

This is a conceptual innovation, not merely an empirical scaling observation, for three reasons:

First, it establishes a new axis for scaling laws research. The field's scaling laws have historically been about pretraining: how does loss scale with model size, data size, and compute? The paper argues that there is a second, equally important scaling axis: within a fixed MLLM architecture family, how does the potential representation quality—the best achievable performance after optimal contrastive refinement—scale with generative quality? This is analogous to asking not "how good can this model get at generation?" but "what is the ceiling on how good this model's embeddings can be, given how good it already is at generation?" No prior work has posed this question, let alone provided empirical evidence and a theoretical bound.

Second, it is formalized as an information-theoretic upper bound, not just a correlation. The PAC-Bayesian analysis (Section 5.2) derives Theorem 1, which decomposes the expected population contrastive risk into a generative bottleneck term (log N − I_P(X; Y)), an optimization inefficiency gap (ε_P), and a complexity penalty (√(KL(Q‖P) + log(1/δ))/2n). The critical theoretical move is Corollary 1: substituting the approximation I_P(X; Y) ≈ H(Y) − L_g(P) shows that the bound tightens as the generative loss L_g(P) decreases. A better generative model (lower L_g) directly lowers the ceiling on achievable contrastive performance. This is not a loose intuition—it is a mathematical relationship linking a measurable quantity (generative cross-entropy loss) to a bound on a different measurable quantity (contrastive population risk).

The distinction from prior PAC-Bayesian analyses in deep learning is important. Most PAC-Bayes bounds are used to justify regularization strategies (e.g., flat minima generalize better) or to bound the gap between train and test error. This paper uses the bound for a different purpose: to identify the generative prior as the fundamental bottleneck, not the contrastive training procedure. The complexity penalty term provides formal justification for LoRA (keeping KL(Q‖P) small), but the novel contribution is the generative bottleneck term, which directly encodes the paper's central thesis.

Third, it is empirically validated through a causal intervention (SeaDoc), not just correlation. The cross-sectional evidence in Figure 6—where better generative performance on OCR, video, and audio benchmarks correlates with better post-CL representation performance—establishes a consistent pattern across backbones and modalities. But correlation does not establish causation: it could be that larger models happen to be better at both, or that some third factor (pretraining data quality, architecture) drives both.

The SeaDoc experiment (Section 5.3) provides the causal test. By taking a single backbone (Qwen2.5-VL-3B) and manipulating its generative capability through additional SFT before contrastive learning, the experiment isolates the effect of generative quality on downstream representation quality. The result—that OCR+Captions SFT improves SeaDoc retrieval while OCR-only SFT degrades it—demonstrates that (a) generative improvement can enhance representation quality, but (b) only when the generative improvement preserves cross-modal alignment. This is precisely what GRSL predicts: the generative bottleneck term depends on I_P(X; Y), the mutual information captured by the generative prior. OCR-only SFT at low resolution reduces I_P(X; Y) by overfitting to spurious correlations; OCR+Captions SFT preserves or increases it by combining domain-specific learning with general alignment maintenance.

The practical implication is a reorientation of the research agenda: to improve multimodal embeddings, invest in generative pretraining (better architectures, more pretraining data, continued generative training on target domains), not in scaling contrastive data. This is a direct consequence of the bound—the generative bottleneck term is the hard floor, and contrastive training can only approach it, not break through it.

---

Innovation 4: Minimal Multimodal Data as Calibration, Not Alignment — and the Identification of the Overfitting Risk

A subtle but practically significant innovation in LCO-EMB is the deliberate decoupling of representation refinement (achieved through text-only data) from task-format calibration (achieved through minimal multimodal data). Prior MLLM-based embedding models—GME with ~8M pairs, mmE5 with ~2.1M pairs—treated multimodal data as the primary training signal, implicitly assuming that more cross-modal contrastive data would improve alignment. LCO-EMB flips this: the text-only data does the heavy lifting of projecting representations into a similarity-matching space, and the ~94k multimodal triplets serve only to align the similarity space with the specific formats of downstream evaluation tasks.

This is not a small data efficiency trick. It is a conceptual distinction between two functions that prior work conflated. The text-only data (all-NLI, Scale-1M) addresses the geometry of the representation space—making it isotropic, sharpening discriminability, establishing that cosine similarity corresponds to semantic relatedness. The multimodal data addresses the format—ensuring that when the evaluation asks "given this English query, find the correct document page image," the similarity space is oriented correctly for that specific pairing pattern.

Evidence for this distinction comes from the task-level breakdowns in Table 1. The text-only variants already outperform Voyage Multimodal 3 and are competitive with GME on most tasks. The multimodal supplement provides the largest gains on tasks with distinctive formats: Document Understanding jumps from 62.2 to 73.0 (3B VL), Zero-shot Classification from 68.5 to 74.1 (7B Omni), and Compositionality from 40.4 to 51.0 (7B VL). These are precisely the tasks where the evaluation protocol differs most from the text-only training distribution—document retrieval requires matching query text to page images, zero-shot classification requires matching image embeddings to label-text embeddings, and compositionality requires fine-grained image-text element matching. The text-only training has already made the space good for similarity matching; the multimodal data fine-tunes which similarity patterns are rewarded.

The paper also identifies a previously undocumented risk of multimodal over-calibration in Appendix A: "large-scale visual document data, when not balanced with text and other task datasets, can degrade overall task generalization." This is why only ~23k visual document triplets are included, and why the synthetic data (~60k) covers diverse formats. The finding suggests that multimodal contrastive data, when over-represented, can distort the embedding space toward task-specific patterns at the expense of general representational quality—a form of contrastive overfitting that the field has not systematically studied.

This innovation has immediate practical implications for practitioners building multimodal embedding models: (1) invest first in text-only training, which provides the geometric foundation; (2) add multimodal data sparingly and in balanced proportions across task formats; (3) monitor not just aggregate benchmarks but task-level performance to detect over-calibration. The paper's specific recipe (~276k text + ~94k multimodal, ~3:1 ratio) provides a starting point, but the principle—decouple geometry refinement from format calibration—is the transferable insight.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation benchmark is MIEB-Lite, the official lightweight subset of the Massive Image Embedding Benchmark (MIEB; Xiao et al., 2025), comprising 51 tasks across 8 categories: Visual STS (English, cross-lingual, multilingual), Document Understanding/Visual Document Retrieval, Image Linear Probing, Compositionality Evaluation, Vision-centric QA, Retrieval (English and multilingual), Zero-shot Classification, and Clustering. A secondary compact subset, MIEB-Sub18, consisting of 18 tasks from MIEB considered most challenging to image-text embedding models—focused on visual text representation, multilingual understanding, and interleaved encodings—is used for rapid iteration and ablation. For audio-text embedding tasks, the paper evaluates on AudioCaps (Kim et al., 2019) and Clotho (Drossos et al., 2019) datasets using Recall@1. For video-text embedding tasks, the paper evaluates on MSR-VTT (Xu et al., 2016) and ActivityNet (Caba Heilbron et al., 2015) using Recall@1. For the Generation-Representation Scaling Law validation, a newly constructed benchmark, SeaDoc, provides 1,001 English queries for cross-lingual visual document retrieval over a 5,055-page corpus in four Southeast Asian languages (Thai, Vietnamese, Malay, Lao), evaluated with nDCG@10.

• Base model(s). The paper uses three MLLM families: Qwen2.5-VL-Instruct (3B and 7B; Bai et al., 2025) for vision-language experiments, Qwen2.5-Omni (3B and 7B; Xu et al., 2025) for omnimodal experiments (supporting text, image, video, and audio), and LLaVA-Next (8B; Liu et al., 2024) for the scaling law analysis in Section 5.1. These models are chosen because they "conform to the standard architecture of modality-specific encoders, a projector, and a language decoder" (Section 4.1), making the decoder isolation procedure directly generalizable. The 3B and 7B variants of the Qwen models enable within-family scaling comparisons, while Qwen2.5-Omni additionally enables audio and video evaluation.

• Metrics. The paper reports a diverse set of metrics aligned with MIEB's task categories: nDCG@10 for retrieval tasks (measuring ranking quality of retrieved items); nDCG@5 for document understanding tasks; Spearman correlation for Visual STS (comparing embedding-based similarity scores against human annotations); accuracy for linear probing, zero-shot classification, compositionality evaluation, and vision-centric QA; NMI (Normalized Mutual Information) for clustering tasks; and Recall@1 for audio-text and video-text retrieval. The MIEB-Lite average score (reported as "Mean" in Table 1) is the arithmetic mean across all 51 tasks. For SeaDoc, nDCG@10 is the primary metric. Generative capability in Section 5.1 is measured by task-specific benchmark averages: for OCR, the average of TextVQA (Singh et al., 2019), DocVQA (Mathew et al., 2021), OCRBench (Liu et al., 2024), and ChartQA (Masry et al., 2022); for video, the average of Video-MMEw/sub (Fu et al., 2025) and MVBench (Li et al., 2024); for audio, the average of MMAU (Sakshi et al., 2025) and VoiceBench (Chen et al., 2024).

• Baselines. The paper compares against two categories of baselines. CLIP-style encoder baselines: CLIP-ViT-bigG (Radford et al., 2021; 2B parameters, Laion variant), which represents the dominant contrastive pretraining paradigm, and SigLIP-so400m (Zhai et al., 2023; 9B parameters), a strong derivative with sigmoid loss. MLLM-based embedding baselines: VLM2Vec-LoRA (Jiang et al., 2025; trained on 662k samples), E5-V (Jiang et al., 2024; trained on 276k text-only NLI pairs), Voyage Multimodal 3 (Voyage AI, 2024; proprietary, commercial model optimized on PDF-text pairs), mmE5 (Chen et al., 2025; 11B, trained on 2.1M synthetic samples), and GME (Zhang et al., 2024; 7B, trained on 8M samples, the prior state-of-the-art on MIEB). Baseline scores are obtained from the latest MIEB leaderboard.

• Generation budget / compute accounting. The paper measures compute in GPU hours on H20 GPUs for training cost comparisons (Table 3). For data efficiency, the key metric is the number of training triplets: LCO-EMB's text-only variant uses ~276k triplets (all-NLI), its multimodal variant uses ~370k triplets (276k text + 94k multimodal), compared to GME's ~8M, mmE5's ~2.1M, and VLM2Vec's 662k. Training cost is included for the training strategy ablation: text-only LoRA on Qwen2.5-VL-3B requires ~4.7 GPU hours, full fine-tuning requires ~8.5 hours, and CLIP-style contrastive training on 800K PixmoCaps pairs requires ~453 hours, establishing a ~100× compute advantage for the LoRA approach.

• Cross-validation / statistical protocol. The paper does not employ formal cross-validation for the MIEB evaluations—performance is reported on the standard MIEB-Lite test tasks using the official MIEB codebase. For the SeaDoc construction, a three-stage quality control process is applied: (1) Qwen2.5-7B-Instruct filters functional pages, (2) the same model scores annotations for Quality and Groundedness on a 10-point scale with retentions thresholds of ≥9 and =10 respectively, and (3) in-house linguists conduct final review. The SeaDoc training split (4k seed images) is constructed to have no overlap with the 1,001 test queries. No statistical significance testing or confidence intervals are reported for any results.

Main Quantitative Results

MIEB-Lite Benchmark Performance (Full Evaluation)

The paper's headline result is that LCO-EMB's multimodal variants set a new state-of-the-art on the MIEB-Lite benchmark (Table 1, Figure 4). The top-performing configuration, LCO-EMB-Omni (M, 7B), trained on only ~370k triplets, achieves an average score of 68.8% across the full 51-task MIEB-Lite suite (which includes multilingual tasks), and 67.6% on the English-only subset. This outperforms the prior state-of-the-art, GME (7B), which achieves 64.5% on the full suite and 63.4% on English-only despite being trained on ~8M samples—approximately 21× more data. The paper notes this explicitly: "Remarkably, despite using only ∼0.37M training pairs—about 21× less data than GME (∼8M)—our multimodal variants set a new state-of-the-art on MIEB" (Section 4.2).

The LCO-EMB-VL (7B) multimodal variant achieves 67.6% on the full MIEB-Lite, narrowly behind the Omni variant. The text-only variants perform competitively despite zero multimodal training data: LCO-EMB-VL (T, 7B) achieves 60.4% on the full suite, outperforming the proprietary Voyage Multimodal 3 (58.1%) and the open-source E5-V (51.9%). Even the 3B text-only variant (58.0%) surpasses Voyage Multimodal 3 on the full suite.

Breaking down by task category (Table 1), LCO-EMB shows particular strength on tasks where MLLM backbones provide advantages: Visual STS (English: 86.6 VL-7B, 86.2 Omni-7B vs. 81.8 for GME; Cross-lingual: 63.1 VL-7B, 64.3 Omni-7B vs. 62.4 for GME; Multilingual: 83.3 VL-7B, 82.4 Omni-7B vs. 75.4 for GME), Document Understanding (75.6 VL-7B, 75.4 Omni-7B vs. 86.1 for GME—the one task where GME retains a clear lead, attributed to GME's extensive document-specific training), and Vision-centric QA (65.2 VL-7B, 70.5 Omni-7B vs. 55.4 for GME). On tasks where CLIP-style models traditionally excel—Clustering (76.0 VL-7B vs. 80.8 for CLIP-ViT-bigG), Linear Probing (72.5 VL-7B vs. 78.8 for SigLIP-so400m), and Zero-shot Classification (66.8 VL-7B vs. 73.8 for SigLIP-so400m)—LCO-EMB narrows but does not fully close the gap. However, on Compositionality, the 7B VL variant (51.0) substantially outperforms CLIP-style baselines (35.0 and 32.8) and prior MLLM-based models (GME: 52.2, with LCO-EMB within striking distance despite the massive data disparity).

Text-Only Training Performance (MIEB-Sub18 Ablation)

The text-only variants of LCO-EMB are evaluated on MIEB-Sub18 (Figure 5) to isolate the core contribution of language-centric contrastive learning without the multimodal calibration stage. Across all six evaluation categories, the text-only LCO-EMB variants consistently outperform the leading open-source baseline (E5-V) and the proprietary baseline (Voyage Multimodal 3). Specific numbers from Figure 5:

• Linear Probe (Accuracy): LCO-EMB-VL-7B achieves 58.61%, compared to E5-V's 37.59% and Voyage Multimodal 3's 31.50%. The paper highlights that this category is where MLLM-based embeddings typically lag behind CLIP-style models.

• Visual STS (Spearman Correlation): LCO-EMB-VL-7B achieves the highest at 84.69 (English), 85.23 (Cross-lingual), and 83.48 (Multilingual). Compared to Voyage Multimodal 3's 80.13 (English), 69.78 (Cross-lingual), and 74.00 (Multilingual), the cross-lingual and multilingual gains are particularly substantial—a +15.45 and +9.48 point advantage respectively.

• Document Understanding (nDCG@5): LCO-EMB-VL-7B scores 65.36, slightly behind Voyage Multimodal 3's 66.50 but substantially ahead of E5-V's 55.47. The paper notes that "Voyage-M3 is a commercial model explicitly optimized on PDF–text pairs for document understanding tasks," making the text-only LCO-EMB's competitive performance notable.

• Multilingual Image Retrieval (nDCG@10): LCO-EMB-VL-7B achieves 64.81, compared to E5-V's 49.46 and Voyage Multimodal 3's 48.26—a margin of +15.35 and +16.55 points respectively.

The paper quantifies the overall advantage: "On average across all evaluation categories, the text-only variants of LCO-EMB have outperformed E5-V and Voyage-Multimodal-3 by 21.69 and 13.00 points" (Section 4.3). The LCO-EMB-Omni variants track closely with the VL variants across all categories, with the 7B versions consistently outperforming the 3B versions.

Training Strategy Comparison (Efficiency and Effectiveness)

Table 3 provides the critical comparison between LoRA-based text-only training and alternative training strategies on Qwen2.5-VL 3B and 7B backbones. The results demonstrate both the effectiveness and the efficiency of the LoRA approach:

Qwen2.5-VL-3B:

• Native (no fine-tuning): 51.49 average across MIEB-Sub18 tasks.
• CLIP-style CL (multimodal, ~453 GPU hours): 53.38 average. This underperforms text-only LoRA despite requiring ~100× more compute.
• Linear Projection (text-only, ~4.5 GPU hours): 52.69 average. Marginal improvement over native embeddings, showing that adding a trainable linear layer without modifying internal representations provides little benefit.
• Full Fine-tuning (text-only, ~8.5 GPU hours): 62.10 average. A reasonable result but notably inferior to LoRA.
• LoRA (text-only, ~4.7 GPU hours): 66.28 average. The best result with the lowest compute cost among trainable methods.

Qwen2.5-VL-7B:

• Native: 52.92 average.
• CLIP-style CL (multimodal, ~550 GPU hours): 50.02 average—actually worse than the native model, indicating disruption of pretrained representations.
• Linear Projection (text-only, ~8.8 GPU hours): 56.22 average.
• Full Fine-tuning (text-only, ~17.3 GPU hours): 66.49 average.
• LoRA (text-only, ~9.3 GPU hours): 71.98 average. The gap between LoRA and full fine-tuning expands from +4.18 points on 3B to +5.49 points on 7B, suggesting that preservation of pretrained structure becomes more important for larger models.

The task-level breakdown reveals that LoRA's advantage over full fine-tuning is largest on Multilingual Image Retrieval (+12.59 points for 7B) and Visual STS-Cross-lingual (+6.21 points for 7B), while both methods perform similarly on Visual STS-English (85.05 vs. 83.15), which is closely aligned with the NLI training distribution.

Dataset Impact and Model Ensemble (Table 2)

Table 2 compares the performance of LCO-EMB (Qwen2.5-VL-7B) fine-tuned on all-NLI versus Scale-1M, and the ensemble of both via model soup:

• all-NLI only: 71.17 average. Excels at Visual STS (English: 84.69, Cross-lingual: 85.23, Multilingual: 83.48) and Document Understanding (65.36).
• Scale-1M only: 71.11 average. Leads on Linear Probing (58.61 vs. 51.86) and Multilingual Image Retrieval (64.81 vs. 56.37).
• Model Soup Ensemble: 72.17 average. Achieves the best overall performance, combining the complementary strengths. The ensemble gains are modest but consistent, with the average improving by roughly 1 point over either individual checkpoint.

This demonstrates that the two text-only datasets optimize complementary aspects of the representation space—NLI sharpens similarity discrimination (benefiting STS and document tasks), while Scale-1M's descriptive and multilingual text activates image-relevant representations and cross-lingual alignment.

Audio and Video Extension

The paper reports audio-text and video-text retrieval results for the Qwen2.5-Omni backbone in Figure 6 (as part of the scaling law analysis, with specific Recall@1 values inferable from the plot axes). For video-text retrieval (V2T Recall@1), the post-CL performance ranges from approximately 38% (Omni-3B) to 46% (Omni-7B), with VL variants achieving intermediate scores. For audio-text retrieval (A2T Recall@1), both Omni variants achieve approximately 23.6-24.3% Recall@1. The Scaling Law figures plot these against generative benchmark performance, with the positive correlation establishing GRSL across modalities. Quantitative comparisons against dedicated audio/video embedding baselines are not provided—the audio and video results serve primarily as evidence for the cross-modal generality of LCO-EMB and the scaling law rather than as standalone SOTA claims.

SeaDoc Experiment Results (Figure 7)

The SeaDoc experiment tests whether enhancing generative capability before contrastive learning improves representation quality. Using Qwen2.5-VL-3B, the results in Figure 7 (nDCG@10):

• Baseline (no additional SFT): text-only contrastive learning on the pretrained model achieves a baseline nDCG@10 score. The high-resolution variant (Baseline-H) achieves higher performance than the low-resolution variant due to better OCR capability in the base model.
• OCR-only SFT at low resolution: Performance collapses below the baseline. The paper describes this as "significant capability collapse" attributed to overfitting, aligning with "observations in recent multimodal reasoning research that continual SFT can lead to overfitting and degrade models' generalization capability" (Section 5.3.3).
• OCR-only SFT at high resolution: Performance improves over low-resolution SFT but remains below baseline. Higher resolution allows better grounding of generated output in visual text rather than spurious visual cues, partially mitigating the collapse.
• OCR+Captions SFT: Incorporating general-domain PixmoCaps captions alongside OCR data yields the best performance, exceeding the baseline at both resolutions. The general-domain captioning data "helps preserve the latent image-text alignment learned by MLLMs during pre-training," which is "crucial for its effective exploitation by the subsequent text-only contrastive finetuning process."

This is the paper's primary causal validation of GRSL: manipulating generative quality through continued pretraining changes the representation quality ceiling, but only when the generative improvement preserves cross-modal alignment. The magnitude of the improvement over baseline is not explicitly stated as a single number in the text but is visible in Figure 7.

Generation-Representation Scaling Law (Figure 6)

Figure 6 plots the relationship between generative benchmark performance (X-axis) and post-contrastive-learning representation performance (Y-axis) across five backbones (LLaVA-Next-8B, Qwen2.5-VL-3B, Qwen2.5-VL-7B, Qwen2.5-Omni-3B, Qwen2.5-Omni-7B) on three modality groups:

• OCR-reliant tasks: A "consistently positive correlation" is observed, with OCR generation performance ranging from ~65 to ~80 and post-CL embedding performance ranging from ~66 to ~74. The 7B models (VL-7B, Omni-7B) occupy the upper-right region, the 3B models occupy the lower-left, and LLaVA-Next-8B sits in between.

• Video-text tasks: Video generation performance spans ~66-71 on the X-axis, and video-text retrieval Recall@1 spans ~38-46 on the Y-axis. Again, the pattern is monotonic: stronger generators yield better retrievers.

• Audio-text tasks: Audio generation performance spans ~65-71, and audio-text retrieval Recall@1 spans ~23.6-24.3. The correlation is positive but the range is narrower than for OCR and video.

The consistent monotonic relationship across modalities and backbones is the empirical basis for GRSL. The paper does not fit a parametric scaling law (no power-law exponent or functional form is estimated); the correlation is presented as qualitative evidence that motivates the theoretical formalization in Section 5.2.

Ablation Studies and Robustness Checks

LoRA hyperparameters (rank and alpha): Table 4 systematically varies LoRA rank ($r \in \{8, 64, 256\}$) and alpha ($\alpha \in \{16, 128, 512\}$) for Qwen2.5-VL-7B, with several findings:

• Tasks aligned with training distribution are insensitive: Visual STS performance varies minimally across configurations (84.98-85.89 for English, 83.39-83.98 for Multilingual), showing that training objective-aligned tasks are robust to LoRA settings.
• Higher alpha degrades OOD task performance: Compositionality (43.07 at $\alpha=128$, $r=256$ vs. 55.64 at $\alpha=16$, $r=64$) and Vision-centric QA (43.40 at $\alpha=128$, $r=64$ vs. 55.64) show "significant performance degradation" at larger alpha values. The paper interprets this as evidence that aggressive LoRA modulation disrupts pretrained knowledge needed for generalization.
• Extreme settings cause training collapse: At $r=256, \alpha=512$, the model experiences "unrecoverable loss spikes in training," resulting in catastrophic performance drops (V-STS English: 5.90, average: 19.31, compared to ~67 for stable configurations).
• No single optimal setting exists: "There doesn't exist a global optimal setting of LoRA hyperparameters" (Appendix D). The paper acknowledges this as a limitation and leaves a more comprehensive empirical and theoretical study of the rank-alpha tradeoff to future work.

Text-only vs. multimodal training data: Table 1 provides the most comprehensive ablation by comparing text-only (T) and multimodal (M) variants across all MIEB-Lite categories. Key observations:

• On tasks closely aligned with NLI training distribution, the multimodal supplement provides small or negligible gains: Visual STS-English shows 88.4 (7B VL-T) vs. 86.6 (7B VL-M)—the text-only variant actually scores higher.
• On tasks with distinctive multimodal formats, the multimodal supplement provides large gains: Document Understanding jumps from 66.0 (7B VL-T) to 75.6 (7B VL-M), +9.6 points; Zero-shot Classification from 49.1 to 66.8, +17.7 points; Multilingual Retrieval from 31.8 to 36.4, +4.6 points.
• The 3B to 7B scaling within each variant type shows consistent improvements: VL-T gains from 56.4 to 57.5 (+1.1 points average); VL-M gains from 61.6 to 66.2 (+4.6 points). The larger model benefits more from the multimodal supplement.

Model backbone comparison (VL vs. Omni): Table 1 shows Qwen2.5-Omni variants outperforming Qwen2.5-VL variants at the same scale for multimodal configurations: Omni-3B (M) achieves 64.8 vs. VL-3B (M)'s 61.6; Omni-7B (M) achieves 67.6 vs. VL-7B (M)'s 66.2. The Omni advantage is attributed to its omnimodal pretraining, providing stronger latent alignment. However, for text-only variants, the VL backbone slightly edges Omni at 7B (57.5 vs. 56.4 in Figure 4 average), though the paper does not discuss this reversal.

Dataset composition and scale effects: Several implicit ablations emerge from comparing across models in Table 1 and Figure 4:

• Data quantity is not the primary driver: LCO-EMB-VL (T, 7B) trained on 276k text-only pairs outscores VLM2Vec-LoRA (trained on 662k multimodal pairs, 49.1 vs. 46.0 on English-only MIEB-Lite). LCO-EMB-VL (M, 7B) trained on 370k pairs outscores GME (trained on 8M pairs, 67.6 vs. 64.5 on full MIEB-Lite). More data helps only if it aligns with the right training strategy.
• Text-only data outperforms multimodal data of comparable scale: LCO-EMB-VL (T, 3B) on 276k text-only pairs achieves 58.0 on full MIEB-Lite, while VLM2Vec-LoRA on 662k multimodal pairs achieves 46.0. The 28.5× data advantage for VLM2Vec is reversed by the training strategy.
• Scale-1M vs. all-NLI complementarity (Table 2): As discussed in Results, the two text-only datasets have complementary strengths that can be combined via model soup for a 1.0 point average improvement.

CLIP-style multimodal training as a negative result: Table 3 reports that CLIP-style contrastive fine-tuning on 800K PixmoCaps image-caption pairs significantly underperforms text-only LoRA across both backbones (3B: 53.38 vs. 66.28; 7B: 50.02 vs. 71.98). For the 7B backbone, CLIP-style training actually degrades performance below the native model (50.02 vs. 52.92). This negative result is crucial because it demonstrates that naïvely applying the dominant CLIP training paradigm to MLLM backbones is counterproductive—the contrastive objective conflicts with the generative pretraining objective, and full-parameter multimodal training disrupts the latent alignment that LCO-EMB preserves.

LoRA vs. Full Fine-tuning (Table 3): The consistent superiority of LoRA over full fine-tuning (3B: 66.28 vs. 62.10; 7B: 71.98 vs. 66.49) is presented as evidence that preserving pretrained structure is essential. The paper frames this as "an objective mismatch: contrastive loss deviates from the model's pretraining objective, and full fine-tuning consequently induces larger perturbations to the pretrained parameters, which are more likely to disrupt the established cross-modal alignment."

Resolution scaling in SeaDoc (Figure 7): The SeaDoc experiment includes an implicit ablation on image resolution, with high-resolution variants (2,621,440 pixels) consistently outperforming low-resolution variants (262,144 pixels) for OCR-intensive tasks. The paper notes this is because "text in visual documents is typically small; training with higher resolution allows for better grounding of the generated output in the visual text of the source image, as opposed to overfitting to example-level visual cues."

Generative data mixture in SeaDoc (Figure 7): The comparison between OCR-only SFT and OCR+Captions SFT demonstrates that the composition of generative training data matters critically. OCR-only training causes "significant capability collapse," while adding general-domain captions preserves latent alignment and improves final retrieval performance. This is a non-obvious finding: one might expect domain-specific generative training to always help, but it can hurt if it narrows the model's representational diversity.

Model soup effectiveness (Table 2): The 1.0 point average improvement from weight-averaging two independently fine-tuned checkpoints—despite no additional training—provides evidence that both fine-tuning runs converge to a shared basin in the loss landscape. This is consistent with the paper's thesis that contrastive learning is a projection operation on a stable pretrained representation space, not a radical restructuring.

Critical Assessment

Do the experiments support the claim that latent cross-modal alignment exists and is activated by text-only contrastive learning?

The anisotropy (Figure 1) and kernel alignment (Figure 2) analyses provide strong, convergent evidence for this claim. The multi-modality anisotropy analysis (text, image, audio, video on Qwen2.5-Omni-3B) shows that text-only contrastive learning makes all four modalities more isotropic, which could not happen if the decoder's representation spaces were separate. The kernel alignment analysis (Figure 2) shows that cross-modal structural alignment improves after text-only CL, and that larger models exhibit stronger alignment both before and after—providing a clean dose-response relationship.

However, these analyses are conducted on a single backbone for the multi-modality case (Qwen2.5-Omni-3B) and a single model family for the kernel analysis (Qwen2.5-VL-3B and 7B). The paper would be strengthened by showing the same diagnostic patterns on LLaVA-Next-8B or other MLLM architectures (e.g., LLaVA-OneVision, InternVL) to demonstrate that latent alignment is a general property of MLLM pretraining rather than a peculiarity of Qwen models. The anisotropy analysis is qualitative—Figure 1 shows curves but no summary statistic quantifying the degree of cross-modal generalization (e.g., correlation between text anisotropy reduction and image anisotropy reduction across layers). The scale of the effect is visually apparent but not numerically characterized.

The text-only to non-text generalization of representation quality (Table 1, text-only variants achieving competitive multimodal performance) provides convergent behavioral evidence. However, the ablations do not include a condition where the decoder is trained on text-only data but the encoders are randomly initialized—such a control would rule out the possibility that the generalization is due to the projector learning identity-like mappings during pretraining rather than genuine latent alignment.

Do the experiments support the claim that LCO-EMB achieves state-of-the-art performance with dramatically less data?

This claim is strongly supported by Table 1 and Figure 4. LCO-EMB-Omni (M, 7B) achieves 68.8% on full MIEB-Lite versus GME's 64.5%, with ~21× less training data (~370k vs. ~8M triplets). The text-only variant (T, 7B at 60.4%) outperforms VLM2Vec (46.0% on 662k samples) and Voyage Multimodal 3 (58.1%).

A genuine weakness: the data comparison is somewhat asymmetric. GME's 8M pairs are primarily multimodal (image-text, interleaved), while LCO-EMB's ~370k pairs are 75% text-only. The data efficiency claim would be cleaner if the paper reported an experiment with text-only data scaled up to match GME's volume—would LCO-EMB continue to improve with more text-only data, or would it plateau? The current results suggest diminishing returns might already be setting in (7B text-only achieves 60.4% on 276k pairs, only marginally better than 3B at 58.0%), but without an explicit data scaling curve, the shape of the scaling function is unknown.

Additionally, the SOTA comparison against GME is partially confounded by model architecture differences: LCO-EMB uses Qwen2.5 backbones (released 2025), while GME uses an earlier Phi-3-Vision backbone. Stronger pretraining in the newer Qwen models could contribute to the performance gap independent of the training methodology. A controlled comparison training LCO-EMB and GME on the same backbone would isolate the method's contribution.

Do the experiments support the Generation-Representation Scaling Law?

The evidence for GRSL comes in two forms: cross-sectional (Figure 6) and interventional (Figure 7, SeaDoc). Both have strengths and weaknesses.

Cross-sectional evidence (Figure 6): The monotonic relationship between generative and representation performance across five backbones and three modality groups is visually compelling, but the sample size is small (five data points per plot). With only five backbones—three from the Qwen family (VL-3B, VL-7B, Omni-3B, Omni-7B) and one LLaVA variant—the correlation could be driven by model scale (larger models are better at everything) rather than a specific generative-to-representation transfer. The paper does not compute a correlation coefficient, fit a scaling law parameter, or test whether the relationship is better explained by a simple parameter-count model. A critical missing experiment would be to compare models matched on parameter count but differing in generative quality (e.g., different checkpoints during pretraining, or different training data mixtures at the same scale).

SeaDoc intervention (Figure 7): This is the paper's strongest causal evidence but has important limitations. The experiment shows that continual SFT on OCR+Captions improves retrieval over the baseline, consistent with GRSL. However, the magnitude of improvement is not clearly quantified in the text, and Figure 7 uses nDCG@10 on a scale of ~24-38, making the absolute gains appear modest. The paper does not report the generative performance improvement from the SFT stage—did OCR+Captions SFT actually improve generative OCR metrics? Without showing that generative quality increased, the causal chain (better generation → better representation) is incomplete; an alternative explanation is that the additional training data (even if applied via SFT rather than contrastive learning) simply provided useful regularization or broader exposure.

The OCR-only SFT collapse is striking but is presented as supporting evidence (alignment preservation matters) rather than as a challenge to GRSL's generality. If OCR-only SFT reduces representation quality, it means the relationship between generative and representation quality is not monotonic—it depends on how generative quality is improved. GRSL as stated ("representational capabilities gained through contrastive refinement scales positively with the MLLM's generative capabilities") would predict that any generative improvement helps, which the OCR-only result refutes. The paper implicitly acknowledges this by emphasizing that alignment preservation is necessary, but the law's statement should arguably include this condition.

Do the experiments support the theoretical bound (Theorem 1)?

The PAC-Bayesian analysis in Section 5.2 provides a formal framework for understanding GRSL, but the empirical validation is indirect. The bound decomposes representation quality into three terms: generative bottleneck, optimization inefficiency, and complexity penalty. The experiments demonstrate that LoRA (which keeps the complexity penalty small) outperforms full fine-tuning (Table 3), consistent with the bound's prediction. The experiments also show that better generative backbones yield better representations (Figure 6), consistent with the generative bottleneck term.

However, the bound is not directly tested: the paper does not estimate the mutual information $I_P(X; Y)$ for different backbones, does not measure the KL divergence $\text{KL}(Q\|P)$ for different training strategies, and does not verify that the quantitative relationship between generative loss and contrastive risk matches the bound's functional form. The bound serves as a conceptual framework and a source of testable predictions, but the empirical validation is qualitative (direction of effects) rather than quantitative (magnitude and functional form). The paper acknowledges this implicitly by not presenting numerical evaluations of any bound terms.

What experiments are missing that would strengthen the paper?

Cross-architecture validation of latent alignment: The anisotropy and kernel alignment diagnostics should be replicated on non-Qwen architectures (LLaVA-OneVision, InternVL2, PaliGemma) to establish generality of the latent alignment property.

Data scaling curves: A systematic sweep of text-only training data volume (e.g., 10k, 50k, 100k, 276k, 1M pairs) would reveal whether text-only contrastive learning is approaching diminishing returns at ~276k pairs or whether further scaling would yield continued improvements. Similarly, a fixed text-only base with varying multimodal supplement sizes would characterize the calibration effect more precisely.

Generative quality measurement in SeaDoc: The SeaDoc experiment should report the change in generative OCR metrics (e.g., OCRBench score) from the SFT stage, not just the downstream retrieval performance, to complete the causal chain.

Controlled comparison at matched backbone: Training LCO-EMB and GME/mmE5 on the same backbone (e.g., Qwen2.5-VL-7B) with the same dataset scale would isolate the method's contribution from the backbone's contribution.

Direct evaluation of the PAC-Bayes bound terms: Estimating $I_P(X; Y)$, $\text{KL}(Q\|P)$, and the empirical contrastive risk for different backbones and training strategies would provide a quantitative test of Theorem 1, moving it from conceptual framework to validated theory.

Statistical significance: The paper reports point estimates without confidence intervals. With a 500-question test set for MIEB (some categories have as few as 2 tasks), the variability of the reported averages is unknown, making it difficult to assess whether differences of 1-2 points (e.g., between model soup and individual checkpoints in Table 2) are reliable.

Generalization beyond MIEB: All vision-language results are on MIEB/MIEB-Lite. While MIEB is comprehensive (130 tasks in the full version, 51 in Lite), it is a specific benchmark suite. Evaluation on additional benchmarks (e.g., ARO for compositionality, XM3600 for multilingual retrieval, specific document understanding benchmarks beyond Vidore) would strengthen the generality claim. The audio and video results are limited to Recall@1 on a few datasets and serve primarily as scaling law evidence rather than standalone embedding quality benchmarks.

Where do the claims hold conditionally?

The SOTA claim holds on MIEB-Lite for Qwen2.5-based backbones but may not transfer to other model families without replication. The data efficiency claim is clearly supported at the specific data volumes tested (~276k text, ~370k total) but the shape of the scaling function beyond these points is unknown. The GRSL claim holds as a positive correlation across the tested backbones, but the SeaDoc experiment reveals an important conditional: generative improvement enhances representation quality only when it preserves (or enhances) cross-modal alignment; narrow domain-specific generative training can be counterproductive. The theoretical bound provides a qualitative framework consistent with empirical patterns but has not been quantitatively validated.

6. Limitations and Trade-offs

6.1 The Latent Alignment Hypothesis Is Validated on a Single MLLM Family for Multi-Modality Diagnostics

The assumption or constraint. The paper's foundational empirical claim—that text-only contrastive fine-tuning reduces anisotropy and improves kernel alignment for non-text modalities because MLLMs inherently possess latent cross-modal alignment—is supported by diagnostic analyses conducted overwhelmingly on Qwen-family models. The multi-modality anisotropy analysis (Figure 1) uses exclusively Qwen2.5-Omni-3B. The kernel alignment analysis (Figure 2) uses Qwen2.5-VL-Instruct at 3B and 7B scales. The scaling law analysis (Figure 6) adds LLaVA-Next-8B, but only for the cross-sectional correlation, not for the core anisotropy/kernel diagnostic. No other MLLM architectures (e.g., LLaVA-OneVision, InternVL2, PaliGemma, Phi-3-Vision, Llama-3.2-Vision) are tested for the latent alignment property that motivates the entire method.

The paper effectively treats "the standard architecture of modality-specific encoders, a projector, and a language decoder" (Section 4.1) as a sufficient condition for latent alignment, but this is an untested generalization. The Qwen family may have specific pretraining properties—such as the proportion of multimodal data during generative training, the use of instruction tuning, or architectural details of the projector—that create unusually strong latent alignment compared to other MLLMs. The paper does not discuss whether different pretraining recipes (e.g., frozen vision encoders vs. co-trained encoders, different multimodal data mixtures, different projector designs) would produce different degrees of latent alignment.

The consequence. If latent alignment is weaker or absent in other MLLM families, LCO-EMB's text-only training strategy could fail to generalize to non-text modalities. Practitioners using backbones other than Qwen2.5 would have no way to predict, without running the full diagnostic pipeline themselves, whether text-only contrastive fine-tuning will produce competitive multimodal embeddings or collapse to near-random performance on visual tasks. The paper provides no a priori indicators of which backbones will exhibit strong latent alignment beyond model scale (larger is better within a family, per Figure 2), but scale is confounded with model architecture and training recipe.

Furthermore, if latent alignment is a property that varies continuously across backbones, the optimal training strategy (ratio of text-only to multimodal data, LoRA rank and alpha, whether to even attempt text-only training) may be backbone-dependent. The paper's recipe of ~276k text + ~94k multimodal pairs with LoRA rank 64, alpha 16 may be tuned to the specific alignment characteristics of Qwen models and could be suboptimal for others.

What evidence exists in the paper. The Qwen backbone dominance is apparent from Table 1 and Figure 4: LCO-EMB is evaluated on Qwen2.5-VL (3B, 7B) and Qwen2.5-Omni (3B, 7B). LLaVA-Next-8B appears in Figure 6 for the scaling law analysis but not in the main MIEB results. The anisotropy diagnostic (Figure 1) is exclusively Qwen2.5-Omni-3B. The kernel alignment diagnostic (Figure 2) is exclusively Qwen2.5-VL-Instruct. The training strategy comparison (Table 3) is exclusively Qwen2.5-VL. The paper does not discuss or acknowledge this architectural concentration as a limitation.

Mitigation status. The paper does not address this limitation. It does not report experiments on non-Qwen backbones for the primary MIEB results, nor does it discuss whether the latent alignment property might be architecture-dependent. The inclusion of LLaVA-Next-8B in Figure 6 provides a single data point outside the Qwen family, but LLaVA-Next shares architectural DNA with Qwen (both use a ViT vision encoder + LLM decoder architecture) and was likely trained with similar multimodal data sources, so it may not represent a meaningfully independent test.

6.2 The Difficulty Estimation Methodology Is Absent—But Analogous Costs Make the Practical Deployment Equation Unclear

The assumption or constraint. The paper builds LCO-EMB on the insight that cross-modal alignment is latent in pretrained MLLMs and only requires lightweight activation. However, the paper provides no mechanism for a practitioner to determine, before training, whether a given MLLM backbone possesses sufficient latent alignment to benefit from text-only contrastive fine-tuning, nor how much multimodal calibration data will be needed. The diagnostic tools developed in Section 2 (anisotropy measurement, kernel similarity analysis) serve as post-hoc validation of the hypothesis for the specific backbones studied, but the paper does not operationalize them as a pre-training screening procedure.

This creates a practical deployment dilemma. A practitioner with a new MLLM backbone faces a choice: (a) trust that latent alignment exists and deploy text-only LoRA training, hoping for cross-modal generalization; (b) run the full diagnostic pipeline (anisotropy across all modality pairs with paired datasets like Pixmo Cap, AudioCaps, MSR-VTT; kernel alignment across layers) before committing to a training strategy; or (c) skip diagnosis and train with multimodal data from the start, which the paper shows can be counterproductive (CLIP-style training degraded performance below native embeddings on Qwen2.5-VL-7B, Table 3). Option (a) risks wasted compute if the backbone lacks alignment; option (b) requires curating paired datasets and running expensive forward passes through the full MLLM for diagnosis—costs not accounted for in the headline training efficiency numbers; option (c) may yield worse results.

The analogous problem in the reference example paper on test-time compute scaling was difficulty estimation: picking the optimal inference strategy required knowing the prompt's difficulty, but estimating difficulty cost 2048 forward passes—a cost not included in the reported compute savings. LCO-EMB faces a similar unaccounted cost: the diagnosis required to apply the method correctly is not part of the method itself.

The consequence. The reported efficiency gains—~4.7 GPU hours for text-only LoRA training on 3B models, ~21× less data than GME—are accurate given the right backbone, but the cost of determining whether a backbone is right is absent from the ledger. If a practitioner must run the full anisotropy and kernel alignment diagnostics on each candidate backbone (sampling thousands of examples across all modality pairs, performing forward passes at every layer), the practical cost of applying LCO-EMB could be substantially higher than the training cost alone. For a single backbone, this might be negligible; for systematic backbone selection across many candidates, it could dominate the budget.

Moreover, the paper's finding that CLIP-style training degrades performance on good backbones (Table 3, 7B dropping from 52.92 to 50.02) means that choosing the wrong strategy is actively harmful. Without diagnostic indicators, the practitioner risks not just inefficiency but value destruction.

What evidence exists in the paper. The paper's diagnostic results themselves demonstrate the cost: Figure 1 requires encoding text, image, audio, and video inputs through Qwen2.5-Omni-3B across all 36 layers for three separate paired datasets (Pixmo Cap, AudioCaps, MSR-VTT), computing pairwise cosine similarities among all samples for anisotropy estimation. Figure 2 similarly requires encoding paired image-text samples through Qwen2.5-VL at every layer for both 3B and 7B variants, computing nearest-neighbor structures. The paper does not report the computational cost of these diagnostic procedures, nor does it discuss how they could be streamlined for practical use.

The training strategy ablation (Table 3) reveals that the wrong choice is costly: CLIP-style training on 800K PixmoCaps pairs consumes ~453 GPU hours (3B) and ~550 GPU hours (7B) yet underperforms simple text-only LoRA (~4.7 and ~9.3 hours respectively). Without diagnostics, a practitioner might default to the CLIP-style approach—the dominant paradigm—and waste ~100× the compute for worse results.

Mitigation status. Not addressed. The paper provides the diagnostic tools as an analysis apparatus to justify the method, not as a practical screening procedure that practitioners can use before training. There is no discussion of operationalizing the diagnostics (e.g., training a lightweight classifier to predict alignment quality from a small number of forward passes, or developing heuristic indicators that correlate with latent alignment strength). The paper does not acknowledge this as a deployment cost or suggest how to manage it.

6.3 The Generation-Representation Scaling Law Is a Correlation Across Five Models, Not a Validated Causal Law with Quantified Parameters

The assumption or constraint. The paper frames GRSL as a "law"—a term implying a reliable, quantitative, causal relationship—but the empirical evidence consists of five data points per modality group in Figure 6 (LLaVA-Next-8B, Qwen2.5-VL-3B, Qwen2.5-VL-7B, Qwen2.5-Omni-3B, Qwen2.5-Omni-7B) with no fitted parametric function, no correlation coefficient, and no confidence intervals. The SeaDoc experiment provides a causal intervention, but only for a single backbone (Qwen2.5-VL-3B) on a single task (visual document retrieval in Southeast Asian languages), and the magnitude of the improvement from OCR+Captions SFT over baseline is not numerically quantified in the text.

Several critical questions about GRSL remain unanswered by the empirical evidence:

• Functional form: Is the relationship log-linear, power-law, or something else? Without a fitted curve, the scaling behavior at extrapolated generative qualities (e.g., applying LCO-EMB to a 70B MLLM) is unpredictable.
• Saturation: Does representation quality saturate as generative quality increases, or does it continue to improve? The five data points span a narrow generative quality range and show roughly linear relationships, but the sample is too small to detect curvature.
• Causal direction: The SeaDoc experiment shows that generative improvement can cause representation improvement, but the paper cannot rule out that both are driven by a common latent factor (e.g., pretraining data quality, architectural efficiency) that the cross-sectional correlation reflects.
• Generality: The OCR-only SFT condition in SeaDoc causes representation degradation, showing that the relationship between generative and representation quality is not monotonic—it depends on how generative quality is improved and whether alignment is preserved. The "law" as stated ("representational capabilities gained through contrastive refinement scales positively with the MLLM's generative capabilities") is falsified by the OCR-only result; the actual relationship requires the qualifier "when cross-modal alignment is preserved or enhanced."

The consequence. Practitioners considering whether to invest in continued generative pretraining to improve their embedding models lack the quantitative guidance that a properly specified scaling law would provide. How much generative improvement (e.g., in terms of perplexity reduction or benchmark score increase) translates to how much representation improvement? Is the relationship efficient enough to justify the cost of additional generative pretraining, or would resources be better spent on more contrastive data or better training recipes? The paper provides qualitative evidence that the direction exists but no quantitative purchasing power to make resource allocation decisions.

The PAC-Bayesian analysis (Section 5.2) provides a theoretical framework that could answer these questions—the bound links generative mutual information I_P(X; Y) to contrastive population risk—but the paper does not estimate any bound terms, leaving the theory disconnected from the empirical data. Practitioners cannot look at the bound and predict, for a given generative loss reduction, how much their embedding benchmark scores will improve.

What evidence exists in the paper. Figure 6 (three scatter plots, five points each) constitutes the entirety of the cross-sectional evidence for GRSL. The SeaDoc experiment (Figure 7) provides a single causal intervention with qualitative results (bars showing nDCG@10 under different conditions). No quantitative scaling law parameters are estimated. The theoretical bound (Theorem 1) is derived but not empirically evaluated—mutual information I_P(X; Y), KL divergence KL(Q‖P), and the inefficiency gap ε_P are never estimated for any backbone.

The paper's language around GRSL is appropriately hedged in some places ("we identify a Generation-Representation Scaling Law, indicating that multimodal representational capabilities... scales positively with the MLLM's generative capabilities," Abstract—note the qualitative "positively" rather than a functional form) but overclaims in others ("This suggests that improving generative abilities evolves as an effective paradigm for enhancing representation quality," Abstract—this is a causal claim that the SeaDoc experiment only partially supports, given its single-backbone, single-task scope).

Mitigation status. Partially addressed by the theoretical analysis in Section 5.2, which provides a framework for quantifying the relationship. The bound decomposes representation quality into generative bottleneck, optimization inefficiency, and complexity penalty—a structure that, if empirically estimated, could produce quantitative scaling predictions. However, the paper does not perform these estimations. The authors acknowledge that "an optimal rank and alpha likely exist for models of each size" (Appendix D) and leave "a more comprehensive empirical analysis and theoretical study to quantify this relationship for future work," but this refers to LoRA hyperparameters rather than the GRSL relationship itself. The GRSL's status as a quantified law remains aspirational rather than established.

6.4 Hard Problems and Out-of-Distribution Tasks Remain Unsolved—The Method Amplifies But Does Not Create Capability

The assumption or constraint. LCO-EMB, like all contrastive refinement methods, can only improve representations within the representational capacity established by the backbone's generative pretraining. The theoretical bound (Theorem 1) encodes this explicitly: the generative bottleneck term log N − I_P(X; Y) sets a floor below which contrastive fine-tuning cannot push the population risk. If the backbone MLLM has fundamentally poor representations for a certain task class—because its pretraining data lacked that type of multimodal content, or because the architecture is ill-suited to it—no amount of text-only contrastive learning or multimodal calibration will create the missing capability.

This is a fundamental limitation, analogous to the finding in the reference paper on test-time compute scaling that the hardest questions (difficulty bin 5) show near-zero improvement regardless of how much inference compute is allocated—test-time strategies amplify existing capability but do not create it from nothing. LCO-EMB faces the same boundary: representation quality after contrastive refinement is bounded above by what the generative prior encodes.

The paper's own results reveal specific capability gaps that persist even in the best LCO-EMB variants:

• Clustering: The best LCO-EMB variant (7B Omni, M) achieves 80.0 on the Clustering NMI metric, compared to 80.8 for CLIP-ViT-bigG (Table 1). This narrow gap (~0.8 points) hides the fact that CLIP achieves this with a 2B vision encoder versus LCO-EMB's 7B MLLM—the MLLM has ~3.5× more parameters but has not closed the gap, suggesting a structural limitation in how MLLM representations capture category-level visual similarity structures.
• Linear Probing: The best LCO-EMB variant (7B Omni, M) achieves 74.1, compared to 78.8 for SigLIP-so400m (Table 1). MLLM-based embeddings systematically underperform CLIP-style models on linear probing, a finding consistent with the MIEB benchmark study that the paper itself cites.
• Document Understanding vs. GME: LCO-EMB's best Document Understanding score (75.6, 7B VL, M) trails GME's 86.1 by 10.5 points (Table 1). Despite achieving higher overall MIEB-Lite averages, LCO-EMB cannot match GME on document-specific tasks, likely because GME's training data included massive document-specific pairs that LCO-EMB's minimal calibration set (~23k visual document triplets) cannot compensate for.

The consequence. For applications where the backbone MLLM has weak inherent representations—low-resource languages not well-represented in pretraining, specialized domains (medical imaging, remote sensing, industrial inspection) where the MLLM's pretraining data is sparse, or tasks requiring representational properties that generative pretraining does not optimize (class-separability for linear probing, hierarchical clustering structure)—LCO-EMB provides limited benefit over the base model and may underperform CLIP-style models trained from scratch on domain-specific paired data. The paper's data efficiency advantage (~21× less than GME) is partly enabled by leveraging the backbone's pretrained knowledge—but this leverage fails when the backbone lacks that knowledge.

The SeaDoc experiment demonstrates this problem directly: OCR-only SFT without general-domain caption preservation causes "significant capability collapse" because the narrow SFT data erodes the general cross-modal alignment that LCO-EMB depends on. This means that domain adaptation—improving a general MLLM for a specific task through continued generative training—requires careful data balancing to preserve the latent alignment structure. There is no guarantee that such balancing is possible for arbitrary target domains, and the paper provides no methodology for determining the right mixture.

What evidence exists in the paper. Table 1 shows clustering and linear probing scores where LCO-EMB trails CLIP-style baselines. The Document Understanding gap with GME (86.1 vs. 75.6) is evident in the same table. The SeaDoc experiment (Figure 7, OCR-only SFT collapse) demonstrates the fragility of domain adaptation. The theoretical bound (Theorem 1, generative bottleneck term log N − I_P(X; Y)) formally encodes the limitation: I_P(X; Y) is determined by the generative prior and cannot be increased through contrastive fine-tuning alone.

Mitigation status. The paper acknowledges this limitation implicitly through the theoretical bound, which formalizes the generative bottleneck, but does not discuss it as a practical constraint on LCO-EMB's applicability. The recommended mitigation—improve generative capability through continued pretraining, then apply contrastive refinement—is the SeaDoc experiment's logic, but the OCR-only collapse shows this is not straightforward. The paper does not provide systematic guidance on when continued generative pretraining will help versus hurt, what data mixtures preserve alignment, or how to diagnose whether a target domain is within the backbone's capability range before investing in training.

6.5 Sequential Dependency and Latency: The Training Is Lightweight, but the Inference Architecture Is As Heavy As the Original MLLM

The assumption or constraint. LCO-EMB produces embeddings by running the full MLLM forward pass: modality-specific encoder → projector → language decoder → last hidden state extraction. While the training is lightweight (~4.7 GPU hours for text-only LoRA on 3B), the inference cost is identical to running the full backbone MLLM for each input being embedded, because the LoRA adapters are merged into the weights and the forward pass goes through all layers.

This creates a significant mismatch between training efficiency and inference cost that the paper does not discuss. A CLIP-style model produces embeddings by running a single vision encoder (for images) or text encoder (for text)—each typically much smaller than an MLLM's language decoder. For example, CLIP-ViT-bigG has a vision encoder with ~1B parameters and a text encoder with ~300M parameters, producing an image embedding with a forward pass through ~1B parameters. LCO-EMB based on Qwen2.5-VL-7B requires running the vision encoder plus the entire 7B-parameter language decoder for a single image embedding—a forward pass through roughly an order of magnitude more parameters.

For retrieval applications where the corpus must be pre-embedded (e.g., embedding millions of images for nearest-neighbor search), this per-item inference cost multiplies across the corpus size. The paper's headline efficiency numbers (~4.7 GPU hours training, ~21× less data) address the training side but not the inference side. A practitioner choosing between LCO-EMB (7B) and a CLIP-style model (2B total) for a large-scale retrieval system must weigh the accuracy gains against the per-item embedding cost, which could differ by 5-10×.

Additionally, LCO-EMB's inference involves a serial dependency that CLIP avoids: the vision encoder processes the image, the projector maps features to the decoder's embedding space, and then the decoder must autoregressively generate tokens (or at minimum process the visual features through all transformer layers) before the last hidden state can be extracted. While the paper does not report inference latency numbers, the decoder forward pass for even a single token typically requires multiple milliseconds on GPU hardware, versus the vision encoder's single forward pass. For real-time applications requiring low-latency embedding (e.g., interactive search, live video processing), this overhead may be prohibitive.

The consequence. LCO-EMB's inference cost undermines its attractiveness for latency-sensitive or large-scale deployments. The paper's emphasis on training efficiency (~4.7 GPU hours, ~276k text pairs) tells only half the story: if embedding a million-image corpus costs 5-10× more in GPU-hours than a CLIP alternative, the total cost of ownership (training + inference) may favor the CLIP approach despite its higher training cost, especially for applications with high inference-to-training ratios (many items embedded, few training runs).

Moreover, the inference cost is particularly relevant given the paper's own comparison point: the text-only LCO-EMB (T, 7B) outperforms Voyage Multimodal 3 (a proprietary model whose architecture and inference cost are unknown) but might not outperform it on an accuracy-per-inference-FLOP basis if LCO-EMB requires significantly more compute per embedding. The paper's argument for efficiency is one-sided (training only) and does not address the deployment economics that practitioners care about.

What evidence exists in the paper. The inference cost is never measured, discussed, or compared to baselines. The training cost comparisons (Table 3, GPU hours) are the only compute metrics reported. The paper does not provide throughput numbers (embeddings per second), latency measurements, or parameter-count-to-embedding-cost ratios. The LCO-EMB architecture description (Section 3, Figure 3, Section 4.1) makes clear that the full MLLM is used at inference, but the cost implications are not analyzed.

Mitigation status. Not addressed. The paper does not discuss inference cost, latency, or throughput as a tradeoff against training efficiency. There is no mention of potential mitigations such as distillation (training a smaller student model to mimic LCO-EMB embeddings), early exit from the decoder (extracting embeddings from intermediate layers rather than the final hidden state), or architecture modifications that reduce the decoder's inference cost for embedding tasks. The limitation is entirely absent from the paper's discussion.

6.6 The Method Requires Access to the Full MLLM—Including the Language Decoder—Which May Be Impractical for Proprietary or API-Accessed Models

The assumption or constraint. LCO-EMB fundamentally requires white-box access to the full MLLM, specifically the ability to: (1) extract the language decoder from the full architecture, (2) inject LoRA adapters into specific decoder layers, (3) fine-tune the decoder on text-only data with a contrastive objective, (4) merge the LoRA weights back into the decoder, and (5) reinsert the decoder into the full MLLM. Each of these steps requires direct access to model weights, the training code, and the ability to run forward and backward passes through the decoder independently.

This constraint excludes all MLLMs accessed via API (GPT-4V, Gemini 2.5 Flash—ironically used to construct SeaDoc annotations—Claude 3.5, Reka, etc.), as well as any open-weight model where the practitioner lacks the infrastructure to perform fine-tuning (GPU memory requirements for 7B models with LoRA are modest but non-trivial, typically requiring 24-48GB VRAM). It also excludes non-standard MLLM architectures where the decoder cannot be cleanly isolated (e.g., early-fusion architectures, models without a clear encoder-projector-decoder separation).

The paper's methodological insight—that generative pretraining creates latent alignment that contrastive learning activates—could in principle apply to API-accessed models: their embeddings likely also exhibit latent cross-modal alignment. But LCO-EMB as a method cannot be applied to them, because the fine-tuning step requires weight access.

The consequence. A large fraction of the MLLM ecosystem is off-limits to LCO-EMB. Practitioners using commercial MLLM APIs (which represent a substantial portion of production deployments, especially for organizations without in-house model training infrastructure) cannot benefit from the paper's data-efficient training recipe. They are limited to either: (a) using the API model's native embeddings (which the paper shows, in Table 3, are substantially worse than LCO-EMB—Qwen2.5-VL-7B native averages 52.92 vs. 71.98 for text-only LoRA), (b) using a separate embedding model not integrated with their MLLM, or (c) training a separate open-weight MLLM with LCO-EMB and accepting the associated hosting costs.

Moreover, the open-weight requirement constrains model selection. The paper demonstrates strong results on Qwen2.5 backbones, but a practitioner whose application benefits from a specific MLLM's capabilities (e.g., a model with superior OCR, better multilingual support, or stronger reasoning) can only use LCO-EMB if that model is open-weight and architecturally compatible. As the MLLM field evolves, the gap between the best proprietary and best open-weight models may widen or narrow unpredictably, and LCO-EMB's applicability is tied to the open-weight frontier.

What evidence exists in the paper. All experiments use open-weight models: Qwen2.5-VL-Instruct, Qwen2.5-Omni, and LLaVA-Next, all available on HuggingFace with standard architectures. The fine-tuning procedure (Section 4.1, Appendix D) assumes full weight access. The paper uses Gemini-2.5-Flash for SeaDoc data construction (Appendix E) but does not attempt to apply LCO-EMB to it. The limitation of API access is not discussed.

Mitigation status. Not addressed. The paper does not discuss whether LCO-EMB's insights could be translated to API-accessible models through alternative techniques—for example, prompt-based embedding extraction (where the model is prompted to produce a task-specific embedding token via few-shot examples), adapter-based methods that operate on the input/output side rather than on internal weights, or distillation from open-weight LCO-EMB models to proprietary ones. The paper takes no position on the closed-source vs. open-source applicability divide, leaving practitioners to infer that LCO-EMB is an open-weight-only method.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the landscape of multimodal representation learning by inverting the assumed causal relationship between generative pretraining and contrastive alignment. The core conceptual shift is the recognition that cross-modal alignment is not something contrastive learning creates from scratch—it is something generative pretraining establishes, which contrastive learning then activates and refines. This is a genuine paradigm shift, not an incremental refinement, because it reorients the field's understanding of where representation quality comes from and, consequently, where research investment should be directed.

The shift has several concrete dimensions:

From data-scaling to backbone-quality as the primary lever. The dominant narrative in multimodal embedding research has been that more contrastive data produces better representations—a narrative inherited directly from CLIP and reinforced by the Chinchilla-era emphasis on data scaling. GME trained on ~8M pairs; mmE5 on ~2.1M; VLM2Vec on 662k. The assumption was that scaling this number was the path forward. LCO-EMB achieves state-of-the-art with ~370k pairs (~21× less than GME), and its text-only variant surpasses proprietary models using zero multimodal pairs. The takeaway is not merely "we can be more data-efficient," but rather "the data we thought was doing the heavy lifting was actually doing light calibration on a structure established elsewhere." This reframes the bottleneck: if you want better multimodal embeddings, invest in better generative MLLMs—either by waiting for improved pretrained models from the frontier or by continuing generative pretraining on your target domains (as demonstrated by SeaDoc). Scaling contrastive data beyond calibration needs becomes a secondary optimization.

From contrastive learning as alignment mechanism to contrastive learning as projection operator. This is the paper's most subtle and consequential reframing. Under the CLIP paradigm, contrastive learning is the mechanism of cross-modal alignment: the InfoNCE loss pulls matching image-text pairs together and pushes non-matching pairs apart, and this process is what teaches vision and text encoders to inhabit a shared space. The paper argues—and empirically demonstrates—that in MLLMs, contrastive learning serves a more modest function: it projects representations that are already aligned (by the generative objective) into a similarity-matching subspace where cosine similarity corresponds to semantic relatedness. This is why text-only NLI data works for multimodal embedding: NLI teaches the model that entailment pairs should have high cosine similarity and contradiction pairs should have low cosine similarity, and this geometric correction propagates to all modalities because the underlying space is shared.

The practical consequences of this reframing are already visible in the training cost numbers (Table 3): text-only LoRA training on ~276k pairs takes ~4.7 GPU hours and achieves 66.28 average on MIEB-Sub18; CLIP-style contrastive training on 800K image-caption pairs takes ~453 GPU hours and achieves 53.38. The 100× compute reduction is not an engineering optimization—it is direct evidence that the field has been misallocating compute by treating contrastive learning as the alignment mechanism when it is actually a projection operator operating on pre-aligned representations.

Reconciling conflicting narratives about MLLM-based embeddings. Prior to this work, the field had documented that MLLM-based embedding models work well (E5-V, VLM2Vec, mmE5, GME) but had no explanation for why. This created an uneasy situation where the empirical results were compelling but practitioners had no principled way to choose among the proliferating training recipes, backbones, and data mixtures. Some approaches used massive multimodal data (GME, mmE5); others used text-only data (E5-V). Some used full fine-tuning; others used parameter-efficient methods. The lack of a mechanistic understanding meant that each new model was essentially a shot in the dark guided by intuition and resource availability.

The paper resolves this fragmentation by providing a unified explanatory framework: all these approaches work because they are, to varying degrees, projecting pre-existing generative alignment into a similarity-matching space. Their differences in data volume, modality mix, and training strategy reflect different (and sometimes suboptimal) choices about how to perform this projection. The framework explains why E5-V's text-only approach works (the alignment is already there; text-only data is sufficient for geometric correction), why GME's 8M pairs are partially redundant (most of those pairs are doing calibration, not alignment), and why full fine-tuning underperforms LoRA (it disrupts the latent alignment during the projection step). The paper converts a confusing landscape of empirical results into a coherent picture with a clear optimization target: preserve the latent alignment, refine it minimally, calibrate to task formats sparingly.

Reorienting the research agenda toward generative pretraining as the foundation for representation learning. The most significant long-term impact of this work may be that it makes generative pretraining—not contrastive data collection—the central investment for multimodal representation quality. If the Generation-Representation Scaling Law holds broadly, then improving representation quality is primarily a matter of improving the backbone MLLM's generative capabilities: better pretraining data, larger models, continued generative training on target domains, instruction tuning that enhances multimodal reasoning. Contrastive refinement becomes a lightweight post-processing step rather than the main event.

This has implications for how organizations allocate research budgets. A team building a multimodal retrieval system faces a choice: invest in curating millions of domain-specific image-text pairs for contrastive training, or invest in continued generative pretraining of their MLLM on domain-specific data, followed by lightweight text-only contrastive refinement. The paper's results strongly suggest the latter is more efficient, with the SeaDoc experiment providing a concrete validation: OCR+Captions generative SFT before contrastive learning improves retrieval more than contrastive learning alone, while OCR-only SFT (which disrupts general alignment) degrades it. The practical recipe is clear: improve the generative model's multimodal understanding, then apply LCO-EMB's lightweight refinement.

Making parameter-efficient fine-tuning a principled choice rather than a resource compromise. LoRA has often been treated as a concession to limited compute—a way to approximate full fine-tuning when you cannot afford to update all parameters. The paper repositions LoRA as the preferred approach for a specific reason grounded in the method's theory: it preserves the latent cross-modal alignment established during generative pretraining. Full fine-tuning, by allowing unrestricted parameter updates, can disrupt this alignment even if the contrastive objective is correct—a phenomenon the paper calls "objective mismatch." The PAC-Bayesian analysis provides formal justification: the KL(Q‖P) term in the generalization bound penalizes posteriors that stray far from the generative prior, and LoRA keeps this term small by construction.

This reframing matters because it changes the default training strategy for practitioners. Under the old view, you would use full fine-tuning if you could afford it and fall back to LoRA if you could not. Under the paper's view, you use LoRA (or other parameter-efficient methods) because full fine-tuning destroys the very property—latent alignment—that makes MLLM-based embeddings effective. The ablation in Table 3 confirms this: full fine-tuning consistently underperforms LoRA across both 3B and 7B scales, with the gap widening for larger models (4.18 points on 3B, 5.49 points on 7B). The practical implication is that LoRA is not a compromise; it is the correct algorithmic choice given the structure of the problem.

Follow-Up Research This Work Enables

Estimating and empirically validating the PAC-Bayesian bound terms across backbones and training strategies. The paper's theoretical framework (Theorem 1, Section 5.2) decomposes representation quality into a generative bottleneck term (log N − I_P(X; Y)), an optimization inefficiency gap (ε_P), and a complexity penalty (√(KL(Q‖P) + log(1/δ))/2n). This decomposition is conceptually powerful but entirely unevaluated empirically—the paper never estimates I_P(X; Y), KL(Q‖P), or ε_P for any backbone or training configuration.

A direct follow-up would estimate the mutual information I_P(X; Y) for multiple MLLM backbones (Qwen2.5-VL 3B, 7B; Qwen2.5-Omni 3B, 7B; LLaVA-Next 8B; and ideally non-Qwen backbones like InternVL2, Phi-3-Vision) using standard mutual information estimation techniques (MINE, InfoNCE lower bound, or conditional entropy estimation via the generative loss approximation I_P(X; Y) ≈ H(Y) − L_g(P) that the paper itself suggests). Simultaneously, measure the KL divergence KL(Q‖P) between LoRA-adapted and pretrained weights for different LoRA configurations (ranks 8, 64, 256; alphas 16, 128). Then evaluate whether the bound's predicted ordering of backbones and training strategies matches the empirical ordering on MIEB-Lite. The critical test would be: does the bound actually predict which backbones and training strategies will perform best, or does it merely provide a post-hoc rationalization? A quantitative correlation between predicted bound values and observed representation performance would elevate the theory from conceptual framework to validated predictive tool.

A stress-test would examine cases where the bound might fail: for instance, measuring I_P(X; Y) on a backbone deliberately degraded through partial untraining (e.g., fine-tuning on random labels) and checking whether the representation quality after contrastive refinement drops in proportion to the I_P reduction. If the bound holds quantitatively, it provides a principled way to screen backbones before training—measure generative mutual information, predict best-case representation quality, and decide whether contrastive refinement is worth the compute.

Cross-architecture replication of the latent alignment diagnostics to establish generality. The paper demonstrates latent cross-modal alignment (anisotropy generalization, kernel alignment improvement after text-only CL) exclusively on Qwen-family models: Qwen2.5-Omni-3B for the multi-modality anisotropy analysis (Figure 1) and Qwen2.5-VL-Instruct 3B/7B for the kernel alignment analysis (Figure 2). This is a critical gap because the entire LCO-EMB framework rests on the assumption that MLLMs generically acquire latent alignment during generative pretraining—but only one architectural family has been tested.

A systematic replication study would run the full diagnostic pipeline (anisotropy before/after text-only LoRA CL on paired datasets like Pixmo Cap, AudioCaps, MSR-VTT; layer-wise vision-language kernel alignment) across a diverse set of MLLM architectures: LLaVA-OneVision (different vision encoder, different training recipe), InternVL2 (different projector design), PaliGemma (different decoder architecture), Phi-3-Vision (smaller scale, different pretraining data), and Llama-3.2-Vision (Meta's architecture). For each backbone, measure: (a) the degree of cross-modal anisotropy generalization (correlation between text anisotropy reduction and image anisotropy reduction across layers), (b) the absolute kernel alignment scores before and after CL, and (c) the final MIEB-Lite performance of the text-only LCO-EMB variant.

The key research question is whether latent alignment is a universal property of the encoder-projector-decoder architecture trained with a generative objective, or whether it depends on specific pretraining choices (ratio of multimodal to text-only data, whether the vision encoder was frozen or co-trained, the projector's capacity, the use of instruction tuning). A negative result—finding an MLLM where text-only CL does not generalize to non-text modalities—would be equally valuable, because it would identify the pretraining conditions necessary for latent alignment and bound LCO-EMB's applicability. The paper's own finding that the 7B Qwen-VL shows stronger kernel alignment than the 3B variant (Figure 2) hints that scale matters, but scale is confounded with training recipe; a controlled study varying architecture while holding scale constant would disentangle these factors.

Fitting and validating parametric scaling laws for the Generation-Representation relationship. The paper presents GRSL as a qualitative correlation across five backbones (Figure 6) but does not fit a parametric function (power law, log-linear, or otherwise) that would enable quantitative prediction. The field needs to know whether GRSL follows the functional forms familiar from pretraining scaling laws (e.g., representation performance ∝ (generative performance)^α with α ~ 0.3-0.5) or whether the relationship saturates rapidly, making generative improvements beyond a certain point yield diminishing representation returns.

A rigorous scaling law study would evaluate LCO-EMB on a much larger set of MLLM checkpoints: within a single architecture family, train (or obtain) checkpoints spanning a wide range of generative qualities—through continued pretraining at different data volumes, model sizes spanning two orders of magnitude (e.g., 0.5B, 1B, 3B, 7B, 14B, 32B variants of the same architecture), or different stages of a single training run. For each checkpoint, measure generative benchmark performance (OCR, video, audio suites as in Figure 6) and representation performance after applying the identical LCO-EMB text-only training recipe. Fit candidate functional forms (power law, broken power law, logarithmic, sigmoid) and assess which best predicts held-out checkpoints.

The SeaDoc experiment provides the template for causal validation but on a single backbone and single task. A comprehensive study would replicate the causal intervention—continued generative pretraining before contrastive refinement—across at least three different domains (e.g., medical imaging with domain-specific generative data, multilingual document understanding, video action recognition) and at least two backbone scales per domain. The question is whether the generative-to-representation transfer efficiency (how many representation points you gain per unit of generative improvement) is consistent across domains or domain-specific. If it varies substantially, practitioners need domain-specific scaling coefficients to decide whether continued generative pretraining is cost-effective for their application.

Combining generative and contrastive objectives during joint training. The paper treats generative pretraining and contrastive refinement as sequential stages: first pretrain (or continue pretraining) with a generative objective, then apply contrastive fine-tuning. The limitations section acknowledges that "one can also jointly train generative loss and contrastive loss to maintain a model's knowledge (through continual generative training), and enhance its representational power (through continual contrastive learning)" but leaves this as future work due to computational cost.

A direct follow-up would implement joint training with a combined objective L_total = L_contrastive + λ·L_generative, where L_contrastive is the InfoNCE loss on text-only (or mixed) data as in LCO-EMB, and L_generative is the standard cross-entropy next-token prediction loss on multimodal data. The key hyperparameter λ controls the balance between representation refinement and generative knowledge preservation. The research questions are: (1) Does joint training outperform the sequential LCO-EMB pipeline, either in final representation quality or in robustness to domain shift? (2) Does it eliminate the need for careful generative data mixture balancing (the OCR-only vs. OCR+Captions phenomenon in SeaDoc), since the contrastive loss provides a regularizing signal that prevents generative overfitting? (3) How does the optimal λ vary with backbone scale and domain? A strong study would evaluate on a setting where sequential LCO-EMB struggles—perhaps a domain where finding the right generative data mixture for alignment preservation is difficult—and test whether joint training is more robust.

Probing the "learn less, forget less" mechanism: what exactly does LoRA preserve that full fine-tuning destroys? The paper argues that LoRA outperforms full fine-tuning because it preserves latent cross-modal alignment (Table 3, LoRA at 71.98 vs. full FT at 66.49 on 7B). But "preserves alignment" is a high-level description—what specific representational properties are being preserved?

A mechanistic study would compare LoRA and full fine-tuning checkpoints at the level of individual attention heads, layer representations, and cross-modal alignment scores. For each checkpoint, measure: (a) per-layer cross-modal kernel alignment (as in Figure 2) to identify which layers lose alignment under full fine-tuning; (b) the degree of change in attention patterns from the pretrained model using standard metrics like attention KL divergence; (c) the representations' performance on diagnostic tasks that isolate specific capabilities (e.g., visual grounding of text mentions, spatial relationship encoding, OCR accuracy in embedding space). The hypothesis is that full fine-tuning disproportionately disrupts the middle-to-late decoder layers where cross-modal integration occurs, while LoRA's low-rank constraint prevents these layers from deviating substantially. Identifying the specific layers and attention mechanisms that encode cross-modal alignment would provide architectural insights for designing even more targeted fine-tuning strategies—perhaps applying LoRA only to the layers most responsible for representational quality while leaving cross-modal integration layers completely frozen.

Practical Applications and Downstream Use Cases

Cost-efficient multimodal retrieval for organizations with limited multimodal data. The most immediate practical application of LCO-EMB is for teams that need high-quality multimodal embeddings but lack access to large-scale multimodal paired datasets. The paper demonstrates that text-only NLI data (publicly available, well-studied, and computationally cheap to process) plus LoRA fine-tuning on an off-the-shelf MLLM produces embeddings that outperform the proprietary Voyage Multimodal 3 (Table 1: text-only 7B VL at 60.4% vs. Voyage-M3 at 58.1% on full MIEB-Lite). For a startup or academic lab building a multimodal search system—for instance, a visual product search engine where users upload images and expect results in their local language—the recipe is concrete: download Qwen2.5-VL-7B from HuggingFace, fine-tune the language decoder with LoRA on all-NLI (276k text triplets, ~9.3 GPU hours on H20s, costing roughly $30-50 at current cloud GPU prices), and deploy. The resulting model handles English and multilingual queries over image corpora, supports visual document retrieval, and achieves competitive zero-shot classification—all without collecting or labeling a single image-text pair.

The economic advantage over the traditional approach is substantial. Training a CLIP-style model from scratch requires hundreds of millions of image-text pairs and thousands of GPU-hours; even fine-tuning an existing CLIP model on domain-specific pairs requires curating those pairs—expensive for domains like multilingual document understanding where paired data is scarce. LCO-EMB decouples representation quality from multimodal data volume: the generative pretraining provides the cross-modal understanding, and the text-only contrastive fine-tuning provides the similarity-matching projection. For domains where the base MLLM already has reasonable generative performance (which is increasingly true for common visual concepts, documents, and scenes given the breadth of modern MLLM pretraining data), the cost of deploying high-quality multimodal retrieval drops by roughly two orders of magnitude in training compute and effectively to zero in multimodal data curation cost.

Domain-specific embedding improvement through targeted generative pretraining. The SeaDoc experiment (Section 5.3) provides a template for organizations that need strong multimodal embeddings in specialized domains poorly covered by general MLLM pretraining. The recipe is: (1) collect domain-specific multimodal data where the generative task is well-defined (e.g., medical images with radiology reports, engineering diagrams with specifications, legal documents with structured metadata); (2) continue generative pretraining on this domain data mixed with general-domain data to improve domain-specific generative capability while preserving cross-modal alignment; (3) apply LCO-EMB's text-only contrastive refinement; (4) optionally add a small amount of domain-specific multimodal calibration data (~10^3-10^4 pairs) to align the embedding space with the target task format.

The SeaDoc numbers provide rough magnitude estimates: OCR+Captions SFT improved retrieval nDCG@10 from the baseline's ~30-32 range to ~34-38 range (Figure 7, approximate reading), representing a ~10-20% relative improvement. While these gains are modest in absolute terms, the approach is complementary to contrastive data scaling—improving the backbone makes all subsequent contrastive training more effective, whereas adding more contrastive data hits diminishing returns bounded by the backbone's capability ceiling. For an organization with access to domain-specific training data but not domain-specific paired retrieval data (a common situation in specialized fields where generative annotations exist but retrieval-specific query-document pairs are expensive to create), the generative-first approach is uniquely practical.

Rapid prototyping and benchmarking of MLLM backbones for representation tasks. The paper's diagnostic toolkit (anisotropy measurement, kernel alignment analysis) enables a new workflow for evaluating MLLMs as embedding backbones before committing to a full training pipeline. A research team considering whether to build an embedding model on a newly released MLLM can: (1) run the anisotropy diagnostic on a few thousand paired samples across modalities to check whether text-only CL generalizes geometrically; (2) measure kernel alignment across layers to assess the strength of latent cross-modal structure; and (3) based on these diagnostics, predict whether the backbone is suitable for text-only LCO-EMB (strong alignment) or requires more multimodal calibration data (weak alignment). The diagnostics cost a few GPU-hours—negligible compared to a full training run—and provide a principled basis for backbone selection that currently does not exist.

This workflow is particularly valuable given the accelerating pace of MLLM releases. New models appear monthly from multiple labs (Qwen, LLaVA, InternVL, Phi, PaliGemma, Llama-Vision), each with different architectures, pretraining data mixtures, and capabilities. Without diagnostics, practitioners rely on benchmark numbers from generative tasks (MMMU, MME, MMBench) to guess which model will produce the best embeddings—a guess that the paper shows can be misleading (CLIP-style training on a strong generative backbone can degrade embedding quality, Table 3). The diagnostic toolkit provides a direct measurement of the property that matters for embedding performance—latent alignment—enabling informed backbone selection.

When to Prefer This Method

The paper provides sufficient evidence to articulate clear decision boundaries for when LCO-EMB should be preferred over alternatives. The following decision rules are grounded in specific empirical results from Tables 1, 3, and Figures 4, 7.

Prefer text-only LCO-EMB (LoRA on language decoder, all-NLI data) when:

• You have access to an open-weight MLLM with the standard encoder-projector-decoder architecture (Qwen2.5-VL, Qwen2.5-Omni, LLaVA-Next, or architecturally similar models).
• The backbone's generative pretraining is reasonably strong for your target domain (the model was not pretrained exclusively on a narrow distribution; it has seen diverse multimodal data).
• You need competitive multimodal embeddings (MIEB-Lite average: 57.5-60.4% for 7B backbones, outperforming Voyage Multimodal 3 at 58.1%) but have zero or minimal multimodal paired data for training.
• Your development budget is constrained: text-only LoRA training costs ~4.7-9.3 GPU hours (Table 3) and requires no multimodal data collection.
• You need strong cross-lingual multimodal alignment—LCO-EMB text-only variants achieve massive gains over CLIP-style baselines on multilingual image retrieval (64.81 vs. 48.26 for Voyage-M3 on MIEB-Sub18, Figure 5) because the generative backbone already encodes multilingual knowledge.

Prefer multimodal LCO-EMB (all-NLI + ~94k multimodal calibration triplets) when:

• You have a small budget for collecting (~10^4-10^5) multimodal pairs and need SOTA performance on benchmarks like MIEB (68.8% for Omni-7B, ~21× less data than GME at 64.5%, Table 1).
• Your downstream tasks include specific formats (document retrieval, zero-shot classification, interleaved VQA) where the evaluation protocol differs from similarity matching in the open domain—the multimodal calibration data aligns the embedding space with these formats.
• You can balance your multimodal data across task types (~23k visual document, ~3k retrieval/compositionality, ~60k synthetic for instruction following, plus multilingual text) to avoid the over-calibration that the paper identifies—too much domain-specific multimodal data degrades generalization.

Prefer continued generative pretraining + LCO-EMB (SeaDoc template) when:

• Your target domain is specialized (low-resource languages, specific document formats, domain-specific visual concepts) and poorly covered by the base MLLM's pretraining data.
• You have access to domain-specific data that supports generative training (image-to-text pairs, OCR training data, captioning data) even if you lack domain-specific retrieval pairs.
• You can mix domain-specific generative data with general-domain data (e.g., PixmoCaps) to preserve cross-modal alignment during continued pretraining—OCR-only SFT causes performance collapse (Figure 7).
• The expected gain is modest but real (~10-20% relative improvement in retrieval nDCG on SeaDoc) and the alternative (collecting domain-specific retrieval pairs) is prohibitively expensive.

Prefer CLIP-style contrastive training from scratch instead when:

• You do not have access to an open-weight MLLM (API-only access, proprietary models).
• Your target tasks are exclusively linear probing and zero-shot classification on standard benchmarks, where CLIP-style models still hold the SOTA (SigLIP-so400m at 78.8 linear probing vs. LCO-EMB at 74.1, Table 1).
• Inference cost dominates your budget: LCO-EMB requires running the full MLLM decoder (7B+ parameters) for every embedding, while CLIP-style vision encoders are typically 300M-1B parameters. For embedding a corpus of millions of images, the 5-10× per-item inference cost of LCO-EMB may outweigh its training efficiency advantages (though the paper does not quantify this tradeoff explicitly, the architectural difference is evident).

Prefer full fine-tuning of MLLMs never. The paper's evidence (Table 3) is unambiguous: full fine-tuning consistently underperforms LoRA (71.98 vs. 66.49 on 7B, 66.28 vs. 62.10 on 3B) while requiring more compute (~2× GPU hours for 7B). The degradation is attributed to disruption of latent cross-modal alignment. Unless a future study demonstrates a domain or backbone where full fine-tuning reverses this pattern, LoRA should be the default and full fine-tuning avoided for MLLM-based embedding training.