SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Model

ArXiv: 2510.12709

🎯 Pitch

A single embedding model unifying vision, text, and audio achieves a +0.5% 7‑day lifetime gain in Douyin’s recommender system—but only when the training explicitly distills behavioral patterns alongside semantic content. Without joint content‑aware and collaboration‑aware training, omni‑modal fusion alone fails to improve online metrics.

---

1. Executive Summary

This paper introduces SAIL-Embedding, an omni-modal embedding foundation model that unifies vision, text, and audio modalities into a single representation space for large-scale recommendation and retrieval. Evaluated on a diverse suite of 21 item-to-item and 9 query-to-item benchmarks derived from real-world Douyin scenarios — spanning content understanding, search, and collaborative perception — SAIL-Embedding employs dynamic hard negative mining (adaptively selecting challenging negatives per dataset based on an F1-optimal similarity threshold) and adaptive multi-source data balancing (learning dataset-specific sampling weights via Sinkhorn-based similarity alignment between training and validation distributions) alongside a multi-stage training scheme that includes content-aware progressive training (a three-phase curriculum transitioning from large-scale diverse data to domain-specific hard-negative datasets) and collaboration-aware recommendation enhancement training (distilling user behavioral patterns through sequence-to-item and ID-to-item alignment). The model achieves state-of-the-art results across both retrieval paradigms — for instance, improving Recall@50 on the Search-q2i short video benchmark to 86.53% (versus 78.53% for the prior VLM-based approach) and delivering a 4× improvement in discriminability on ID-based retrieval benchmarks after recommendation enhancement — while in online A/B tests on the Douyin recommender system, the model yields a +0.5% 7-day Lifetime (LT) gain in the Douyin-Selected scenario and a +0.1% AUC improvement in the feed ranking model, establishing that omni-modal fusion with behaviorally-distilled representations improves downstream recommendation metrics only when both content-aware semantic learning and collaboration-aware interest modeling are jointly optimized.

2. Context and Motivation

The Core Problem: Industrial Embedding Models Require Omni-Modal Understanding That Current Approaches Cannot Deliver

The central problem this paper tackles is deceptively straightforward to state but enormously complex to solve: how do you build a single embedding model that can ingest arbitrary combinations of vision, text, and audio modalities — each with varying quality and availability — and produce representations that are simultaneously useful for content understanding, cross-modal search, and behavioral recommendation? This matters because short-video platforms like Douyin present a uniquely challenging information environment where every item (video, live stream, ad) carries dense, heterogeneous signals across multiple modalities, and the downstream use cases span fundamentally different objectives: retrieving visually similar content (content understanding), matching user text queries to relevant videos (cross-modal search), and recommending items based on collaborative consumption patterns (recommendation). A model that excels at one of these but fails at another is economically non-viable in a production system where these capabilities must coexist.

The practical stakes are enormous. Recommendation systems power the core engagement loop for platforms serving billions of users, and the quality of item representations directly constrains every downstream stage: recall (retrieving candidate items from a massive corpus), pre-ranking (filtering candidates efficiently), ranking (precision scoring), and re-ranking (contextual adjustment). The authors quantify this explicitly through their online experiments (Table 6), showing that embedding improvements cascade through the entire pipeline: +0.01% LT30 gains at recall, pre-rank, and re-rank stages, +0.1% finish AUC at the ranking stage, and cumulative +0.05% LT30 in cold-start scenarios. These are not abstract metrics — Lifetime (LT) measures actual user retention, and even fractions of a percent represent millions of users on a platform of Douyin's scale. The paper is thus motivated by a direct business imperative: can a unified embedding model replace multiple single-purpose models, and can it do so while improving every downstream stage rather than trading off one for another?

However, the gap between this vision and the state of existing embedding approaches is substantial, and the paper identifies three specific failure modes that prior work cannot address.

Limitation 1: Most Models Support Only Two Modalities, but Real-World Items Carry Richer Signals

The paper situates its work within two dominant architectural paradigms (Section 2, Figure 2), both of which share a common limitation: they are designed for image-text or video-text pairs, not for the omni-modal reality of industrial content.

The dual-tower paradigm (Figure 2a), epitomized by CLIP and its descendants (SigLIP, AudioCLIP, CLIP4Clip), employs separate encoders for each modality — a Vision Transformer for images, a text encoder for captions — and trains them to maximize cosine similarity between paired samples via contrastive learning. This architecture is structurally clean, precomputable (encode all items once, cache embeddings), and scales efficiently to billion-item corpora. However, as the paper notes in Section 2.1, these models "generally fuse modalities only in the final embedding space, which limits their ability to capture fine-grained token-level interactions, temporal dynamics, or higher-level multimodal reasoning." More fundamentally, they are architecturally not designed to ingest three or more modalities simultaneously — audio is an afterthought (AudioCLIP adds a branch), and there is no mechanism for the modalities to interact before the final projection. On Douyin, where a single video contains cover frames, keyframes, titles, OCR text, ASR transcripts, background music, and author tags, this shallow fusion is simply insufficient. The paper gives a concrete example in the introduction: "omitting or misinterpreting even one [modality] can severely impact downstream tasks such as recommendation and search, ultimately degrading user experience."

The MLLM-based paradigm (Figure 2b), represented by VLM2Vec, GME, mmE5, and MoCa, addresses some of these limitations by routing modality-specific encodings through a shared large language model backbone, enabling cross-modal attention and deeper semantic fusion. These models achieve stronger representational capacity because the LLM's self-attention layers can model token-level interactions across modalities — a video frame patch can attend to the corresponding ASR transcript token, creating representations that capture cross-modal semantics rather than just post-hoc alignment. However, the paper argues (Section 2.2) that "most existing methods remain constrained to image and text modalities, falling short of fully supporting omni-modal understanding." Even VLM2Vec-v2, which extends the paradigm to videos and visual documents, does not incorporate audio. This matters because audio carries information that is both orthogonal to and complementary with vision and text: the emotional tone of background music signals the intended mood of a video, ASR transcripts capture spoken information that may not appear in titles or OCR, and ambient sounds can disambiguate visual content (e.g., distinguishing a cooking video from a food review based on sizzling sounds).

Why this limitation exists is instructive. Adding modalities to an LLM-based fusion model is not architecturally trivial — each new modality requires a separate encoder, an alignment mechanism to project its features into the LLM's token space, and careful training to prevent the stronger modalities (typically text, which the LLM is pre-trained on) from dominating the weaker ones. The paper's architecture (Figure 2c) explicitly addresses this by treating non-text modalities as a "foreign language" (Section 3.2.1): each modality gets its own encoder (ViT for vision, CLAP for audio), tokens are aligned to the LLM's embedding dimension, and the full multimodal sequence is concatenated before being fed into the fusion Transformer. This design choice is consequential because it means the model can handle arbitrary modality combinations at inference time without architectural changes — if audio is missing (as it often is in deployment), the model simply receives an empty audio token sequence, and the LLM's attention mechanism learns to route information from available modalities accordingly.

Limitation 2: Training Instability and Optimization Challenges at Industrial Scale

The paper identifies a second critical gap that is rarely discussed in academic literature but is paramount in deployment: training multimodal LLM-based embedding models at the scale of billions of samples is fundamentally unstable, and standard practices from academic benchmarks do not transfer. This instability manifests in several specific ways that the paper addresses through its training strategies:

Heterogeneous data mixing creates gradient variance. In multi-domain industrial training, data comes from dozens of sources with vastly different characteristics — user behavior-based pairs (Copair-i2i, 2.9B samples), search click-based pairs (Search-q2i, 1.8B samples), classification pairs (CLS, 3.1B samples), and smaller specialized datasets (Hashtag-i2i, 0.05B samples). When these are naively mixed within each training batch (the "conventional training method" in Figure 6b), the effective batch size for each dataset becomes small, gradient estimates become noisy, and optimization oscillates. The paper's stochastic specialization training (Section 3.3.3) addresses this by sampling from a single dataset per iteration based on learned sampling weights, yielding larger per-domain effective batch sizes while maintaining global coverage. This is not merely an engineering convenience — it is an optimization necessity at the scale of 10B+ samples that directly impacts whether the model converges at all.

Manual data mixing ratios are arbitrary and suboptimal. The paper criticizes the prevailing practice of "manually assigned dataset mixing ratios, determined by subjective expertise and task intuition" (Section 3.1.3). In academic settings, where training sets are homogeneous (e.g., LAION for CLIP) and evaluation sets are standardized (e.g., ImageNet zero-shot), a uniform mixing ratio is a reasonable default. In industrial settings, where downstream tasks have different importance weights and data sources have different reliability, manual tuning becomes combinatorially intractable. The paper's adaptive multi-source data balancing framework (Section 3.1.3, Figure 3) replaces human heuristics with a data-driven procedure: embed validation samples from target downstream tasks, embed training samples from each source dataset, cluster both, compute Sinkhorn-regularized similarity scores between validation and training clusters, and derive sampling weights via softmax normalization. This ensures that training data sources more semantically similar to the target tasks receive higher sampling probability, while still preserving diversity through the soft-weighting mechanism rather than hard filtering.

Verifier-free contrastive learning struggles with fine-grained distinctions. Standard contrastive learning with random in-batch negatives is computationally efficient but educationally vacuous: the model learns to distinguish a cooking video from a sports video easily, but fails to distinguish a Sichuan cooking video from a Hunan cooking video — exactly the kind of fine-grained distinction that matters for recommendation. The paper's dynamic hard negative mining (Section 3.1.2) addresses this by adaptively selecting negatives that are semantically close to the positives. The key insight is that the optimal similarity threshold for defining "hard" negatives varies by dataset and task — a threshold that works for broadly-tagged items might admit too many false negatives for finely-grained categories. By computing the F1-optimal threshold per dataset (the threshold that maximizes the tradeoff between precision and recall in the binary classification of positive vs. negative pairs based on cosine similarity), the method tailors the hardness criterion to each data distribution, ensuring that the model is consistently challenged at the appropriate difficulty level throughout training.

Modality imbalance causes text to dominate. LLMs are pre-trained on text, and their internal representations are optimized for natural language. When visual and audio tokens are concatenated alongside text tokens, the model can learn to ignore the non-text modalities entirely — achieving low contrastive loss by attending only to the rich text signal — a phenomenon the paper terms "modality imbalance." The paper counters this through several mechanisms: the multimodal In-Context Learning (mICL) loss (Equation 4) applies separate contrastive objectives within each modality, forcing visual and textual embeddings to maintain discriminability even when used in isolation; the late fusion mechanism (Equation 5) learns a gated combination of visual and multimodal embeddings, ensuring that visual fidelity is preserved in the final representation rather than being washed out by the text-heavy fusion; and the dataset-driven pattern matching (Section 3.3.4) dynamically constructs all valid query-target modality pairs (ITC, IIC, VTC, VVC, TTC, OOC) and optimizes across all of them, preventing any single modality from dominating the loss landscape.

Limitation 3: The Industrial Domain Gap — Content Similarity ≠ Recommendation Relevance

This is perhaps the paper's most consequential insight and the primary motivation for its most novel contribution (collaboration-aware recommendation enhancement training): embeddings that excel at content understanding (grouping items by visual similarity, thematic coherence, or semantic relatedness) are fundamentally misaligned with recommendation objectives (grouping items by user co-consumption patterns). The paper demonstrates this tension implicitly through its benchmarks: the same model (SAIL-Embedding without CRE) achieves different relative performance on content-understanding i2i tasks versus collaborative-perception i2i tasks (Table 2). For example, on Film-i2i (content understanding: items grouped by movie IP), recall@50 is 89.08%, while on Copair-i2i (collaborative perception: items grouped by user co-consumption), recall@50 is 69.17%. These are fundamentally different grouping principles — one based on semantic content, the other on behavioral patterns — and a model optimizing only for the former will necessarily underperform on the latter.

This gap has deep roots in the design of existing multimodal embedding models. Both dual-tower and MLLM-based approaches are trained on content-derived supervision: image-caption pairs, video-text pairs, or item-tag pairs. The supervision signal always encodes what the item is (its content, its theme, its category) but never how users interact with it (who consumes it, in what sequence, alongside what other items). The consequence is that two videos can be content-dissimilar — a makeup tutorial and a gaming livestream — yet highly correlated in user behavior because they share an audience demographic, creator style, or consumption context. A pure content embedding would place these items far apart in the representation space; a collaborative filtering signal would pull them together because users who watch one tend to watch the other. Neither signal alone is sufficient, and the paper's key argument is that prior work doesn't even attempt to reconcile them.

The paper's approach to this gap is its collaboration-aware recommendation enhancement (CRE) training (Section 3.3.5), which comes in two complementary forms:

Sequence-to-item distillation (Figure 7a) attempts to inject user behavioral patterns directly into the embedding space. The intuition is that a user's historical viewing sequence encodes their interests, and the embedding of the target item (the next video they watch) should be predictable from the embeddings of the preceding items in their sequence. This is not standard collaborative filtering — it's not about learning item-item co-occurrence matrices or user embedding vectors. Instead, it's about teaching the multimodal embedder that certain content patterns, when placed in specific sequences, predict consumption. The paper constructs sequence data from four perspectives (Section 3.3.5): content-aware single-peak interest (items with at least three positive behaviors, filtered by content similarity), content-aware multi-peak interest (items with any positive behavior, filtered by Jaccard coefficient on behavior labels), and two collaboration-aware variants that additionally filter by clustered user interaction distributions. The diversity of these sequence construction strategies ensures the model learns both tight thematic clusters and broad interest patterns.

ID-to-item distillation (Figure 7b) takes a more direct approach: it aligns the multimodal embedding of an item with the ID-based embeddings already used in the production recommender system. These ID embeddings are trained on massive user interaction logs and encode collaborative signals with high fidelity — they know which items are consumed together not because they look similar, but because millions of users have exhibited that pattern. By distilling this knowledge into the multimodal embedder, the model gains access to behavioral patterns that no amount of content-based training could uncover. The paper is careful to note that this distillation is performed alongside an auxiliary i2i retrieval task (Section 3.3.5) to prevent the representation from "overly sacrificing content-aware perception capabilities" — the goal is augmentation, not replacement.

The effectiveness of this approach is validated quantitatively in Table 4, where CRE improves the VLM-based baseline on the Gid-i2i benchmark (constructed from ID-embedding relevance) by an average of 3.78% across Recall@50 and Recall@100, and improves SAIL-Embedding's performance on the same benchmark by 7.23% and 7.31% respectively. Equally important is what happens on the Copair-i2i benchmark (constructed from content-based representation similarity): performance slightly degrades (from 69.17% to 66.83% at Recall@50 for SAIL-Embedding with CRE). This tradeoff — better collaborative perception at a small cost to content understanding — is explicitly acknowledged by the authors: "We consider this variation tolerable to balance the differing application requirements of content-oriented and collaborative behavior-oriented scenarios in industrial settings." This is a mature engineering perspective: in production, you cannot optimize for everything simultaneously, and the paper makes a conscious choice about what to prioritize.

Where Existing Approaches Fall Short: A Systematic Breakdown

The paper's critique of prior work is distributed across Sections 1, 2, and implicitly through its experimental comparisons (Tables 2 and 3). Synthesizing these arguments:

Unimodal text embedding models (Doubao-Embedding, Qwen3-Embedding-4B/8B) are strong on text queries (Table 3, Short Video-q2i Recall@50 of 71.73–72.42) but fundamentally cannot leverage visual and audio signals. On tasks where the query or item has rich multimodal content, their performance plateaus because the text modality alone is insufficient to capture visual aesthetics, audio mood, or the holistic "feel" of a video. The paper's data design implicitly acknowledges this: Search-q2i pairs are constructed from user click behavior on search results, meaning the items users click on are often chosen based on visual appeal (the cover frame, the preview) that the search text alone cannot convey.

CLIP-based dual-tower models scale well and are efficient at retrieval time, but the paper's results show they consistently underperform on tasks requiring deep semantic reasoning. On Film-i2i (Table 2), the CLIP-based model achieves Recall@50 of 75.98% versus SAIL-Embedding's 89.08% — a 13 percentage-point gap. The limitation is not just about modality count (CLIP has vision and text) but about fusion depth. CLIP aligns modalities only in the final embedding space, whereas SAIL-Embedding's LLM backbone performs cross-modal attention at every layer, allowing it to learn that certain combinations of visual features and audio patterns jointly signal a particular genre or creator style.

Standard VLM-based models (GME, VLM2Vec) improve over CLIP by introducing deeper fusion, but they remain image-text models at their core. The paper's VLM baseline (which appears to be a GME-like architecture fine-tuned on the same data) underperforms SAIL-Embedding on most i2i tasks, particularly in search and collaborative perception scenarios. For instance, on Copair-i2i Recall@50 (Table 2), the VLM-based model achieves 66.06% versus SAIL-Embedding's 69.17%, and on RSDF-i2i (a collaborative-perception task using recommender system dense features), the gap is 48.44% versus 59.06% — a 10.6 percentage-point improvement. The paper attributes this to the audio modality (Section 5.4.1): "Compared to the VLM-based model, our model verifies the importance and effectiveness of the audio modality in multimedia retrieval driven by short videos."

None of these models address the recommendation alignment problem. This is the paper's central differentiator. CLIP, VLM2Vec, GME, mmE5 — all are trained to produce embeddings that reflect what content is, not what users do. The paper's CRE training is, to the authors' knowledge, the first systematic attempt to inject collaborative filtering signals into an LLM-based multimodal embedding model through distillation from production recommender system embeddings. This is not an incremental improvement — it's a fundamentally different supervision signal that changes what the embedding space represents.

How This Paper Positions Itself

The paper constructs its identity through a series of deliberate design choices that collectively distinguish it from prior work:

It is an omni-modal model, not a multimodal one. The distinction matters. Multimodal models typically assume a fixed set of modalities (usually two: image + text) and are trained on datasets where those modalities are always present and paired. Omni-modal models, as the paper defines them through its architecture (Figure 2c), can accept arbitrary modality combinations at inference time, handle missing modalities gracefully (Section 3.2.2 mentions "random dropping of partial fields in practice to accommodate scenarios where original fields are missing during online deployment"), and support query-target pairs where the query might be pure text while the target contains all three modalities (Search-q2i in Table 1) — or vice versa. The dataset-driven pattern matching (Section 3.3.4) is the mechanism that enables this flexibility, dynamically constructing valid modality-to-modality contrastive pairs at training time based on what modalities each sample actually contains.

It is an embedding model, not a generative model. Unlike many recent MLLM works that focus on visual question answering, captioning, or instruction-following, SAIL-Embedding's sole purpose is producing fixed-dimensional dense vectors. This constraint shapes every design choice: the mean pooling over output tokens (rather than using a special CLS token or decoder output), the tanh normalization to bound embeddings for efficient similarity computation, the Matryoshka representation learning to produce usable sub-embeddings at multiple dimensionality levels (1536, 768, 128) for different deployment constraints (Section 3.3.2, Equation following Lnce-mrl definition). The paper is not trying to build a general-purpose multimodal assistant; it's building a production embedding service that must run efficiently at massive scale.

It is a recommendation-first model, not a retrieval-first one. This is the paper's most distinctive positioning. While evaluated on retrieval benchmarks (i2i, q2i), the ultimate downstream application is recommendation, and the training pipeline is designed accordingly. The data is constructed from user behaviors (clicks, searches, consumption sequences, co-occurrence), not from web-crawled image-text pairs. The training stages progress from content understanding toward collaborative perception (Figure 6a), reflecting a deliberate curriculum. The evaluation framework (Section 4.2) includes metrics like group-wise clustering consistency (NMI) and bijective alignment that are specifically designed to assess whether embedding spaces will transfer well to recommendation stages. And the online experiments (Table 6) measure recommendation-specific metrics (LT, AUC) rather than retrieval recall alone.

It is scalable by design. Every training component is chosen not just for accuracy but for its ability to handle 10B+ samples across heterogeneous data sources. The stochastic specialization training reduces inter-domain communication overhead. The adaptive data balancing automates what would otherwise require months of manual tuning. The dynamic hard negative mining eliminates the need for dataset-specific negative sampling configurations. The progressive training curriculum stabilizes optimization when naive end-to-end training would diverge. These are not theoretical niceties — they are engineering necessities for training models at this scale, and their inclusion signals that the paper is written by practitioners who have actually deployed such systems.

The paper is self-aware about its limitations. Notably, the paper does not claim to have "solved" multimodal recommendation. The hard problems (difficulty bin 5 in the analyzed paper's taxonomy) remain: CRE training improves collaborative perception at a small cost to content understanding (Table 4, Copair-i2i degrades slightly). The paper acknowledges this tradeoff explicitly and frames it as a necessary balance in industrial settings. The implication is that future work should not blindly pursue "better content understanding" or "better collaborative signals" in isolation, but should develop principled methods for trading off between these objectives based on deployment context — a more nuanced research agenda than simply scaling up existing approaches.

In summary, the paper positions SAIL-Embedding as filling a specific gap that no prior work addresses: an embedding model that (1) supports omni-modal inputs with deep fusion, (2) trains stably at the scale of tens of billions of industrial samples, and (3) produces representations that serve both content understanding and behavioral recommendation through explicit distillation from collaborative filtering signals. Each of these three axes — modality breadth, training robustness, and recommendation alignment — is necessary for deployment in platforms like Douyin, and no prior model satisfies all three simultaneously. The paper's contributions are thus less about any single architectural or algorithmic novelty and more about the integration and co-design of multiple techniques to achieve a specific, demanding production requirement.

3. Technical Approach

3.1 Reader Orientation

SAIL-Embedding is a production-scale neural network that converts any combination of video frames, text descriptions, and audio signals into a single fixed-length vector that captures both content semantics (what the item is about) and collaborative signals (how users interact with it). The system solves the problem of building a single embedding model that simultaneously serves three fundamentally different application needs — content-based similarity search (find videos that look/sound similar), cross-modal retrieval (find videos matching a text query), and behavioral recommendation (find videos a user will likely watch next) — by training in progressive stages that transition from general multimodal understanding to recommendation-specific behavioral distillation, all built on an architecture that fuses arbitrary modality combinations through a shared large language model backbone.

3.2 Big-Picture Architecture (Diagram in Words)

The system has six major processing stages that convert raw multimodal items into embedding vectors:

1. Modality-Specific Encoders: Raw inputs from three modalities — vision (video frames), audio (soundtrack and spoken content), and text (titles, tags, OCR, ASR transcripts, author names) — are independently processed by dedicated encoder networks. The vision encoder (a ViT backbone with a Perceiver bottleneck) converts sampled video frames into condensed visual tokens. The audio encoder (a CLAP model) converts variable-length audio clips into a single acoustic token. The text modality is tokenized directly into the LLM's vocabulary.

2. Modality Alignment Layer: The visual and audio tokens, which live in different dimensional spaces from the LLM's text embeddings, are projected through learned adapters into a common 1536-dimensional space compatible with the language model. This is where the "foreign language" metaphor from Section 3.2.1 operates: non-text modalities are translated into the LLM's native token language.

3. Instruction-Guided Token Assembly: Task-specific instruction prompts are prepended to define the embedding objective (e.g., "find an image-text-audio triplet that shares the same theme"), and all modality tokens are concatenated into a single multimodal sequence. Instructions differ between query and target samples to enable asymmetric retrieval (text queries retrieving video targets, etc.).

4. LLM-Based Cross-Modal Fusion: The concatenated token sequence passes through the full Transformer backbone of a pre-trained large language model with bidirectional attention, enabling every token to attend to every other token regardless of modality. This is where deep fusion happens — a video frame patch can attend to an ASR transcript token to create cross-modal representations.

5. Embedding Extraction: The output hidden states from all token positions are normalized via tanh activation (bounding values to [-1, 1]) and averaged via mean pooling into a single fixed-dimensional vector (1536 dimensions by default, with trainable sub-embeddings at 768 and 128 dimensions via Matryoshka Representation Learning).

6. Contrastive Training with Multi-Source Supervision: Query and target embeddings are contrasted against each other using a composite loss function combining Noise-Contrastive Estimation (for global alignment), COSENT (for fine-grained ranking), modality-specific in-context learning losses (to prevent modality imbalance), and late fusion losses (to preserve visual fidelity). The training data is sampled according to learned dataset weights and dynamically augmented with hard negatives selected per-dataset at F1-optimal similarity thresholds.

Information flows through the system as follows: raw multimodal item data enters → modality encoders produce initial token sequences → adapter projections align dimensions → instruction prompts set task context → tokens are assembled into one sequence → LLM Transformer fuses cross-modal information → mean pooling extracts the embedding → contrastive loss updates all components end-to-end based on the current training stage (content-aware progressive training → collaboration-aware recommendation enhancement).

3.3 Roadmap for the Deep Dive

I will explain the components in this order to build understanding from data to architecture to optimization:

• First, the data preparation pipeline (Section 3.1 of the paper), because the training data's scale (10B+ samples), heterogeneity (i2i, q2i, classification), and construction philosophy fundamentally shape every downstream design choice. I will cover how recommendation-aware pairs are constructed, how dynamic hard negative mining adaptively selects challenging negatives per dataset, and how adaptive multi-source data balancing learns sampling weights automatically.

• Second, the architecture components (Section 3.2), because you need to understand the model's structure before you can understand how it is trained. I will cover the text tokenizer and instruction design, the vision encoding module with its Perceiver bottleneck, the audio encoding module with its length-adaptive pipeline, and the LLM-based fusion module with bidirectional attention.

• Third, the training strategies (Section 3.3), which is the paper's most technically dense section. I will cover content-aware progressive training (the three-stage curriculum), the composite loss function (with detailed breakdown of NCE, COSENT, mICL, and late fusion losses, plus Matryoshka representation learning), stochastic specialization training (single-dataset-per-batch sampling), dataset-driven pattern matching (dynamic modality-to-modality contrastive pairing), and finally the collaboration-aware recommendation enhancement training (sequence-to-item and ID-to-item distillation) which is the paper's most novel contribution.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an industrial systems paper whose core idea is that an omni-modal embedding model, when trained with a carefully staged curriculum that transitions from broad multimodal understanding to behavior-specific collaborative filtering distillation, can serve as a unified representation backbone for content understanding, cross-modal retrieval, and recommendation simultaneously — provided that training instability, modality imbalance, and the content-recommendation alignment gap are explicitly addressed through architectural co-design and optimization strategies.

---

Data Preparation and Preprocessing

Recommendation-Aware Data Construction

The paper constructs a massive training corpus of over 10 billion samples organized into three meta-task categories (Table 1): item-to-item retrieval (i2i), query-to-item retrieval (q2i), and classification. Each meta-task is designed to teach the model a specific type of semantic relationship, and the data construction philosophy reflects the dual goals of content understanding and collaborative perception.

Item-to-Item Retrieval (i2i) tasks pair two items (both with omni-modal information: Vision + Text + Audio) based on different notions of relatedness, capturing both content similarity and behavioral co-occurrence. The seven i2i sub-tasks and their construction rationales are:

• Copair-i2i (2.9B pairs): Pairs formed from user consumption behavior — if many users watch video A and then watch video B, the pair (A, B) becomes a positive training example. This directly encodes collaborative filtering signals: the model learns to place items that are co-consumed near each other in embedding space, even if their content differs substantially. This is the largest i2i dataset and carries the strongest behavioral signal.

• Search-i2i (0.6B pairs): Pairs formed from user search behavior — if a user searches for query Q, sees item A in the results, and clicks on it, then A is paired with other items that were also clicked for query Q. This encodes a different behavioral signal than Copair: search queries reveal explicit user intent, and items paired through search share relevance to a common information need.

• Live-i2i (1.4B pairs): Pairs from live streaming behavioral co-occurrence, analogous to Copair but for the live streaming domain. The separation between video and live streaming pairs allows the model to learn domain-specific behavioral patterns — live streaming consumption dynamics (real-time interaction, creator-audience relationships) differ from on-demand video consumption.

• Summary-i2i (1.1B pairs): Pairs formed by n-gram-based filtering — items that share the same automatically extracted key phrases or n-grams in their content descriptions are paired. This is a content-based signal: it teaches the model that items expressing similar themes in their textual descriptions should be close in embedding space.

• Hashtag-i2i (0.05B pairs): Pairs based on user-submitted hashtags — items tagged with the same hashtag by creators are paired. This is a weak content signal (hashtags are often noisy, trending, or promotional) but captures community-defined categorization.

• RSDF-i2i (0.3B pairs): Pairs formed by computing cosine similarity between items' dense feature vectors from the production recommender system (Recommendation-Side Dense Features), then thresholding to retain high-similarity pairs. This is a direct distillation of the existing recommender's representation space into the training data.

• ID-i2i (0.08B pairs): Pairs formed by clustering items based on their ID embeddings from the recommender system. This captures very coarse collaborative structure — items in the same ID cluster tend to share audience demographics.

The diversity of these construction methods is intentional: no single pairing strategy captures all relevant relationships. Copair captures co-consumption, Search captures intent relevance, Summary captures topical similarity, RSDF captures the recommender's internal similarity metric, and ID captures audience structure. The model must learn to represent all of these simultaneously, which is why the training strategies (adaptive balancing, stochastic specialization) are necessary — naively mixing them would cause the strongest signal (Copair's 2.9B pairs) to dominate.

Query-to-Item Retrieval (q2i) tasks pair a text-only query with an omni-modal target item, teaching the model cross-modal alignment where the query side lacks visual and audio information:

• Search-q2i (1.8B pairs): Pairs formed from search query text and items that users clicked on in the search results. The query is pure text (what the user typed), and the target is a full multimodal item (what the user chose to watch). This teaches the model that text queries about "cooking tutorial" should map to video embeddings that contain cooking content across all three modalities.

• Score-q2i (0.6B pairs): Pairs with labeled similarity scores, constructed by applying rule-based and LLM-based methods to generate fine-grained relevance judgments between queries and targets. These scores enable training with ranking losses (COSENT) rather than just binary positive/negative contrastive signals, allowing the model to learn that some query-item mismatches are worse than others.

Classification tasks (3.1B samples) pair an omni-modal item with text labels representing hierarchical tags, following the UniCL and iCLIP paradigm of transforming classification datasets into item-label text pairs. This teaches the model to produce embeddings where an item's vector is close to the vectors of its category labels. The paper mentions these span "user motivation" and "typical image-text classification" — for instance, labeling whether a video expresses "finding a gaming partner" versus "showing off gameplay highlights."

The scale of each dataset matters for understanding the training dynamics: Copair-i2i (2.9B) and CLS (3.1B) dominate in volume, meaning the model will see far more behavioral co-occurrence and classification pairs than, say, Hashtag-i2i (0.05B). This is why the adaptive multi-source data balancing (Section 3.1.3) is critical — without it, the small datasets would contribute negligibly to the loss, and the model would fail to learn the niche relationships they encode.

Dynamic Hard Negative Mining

Standard contrastive learning uses random in-batch negatives: for a positive pair (query, target), all other targets in the same training batch serve as negatives. This is computationally efficient but pedagogically weak — random negatives are typically trivially distinguishable from positives (e.g., a cooking video and a car repair video share almost no semantic overlap), so the model learns a coarse discrimination boundary that fails on fine-grained distinctions.

Hard negative mining addresses this by deliberately selecting negatives that are semantically similar to the positives — samples that the model might plausibly confuse. The challenge is defining "hard" appropriately: a similarity threshold that works for one dataset may admit too many false negatives for another, where false negatives are items that appear in the negative set but are actually relevant to the query (e.g., two cooking videos that use different hashtags but share the same cuisine).

The paper's dynamic hard negative mining strategy (Section 3.1.2) adapts the hardness threshold per dataset using an F1-optimization procedure. The algorithm proceeds as follows:

Step 1: Collect positive and negative similarity scores. For each dataset, all positive pairs $P = \{(q_i, t_i)\}_{i=1}^N$ provide positive similarity scores $s_{ii} = \cos(q_i, t_i)$. All cross-product pairs that are NOT positives form the negative set $\mathcal{N} = \{(q_i, t_j) \mid i \neq j, (q_i, t_j) \notin P\}$, providing negative similarity scores $s_{ij} = \cos(q_i, t_j)$. This creates a labeled dataset of (similarity score, binary label) pairs where label = 1 for positives and 0 for negatives.

Step 2: Define a threshold-based classifier. For any candidate threshold $\lambda$, the predicted label for a pair with similarity $s_{ij}$ is:

$\hat{y}_{ij}(\lambda) = \begin{cases} 1, & s_{ij} \geq \lambda \\ 0, & s_{ij} < \lambda \end{cases}$

where $\hat{y}_{ij}(\lambda)$ is the binary prediction indicating whether the pair is classified as positive (1) or negative (0) based on whether its cosine similarity exceeds the threshold $\lambda$.

What this computes: For a given threshold $\lambda$, every pair in the dataset is assigned a binary prediction — pairs with similarity at or above $\lambda$ are predicted positive, those below are predicted negative. This creates a simple decision rule that can be evaluated against the ground-truth labels.

Why this form: The cosine similarity between embeddings is the natural decision variable because it is exactly what the contrastive loss will later optimize. By setting the hardness threshold in the same metric space that training operates in, the selected negatives are guaranteed to be "hard" in the sense that matters for optimization — they have high cosine similarity to the query, meaning the model must learn to push them apart in precisely the space where they are close.

Step 3: Compute precision and recall at each threshold. For each candidate $\lambda$, count true positives (pairs with $s \geq \lambda$ that are actually positive), false positives (pairs with $s \geq \lambda$ that are actually negative), and false negatives (pairs with $s < \lambda$ that are actually positive). From these:

$\text{Precision}(\lambda) = \frac{\text{True Positives}}{\text{True Positives} + \text{False Positives}}$

$\text{Recall}(\lambda) = \frac{\text{True Positives}}{\text{True Positives} + \text{False Negatives}}$

where Precision measures what fraction of pairs above the threshold are truly positive, and Recall measures what fraction of all truly positive pairs are above the threshold.

Step 4: Select the F1-optimal threshold. The F1 score, the harmonic mean of precision and recall, is computed for each $\lambda$:

$\text{F1}(\lambda) = 2 \cdot \frac{\text{Precision}(\lambda) \cdot \text{Recall}(\lambda)}{\text{Precision}(\lambda) + \text{Recall}(\lambda)}$

and the optimal threshold is $\lambda^* = \arg\max_\lambda \text{F1}(\lambda)$.

Step 5: Construct the hard negative set. Once $\lambda^*$ is determined, hard negatives are selected as samples whose similarity scores are below $\lambda^*$ (so they are not classified as positives) but among the highest below this threshold:

$\mathcal{H} = \{x_i \mid s(x_i) < \lambda^*, s(x_i) \text{ is among the highest below } \lambda^*\}$

where $\mathcal{H}$ is the set of hard negatives, $s(x_i)$ is the similarity score of candidate $x_i$, and the selection criterion picks negatives that are as close as possible to being positives without crossing the threshold.

What this computes: The final output is, for each query in the dataset, a set of negative items that are semantically similar to the positive item but not similar enough to be considered truly positive by the F1-optimal classifier. These are the "confusable" items — the ones the model will benefit most from learning to distinguish.

Why this form: Using F1 as the optimization criterion balances precision and recall. A threshold that maximizes precision alone would be too high (admitting only extremely similar negatives, which might be genuine false negatives — items that should be positive but were mislabeled in the data). A threshold that maximizes recall alone would be too low (admitting many obvious negatives, defeating the purpose of hard mining). The F1 score penalizes both types of error, producing a threshold that admits challenging but genuinely negative samples. The "among the highest below" selection criterion ensures that within the negative set, the hardest examples (closest to the threshold) are prioritized, maximizing the educational value of each negative.

Integration into contrastive loss. During training, each query $q_i$ is contrasted against its positive target $t_i$, its hard negatives from $\mathcal{H}$, and random in-batch negatives from $\mathcal{N}$:

$\mathcal{L}_{\text{contrast}} = -\sum_{i=1}^N \left[ \log \frac{\exp(s_{ii}/\tau)}{\exp(s_{ii}/\tau) + \sum_{(q_i, t_j) \in \mathcal{H}} \exp(s_{ij}/\tau) + \sum_{(q_i, t_j) \in \mathcal{N}} \exp(s_{ij}/\tau)} \right]$

where $s_{ii}$ is the cosine similarity of the positive pair $(q_i, t_i)$, $\tau$ is a learnable temperature parameter (initialized differently per task and per dataset to account for distributional variations in similarity scales), $\mathcal{H}$ provides hard negatives that challenge the model to make fine-grained distinctions, and $\mathcal{N}$ provides random negatives that maintain the broad discrimination boundary.

What this computes: The standard InfoNCE loss, but with the negative set augmented by deliberately chosen hard negatives. For each query, the numerator encourages the positive pair to have high similarity. The denominator penalizes the model if any negative — either hard or random — has similarity approaching or exceeding that of the positive. The temperature $\tau$ controls the sharpness of the softmax distribution: lower temperatures make the model more sensitive to small similarity differences, higher temperatures smooth the distribution.

Why this form: The InfoNCE objective is the standard for contrastive representation learning because it maximizes a lower bound on mutual information between query and target representations. Augmenting with hard negatives makes this bound tighter — random negatives provide a weak signal (they're easy to push apart), but hard negatives force the model to use fine-grained features to separate confusable pairs. This is particularly important in recommendation settings where the difference between a video a user will watch and one they will skip often lies in subtle details (creator style, editing technique, mood) that random negatives do not train the model to capture.

The key operational insight is that $\lambda^*$ is computed before training begins (or periodically during training), not dynamically within each batch. This means the hard negative set is predetermined for each dataset, enabling efficient sampling during training without recomputing similarities on the fly.

Adaptive Multi-Source Data Balancing

When training on dozens of datasets with different sizes and relevance to downstream tasks, the naive approach is to assign a fixed mixing ratio — e.g., "30% Copair data, 20% Search data, 10% Classification data, etc." The paper argues this is problematic for three reasons: (a) manual ratios are based on intuition, not evidence; (b) the optimal ratio changes as training progresses; and (c) a fixed ratio cannot adapt to differences between datasets in terms of their semantic alignment with the target tasks.

The adaptive multi-source data balancing framework (Section 3.1.3, Figure 3) learns dataset-specific sampling weights by measuring the semantic similarity between each training dataset and a set of high-quality benchmark validation datasets that represent the downstream tasks. The procedure operates as follows:

Step 1: Construct validation subsets. From the training data, validation subsets are extracted that have distributions similar to the downstream test tasks. These are not held-out test sets but rather curated subsets that represent the type of relationship the model should excel at. This step is critical because the balancing framework will literally measure "how much does training on Dataset A help performance on these validation samples?" — so the validation samples must be representative of the deployment target.

Step 2: Embed both training and validation data. Using an early (preliminarily trained) version of the SAIL-Embedding model, embed all samples from both the training datasets and the validation subsets. To reduce computational cost, approximately 10,000 samples are randomly selected per dataset. This step produces dense vectors that capture the current model's representation of each dataset's content.

Step 3: Cluster and compute centroids. Within each dataset (training and validation), apply k-means clustering to the embeddings to obtain cluster centroids. These centroids represent the major semantic themes within each dataset — for instance, a dataset of cooking videos might have centroids corresponding to "baking," "stir-frying," "food review," etc. The clustering step abstracts away from individual sample noise and captures distribution-level patterns.

Step 4: Compute cross-dataset similarity matrices. For each training-validation dataset pair, compute a cosine similarity matrix $C$ between their cluster centroids. If training dataset $M_i$ has $k_i$ clusters and validation dataset $N_j$ has $k_j$ clusters, then $C \in \mathbb{R}^{k_i \times k_j}$ where $C_{ab} = \cos(c_a^{(i)}, c_b^{(j)})$ is the cosine similarity between the $a$-th cluster centroid of training dataset $i$ and the $b$-th cluster centroid of validation dataset $j$.

Step 5: Reduce the similarity matrix to a scalar via Sinkhorn algorithm. This is the key computational step. The Sinkhorn algorithm (Cuturi, 2013) computes an optimal transport plan between two distributions. Applied here:

• The distance matrix is $D = 1 - C$ (converting cosine similarities to distances, where identical centroids have distance 0 and orthogonal centroids have distance 1).
• The Sinkhorn algorithm solves for a transport matrix $P$ that minimizes $\sum_{a,b} P_{ab} D_{ab}$ subject to marginal constraints (each cluster in the training dataset must "send" probability mass summing to $1/k_i$, each cluster in the validation dataset must "receive" probability mass summing to $1/k_j$), with an entropic regularization term that smooths the transport plan.
• The scalar similarity score is then $\sum_{a,b} P_{ab} C_{ab}$ — the weighted sum of cosine similarities under the optimal transport plan, which effectively measures how much the cluster distributions of the two datasets overlap.

What this computes: The Sinkhorn-based similarity is a principled measure of distributional overlap between two datasets that accounts for the fact that clusters may have different sizes and importance. If all training clusters closely match all validation clusters, the similarity is high. If only some clusters match or the match is partial, the similarity is appropriately reduced. If the clusters are completely disjoint, the similarity is low.

Why this form: Simple alternatives like averaging pairwise centroid similarities or using the maximum similarity per cluster fail to account for the distributional structure. Averaging treats all cluster pairs equally, even if some training clusters have no corresponding validation cluster. Maximum similarity is optimistic — if one training cluster matches well but nine others are irrelevant, the average would be low (which might be correct) but the maximum would be misleadingly high. The Sinkhorn algorithm properly weights the contribution of each cluster by how much probability mass it accounts for in the overall distribution, providing a holistic measure of dataset similarity.

Step 6: Apply "fusion first" strategy for modality combination. The similarity computation is performed for each modality separately — producing vision similarity, text similarity, audio similarity, and fusion similarity (where "fusion" means concatenated multimodal embeddings). The "fusion first" strategy prefers the fusion similarity when multimodal embeddings are available, falling back to individual modality similarities when they are not. This ensures that the balancing weights reflect the model's actual multimodal capability rather than single-modality proxies.

Step 7: Normalize to sampling weights. The similarities between each training dataset $M_i$ and each validation benchmark $N_j$ form an $m \times n$ matrix (where $m$ is the number of training datasets and $n$ is the number of validation benchmarks). Averaging across benchmarks (the $n$ dimension) gives a single importance score for each training dataset. Applying softmax normalization converts these importance scores into a probability distribution over training datasets:

$w_i = \frac{\exp(s_i / \tau_{\text{bal}})}{\sum_{j=1}^m \exp(s_j / \tau_{\text{bal}})}$

where $w_i$ is the sampling weight for training dataset $i$, $s_i$ is its average similarity to all validation benchmarks, and $\tau_{\text{bal}}$ is a temperature parameter controlling the sharpness of the distribution (not explicitly parameterized in the paper but implied by the softmax operation).

What this computes: A probability distribution over training datasets where datasets with high semantic similarity to the validation benchmarks receive higher sampling probability, and datasets with low similarity receive lower probability (but not zero — the softmax ensures all datasets have some chance of being sampled, preserving distributional diversity).

Why this form: Hard filtering (removing datasets with similarity below a threshold) would risk overfitting to the validation benchmarks and losing coverage of edge cases. Soft weighting via softmax preserves the contribution of all datasets while appropriately prioritizing those most relevant. The temperature can be tuned: a high temperature produces near-uniform weights (risk of training on irrelevant data), while a low temperature approaches hard selection (risk of distribution collapse). The paper does not specify how this temperature is chosen, but the adaptive nature of the framework means the weights can be recomputed periodically as training progresses and the model's representation space evolves.

---

Architecture Design

Overall Architecture Philosophy

SAIL-Embedding treats the LLM as a universal fusion engine. Rather than designing separate cross-modal attention mechanisms for each modality pair (vision-text, audio-text, vision-audio), the architecture routes all modality signals through the LLM's existing self-attention layers, leveraging the LLM's pre-trained ability to model token-level interactions. The key design metaphor is "foreign language" (Section 3.2.1): just as an LLM can process text in English, Chinese, or code by mapping tokens to a shared embedding space, SAIL-Embedding extends this capability to visual and audio "languages" by training encoders that translate these modalities into the LLM's native token format.

The architecture is initialized from a pre-trained LLM (likely Qwen2-based, given the references to Qwen architectures and the Qwen3 technical report citation) that has been "warmed up" following the SAIL-VL training procedure (reference [13]), which presumably involves continued pre-training on multimodal data to adapt the purely text-based LLM to handle non-text tokens.

Input modalities and their representations:

• Text modality: Raw text fields (title, OCR, ASR, nickname, tags) are cleaned, deduplicated, tokenized into the LLM's vocabulary, and mapped to word embeddings via a trainable embedding layer. The paper notes that partial fields are randomly dropped during training to simulate missing modalities at deployment time — a crucial detail for production robustness, since real-world items frequently lack OCR text, ASR quality varies, and tags may be incomplete.

• Visual modality: Sampled video frames are processed through a Vision Transformer backbone (SAIL-ViT, reference [70]) with patch size 14×14 and uniform resolution resizing, producing a sequence of spatio-temporal patch tokens. These tokens are then compressed through a Visual Perceiver module (detailed below).

• Audio modality: Variable-length audio clips are processed through a CLAP model (Contrastive Language-Audio Pre-training, reference [15]) that extracts a single acoustic embedding per clip, with length-dependent preprocessing (detailed below).

All tokens are then concatenated into the sequence $\mathbf{T} = [\mathbf{T}_a; \mathbf{T}_v; \mathbf{T}_t]$ where $[\cdot; \cdot]$ denotes concatenation along the token dimension, and fed into the LLM Transformer.

Text Tokenizer and Instruction Design

The text modality carries the richest information on Douyin because it includes both creator-supplied metadata and automatically extracted content. The five text fields used are:

1. Title: The video's title text, which is typically a short description authored by the creator to attract viewers. Titles often carry the primary topic signal and emotional framing.

2. OCR Text: Characters recognized from video frames via optical character recognition. This captures on-screen text that may contain critical information not present in the title — for instance, a cooking video might have ingredient quantities overlaid on the frame that the title only describes qualitatively.

3. ASR Text: Speech transcribed from the audio track via automatic speech recognition. This captures spoken content that may differ substantially from the title — a creator might narrate their process while the title is a catchy hook.

4. Nickname: The content creator's display name. This provides creator identity information, enabling the model to potentially learn creator-specific styles and audience patterns.

5. Tags: Hierarchical text labels generated by automatic tagging models. These provide structured category information (e.g., "# Sports; # Comprehensive Sports; # Yoga") that complement the unstructured text fields.

These fields are concatenated into a single text sequence with field-specific prefixes or separators (the exact formatting is not specified but can be inferred from the prompt templates in Figure 5). The processing includes cleaning (removing noise, normalizing whitespace), deduplication (removing redundant information across fields), and random field dropping (to simulate deployment where fields may be missing).

Instruction prompts (Figure 5) serve a critical function beyond simple task specification: they teach the model to produce asymmetric embeddings for query and target sides. In i2i retrieval, both query and target are full multimodal items, and their instructions mirror each other — "find an image-text-audio triplet that shares the same theme" for the query side and a matching instruction for the target side. In q2i retrieval, the query is text-only, so the query instruction says "based on the user's query keywords... extract [videos] that best match the user's intent," while the target instruction describes the video content. In classification (CLS), the query side receives the item with its modalities and is asked to "find the hierarchical label that best represents the content theme," while the target side receives only the text label and is asked to "find the live screenshots and their text and audio that match the theme."

This asymmetry is crucial because it means the model learns to produce embeddings that mean "this is a query looking for answers" versus "this is an answer matching queries," enabling retrieval without requiring symmetric architectures. The instructions follow the chat template format with <|im_start|>system, <|im_start|>user, and <|im_start|>assistant tokens, consistent with the Qwen chat model format.

Vision Encoding Module

Video data presents a unique challenge: a typical Douyin video might be 15–60 seconds long, and sampling frames at a regular interval can produce dozens or hundreds of frames, each of which is tokenized into hundreds of patches by the ViT. Even at a moderate resolution and frame rate, this can produce thousands of visual tokens per video — far too many to feed into an LLM with quadratic self-attention complexity.

The Visual Perceiver module (Section 3.2.3) addresses this by acting as a learned bottleneck between the ViT encoder and the LLM fusion module. The architecture follows the Perceiver design (Jaegle et al., 2021): a small set of $N_q = 16$ learnable latent query tokens are concatenated with the full sequence of visual patch tokens, passed through a Transformer block (with self-attention), and only the query token outputs are retained. The latent dimension is 1024, the number of Transformer layers in the Perceiver is 6, and self-attention is enabled (meaning the query tokens can attend to each other as well as to the visual tokens).

How it works mechanically:

1. The ViT backbone processes each sampled frame independently, producing a sequence of patch tokens per frame. These sequences are concatenated across frames, yielding a long sequence of visual tokens (length proportional to number of frames × patches per frame).

2. The 16 learnable query tokens — which are randomly initialized and trained — are prepended to this sequence. These tokens have no inherent meaning at initialization; they are "slots" that will learn to extract the most salient visual information.

3. The combined sequence passes through 6 Transformer layers with self-attention. The query tokens attend to all visual tokens (and to each other), learning to aggregate information from the visual tokens into compact representations. The visual tokens also attend to the query tokens, so the compression is bidirectional — the visual tokens can "decide" what information is most important to communicate to the queries.

4. After the Transformer layers, only the 16 query token embeddings are kept. The original visual tokens are discarded, having served their purpose of providing information to the queries.

What this computes: A fixed-size (16 tokens × 1024 dimensions = 16,384 values) compressed representation of the visual content, regardless of how many frames or patches were in the original input. This is a dramatic reduction — from potentially thousands of tokens to exactly 16 — making the visual modality tractable for the LLM fusion module.

Why this form: The Perceiver architecture is specifically designed for problems where the input has high token count (images, video, point clouds) but the information content is much lower-dimensional. The learnable query tokens act as an information bottleneck — the Transformer is forced to compress the relevant visual information into the query tokens because the original tokens will be discarded. The number of layers (6) provides sufficient depth for the queries to iteratively refine their extracted information: early layers might capture low-level features (edges, colors), while later layers can reason about higher-level semantics (objects, actions, scene composition). The dimension of 1024 provides sufficient capacity per token to represent rich visual semantics while keeping the total contribution to the LLM sequence manageable.

The alternative — simply subsampling frames or reducing ViT resolution — would lose fine-grained visual details that might be critical for distinguishing similar-looking videos. The Perceiver approach makes the compression learned and task-dependent: the query tokens learn to extract whatever visual information is most useful for the downstream contrastive objectives, rather than being constrained by a hand-designed compression scheme.

Audio Encoding Module

The audio modality processing must balance two competing constraints: (1) extraction of semantically meaningful acoustic features, and (2) computational efficiency for online deployment where encoding latency matters. The paper empirically selects CLAP (Contrastive Language-Audio Pre-training, Elizalde et al., 2023) over alternatives like Whisper (Radford et al., 2023) or Qwen-Audio (Chu et al., 2023), citing that CLAP is "faster than models like Whisper" and balances "the efficiency of online deployment with mitigating the long-tail distribution issue in post-sampling audio sequence lengths."

The key design challenge is handling variable-length audio. Short videos can range from a few seconds (a quick clip) to several minutes (a detailed tutorial), and the audio encoder must produce a fixed-size representation regardless. The paper's solution (Section 3.2.4) uses a threshold of 10 seconds and two different processing paths:

For short audio (≤ 10 seconds): Apply a repeat-and-pad operation to normalize the waveform length before feeding it to CLAP. This ensures that very short clips (e.g., 2-second sound effects) produce valid inputs to the encoder despite being below the expected input length. The output is a single 1 × 512 dimensional audio token.

For long audio (> 10 seconds): Segment the waveform into consecutive non-overlapping 10-second chunks. Each chunk is independently processed through CLAP to produce a 1 × 512 embedding. The chunk embeddings are then aggregated via mean pooling into a single 1 × 512 representation.

Why 10 seconds? The paper does not explicitly justify this threshold, but it likely relates to CLAP's training: CLAP was trained on AudioSet-style clips that are typically around 10 seconds, so its representations are optimized for this duration. Segmenting at 10 seconds ensures each chunk falls within CLAP's effective operating range.

Dimensionality alignment: The raw CLAP output dimension is 512. This is mapped through a learned adapter (likely a linear projection or small MLP) to the LLM's hidden dimension of 1536. The paper specifies this in Section 4.1: "Audio tokens extracted from the CLAP model have an original dimension of 512 and are mapped via an adapter to a 1536-dimensional alignment hidden state."

The single-token design choice: Notably, the audio processing produces exactly one token per item (after mean pooling for long audio). This is in stark contrast to the visual modality, which produces 16 tokens through the Perceiver. The justification is likely that audio in short videos carries relatively low information density compared to vision — background music, ambient sounds, and speech (which is already captured by ASR text) provide complementary but not primary signals. A single token forces the model to compress all audio information into one representation, which may be appropriate if audio serves as a "mood signal" rather than a detailed content carrier.

However, this design choice also means the model cannot learn fine-grained temporal alignment between audio events and visual content — it cannot associate a specific sound effect with a specific visual action because the audio is compressed to a single global token. This is a conscious tradeoff of temporal resolution for computational efficiency.

Fusion Module

The fusion module is the LLM backbone itself — a pre-trained Transformer language model with bidirectional attention that processes the concatenated multimodal token sequence. The paper specifies that the LLM is warmed up following reference [13] (the SAIL-VL training procedure), which means it has already been exposed to multimodal tokens during a prior training phase and is not starting from a purely text-based initialization.

Bidirectional attention: The paper explicitly replaces the LLM's default causal attention (where each token can only attend to previous tokens) with bidirectional attention (where each token can attend to all other tokens). The rationale (Section 4.4.5, ablation results in Figure 9): "the bidirectional attention captures global multimodal semantic dependencies, mitigating the semantic bias potentially introduced by the causal attention. This makes it more suitable for the representation embedding scenarios."

This is a consequential design choice. Causal attention is natural for autoregressive generation (predicting the next token), but for embedding extraction, the model needs to produce representations that incorporate information from the entire multimodal context. A causal attention model would mean that early tokens in the sequence cannot attend to later tokens, creating an asymmetry where text tokens at the end of the sequence can "see" visual tokens but visual tokens at the start cannot "see" later text tokens. Bidirectional attention removes this asymmetry, enabling symmetric information flow across all modality tokens.

Embedding extraction: After the multimodal sequence passes through all Transformer layers, the output hidden states for all token positions undergo two post-processing steps:

1. tanh normalization: $\mathbf{H}_{\text{norm}} = \tanh(\mathbf{H})$ where $\mathbf{H}$ is the matrix of hidden states. The tanh function squashes each value to the range [-1, 1], ensuring embeddings are bounded — which improves numerical stability during similarity computation and can help with quantization for deployment.

2. Mean pooling: $\mathbf{z} = \text{Meanpool}(\mathbf{H}_{\text{norm}})$ where $\mathbf{z} \in \mathbb{R}^{1536}$ is the final embedding vector, computed by averaging the normalized hidden states across all token positions (or possibly across a subset of positions — the paper doesn't specify whether special tokens are excluded).

Why mean pooling over CLS token or last token? Mean pooling aggregates information from all token positions, giving equal weight to every modality token. This is important because different tasks require attending to different modalities — for search, the text tokens are most important; for content understanding, visual tokens dominate; for recommendation, the combined signal matters. A CLS token approach would force the model to route all task-relevant information through a single learned position, which can create a bottleneck. Mean pooling provides a simple, task-agnostic aggregation that preserves information from all positions.

Matryoshka Representation Learning (MRL): The full 1536-dimensional embedding is optimized alongside nested sub-embeddings at 768 and 128 dimensions. The sub-embeddings are created by simply slicing the first 768 or 128 dimensions of the full embedding:

$\mathcal{L}_{\text{nce-mrl}} = \mathcal{L}_{\text{nce}}(\mathbf{z}_{128}) + \mathcal{L}_{\text{nce}}(\mathbf{z}_{768}) + \mathcal{L}_{\text{nce}}(\mathbf{z}_{1536})$

where $\mathbf{z}_d$ denotes the sub-embedding formed by taking the first $d$ dimensions of the full 1536-dimensional vector $\mathbf{z}$, and $\mathcal{L}_{\text{nce}}$ is the contrastive loss applied independently to each dimensionality.

What this computes: Three separate contrastive losses, each applied to embeddings of different dimensionalities, with all three losses summed and optimized jointly. The model must learn to produce 128-dimensional embeddings that are approximately as discriminative as 768-dimensional ones, which in turn must be as good as 1536-dimensional ones.

Why this form: In production, different deployment stages have different latency-storage tradeoffs. Recall (retrieving candidates from a billion-item corpus) requires fast similarity search, often using approximate nearest neighbor methods that scale with embedding dimensionality — 128-dimensional embeddings enable orders-of-magnitude faster search than 1536-dimensional ones. Ranking (scoring a few hundred candidates) can afford higher-dimensional embeddings for better precision. MRL enables a single model to serve both needs by producing nested embeddings where the information is "front-loaded" — the most important discriminative features are placed in the first 128 dimensions, with progressively finer-grained features in higher dimensions. The slicing approach (rather than training separate models or using separate projection heads) ensures the embeddings are truly nested, which simplifies deployment.

---

Training Strategies

Content-Aware Progressive Training

Training a general-purpose LLM to become a specialized embedding model cannot be done in a single stage because the data distribution and training objectives change substantially between early phases (broad multimodal understanding) and late phases (fine-grained hard negative discrimination and recommendation alignment). The paper's progressive training framework (Section 3.3.1, Figure 6a) divides training into three stages with decreasing dataset diversity and increasing data quality:

Stage I: Base Model Training. The model is trained on large-scale, diverse datasets — essentially all the data described in Table 1 at full scale. This stage aims to "acquire fundamental multimodal representation capabilities" (Section 3.3.1). The diversity is critical here because it prevents the model from overfitting to any single pairing strategy or modality. The model learns general relationships: text queries should map to relevant videos, similar-looking videos should be close in embedding space, items with the same tags should cluster together.

Stage II: Fine-tuning on Task-Aligned Subsets. The model is fine-tuned on a selected subset of datasets that are "more aligned with the target downstream tasks." The paper does not specify the exact selection criterion, but it likely uses the adaptive data balancing framework (Section 3.1.3) to identify which training datasets are most similar to the validation benchmarks, and up-weights or exclusively uses those datasets. This stage narrows the model's focus from general multimodal understanding to the specific types of relationships that matter for the deployment tasks (e.g., focusing on Search-q2i data if query-to-item retrieval is the primary downstream task).

Stage III: Refinement with Hard Negatives. A final fine-tuning stage is performed using "hard negatives and a clean subset" of data. This stage uses the dynamic hard negative mining (Section 3.1.2) to construct challenging training examples that force the model to make fine-grained distinctions. The "clean subset" likely refers to data that has been filtered for quality — removing noisy pairs where the relationship is weak or ambiguous, and focusing on high-confidence positive pairs where the semantic or behavioral connection is strong.

Why progressive rather than joint training? The paper's ablation (Figure 9) shows that progressive training improves over the "all-in-one training procedure used by the baseline." The rationale is that "progressive training enables the model to acquire substantial domain knowledge in the early phases while establishing robust task dependencies and activating corresponding representational knowledge in subsequent phases." In plainer terms: if hard negatives are introduced too early, the model hasn't yet learned basic multimodal alignment and will struggle to distinguish fine-grained differences because it can't even align modalities coarsely. The progressive curriculum ensures the model masters easy distinctions before tackling hard ones — a standard curriculum learning principle applied to the data scale axis.

Loss Definition

The training objective is a composite of four complementary losses, each designed to address a specific failure mode in multimodal embedding learning. The overall loss is:

$\mathcal{L} = \mathcal{L}_{\text{nce-mrl}} + \lambda \mathcal{L}_{\text{cosent}} + \alpha \mathcal{L}_{\text{micl}} + \beta \mathcal{L}_{\text{lf}}$

where $\lambda$, $\alpha$, and $\beta$ are balancing hyperparameters that control the relative importance of each auxiliary loss relative to the primary NCE loss.

Component 1: Noise-Contrastive Estimation (NCE) Loss with MRL

The foundational contrastive loss for global alignment between query and target embeddings:

$\mathcal{L}_{\text{nce}} = -\log \frac{\exp(\cos(\mathbf{e}_q, \mathbf{e}_t^+) / \tau)}{\exp(\cos(\mathbf{e}_q, \mathbf{e}_t^+) / \tau) + \sum_{i=1}^B \exp(\cos(\mathbf{e}_q, \mathbf{e}_t^-) / \tau)}$

where $\mathbf{e}_q$ is the query embedding, $\mathbf{e}_t^+$ is the positive target embedding (the item that should match the query), $\mathbf{e}_t^-$ are negative target embeddings from the batch (other items that do not match the query), $\cos(\cdot, \cdot)$ is the cosine similarity, $\tau$ is a learnable temperature parameter, and $B$ is the batch size.

What it computes: For a batch of $B$ query-target pairs, the loss encourages each query to be most similar to its paired target and dissimilar to all other targets in the batch. The softmax normalization creates competition: if a negative target has higher similarity to the query than the positive target does, the loss spikes — the model is heavily penalized for confusing items.

Why this form: The InfoNCE objective with in-batch negatives is the standard in contrastive representation learning because it scales well — you get $B-1$ negative examples for free per positive pair, where $B$ can be thousands with large-batch training. The learnable temperature $\tau$ is initialized differently per task and per dataset "to account for distributional variations" — some datasets naturally have higher within-pair similarity (e.g., Summary-i2i where items share n-grams) and need a higher temperature to avoid over-penalizing, while others have lower similarity and need a lower temperature to maintain discriminative pressure.

The NCE loss is extended with Matryoshka Representation Learning (MRL) as described earlier:

$\mathcal{L}_{\text{nce-mrl}} = \mathcal{L}_{\text{nce}}(\mathbf{z}_{128}) + \mathcal{L}_{\text{nce}}(\mathbf{z}_{768}) + \mathcal{L}_{\text{nce}}(\mathbf{z}_{1536})$

ensuring that all three dimensionality levels (128, 768, 1536) are optimized simultaneously.

Component 2: COSENT Loss for Fine-Grained Ranking

For text-oriented pre-training datasets (especially Score-q2i with its labeled similarity scores), a ranking loss enforces ordinal consistency — pairs with higher labeled similarity should have higher cosine similarity in the embedding space than pairs with lower labeled similarity. The COSENT loss (Su, 2022, reference [51]) is:

$\mathcal{L}_{\text{cosent}} = \log \left( 1 + \sum_{\text{sim}(i,j) > \text{sim}(k,l)} \exp\left( \frac{\cos(\mathbf{e}_k, \mathbf{e}_l) - \cos(\mathbf{e}_i, \mathbf{e}_j)}{\tau} \right) \right)$

where the sum runs over all pairs of pairs $(i,j)$ and $(k,l)$ where the labeled similarity of $(i,j)$ is greater than that of $(k,l)$, $\mathbf{e}_i, \mathbf{e}_j$ are embeddings for pair $(i,j)$, $\mathbf{e}_k, \mathbf{e}_l$ are embeddings for pair $(k,l)$, and $\tau$ is a temperature parameter.

What it computes: For every pair of pairs where the first pair should be more similar than the second according to ground-truth labels, the loss penalizes the model if the embedding cosine similarity violates this ordering — specifically, if $\cos(\mathbf{e}_k, \mathbf{e}_l) > \cos(\mathbf{e}_i, \mathbf{e}_j)$ when the labels say the opposite. The exponential term means that large ranking violations (where the higher-labeled pair has much lower embedding similarity than the lower-labeled pair) are penalized exponentially more than small violations.

Why this form: Standard contrastive loss treats all negatives equally, but in many retrieval scenarios, some negatives are "more negative" than others. For example, for the query "cooking tutorial," a video about car repair is a strong negative (very irrelevant), while a video about restaurant reviews is a weak negative (tangentially related to food). A ranking loss teaches the model about degrees of relevance, which is critical for search applications where result ordering matters. The exponential form of COSENT is specifically designed to be a smooth approximation to the number of ranking errors, making it effective for fine-grained ranking optimization.

Component 3: Multimodal In-Context Learning (mICL) Loss

This loss prevents modality imbalance — the tendency for the text modality to dominate because the LLM is pre-trained on text and "understands" text tokens better than visual or audio tokens. Without intervention, the model can achieve low contrastive loss by attending primarily to text while ignoring other modalities, producing embeddings that are effectively text-only despite having multimodal inputs.

The mICL loss (Equation 4) separates the multimodal embedding into modality-specific components and applies contrastive learning within each modality:

$\mathcal{L}_{\text{micl}} = \mathcal{L}(\mathbf{n}_v, \mathbf{n}_v^+) + \mathcal{L}(\mathbf{n}_t, \mathbf{n}_t^+)$

where $\mathbf{n}_v$ is the visual embedding (extracted from the model output by mean pooling over only the visual token positions, or obtained from a separate projection head), $\mathbf{n}_v^+$ is the positive visual embedding from the target, $\mathbf{n}_t$ is the textual embedding, $\mathbf{n}_t^+$ is the positive textual embedding, and each $\mathcal{L}$ term follows the NCE form (Equation 2) but operates within a single modality.

What it computes: Two separate contrastive losses — one that encourages visual embeddings of paired items to be similar, and one that encourages textual embeddings of paired items to be similar. This forces the model to maintain discriminative visual and textual representations even when the full multimodal embedding is available.

Why this form: Without mICL, the model can "cheat" by producing multimodal embeddings that are essentially text-only — if text carries enough signal to minimize the contrastive loss, there is no gradient pressure to improve visual or audio representations. By explicitly supervising within-modality contrastive objectives, mICL ensures that visual features remain useful even in the presence of strong text signals. This is crucial for deployment scenarios where one modality may be missing (e.g., audio not available, or text fields incomplete) — the model must be able to fall back on whatever modalities are present without catastrophic degradation.

Component 4: Late Fusion Loss

The late fusion mechanism (Equation 5) preserves visual fidelity by learning an adaptive combination of the pure visual embedding with the full multimodal embedding:

$\mathbf{z} = \sigma(\mathbf{W}[\mathbf{v}, \mathbf{n}_m] + \mathbf{b})$

$\hat{\mathbf{n}}_m = \mathbf{z} \odot \mathbf{v} + (1 - \mathbf{z}) \odot \mathbf{n}_m$

where $\mathbf{v}$ is the visual embedding (from the vision encoder, before fusion), $\mathbf{n}_m$ is the multimodal embedding (after LLM fusion), $[\mathbf{v}, \mathbf{n}_m]$ denotes concatenation along the feature dimension, $\mathbf{W}$ and $\mathbf{b}$ are learnable parameters of the gating layer, $\sigma$ is the sigmoid function producing per-dimension weights in $(0, 1)$, $\mathbf{z} \in (0, 1)^{1536}$ is the learned gating vector, and $\odot$ denotes element-wise multiplication.

What it computes: A dimension-specific gated combination of the pure visual representation and the full multimodal representation. For dimensions where $z_i$ is close to 1, the output relies primarily on the visual embedding. For dimensions where $z_i$ is close to 0, the output relies primarily on the multimodal embedding. The sigmoid constraint means the combination is a convex interpolation in each dimension — the model cannot amplify or negate components, only blend them.

Why this form: The multimodal embedding after LLM fusion may lose visual details because the text modality dominates self-attention (the LLM has an inherent bias toward text). The late fusion mechanism provides a "shortcut" for visual information, allowing the final representation to retain visual fidelity even when the fusion process would otherwise wash it out. The per-dimension gating (rather than a scalar gate) allows the model to be selective: some dimensions of the embedding space may benefit from pure visual features (e.g., dimensions encoding color histograms or texture patterns), while others benefit from multimodal integration (e.g., dimensions encoding high-level semantics that require cross-modal reasoning). The late-fusion contrastive loss $\mathcal{L}_{\text{lf}}$ is then applied over $\hat{\mathbf{n}}_m$ to ensure the gated combination remains discriminative.

Why not simply concatenate or average? Concatenating visual and multimodal embeddings would double the dimensionality, defeating the purpose of compression. Simple averaging would give equal weight to both, but the optimal balance likely varies by dimension and by task. The learned gating provides adaptivity — for search tasks where text is the query, the gate might favor multimodal features; for visual similarity tasks, the gate might favor visual features.

Stochastic Specialization Training

The standard multi-dataset training approach (Figure 6b) mixes data from all sources within each batch — a batch of size 256 might contain 100 Copair pairs, 80 Search pairs, 50 Classification pairs, and 26 RSDF pairs, proportional to dataset sizes or manual mixing ratios. The paper identifies two problems with this approach (Section 3.3.3):

1. Small per-domain effective batch size: The contrastive loss operates within-batch — positives are contrasted against other targets in the same batch. If a batch contains only 26 RSDF pairs, the model gets only 25 RSDF negatives, which may not be enough for meaningful contrastive learning, especially for the hard negatives that benefit from larger candidate pools.

2. Inter-domain communication overhead: Different datasets may require different preprocessing, different negative sampling strategies, or different loss configurations. Mixing them in one batch requires the training loop to handle all these variations simultaneously, increasing complexity and potentially introducing synchronization bottlenecks.

Stochastic specialization training (Section 3.3.3, Figure 6c) replaces mixed-batch sampling with single-dataset-per-batch sampling. The procedure is:

1. Define a probability distribution over training datasets using the adaptive multi-source data balancing weights (Section 3.1.3).
2. At each training iteration, sample a single dataset according to this distribution (e.g., with probability 0.3, select Copair; with probability 0.2, select Search-q2i; etc.).
3. Draw the entire batch from the selected dataset.
4. Apply dataset-specific preprocessing, loss configurations, and negative sampling within that batch.

Why this helps: Within a Copair-only batch of size 256, the model gets 255 Copair negatives, providing much richer contrastive signal than a mixed batch's 100 Copair pairs with only 99 Copair negatives. The specialization reduces gradient variance because all samples in the batch come from the same distribution — there's no tension between fitting Copair patterns and fitting Search patterns within the same gradient step. The approach also "exhibits strong scalability: adding a new dataset requires only its dataset-specific configuration, without modifying the overall batching or communication pattern."

The tradeoff: The model does not see all datasets in every iteration. Over the course of many iterations, all datasets are visited in proportion to their sampling weights, but any single batch is domain-specialized. This is acceptable because embedding learning does not require inter-dataset comparisons within a batch — the contrastive loss only needs within-dataset positives and negatives. Cross-dataset generalization comes from visiting different datasets in different iterations, allowing the model parameters to gradually accommodate all domains.

Connection to meta-learning: The paper notes this strategy is "inspired by meta-learning schemes." In meta-learning (specifically MAML-style algorithms), training proceeds by sampling tasks, taking gradient steps on each task, and learning a parameter initialization that generalizes across tasks. Stochastic specialization is analogous if each dataset is viewed as a "task" — the model learns representations that work for any dataset (task), but gradient updates are computed on one task at a time.

Dataset-Driven Pattern Matching

Multimodal embedding models typically define a fixed contrastive objective — e.g., Image-to-Text Contrastive (ITC) in CLIP, where the image embedding is contrasted against text embeddings. However, when training on heterogeneous data where queries and targets have different modality compositions, a fixed objective is insufficient:

• In Copair-i2i, both query and target have V+T+A (all three modalities), so Omni-to-Omni Contrastive (OOC) is appropriate: contrast the full multimodal query embedding against full multimodal target embeddings.
• In Search-q2i, the query is text-only (T) and the target is omni-modal (V+T+A), so Text-to-Omni contrastive is appropriate: contrast text query embeddings against multimodal target embeddings.
• In classification, the query is omni-modal and the target is text-only (the label), so Omni-to-Text contrastive is appropriate.

The dataset-driven pattern matching strategy (Section 3.3.4) dynamically constructs all valid query-target contrastive pairs based on each sample's actual modality composition. The paper generalizes the CLIP objective to a "comprehensive modality-to-modality paradigm" supporting six contrastive types:

• ITC (Image-to-Text Contrastive): Image query vs. text target.
• IIC (Image-to-Image Contrastive): Image query vs. image target.
• VTC (Video-to-Text Contrastive): Video query vs. text target.
• VVC (Video-to-Video Contrastive): Video query vs. video target.
• TTC (Text-to-Text Contrastive): Text query vs. text target.
• OOC (Omni-to-Omni Contrastive): Full multimodal query vs. full multimodal target, covering arbitrary cross-modal pairs without restriction.

How it works mechanically:

1. A configurable data processor standardizes heterogeneous raw inputs (frame paths, caption strings, ASR transcript strings, OCR token lists) into a canonical set of modality tokens: visual tokens, audio tokens, text tokens.
2. For each training sample, the processor identifies which modalities are actually present — some samples may lack audio, some may have incomplete text fields, etc.
3. Based on predefined matching patterns (which encode domain knowledge about what modality pairings are meaningful for each dataset), all valid query-target modality pairs are dynamically constructed.
4. For each valid pair, the model extracts the corresponding modality-specific embeddings (by masking out other modalities at the embedding extraction stage or by using separate projection heads) and applies the contrastive loss.
5. Modality pairings that are not valid (e.g., audio-to-audio contrastive when neither query nor target has meaningful audio) are excluded from the loss calculation.

Why this matters: This dynamic construction "maximizes the utilization of extracted embeddings" — a single forward pass produces embeddings for all modalities, and the pattern matching ensures every meaningful contrastive pair is optimized, rather than pre-committing to a single pairing per dataset. The paper reports that this "improves optimization stability, as reflected by smoother convergence in training loss." The stability improvement likely comes from more consistent gradient signals: when the model is optimized against multiple contrastive pairs per sample, the gradients are averaged across modalities, reducing the variance that would arise from optimizing only one randomly-chosen pairing.

Collaboration-Aware Recommendation Enhancement (CRE) Training

This is the paper's most novel contribution and represents a fundamentally different training phase from the content-aware progressive training that precedes it. CRE addresses the core tension identified in the introduction: embeddings trained for content similarity do not naturally capture collaborative filtering signals, but recommendation systems fundamentally depend on collaborative signals.

The CRE phase has two complementary components: sequence-to-item distillation (teaching the model that user viewing sequences predict the next consumed item) and ID-to-item distillation (aligning multimodal embeddings with the production recommender system's ID-based embeddings). Both are applied after the content-aware training stages, meaning the model first learns general multimodal semantics and then additionally learns behavioral patterns.

Sequence-to-Item Distillation (Figure 7a)

The core idea: a user's historical viewing sequence encodes their interests, and the embedding of the next video they watch should be predictable from the embeddings of the preceding videos. By training the model to make this prediction, the embedding space is infused with behavioral information — items that are frequently consumed in sequence become closer in the embedding space.

Data construction proceeds from four perspectives to capture different interest patterns:

1. Content-aware Single-Peak Interest: From a user's 1000-video viewing history, retain only videos where the user has had at least three positive interaction behaviors (watching, liking, commenting, sharing). This filters to items the user engaged with deeply, representing concentrated interest. Then, further filter based on similarity in content representations — select sequence items whose content embeddings are similar to the target video's content embedding. This constructs sequences where the user's interest is narrowly focused on a specific content type, and the model should learn that within-category sequences are predictive.

2. Content-aware Multi-Peak Interest: Retain any video with at least one positive behavior — a weaker criterion that admits more diverse interests. Then select sequence items based on the Jaccard coefficient of behavior labels with the target item, keeping items with Jaccard > 0.5. This captures sequences where the user explores different content types, and the model should learn that diverse but behaviorally-correlated items predict consumption.

3. Collaboration-aware Single-Peak Interest: Similar to (1) but using label-based clustering and user interaction distributions to select items that satisfy the original user interaction pattern. This introduces collaborative signals explicitly — items are grouped not by content similarity but by audience overlap.

4. Collaboration-aware Multi-Peak Interest: Similar to (2) but selecting items proportional to the distribution of clustered video IDs. This captures broad collaborative patterns where users with diverse interests still exhibit predictable consumption trajectories.

The output is a dataset of (sequence, target) pairs where the sequence contains 10 videos (default sequence length, specified in Section 4.1) and the target is the next video watched.

Training is performed via contrastive learning between the sequence representation and the target item embedding. Two sequence modeling approaches are explored:

• Mean pooling: Each video in the sequence is independently embedded by the (frozen? fine-tuned? — the paper doesn't specify) SAIL-Embedding model, producing 10 embeddings. These are averaged to form the sequence representation. Simple, fast, no additional parameters.

• Sequence encoder: A three-layer Transformer module is trained to ingest the sequence of 10 video embeddings and produce a single sequence representation via a special output token. This approach can learn to weight items in the sequence differently — recent items might be more predictive, or certain types of items might be strong indicators of the next watch.

The paper defaults to mean pooling for efficiency (Section 4.1: "In the seq2item distillation, we default to the mean pooling method").

ID-to-Item Distillation (Figure 7b)

The second component directly transfers knowledge from the production recommender system into the multimodal embedding space. The recommender maintains ID embeddings for each item — dense vectors trained on massive user interaction logs that capture collaborative filtering patterns with high fidelity. These embeddings encode which items are consumed together, by whom, and in what contexts.

The distillation procedure:

1. For each item, obtain its ID embeddings from the recommender system. The paper says they "jointly utilize multiple ID embeddings of each item" — this likely means using embeddings from different ID spaces (item ID, author ID, category ID, etc.) that the recommender maintains.

2. Project the ID embeddings through a learned feature projection layer to align their dimensionality and semantics with the multimodal embedding space.

3. Train the SAIL-Embedding model to minimize the distance between its multimodal embedding of an item and the projected ID embeddings — a standard knowledge distillation objective where the ID embeddings serve as soft targets.

4. Crucially, this distillation is performed alongside an auxiliary i2i retrieval task (the standard contrastive loss on item-to-item pairs). This prevents "the representation distribution from overly sacrificing content-aware perception capabilities" — without the auxiliary task, the model might collapse its representations to simply mimic the ID embeddings, losing all content understanding.

Why both sequence-to-item and ID-to-item? They capture different aspects of collaborative filtering. Sequence-to-item distillation teaches the model about user trajectories — what items tend to be consumed in what order. ID-to-item distillation teaches the model about item-item relationships — what items tend to have overlapping audiences. Together, they provide complementary behavioral signals that move the embedding space from pure content similarity toward a hybrid of content and collaboration.

Experimental evidence for CRE effectiveness (Section 4.4.3):

• On the Gid-i2i benchmark (paired items based on ID embedding relevance), CRE improves SAIL-Embedding's Recall@50 from 52.46% to 59.69% (+7.23 percentage points) and Recall@100 from 59.06% to 66.37% (+7.31 percentage points).

• On the Copair-i2i benchmark (paired items based on content representation similarity), performance slightly degrades from 69.17% to 66.83% at Recall@50. This tradeoff is explicitly acknowledged as "tolerable to balance the differing application requirements."

• The separability analysis (Figure 8c) shows that CRE reduces the overlap between positive and negative similarity distributions and makes the positive distribution more compact — the model becomes more confident about which items are truly related, and the related items cluster more tightly.

• The sequence modeling evaluation (Table 5) shows that after seq2item distillation, Filtered Seq2item Recall@10 improves from 19.35% to 22.02% (+2.67 percentage points), and after additional ID2item distillation, it further improves to 23.77%. On the more challenging Vanilla Seq2item (random sequence items without content filtering), the final improvement is from 10.46% to 13.36% at Recall@10 — a 2.9 percentage-point gain, demonstrating that the model learns to extract predictive signals even from noisy sequences.

A subtle design choice: The paper trains CRE after content-aware training, not jointly from the start. This ordering is important because the ID embeddings and sequence patterns encode the recommender's current behavior, which may include biases (popularity bias, demographic bias, etc.). By first training on content, the model establishes a content-based "anchor" that prevents the CRE phase from pulling the embedding space entirely toward the recommender's potentially biased collaborative signals. The auxiliary i2i task during CRE serves the same purpose — it is a regularizer that says "become more like the recommender, but don't forget how to represent content."

4. Key Insights and Innovations

Innovation 1: Collaboration-Aware Recommendation Enhancement as a New Training Paradigm for Embedding Models

The most intellectually distinctive contribution of this paper is not any single architectural or optimization trick, but the identification and operationalization of a fundamental misalignment that prior work had simply ignored: embeddings optimized for content similarity are structurally incapable of capturing the collaborative signals that drive recommendation systems, and this gap cannot be closed by better content understanding alone. The paper doesn't just observe this — it builds a training paradigm around it, introducing the concept of a collaboration-aware training phase that directly injects user behavioral patterns into the embedding space through two complementary distillation mechanisms.

Prior to this work, the dominant assumption in multimodal embedding research — from CLIP through VLM2Vec to GME — was that better content understanding automatically translates to better downstream task performance. The implicit logic was: if the model truly understands what a video is (its topic, its visual style, its mood), then two videos that users tend to watch together must share some latent content property, and improving content representation will eventually capture this. This paper demonstrates that this assumption is empirically false. The Copair-i2i benchmark (items paired by user co-consumption) and the Gid-i2i benchmark (items paired by ID embedding similarity from the recommender system) measure fundamentally different grouping principles than the Film-i2i or Summary-i2i benchmarks (items paired by shared intellectual property or textual n-grams). A model can excel at one while performing poorly on the other — and indeed, as Table 4 shows, SAIL-Embedding without CRE achieves 69.17% Recall@50 on Copair-i2i but only 52.46% on Gid-i2i, while the CRE-trained version reverses the emphasis, reaching 66.83% and 59.69% respectively. These are not different points on the same spectrum of "understanding" — they are different objectives, and optimizing one degrades the other.

The conceptual move that makes this an innovation rather than an observation is the dual distillation architecture (sequence-to-item and ID-to-item, Section 3.3.5) that treats collaborative signals as a separate knowledge source to be transferred into the embedding model, rather than as something the model should discover on its own from content. This is a fundamental departure from prior work. In standard multimodal embedding training, all supervision comes from relationships that exist in the content itself: image-caption pairs, video-text pairs, item-label pairs. The CRE training instead introduces supervision from user behavior — what users click on, what sequences they watch, what items have overlapping audiences. These signals are not derivable from content; a makeup tutorial and a gaming livestream might share no visual, textual, or auditory similarity, but if the same demographic consumes both, a collaborative filter places them together. The paper's innovation is recognizing that this second signal source requires a different training mechanism — knowledge distillation from a teacher (the recommender's ID embeddings) rather than self-supervised contrastive learning from content pairs — and that both must coexist in the same representation space.

What elevates this beyond an engineering convenience is the deliberate tradeoff acknowledgment. The paper doesn't claim CRE is universally beneficial — it shows that Copair-i2i performance degrades slightly (Table 4: 69.17% → 66.83%), and frames this as "tolerable to balance the differing application requirements." This is intellectually honest in a way that distinguishes it from papers that claim monotonic improvements. The implication is that future embedding models for recommendation should not pursue "better representations" in the abstract but should explicitly manage the content-collaboration tradeoff based on deployment context — a nuanced research agenda that the paper opens but doesn't fully resolve.

The significance of this innovation extends beyond the specific distillation techniques used. What the paper has effectively done is define a new training stage that is as fundamental to recommendation-oriented embedding models as contrastive pre-training is to CLIP. Just as the field now accepts that vision-language models need a contrastive pre-training phase, this paper argues that embedding models for recommendation need a collaboration alignment phase — and it provides the first systematic evidence for what happens when you include it versus when you don't. The 7.23 percentage-point gain on Gid-i2i Recall@50 (Table 4) is not just a metric improvement; it's evidence that a substantial fraction of the information needed for recommendation simply doesn't exist in content-based supervision and must be injected from behavioral sources.

Innovation 2: Difficulty-Adaptive Training Through Dynamic, Per-Dataset Hard Negative Thresholds

The paper's dynamic hard negative mining (Section 3.1.2) appears at first glance to be a minor optimization trick — an automated way to set a similarity threshold for selecting challenging negatives. But the conceptual contribution runs deeper: it replaces a global notion of hardness (the same threshold applied to all data) with a dataset-relative notion, where "hard" is defined relative to the inherent ambiguity of each data source. This reframes hard negative mining from a hyperparameter tuning problem into a statistical property of the data distribution that can be estimated from the data itself.

Prior work on hard negative mining in contrastive learning (and in the retrieval literature more broadly) typically applies a fixed similarity threshold across all training samples: select negatives whose cosine similarity to the query exceeds some value θ, treat θ as a hyperparameter, and tune it on a validation set. This works poorly in multi-source industrial training because different datasets have fundamentally different similarity distributions. In the Copair-i2i dataset (behavioral co-occurrence), positive pairs are noisy — two videos might be paired because a user watched both, but they might be completely unrelated in content, so the average positive similarity might be relatively low (say, 0.4). In the Summary-i2i dataset (n-gram matching), positive pairs share explicit textual overlap, so the average positive similarity might be much higher (say, 0.8). A single threshold of 0.5 would correctly admit hard negatives for Summary-i2i (where anything below 0.5 is clearly negative) but would flood Copair-i2i with false negatives — pairs with similarity 0.45 that are actually genuine behavioral correlations but fall below the arbitrary cutoff.

The paper's F1-optimization procedure (computing the threshold that maximizes the harmonic mean of precision and recall in classifying positive vs. negative pairs by similarity score) is elegant because it defines hardness operationally: a negative is "hard" if it is similar enough to the positive that the model might confuse them, but not so similar that a reasonable classifier would call it a positive. The F1 criterion balances these competing desiderata automatically, producing per-dataset thresholds that reflect each dataset's inherent separability. This is not just automation — it's a principled redefinition of what "hard" means in multi-source training.

The intellectual significance of this reframing is that it decouples hard negative mining from task-specific validation metrics. In prior work, you might tune the hardness threshold to maximize retrieval recall on a held-out set. But in industrial settings with dozens of downstream tasks, this is intractable — optimizing for Search-q2i recall might produce thresholds that hurt Film-i2i performance. By using F1 on the training data's own similarity distribution as the objective, the paper finds a threshold that is optimal for representing that dataset's structure, not for any particular downstream metric. The downstream task performance then follows from having well-structured representations, rather than being directly optimized. This is a subtle but important distinction that moves hard negative mining from a task-specific tuning exercise to a data-characterization step that requires no downstream labels.

The evidence that this matters comes indirectly but persuasively: the progressive training ablation (Figure 9) shows that adding hard negative refinement in Stage III improves average i2i performance from 51.75% to 52.29% (a modest absolute gain, but the paper doesn't isolate the hard negative contribution from the progressive training structure). More importantly, the fact that this method is integrated into a training pipeline that handles 10B+ samples across dozens of datasets — and doesn't require per-dataset manual threshold configuration — is itself evidence of its practical significance. In industrial ML, methods that eliminate manual tuning are often more impactful than methods that achieve +1% on a benchmark.

Innovation 3: Sinkhorn-Based Adaptive Data Balancing as a Distribution-Matching Alternative to Manual Mixing Ratios

The adaptive multi-source data balancing framework (Section 3.1.3, Figure 3) constitutes a genuine methodological contribution to the problem of training on heterogeneous multi-source data — a problem that is ubiquitous in industrial ML but severely underexplored in academic research. The innovation is not the idea of weighting training datasets (which is standard practice), but the specific mechanism for deriving weights: measuring distributional overlap between training and validation datasets via optimal transport (Sinkhorn algorithm applied to cluster centroids) and using this overlap to soft-weight sampling probabilities.

Prior approaches to multi-source training fall into roughly three categories, all of which the paper implicitly critiques: (1) uniform mixing — sample from all datasets with equal probability, regardless of relevance to downstream tasks; (2) heuristic weighting — assign manual ratios based on practitioner intuition ("30% Copair, 20% Search, 10% Classification"); (3) hard filtering — remove entire datasets whose samples fall below some similarity threshold to a validation set. Each approach has a known failure mode. Uniform mixing wastes compute on irrelevant data. Heuristic weighting is untestable and brittle to distribution shifts — the optimal ratio for one downstream task may be different for another, and practitioners typically guess at a compromise. Hard filtering risks distribution collapse: if you train only on data that looks like your validation set, you lose coverage of edge cases and long-tail phenomena that the validation set doesn't represent.

The paper's Sinkhorn-based approach navigates between these failure modes. By computing cluster-level similarity between training and validation datasets (rather than sample-level similarity), it captures distributional affinity — the degree to which two datasets share semantic themes — rather than instance-level duplication. The Sinkhorn algorithm is the critical technical choice here because it produces a properly normalized transport plan that accounts for the fact that clusters have different sizes and importance. A naive alternative like averaging all pairwise centroid similarities would overweight large clusters and underweight small but highly relevant ones. The Sinkhorn solution ensures that every cluster in the validation set "receives" probability mass proportional to its size, and every cluster in the training set "sends" probability mass proportional to its size, yielding a symmetric, balanced measure of overlap.

The intellectual move that distinguishes this from prior work is treating data balancing as a distribution-matching problem rather than a sample-selection problem. The goal is not to find individual training samples that look like validation samples (which leads to overfitting) but to ensure that the training distribution has high coverage of the validation distribution's semantic themes, even if individual samples differ. The soft weighting (softmax normalization of Sinkhorn similarities) preserves this distributional diversity — a training dataset with low similarity to the validation benchmarks is not discarded but down-weighted, so it still contributes its unique semantic themes but doesn't dominate the loss.

The significance of this approach extends beyond the paper's specific use case. Multi-source training with distributionally-aware weighting is a general problem that arises whenever models are trained on data from different domains, different time periods, or different collection pipelines. The paper provides a reusable recipe: embed validation samples, embed training samples, cluster both, compute Sinkhorn similarities, softmax to weights. This is modular (the similarity computation can use any embedding model, not just SAIL-Embedding) and scalable (clustering and Sinkhorn are applied to centroids, not raw samples, making the cost manageable). The paper doesn't ablate this component in isolation (it's part of the progressive training structure in Figure 9), so the direct performance contribution is unclear — but as a conceptual contribution, it provides a principled alternative to the manual tuning that dominates industrial practice.

Innovation 4: Stochastic Specialization Training as a Gradient Variance Reduction Strategy for Multi-Domain Contrastive Learning

Stochastic specialization training (Section 3.3.3) addresses a subtle but practically important problem that arises specifically in contrastive learning with large-scale multi-source data: within-batch domain mixing reduces the effective number of negatives per domain, increasing gradient variance and slowing convergence. The innovation is recognizing that this is not merely an engineering inconvenience but a fundamental optimization challenge, and that the solution — sampling one dataset per batch rather than mixing — is both simple and theoretically justified.

The standard approach to multi-dataset training (Figure 6b) — interleaving samples from all sources in each batch — is so ubiquitous that it's rarely questioned. It emerges naturally from data loader implementations: you concatenate all datasets, shuffle, and draw batches. The paper's critique is that this treats contrastive learning as if it were supervised classification, where cross-domain mixing is harmless because the loss for each sample is independent. But contrastive learning with in-batch negatives creates dependencies between samples — the loss for one Copair pair depends on the other Copair targets in the batch serving as negatives. If only 20% of the batch is Copair data, each Copair query sees only 0.2B Copair negatives (where B is batch size), reducing the contrastive signal strength proportionally.

The paper's stochastic specialization solution — sampling one dataset per batch according to learned weights — is conceptually related to stratified sampling and to meta-learning's task-sampling paradigm, but its application to contrastive multi-domain training appears to be novel. The intuition borrowed from meta-learning is illuminating: in MAML-style algorithms, each inner loop trains on a single task to produce task-specialized gradients, and the outer loop aggregates these to find a shared initialization. Here, each iteration trains on a single dataset to produce domain-specialized contrastive gradients, and the shared model parameters must accommodate all domains over the course of training. The key difference is that there is no inner/outer loop separation — it's standard SGD with domain-specialized batches, which makes it simpler to implement and more scalable.

The significance of this contribution is primarily practical: it enables stable training on 10B+ samples across dozens of heterogeneous datasets without requiring practitioners to manually balance batch composition. The paper doesn't provide an ablation isolating the effect of stochastic specialization versus mixed-batch training, so the quantitative contribution is unclear. However, as a design principle, it has implications for any large-scale contrastive learning system that aggregates data from multiple sources — a setting that is increasingly common as models are trained on web-scale data with diverse provenance. The observation that contrastive learning creates sample dependencies that make domain mixing suboptimal is non-obvious and worth surfacing as a conceptual contribution, even if the experimental evidence for its impact is indirect (it is part of the progressive training stack in Figure 9).

Innovation 5: The Omni-Modal Embedding as a Unifying Abstraction for Heterogeneous Downstream Tasks

The paper's most ambitious conceptual contribution — one that is woven throughout the architecture, training, and evaluation design rather than concentrated in a single section — is the proposition that a single omni-modal embedding can serve as a universal representation for tasks that are traditionally handled by separate, purpose-built models. This is more than a technical claim about a specific architecture; it's an argument about the nature of the representation learning problem in multimodal platforms.

Prior to SAIL-Embedding, the dominant deployment pattern for multimodal embedding models in industrial systems was fragmented: a CLIP-based model for visual similarity search, a separate text embedding model for query-document matching, a collaborative filtering model for recommendation, and task-specific classifiers for tag prediction. Each model was trained on its own data with its own objective, and the representations they produced lived in different spaces with different semantics. The paper's architecture (Figure 2c) and training pipeline argue — and the experimental results demonstrate — that these can be unified. The same 1536-dimensional vector that retrieves visually similar videos (Film-i2i Recall@50: 89.08%) also matches text queries to videos (Short Video-q2i Recall@50: 86.53%), classifies content into hierarchical tags (CLS task, Table 1), and — after CRE training — captures collaborative consumption patterns (Gid-i2i Recall@50: 59.69%).

What makes this a conceptual innovation rather than an obvious consequence of using a larger model is the deliberate co-design of architecture and training to support this unification. The dataset-driven pattern matching (Section 3.3.4) is not just an engineering convenience — it's the mechanism that allows the model to learn ITC, IIC, VTC, VVC, TTC, and OOC objectives simultaneously in a single training run, without committing to any single modality pairing as "primary." The instruction-based prompt design (Figure 5) is not just prompt engineering — it's a mechanism for producing task-conditional embeddings, where the same item can have different representations depending on whether it's serving as a query or a target, and on what task is being performed. The Matryoshka representation learning (Section 3.3.2) is not just a deployment trick — it forces the model to structure its embedding space so that discriminative information is concentrated in the lowest dimensions, enabling graceful degradation when computational constraints demand lower-dimensional representations.

The intellectual thrust of this innovation is that it challenges the assumption that different downstream tasks require fundamentally different representations. The paper doesn't claim that a single embedding is optimal for every task — the CRE tradeoff (Copair degrades slightly when Gid improves) demonstrates that it's not. But it claims — and the evidence supports — that a single embedding can be good enough across tasks that the operational simplification of maintaining one model instead of five outweighs the small per-task losses. This is a systems-level insight that is undervalued in academic research but central to industrial deployment: sometimes the most impactful innovation is not improving each component by 1%, but eliminating four of the five components entirely.

The evaluation framework (Section 4.2) itself embodies this conceptual contribution. By measuring not just retrieval recall but also positive-negative separability, group-wise clustering consistency (NMI), ranking consistency (Kendall's τ), and bijective alignment, the paper defines a multi-dimensional notion of embedding quality that goes beyond any single task metric. This framework is implicitly arguing that a good universal embedding should exhibit structural properties — symmetry, stability, discriminability — that predict transfer to multiple downstream tasks, rather than being evaluated only on the specific tasks it was trained for. This is a more rigorous standard than the typical "our model beats baselines on benchmark X" and represents a methodological contribution to how embedding models should be evaluated.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The evaluation spans 30 benchmarks in total: 21 item-to-item (i2i) and 9 query-to-item (q2i) tasks, all constructed from real-world Douyin scenarios. The i2i benchmarks are grouped into three categories: Content Understanding (e.g., Film-i2i, Brand Vehicle-i2i, Game Tag-i2i, Summary-i2i, Sub-i2i, ID Fusetag-i2i), Search-Based (e.g., Video Search-i2i, Search-i2i), and Collaborative Perception (e.g., RSDF-i2i, Copair-i2i, Live-i2i, Gid-i2i). The q2i benchmarks include retrieval tasks (Short Video-q2i, Live-q2i, Live Summary2i) and classification tasks (Decision-q2i, Longtail-q2i, Search Longtail-q2i, Unbiased-q2i, Unbiased Longtail-q2i, Biased-q2i). Dataset construction details are given in Section 4.3 of the paper; test splits are derived from held-out portions of the training datasets described in Table 1, augmented with out-of-domain benchmarks for generalization assessment.

• Base model(s). The primary model is SAIL-Embedding, built on a pre-trained LLM backbone (Qwen2/3 family, warmed up via the SAIL-VL procedure). For i2i comparisons, two baselines are fine-tuned on the same training data from Table 1: a CLIP-based dual-tower model [47] and a standard VLM-based model [74] (GME-like architecture). For q2i comparisons, three additional unimodal text embedding baselines are included: Doubao-Embedding (ByteDance's internal text embedding model), Qwen3-Embedding-4B, and Qwen3-Embedding-8B [75]. The choice of baselines spans three model paradigms: unimodal text, dual-tower multimodal, and MLLM-based multimodal.

• Metrics. The paper employs a multi-dimensional evaluation framework (Section 4.2) with six metric categories:

  1. Retrieval Recall (Recall@k): The fraction of relevant items retrieved in the top-k results, reported at k ∈ {1, 10, 25, 50, 100}. Applied to all i2i and q2i retrieval benchmarks.
  2. AUC (Area Under the ROC Curve): Used for q2i classification tasks where cosine similarity scores between query and item embeddings serve as binary classification probabilities predicting whether a query-item pair is a positive match. Applied to six classification-oriented q2i benchmarks (Decision-q2i through Biased-q2i in Table 3).
  3. Positive-Negative Separability: The distributional gap between cosine similarity scores of positive pairs and negative pairs, visualized in Figure 8(b,c). Quantifies discriminative capability.
  4. Group-wise Clustering Consistency (NMI): Normalized Mutual Information between cluster assignments of positive and negative samples in the embedding space (Section 4.2, Equation for NMI). Higher NMI indicates that positive and negative samples are distinguished by consistent feature dimensions.
  5. Ranking Consistency (Kendall's τ, Intersection): Measures whether query-target similarity rankings remain stable across representation spaces. Kendall's τ captures pairwise ordering agreement; Intersection measures top-k overlap proportion.
  6. Bijective Alignment Test: For each query, retrieves the most similar target, then from that target retrieves back the most similar query. The proportion of cases where indices match quantifies embedding fidelity and one-to-one mapping consistency.

• Baselines. Five distinct model types serve as comparison points:

  • Doubao-Embedding: ByteDance's internal unimodal text embedding model (q2i tasks only).
  • Qwen3-Embedding-4B and Qwen3-Embedding-8B [75]: State-of-the-art text embedding models at two scales (q2i tasks only).
  • CLIP-based Model [47]: A dual-tower architecture with separate vision and text encoders, fine-tuned on the Table 1 training data. Represents the shallow-fusion paradigm.
  • VLM-based Model [74]: An MLLM-based embedding model (GME-like), fine-tuned on the Table 1 training data. Represents the deep-fusion image-text paradigm, lacking audio modality support and CRE training.
  • SAIL-Embedding variants: The full model is compared against ablations (SAIL-Embedding without CRE, SAIL-Embedding with CRE, and the VLM-based model with CRE applied) to isolate the contributions of omni-modal architecture, audio modality, and collaboration-aware training.

• Generation budget / compute accounting. The paper does not measure inference cost in FLOPs or wall-clock time. Instead, the primary unit of comparison is model architecture and training methodology under fixed data conditions — all models compared in Tables 2 and 3 are trained on the same underlying datasets (Table 1), and the evaluation is on held-out test benchmarks. For the Matryoshka embedding component, the model produces embeddings at three dimensionalities (1536, 768, 128) with a single forward pass, enabling deployment-time accuracy-efficiency tradeoffs without retraining. For online experiments (Table 6), the relevant "budget" is the computational cost of feature extraction and integration into the existing recommender pipeline stages.

• Cross-validation / statistical protocol. The paper does not report k-fold cross-validation or confidence intervals on benchmark results. The evaluation is performed on fixed test splits derived from Table 1 datasets and out-of-domain benchmarks. For the progressive training ablation (Figures 9 and 10), average performance across subsets of i2i and q2i tasks is reported, but the specific subset composition and the number of runs (single vs. multiple seeds) are not specified. The online A/B test results (Table 6) use standard industrial experimentation protocols (A/B testing with statistical significance monitoring), but the paper does not report p-values, confidence intervals, or experiment duration. The lack of variance estimates is a genuine limitation for assessing whether differences between models — particularly small ones like the 0.01% LT gains at individual pipeline stages — are statistically reliable.

Main Quantitative Results

Item-to-Item Retrieval Performance (Table 2)

The headline result across 21 i2i benchmarks: SAIL-Embedding achieves the highest Recall@50 on 15 out of 21 tasks and the highest Recall@100 on 13 out of 21 tasks, with particularly strong advantages in search and collaborative perception scenarios. Performance varies substantially by application category:

Content Understanding i2i: On tasks measuring semantic content similarity, SAIL-Embedding shows strong but not uniformly dominant performance. On Film-i2i (items grouped by movie intellectual property), SAIL-Embedding achieves Recall@50 of 89.08% and Recall@100 of 91.86%, substantially outperforming both the CLIP-based model (75.98% / 80.40%) and the VLM-based model (81.32% / 84.79%). On Brand Vehicle-i2i, SAIL-Embedding reaches 52.03% / 57.34% versus the VLM-based model's 45.97% / 52.22% and the CLIP-based model's 26.54% / 33.15% — a nearly 2× improvement over CLIP. On Store Visit (Restaurant)-i2i, all three models perform similarly (SAIL-Embedding: 98.13% / 98.69%, VLM: 98.75% / 99.07%, CLIP: 95.31% / 96.69%), suggesting near-saturation on this task. On University Sub-i2i, CLIP slightly edges out SAIL-Embedding at Recall@100 (24.46% vs. 24.68% for baseline VLM vs. 21.27% for CLIP), but SAIL-Embedding leads at Recall@50 (20.76% vs. 19.99% for VLM). The pattern suggests that for visually-driven similarity, all deep-fusion models (VLM and SAIL-Embedding) substantially outperform shallow-fusion CLIP, but the marginal benefit of adding audio and omni-modal training over a strong VLM baseline is task-dependent.

Search-Based i2i: These tasks evaluate items paired by search query co-clicks. SAIL-Embedding achieves Recall@50 of 80.50% and Recall@100 of 84.01% on Video Search-i2i, compared to 77.45% / 80.90% for the VLM-based model and 56.72% / 62.23% for CLIP. On the broader Search-i2i benchmark (including non-video items), SAIL-Embedding reaches 68.43% / 73.00% versus the VLM baseline's 65.79% / 70.58% and CLIP's 39.41% / 44.63%. The gap between SAIL-Embedding and the VLM baseline is modest but consistent (2-3 percentage points at Recall@100), while the gap between deep-fusion models and CLIP is enormous (~30 percentage points). This reinforces that LLM-based fusion is the dominant differentiator for search tasks, with omni-modal support providing incremental gains.

Collaborative Perception i2i: These tasks measure behavioral co-occurrence rather than content similarity, and they reveal the paper's most distinctive performance patterns. On Copair-i2i — the largest training dataset (2.9B pairs), constructed from user consumption co-occurrence — SAIL-Embedding without CRE achieves Recall@50 of 69.17% and Recall@100 of 75.57%. The VLM baseline achieves 66.06% / 72.70%. The CLIP baseline reaches 53.47% / 70.60%. On RSDF-i2i (items paired by recommender system dense feature similarity), SAIL-Embedding achieves 52.46% / 59.06% versus the VLM baseline's 42.52% / 48.44% — a gap of approximately 10 percentage points, the largest among all i2i tasks. On Live-i2i (live streaming behavioral co-occurrence), SAIL-Embedding reaches 54.10% / 63.80% versus VLM's 52.55% / 62.60%.

These collaborative perception results are crucial because they isolate the value of the omni-modal architecture on the pre-CRE model. The ~10-point gap on RSDF-i2i between SAIL-Embedding and the VLM baseline cannot be attributed to CRE training (which is separately evaluated in Table 4) — it reflects the model's ability to capture patterns that the image-text-only VLM misses, presumably through audio and the richer text processing (OCR, ASR, tags in addition to titles).

Notable anomalies in Table 2: On Music Play-i2i, the VLM baseline achieves higher Recall@50 and Recall@100 (85.30% / 89.25%) than SAIL-Embedding (84.55% / 88.72%), and on Music Gameplay-i2i, VLM also leads (82.52% / 86.98% vs. 82.08% / 86.72%). The paper does not comment on these reversals. A plausible interpretation: music-related tasks may rely more heavily on visual cues (album art, performance visuals) and title text (song names, artist names), and the VLM baseline — fine-tuned on the same data — may allocate its representational capacity differently, focusing more on visual-textual alignment than on audio, which may actually distract on these specific tasks. This is speculative but highlights that omni-modal fusion is not uniformly beneficial.

Query-to-Item Retrieval Performance (Table 3)

SAIL-Embedding achieves state-of-the-art performance on all 9 q2i tasks, often by substantial margins over both unimodal text embeddings and the VLM-based multimodal baseline. The headline numbers:

Short Video-q2i (retrieval): SAIL-Embedding reaches Recall@50 of 86.53% and Recall@100 of 88.54%. This represents an 8.0 percentage-point improvement over the VLM baseline (78.53% / 80.66%) and a 14.1 percentage-point improvement over the best unimodal text baseline (Qwen3-Embedding-8B at 72.42% / 75.77%). The CLIP-based model achieves 74.16% / 76.21%, placing it between pure text models and deep-fusion multimodal models.

Live-q2i (retrieval): SAIL-Embedding achieves 79.08% / 84.38%, compared to VLM's 73.91% / 79.34% and Qwen3-Embedding-8B's 64.22% / 69.57%. The 5.2 percentage-point gap between SAIL-Embedding and VLM at Recall@50 is the clearest evidence for the audio modality's contribution in live streaming contexts — live content has rich acoustic signals (host speech, background music, audience reactions) that text-only queries benefit from aligning with.

Live Summary2i (retrieval): SAIL-Embedding reaches 84.33% / 88.16%, versus VLM's 81.55% / 85.62% and Doubao-Embedding's 66.31% / 69.31%. The margin over VLM is smaller here (2.8 percentage points at Recall@50), suggesting that when the query is a text summary of the live content (rather than raw user search text), the additional modalities in the target item provide less marginal benefit.

Classification-oriented q2i tasks (AUC metrics): These six tasks measure the model's ability to discriminate matched query-item pairs from mismatched pairs using cosine similarity as a binary classifier score. SAIL-Embedding achieves:

• Decision-q2i: AUC 82.44% (vs. Doubao-Embedding at 74.51%, VLM at 67.34%)
• Longtail-q2i: AUC 91.22% (vs. Doubao-Embedding at 85.84%, VLM at 84.02%)
• Search Longtail-q2i: AUC 83.31% (vs. Doubao-Embedding at 76.86%, VLM at 74.99%)
• Unbiased-q2i: AUC 93.86% (vs. CLIP at 88.62%, Doubao-Embedding at 88.67%)
• Unbiased Longtail-q2i: AUC 89.10% (vs. VLM at 85.86%, CLIP at 85.06%)
• Biased-q2i: AUC 91.65% (vs. VLM at 89.02%, CLIP at 88.65%)

Two patterns emerge from these AUC results. First, the unimodal text embedding models (Doubao-Embedding, Qwen3-Embedding) substantially outperform the multimodal baselines (CLIP, VLM) on several classification tasks — for instance, on Decision-q2i, Doubao-Embedding achieves 74.51% versus VLM's 67.34%, and on Longtail-q2i, 85.84% versus 84.02%. This suggests that the VLM baseline, despite having access to multimodal target information, produces query-target similarity scores that are less discriminative for these classification tasks than pure text embeddings — possibly because the multimodal fusion introduces noise or because the VLM's embedding space is optimized for retrieval-style contrastive objectives rather than fine-grained binary discrimination. SAIL-Embedding's omni-modal architecture and training strategies (particularly the COSENT ranking loss and mICL modality balancing) appear to resolve this, achieving AUCs substantially above both unimodal and multimodal baselines.

Second, the absolute AUC values vary dramatically across tasks — from 82.44% on Decision-q2i to 93.86% on Unbiased-q2i — indicating that query-item discrimination difficulty is highly task-dependent. SAIL-Embedding maintains the largest margins over baselines on Longtail-q2i (+5.38% over Doubao-Embedding) and Search Longtail-q2i (+6.45% over Doubao-Embedding), suggesting that omni-modal fusion is particularly valuable for long-tail queries where text alone carries weak signal and visual/audio context in the target items provides complementary disambiguation.

Collaboration-Aware Recommendation Enhancement Results (Table 4, Figure 8, Table 5)

The CRE training phase is evaluated across multiple dimensions to isolate its effects on both content and collaborative perception capabilities.

Discriminability and separability (Figure 8): The training loss curve with CRE (Figure 8a) shows stable convergence after approximately 20,000 steps, confirming that the distillation objectives do not destabilize optimization. The similarity score distributions for positive and negative pairs (Figures 8b and 8c) reveal two effects of CRE training: (1) the overlap between positive and negative distributions is reduced, particularly in the similarity range around 0.5 where hard negatives concentrate, and (2) the positive distribution becomes more compact (narrower spread), indicating that the model learns to represent related items with higher confidence and tighter clustering. The paper explicitly interprets this as evidence that "by jointly incorporating some ID embeddings, the model learns correlations among positive samples with higher confidence" — the embeddings of items that share collaborative signals become more similar to each other.

Quantitative metrics on embedding quality (Table 4): Four structural metrics are reported for the VLM-based model and SAIL-Embedding, both with and without CRE:

• NMI (Group-wise Clustering Consistency): SAIL-Embedding with CRE achieves 0.65, versus 0.60 without CRE and 0.53 for VLM without CRE. The +5% absolute gain from CRE on SAIL-Embedding (and +8% on VLM with CRE: 0.53 → 0.61) indicates that CRE-trained embeddings produce more consistent cluster structures where positive and negative samples are distinguished by similar feature dimensions. This is particularly relevant for recommendation recall stages, where embedding-based nearest-neighbor retrieval relies on consistent cluster boundaries.

• Kendall's τ (Ranking Consistency): SAIL-Embedding with CRE achieves 77.2 (×10K), versus 73.4 without CRE and 68.6 for VLM without CRE. The +3.8% relative improvement suggests that CRE training produces embedding spaces where the relative ordering of item similarities is more stable — an item that is the 5th most similar to a query will reliably be the 5th most similar across different representation subspaces.

• Intersection (Top-k Overlap): SAIL-Embedding with CRE achieves 31.63%, versus 29.73% without CRE. The +1.9% improvement indicates more consistent top-k retrieval sets.

• Bijective Alignment (Accuracy): SAIL-Embedding with CRE achieves 48.94%, versus 48.28% without CRE. The +0.66% improvement is modest but directionally positive, suggesting that CRE training slightly strengthens the one-to-one mapping property of the embedding space.

Task-specific retrieval after CRE (Table 4, bottom rows): The most revealing results are the divergent effects on Gid-i2i versus Copair-i2i:

• Gid-i2i Recall@50: SAIL-Embedding with CRE reaches 59.69%, up from 52.46% without CRE — a 7.23 percentage-point absolute improvement (13.8% relative). The VLM baseline with CRE shows a similar pattern: 46.34% versus 42.52% without CRE. Gid-i2i is constructed from ID embedding relevance — the same ID embeddings used for distillation — so this improvement directly measures how well the multimodal embedder has absorbed the collaborative signal from the teacher.

• Gid-i2i Recall@100: SAIL-Embedding with CRE reaches 66.37%, up from 59.06% (+7.31 percentage points). VLM with CRE: 52.18% versus 48.44% (+3.74 percentage points). SAIL-Embedding benefits substantially more from CRE than the VLM baseline on this metric, suggesting that the omni-modal architecture provides a richer representational capacity that can better accommodate the additional collaborative signal without destructive interference.

• Copair-i2i Recall@50: SAIL-Embedding with CRE drops to 66.83% from 69.17% without CRE (−2.34 percentage points). VLM with CRE drops to 64.78% from 66.06% (−1.28 percentage points). Both models lose content-based copair recall after CRE training.

• Copair-i2i Recall@100: SAIL-Embedding with CRE drops to 73.10% from 75.57% (−2.47 percentage points). VLM with CRE drops to 70.52% from 72.70% (−2.18 percentage points).

This tradeoff — improved Gid-i2i at the cost of degraded Copair-i2i — is the paper's most important negative result and its clearest demonstration of the content-collaboration tension. The Copair-i2i benchmark was constructed from user consumption co-occurrence but paired via content representation similarity (items with similar multimodal embeddings are paired), while Gid-i2i was constructed from ID embedding similarity (items with similar recommender system ID embeddings are paired). When CRE training aligns the multimodal embedder more closely with ID embeddings, items that have similar multimodal content but different ID embeddings are pushed apart, reducing Copair recall. The paper explicitly accepts this tradeoff: "We consider this variation tolerable to balance the differing application requirements of content-oriented and collaborative behavior-oriented scenarios in industrial settings."

Sequence modeling evaluation (Table 5): After seq2item and ID2item distillation, the model's ability to predict the next video a user will watch from their viewing history is assessed on two benchmarks:

• Filtered Seq2item (sequences filtered by content tag or clustered copair-id consistency with the target): After seq2item distillation, Recall@10 improves from 19.35% to 22.02% (+2.67 percentage points), and Recall@25 improves from 25.46% to 30.12% (+4.66 percentage points). After additional ID2item distillation, Recall@10 reaches 23.77% (+4.42 over baseline) and Recall@25 reaches 32.41% (+6.95 over baseline). The progressive improvement demonstrates that seq2item and ID2item distillation provide complementary sequence understanding capabilities.

• Vanilla Seq2item (sequences with randomly sampled historical items, no content filtering): This is a harder task where the sequence items are less obviously related to the target. After the full distillation pipeline, Recall@10 improves from 10.46% to 13.36% (+2.9 percentage points), and Recall@25 improves from 14.33% to 19.09% (+4.76 percentage points). The absolute numbers are low (predicting the exact next video from 100 candidates is inherently difficult), but the relative improvement is substantial — approximately 28% relative gain at Recall@10 and 33% at Recall@25.

A notable detail: on Filtered Seq2item, seq2item distillation alone slightly decreases Recall@1 (from 8.41% to 8.28%) before ID2item distillation recovers it to 9.04%. The paper doesn't comment on this, but it may indicate that pure sequence-to-item distillation initially disrupts the top-1 precision while improving broader recall, and the subsequent ID-to-item distillation re-stabilizes the top-1 ranking.

Extensive Online Experiments (Table 6)

The online A/B test results across multiple Douyin scenarios and recommender system stages provide real-world validation of the offline benchmark improvements. The paper reports two types of features derived from SAIL-Embedding: dense embeddings (1536-dimensional vectors) and Semantic IDs (SIDs) (discrete tokens obtained by clustering or vector quantization of the embeddings).

The headline online results, organized by scenario and pipeline stage:

Feed (main Douyin recommendation feed):

• Recall stage with SIDs: LT30 +0.01%
• Pre-Rank stage with SIDs: LT30 +0.01%
• Rank stage with SIDs + Embeddings: Finish AUC +0.1%
• Re-Rank stage with SIDs: LT30 +0.01%

Message Pushing:

• Recall stage with SIDs + Embeddings: LT30 +0.03%
• Rank stage with SIDs: LT30 +0.01%

Coldstart (new users/items):

• Recall stage with SIDs + Embeddings: LT30 +0.05%

Douyin-Selected (separate application):

• Recall stage with SIDs: LT7 +0.4%
• Rank stage with Embeddings: LT7 +0.1%

Several patterns are noteworthy. First, SIDs consistently bring more gain than dense embeddings across most stages — the paper hypothesizes that SIDs are easier to use in rule-based methods (such as rule-based decentralization) and can be encoded into trainable embeddings that adapt to the recommendation model, whereas dense embeddings require more sophisticated integration. This is a practical insight: discrete representations, despite losing information relative to continuous embeddings, may be more actionable in existing recommender infrastructure.

Second, the cumulative effect across pipeline stages is substantial. In the Feed scenario, SIDs contribute +0.01% LT30 at recall, pre-rank, and re-rank stages, with the ranking model additionally benefiting from embeddings (+0.1% Finish AUC). While each individual gain is small (0.01% LT30), the paper notes that these gains accrue across sequential stages — a user retained through better recall also benefits from better ranking — though the paper does not report the total cumulative LT gain across all stages.

Third, the coldstart scenario shows the largest LT gain (+0.05% LT30), which aligns with the intuition that multimodal content embeddings are most valuable when behavioral history is sparse. For new items with few user interactions, the embedding provides the primary signal for which existing items are similar.

Fourth, the Douyin-Selected application — a separate recommendation pipeline — sees substantially larger gains: +0.4% LT7 at the recall stage and +0.1% LT7 at the rank stage. The paper does not explain why Douyin-Selected benefits more than the main Feed, but possible reasons include: different baseline performance (more room for improvement), different content distribution (selected/higher-quality content where multimodal signals are more informative), or different integration depth.

The magnitude of the AUC gains (+0.1% Finish AUC in Feed, similar in other scenarios) is consistent with the offline benchmark improvements: SAIL-Embedding shows substantial relative gains over baselines on retrieval tasks (Tables 2 and 3), which translates to modest but economically meaningful improvements in production metrics where the baseline recommender is already highly optimized.

Ablation Pathway (Figures 9 and 10)

The paper conducts a systematic ablation study by incrementally adding components to a baseline and measuring average performance across subsets of i2i and q2i tasks. The progression and cumulative effects:

Starting point: BERT as encoder (text modality only). Average i2i performance: 42.47%. Average q2i performance: 74.16%.

+ LLM as Fusion (replace BERT with LLM): i2i reaches 42.97% (+0.50%). q2i reaches 79.17% (+5.01%). The disproportionate q2i gain suggests that the LLM's pre-trained language understanding is particularly valuable for query-to-item tasks where the query is text and the model must reason about complex search intents.

+ Bidirectional Attention (replace causal with bidirectional): i2i reaches 43.02% (+0.05%). q2i reaches 79.69% (+0.52%). The small but consistent improvement supports the paper's claim that bidirectional attention is better suited for embedding extraction than causal attention, though the effect is modest relative to other components.

+ Task-Specific Instructions (replace unified instruction with per-task prompts): i2i reaches 45.93% (+2.91%). q2i reaches 80.11% (+0.42%). This is the largest single improvement on i2i tasks in the ablation sequence, confirming that instruction-based multi-task conditioning is critical for item-to-item retrieval. The paper's interpretation (Section 4.4.5): "the instruction-based multi-task training paradigm guides the model to focus on distinct feature details."

+ Full Fine-Tuning (replace LoRA with full-parameter optimization): i2i reaches 47.44% (+1.51%). q2i reaches 80.79% (+0.68%). The i2i gain is notably larger than the q2i gain, suggesting that item-to-item retrieval requires more adaptation of the LLM's internal representations than query-to-item retrieval, which may leverage the LLM's pre-trained text understanding more directly.

+ Add Data Diversity (expand training data coverage): i2i reaches 50.08% (+2.64%). q2i reaches 80.80% (+0.01%). The i2i gain is substantial; the q2i gain is negligible. This asymmetry indicates that i2i tasks benefit more from diverse training data (pairing items by different behavioral and content signals), while q2i tasks — already well-served by the Search-q2i and Score-q2i datasets — see diminishing returns from additional data types.

+ COSENT Loss (add fine-grained ranking objective): q2i average increases to 82.49% from 80.80% (+1.69%). The paper reports this as a q2i-only improvement (the i2i bar remains at approximately 50.08% for the subsequent steps, though the exact values after this point are slightly ambiguous in Figure 9). The COSENT loss is specifically designed for ranking scenarios with graded relevance, so its benefit concentrating on q2i tasks is expected.

+ Data Scaling Up (increase total training samples): q2i reaches 83.91% (+1.42%). i2i reaches 51.75% (+1.67%). Both tasks benefit approximately equally from scale, consistent with scaling law expectations, though the absolute increments are modest given the already-large data volume.

+ Progressive Training (replace all-in-one training with three-stage curriculum): i2i reaches 52.29% (+0.54%). q2i reaches 83.92% (+0.01%). The progressive training benefit is concentrated on i2i tasks and is relatively small in absolute terms. This may understate the importance of progressive training if its primary role is enabling stable convergence at scale rather than improving final performance on medium-scale evaluations — the paper notes that the baseline uses "all-in-one training" which may already converge reasonably on the ablation subsets but would fail at full scale.

What the ablation does NOT measure: The individual contributions of dynamic hard negative mining, adaptive multi-source data balancing, stochastic specialization training, and dataset-driven pattern matching are not isolated. These components are part of the "progressive training" stack and are included in the final 52.29% / 83.92% numbers, but their marginal effects cannot be disentangled. The ablation also does not measure the contribution of the audio modality (since the starting BERT baseline is text-only, the progression to omni-modal is not shown as a discrete step). The mICL and late fusion losses are not independently ablated.

Critical Assessment

Do the Experiments Support the Central Claims?

Claim: SAIL-Embedding achieves state-of-the-art performance on item-to-item and query-to-item retrieval.

The evidence for this claim is strong but conditional. On q2i tasks (Table 3), SAIL-Embedding dominates all baselines across all 9 tasks with substantial margins — the claim is cleanly supported. On i2i tasks (Table 2), the picture is more nuanced. SAIL-Embedding achieves the best performance on the majority of tasks, but there are reversals: the VLM baseline leads on Music Play-i2i and Music Gameplay-i2i, and CLIP leads on University Sub-i2i at Recall@100. The claim of "state-of-the-art" across all i2i tasks is therefore not strictly true — SAIL-Embedding is state-of-the-art on most i2i tasks, with music-related tasks as notable exceptions. The paper does not discuss these exceptions, which weakens the completeness of the analysis. A more accurate characterization: SAIL-Embedding is the strongest model on search-oriented and collaboration-oriented i2i tasks, competitive with the VLM baseline on content-understanding i2i tasks, and occasionally outperformed on specific content categories (music).

Claim: Collaboration-aware recommendation enhancement (CRE) training improves recommendation performance by distilling behavioral signals.

The offline evidence supports this claim with a crucial qualification: CRE improves collaborative perception at the cost of content understanding. Table 4 shows Gid-i2i gains of +7.23 and +7.31 percentage points at Recall@50 and Recall@100 for SAIL-Embedding, alongside Copair-i2i losses of −2.34 and −2.47 percentage points. This is not a universal improvement — it's a tradeoff. The paper is transparent about this tradeoff and frames it as acceptable in industrial settings, but it means the claim is conditional: CRE improves recommendation-oriented metrics when the deployment prioritizes collaborative signals over content signals. A system that needs both equally may not benefit from CRE as implemented.

The online experiments (Table 6) provide real-world validation but with thin documentation. The paper reports LT and AUC gains across pipeline stages but does not report: (a) whether the gains are statistically significant (p-values), (b) the experiment duration and sample sizes, (c) whether the gains are sustained beyond the reported LT7/LT30 windows, or (d) whether there were any negative impacts on other metrics (e.g., content diversity, creator exposure, user satisfaction surveys). The 0.01% LT30 gains at individual pipeline stages are small enough that without significance testing and confidence intervals, it's impossible to distinguish real effects from measurement noise. The 0.4% LT7 gain in Douyin-Selected is more convincing but lacks context (baseline LT7, experiment design details).

Claim: The omni-modal architecture with audio modality improves over VLM-based image-text models.

This claim is supported for most tasks but the contribution of audio specifically is not isolated. The VLM baseline and SAIL-Embedding differ in three ways: (1) audio modality support, (2) richer text processing (OCR, ASR, tags in addition to title), and (3) the full training strategy stack (dynamic hard negative mining, adaptive balancing, progressive training, dataset-driven pattern matching, stochastic specialization). The performance gap between SAIL-Embedding and the VLM baseline (e.g., +8.0 percentage points on Short Video-q2i Recall@50 in Table 3) is the combined effect of all three differences. The paper attributes the improvement to audio in Section 4.4.1 ("Compared to the VLM-based model, our model verifies the importance and effectiveness of the audio modality"), but no ablation isolates audio's contribution from the other factors. Running SAIL-Embedding with the audio encoder removed (text+vision only, but with the same training strategies) would cleanly isolate the audio contribution — this experiment is missing.

Claim: Dynamic hard negative mining and adaptive multi-source data balancing improve training robustness.

Neither of these components is independently ablated. Figure 9's ablation pathway does not include steps for "add dynamic hard negative mining" or "add adaptive data balancing." The progressive training step at the end of the ablation incorporates these components, but their marginal effects cannot be separated from the general benefit of staged training. The paper provides conceptual justification and algorithmic descriptions for these methods, but no experimental evidence that they improve over simpler alternatives (fixed threshold hard negative mining, uniform or heuristic data mixing). This is a significant gap — these are presented as key contributions in the methodology (Sections 3.1.2 and 3.1.3), but their empirical value is unproven.

Claim: The model achieves a 4× improvement in discriminability on ID-based retrieval benchmarks after recommendation enhancement.

This claim (paraphrased from the executive summary) overstates the evidence. Table 4 shows a 13.8% relative improvement in Gid-i2i Recall@50 (52.46% → 59.69%) and a 12.4% relative improvement at Recall@100 (59.06% → 66.37%) for SAIL-Embedding with CRE. These are substantial but not "4×" improvements. The 4× figure may refer to a different metric or comparison that is not clearly identified in the paper. This disconnect between the summary claim and the reported numbers is a genuine weakness in the paper's presentation.

Genuine Weaknesses in Experimental Design

Single model family, single platform. All experiments use SAIL-Embedding with a Qwen2/3-based LLM backbone, fine-tuned following the SAIL-VL procedure. The paper claims the model is "representative of the capabilities of many contemporary LLMs" (Section 4), but no alternative LLM backbone is tested. Would the training strategies transfer to a Llama-based or Gemma-based architecture? The dependency on a specific pre-training recipe (SAIL-VL warming) further limits generalizability — practitioners using different LLM families and different multimodal pre-training procedures may not observe the same benefits.

No ablation of audio modality in isolation. As discussed above, the contribution of audio to the performance gap between SAIL-Embedding and the VLM baseline is confounded with text processing richness and training strategy improvements. Running an ablation where the audio encoder and audio tokens are removed from SAIL-Embedding (keeping all other components identical) would directly measure audio's marginal value.

Missing statistical significance reporting. No confidence intervals, standard deviations, or p-values are reported for any offline benchmark results. For the online A/B tests, the paper mentions "LT gain" and "AUC gain" but does not report whether these gains are statistically significant, what the experiment duration was, or what the sample sizes were. The 0.01% LT30 gains at individual pipeline stages are particularly concerning — without significance testing, it is impossible to know whether these represent real effects or noise.

Ablation incompleteness. Several components described as key contributions are not independently ablated: dynamic hard negative mining, adaptive multi-source data balancing, stochastic specialization training, dataset-driven pattern matching, the audio modality, the mICL loss, and the late fusion loss. The ablation pathway (Figures 9 and 10) tests the high-level design choices (LLM fusion, bidirectional attention, instruction design, full fine-tuning, data diversity, COSENT loss, data scale, progressive training) but leaves the paper's more novel methodological contributions untested.

Single evaluation paradigm. All offline benchmarks measure retrieval recall (for i2i/q2i) or similarity-based classification AUC (for q2i classification). While these are appropriate metrics for embedding models, the paper does not evaluate on downstream recommendation tasks in a controlled offline setting — for instance, using the embeddings in a trained recommendation model and measuring held-out user engagement prediction. The online A/B tests partially fill this gap, but the offline evaluation is limited to retrieval and classification, not recommendation per se.

Hard negative mining evaluation gap. The dynamic hard negative mining method selects an F1-optimal threshold per dataset, but the paper does not compare this against simpler alternatives: fixed thresholds, top-k hardest negatives per query, or no hard negative mining at all. Without this ablation, it's unclear whether the adaptive thresholding provides benefit over a well-tuned fixed threshold or whether hard negative mining helps at all in this setting.

Data balancing evaluation gap. The adaptive multi-source data balancing method uses Sinkhorn-based similarity to derive dataset weights, but the paper does not compare against uniform sampling, heuristic weighting, or hard filtering (removing low-similarity datasets). The claim that adaptive balancing "reduces reliance on subjective manual tuning" is a convenience argument, not an accuracy argument, and would be strengthened by showing that adaptive weights outperform manually-tuned weights or that they adapt successfully to distribution shifts.

Experiments That Would Have Strengthened the Paper

Isolated CRE ablation: A direct comparison of SAIL-Embedding with and without CRE, matched on all other factors, across all 21 i2i and 9 q2i benchmarks (not just the four metrics in Table 4). This would map the full content-collaboration tradeoff surface and identify which tasks benefit, which suffer, and by how much. The current evaluation (Table 4) only reports Gid-i2i and Copair-i2i, leaving the effect of CRE on other tasks unknown.

Audio modality ablation: Removing the audio encoder and audio tokens from SAIL-Embedding while keeping the architecture and training strategy otherwise identical, then comparing performance to the full omni-modal model. This would isolate audio's contribution and validate the paper's claim that audio is "important and effective" for multimedia retrieval.

Scaling analysis: Evaluating SAIL-Embedding performance as a function of model size (LLM backbone scale), data volume, and training compute. The paper claims to train on 10B+ samples but does not show whether performance saturates or continues to improve with scale. A scaling curve would contextualize the absolute performance numbers and help practitioners decide whether to invest in more data, larger models, or better training strategies.

Cross-model generalization of CRE: Applying CRE training to a different MLLM-based embedding architecture (e.g., VLM2Vec, GME) and measuring the Gid-i2i vs. Copair-i2i tradeoff. This would test whether the CRE approach is architecture-specific or generalizable to other multimodal embedding paradigms.

Downstream recommendation simulation: Using SAIL-Embedding embeddings as features in a trained recommendation model and measuring held-out user engagement prediction (click-through rate, watch time) in an offline simulation. This would bridge the gap between retrieval/classification benchmarks and the online A/B test results, providing a controlled evaluation of recommendation-specific value.

Hard negative mining alternatives: Comparing F1-optimal dynamic thresholding against: (a) no hard negative mining, (b) a fixed global threshold tuned on a validation set, (c) top-k hardest negatives per query (fixed k), and (d) top-p hardest negatives (selecting the hardest p% of negatives per dataset). This would establish the value of the adaptive thresholding approach relative to simpler baselines.

Adaptive data balancing alternatives: Comparing Sinkhorn-based adaptive weighting against: (a) uniform sampling, (b) sampling proportional to dataset size, (c) manually-tuned fixed ratios, and (d) hard filtering (removing datasets below a similarity threshold). This would validate the distribution-matching approach against practical alternatives.

Long-term online impact: Reporting LT metrics beyond LT7 and LT30 windows (e.g., LT60, LT90) to assess whether the initial gains from better embeddings are sustained or decay over time as user behavior adapts. Also reporting diversity metrics, creator-side metrics, and content freshness to assess whether embedding-driven recommendation creates filter bubbles or homogenization.

Where the Claims Hold Conditionally

"SAIL-Embedding outperforms all baselines": Holds for q2i tasks unconditionally (Table 3). Holds for most i2i tasks but not music-related tasks (Table 2: Music Play-i2i, Music Gameplay-i2i). The conditions for the exceptions are unclear — the paper provides no analysis of why music tasks favor the VLM baseline.

"CRE training improves recommendation performance": Holds when the recommendation objective is aligned with collaborative filtering signals (Gid-i2i, Table 4). Does NOT hold — in fact, performance degrades — when the objective is aligned with content similarity (Copair-i2i, Table 4). The condition is the deployment's emphasis on collaborative vs. content signals.

"Audio modality improves multimodal retrieval": The evidence is suggestive but not conclusive because audio's contribution is confounded with other architectural and training differences. The claim holds only if the performance gap between SAIL-Embedding and the VLM baseline is primarily attributable to audio, which is not directly tested.

"Dynamic hard negative mining and adaptive data balancing improve training": No direct experimental evidence supports or refutes this claim. The methods are components of the overall training pipeline, and their marginal contributions are unknown. The claim is unproven.

"Online gains of +0.5% LT7 and +0.1% AUC": The Douyin-Selected +0.5% LT7 gain (Table 6: 0.4% + 0.1% = 0.5%) is reported but lacks statistical context. The +0.1% AUC gain in the Feed rank model is reported but also lacks significance testing. These claims hold only if the reported numbers are statistically significant and sustained, which the paper does not establish.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For and Dominates the Test-Time Inference Budget

The assumption or constraint. The entire compute-optimal test-time scaling framework — for both search against PRM verifiers and iterative revisions — depends on being able to estimate each prompt's difficulty before deciding how to allocate the inference budget. The paper's method for doing so requires generating 2,048 samples per question and either computing pass@1 (oracle difficulty) or averaging PRM final-answer scores (predicted difficulty). The paper explicitly acknowledges this cost in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The headline efficiency gains — the 4× compute reduction claimed for compute-optimal scaling over best-of-N (Figures 4 and 8) — are computed after difficulty has been estimated, without any amortization of the estimation cost across the test set or across multiple uses of the same prompt. The 2,048-generation difficulty estimation cost is an order of magnitude larger than the largest test-time budgets studied (256–512 generations in most experiments, up to 2,048 in Appendix F). In any realistic deployment where a prompt is seen only once or a small number of times, the total cost would be difficulty estimation + strategy execution, and the former would dominate the latter by a factor of 4–8×. This means the 4× efficiency figure is an upper bound — it represents how much more efficient the strategy execution is, conditional on already having a perfect difficulty estimate. The true end-to-end efficiency, including the cost of obtaining that estimate, could be substantially worse than the best-of-N baseline, not better.

The problem is most acute for the PRM-based predicted difficulty estimation, which the paper promotes as deployable without ground-truth labels. While it avoids the circularity of needing answers, it still requires 2,048 forward passes through both the base model and the PRM — a cost that must be paid for every new prompt. The paper frames this as an "exploration-exploitation tradeoff" (Section 3.2) and suggests future work on cheap difficulty prediction, but provides no empirical evidence that difficulty can be estimated cheaply enough to preserve the efficiency gains.

What evidence exists in the paper. None. The paper never reports the total cost (difficulty estimation + strategy execution) for any configuration. Figures 4 and 8 plot accuracy against "Number of Generations" where the x-axis counts only the strategy execution budget, with difficulty assumed already known. There is no curve showing what happens when the 2,048-generation estimation cost is amortized across a realistic number of queries. The paper does not even report how many distinct prompts the difficulty bins were computed for — the 500-question MATH test set suggests the cost is paid once per question, but in a deployment setting where new questions arrive continuously, this cost recurs.

Mitigation status. The paper acknowledges the limitation explicitly ("our experiments do not account for this cost largely for simplicity") and flags it as a "key avenue for future work" (Section 3.2), but proposes no concrete method for reducing it. The suggestion of training a model to "directly predict difficulty of a question" is mentioned in Section 8 but not developed or evaluated. As the paper stands, the compute-optimal framework is an analysis tool demonstrating what would be possible if difficulty were known cheaply, not a deployable system.

---

Hard Problems (Difficulty Bin 5) Are Completely Unsolved — Test-Time Compute Adds Zero Value

The assumption or constraint. The compute-optimal framework assumes that the base model's proposal distribution contains at least some correct solutions for a given prompt. For difficulty bin 5 — the hardest quintile of MATH problems — this assumption fails. The paper's own data shows that for these problems, the base model's pass@1 rate is near zero, meaning no amount of search or revision can find correct answers because none exist in the candidate pool.

The consequence. Across every method, every budget level, and every allocation strategy studied in the paper, bin 5 accuracy remains essentially flat at 1–3% (Figure 3, right panel; Figure 7, right panel). The FLOPs-matched comparison (Section 7, Figure 9) shows that even at R ≪ 1 (where the smaller model gets an enormous inference budget), test-time compute provides no meaningful improvement on bin 5 — the scaling curve is a flat line near zero, far below the ~14× larger model's performance. This establishes a hard capability boundary: test-time compute can amplify existing capability but cannot create it. For problems that are fundamentally outside the base model's reach — either because they require knowledge the model lacks, or reasoning patterns it has not acquired during pretraining — the entire compute-optimal framework is worthless.

The paper is transparent about this (Section 7 takeaway box), but the practical implication is severe: any deployment of this approach must include a mechanism for detecting when a problem falls into this regime and either escalating to a larger model or declining to answer. The compute-optimal policy for bin 5 is essentially "give up immediately" since no budget allocation helps, but the paper provides no guidance on how to identify such problems before expending the full difficulty estimation budget.

What evidence exists in the paper. The bin 5 results are consistent and unambiguous: Figure 3 (right) shows ~1–3% accuracy for both beam search and best-of-N across all budget levels from 4 to 256 generations. Figure 7 (right) shows ~2–3% accuracy for all sequential-to-parallel ratios at 128 generations. Figure 9 shows the bin 5 line (blue, bottommost) remaining below 5% across all test-time compute budgets, far below the ~14× larger model's star markers. Table 1 implicitly confirms this: the binning procedure places questions with the lowest pass@1 into bin 5, meaning the base model essentially never produces correct answers for these problems.

Mitigation status. None. The paper does not propose any mechanism for handling hard problems beyond acknowledging that pretraining is the only viable approach for them. The difficulty estimation procedure (2,048 samples per question) would still need to be run to discover that a problem is in bin 5, wasting substantial compute before the decision to escalate or abandon is made. The paper's finding that test-time compute cannot help on hard problems is a genuine negative result that bounds the applicability of the entire framework.

---

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate, Making Sequential Revision Chains Unreliable Without Post-Hoc Selection

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target. This means the model never sees examples during training of what to do when the current answer is already correct. At test time, when a revision chain produces a correct answer at step t, the model — conditioned on this correct answer at step t+1 — has no learned behavior for "leave it alone" or "recognize it is correct." Instead, it applies its learned revision policy (make changes) and often converts the correct answer back to an incorrect one. The paper reports (Section 6.1) that:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

The consequence. Pure sequential revision (generate one chain of length N and take the final output) would be extremely unreliable — the last answer in the chain is likely to be wrong not because the model can't reach a correct answer, but because it overshoots past correctness and then revises itself back into error. This makes the revision model dependent on post-hoc selection mechanisms (majority voting across the chain, or verifier-based selection) that pick the best answer from any point in the chain rather than trusting the final output. These selection mechanisms add their own costs: majority voting requires generating and storing all intermediate revisions, and verifier-based selection requires running a separate verifier model over the entire chain.

The 38% reversion rate also means that the effective length of useful revision chains is limited. Even if the model improves its per-step pass@1 throughout the chain (Figure 6, left, shows pass@1 rising from ~18.2% at step 1 to ~24–25% by steps 15–20), the probability that a correct answer at step t survives to step t+1 is at best ~62%. Over a chain of length 4, the probability that a correct answer produced at step 2 is still correct at step 4 is roughly 0.62² = 0.38 — meaning more than half of correct intermediate answers are lost before the chain ends. This explains why the fully sequential strategy performs worse than the balanced sequential-to-parallel ratio on medium-difficulty problems: longer chains have more opportunities for correct answers to be generated but also more opportunities for them to be overwritten.

What evidence exists in the paper. The 38% figure is stated in Section 6.1 but the exact measurement methodology is not detailed. The existence of the problem is confirmed indirectly by the fact that the paper deploys within-chain selection (majority voting or verifier-based) rather than taking the final revision output — a design choice that would be unnecessary if reversion were rare. Figure 7 (left) shows that fully sequential (rightmost point) underperforms the optimal intermediate sequential-to-parallel ratio at budgets of 64, 128, and 256 generations, consistent with long chains suffering from reversion despite beneficial within-chain selection. Figure 6 (left) shows pass@1 per step gradually improving — but this metric counts a step as "correct" if any answer in the chain up to that point is correct under the selection mechanism, not if the raw output at that step is correct, which obscures the reversion problem in the per-step visualization.

Mitigation status. The paper mitigates reversion through post-hoc selection (majority voting or verifier-based chain selection) but does not address the root cause. The training data construction (only incorrect→correct trajectories) is the source of the problem, and the paper does not experiment with alternative constructions that include "already correct → stay correct" examples. The selection mechanisms are imperfect patches: majority voting can fail if the chain produces multiple different incorrect answers that collectively outvote a single correct one, and verifier-based selection is only as reliable as the verifier itself (which has its own over-optimization problems documented in Section 5.3). The ReST^EM experiment (Appendix K, Figure 16) shows that alternative revision training approaches can catastrophically worsen the problem, suggesting the current training recipe is fragile.

---

The ~14× Larger Pretraining Baseline Is Not Compute-Optimally Trained and Uses Only Greedy Decoding, Weakening the FLOPs-Matched Comparison

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters, but (a) the larger model scales only parameters, not training data, departing from Chinchilla-optimal pretraining where data and parameters are scaled equally, and (b) the larger model uses only greedy decoding with no test-time augmentation of its own. The paper acknowledges this in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. The reported advantages of test-time compute over pretraining may be overstated. A Chinchilla-optimal model trained with 14× more total FLOPs, with both data and parameters properly scaled, would likely outperform a parameter-only-scaled model on the same total compute budget. The paper's baseline is weaker than a compute-optimal pretraining baseline would be, making test-time compute look better by comparison.

The greedy decoding choice is equally consequential. The ~14× larger model is evaluated with a single greedy output, while the smaller model is allowed to use sophisticated multi-sample strategies (beam search, sequential revisions with verifier-based selection). This is an asymmetric comparison: the larger model gets one shot at each problem, while the smaller model gets hundreds of shots with learned verification. A fairer comparison would give the larger model a test-time compute budget proportional to its per-token cost — for instance, if the larger model costs 14× more per token, the smaller model might get 14 generations at R ≫ 1 while the larger model gets 1, but at R ≪ 1 the smaller model might get thousands of generations. The paper's FLOPs accounting (Section 7) correctly handles the total FLOPs, but the allocation of those FLOPs only to the smaller model's test-time compute (not shared with the larger model) creates an asymmetry where the smaller model gets a massive inference budget while the larger model gets zero.

What evidence exists in the paper. The FLOPs-matched results in Figure 9 and the bar charts in Figure 1 show the ~14× larger model's greedy performance as horizontal lines (or stars) — a single data point per difficulty bin, rather than a scaling curve. The paper never shows what happens if the larger model is also given test-time compute (e.g., best-of-8 or best-of-16 with a verifier). Section 7 acknowledges the parameter-only scaling departure from Chinchilla optimality but does not provide even a rough estimate of how much this matters. The scaling laws literature (Hoffmann et al., 2022) provides formulas that could be used to estimate the performance of a compute-optimally trained 14× larger model, but the paper does not perform this calculation.

Mitigation status. The paper is transparent about the limitation but does not address it experimentally. The acknowledgment in Section 7 is honest, but the limitation affects the central takeaway of the FLOPs-matched comparison — the claim that test-time compute can substitute for pretraining — and the paper's decision to use a weaker baseline makes the substitution case appear stronger than it may be against a properly optimized pretraining baseline.

---

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*), and the 500-Question Test Set Is Small for Difficulty-Conditioned Strategy Selection

The assumption or constraint. Every experiment in the paper uses the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* as the base model. The paper argues (Section 4) that this model is "representative of the capabilities of many contemporary LLMs" and that MATH is appropriate because it requires multi-step reasoning where test-time compute is expected to help. However, no experiments are conducted on other reasoning domains (code generation, logical reasoning, scientific QA), other model families (GPT, Llama, Gemini), or other model scales.

The consequence. The paper's central findings — that beam search helps on medium problems but hurts on easy ones, that sequential revisions dominate on easy problems, that the 4× efficiency gain is achievable — are all conditional on PaLM 2-S*'s specific output distribution and the MATH benchmark's specific difficulty structure. Several mechanisms identified in the paper could be model-specific:

• PRM over-optimization (beam search degrading easy-problem performance at high budgets) depends on the PRM's calibration properties, which are a function of PaLM 2-S*'s output distribution and the Monte Carlo rollout training procedure. A model with different error patterns might produce a PRM that is more or less susceptible to over-optimization, shifting the difficulty thresholds at which beam search becomes harmful.

• Revision model effectiveness depends on the base model's ability to learn from in-context incorrect examples, which varies substantially across model families (some models are better at self-correction than others, as shown by Huang et al., 2023).

• The difficulty bin thresholds (the pass@1 rates that define quintiles) are specific to PaLM 2-S* on MATH. A stronger or weaker model would have different quintile boundaries, and a different benchmark would have a different distribution of problem types within each bin.

The 500-question MATH test set, when split into five difficulty quintiles of ~100 questions each and then further split by two-fold cross-validation for strategy selection, means the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a very small sample for optimizing over a combinatorial space of strategy hyperparameters (search algorithm, beam width, lookahead depth, revision chain length, sequential-to-parallel ratio). The paper does not report confidence intervals on the compute-optimal scaling curves (Figures 4 and 8), making it impossible to assess whether the observed 4× gain is reliable or whether the selected strategies would generalize to a new set of questions from the same distribution.

What evidence exists in the paper. None beyond the MATH/PaLM 2-S* results themselves. The paper contains no cross-benchmark or cross-model validation. The claim of representativeness (Section 4) is asserted but not supported. The difficulty bin cross-validation protocol is described in Section 3.2, but the paper does not report the variance of strategy performance across the two folds, the stability of the selected strategies, or any measure of how sensitive the compute-optimal curves are to the specific fold split.

Mitigation status. The paper does not address this limitation. Section 8 (future work) does not mention cross-domain or cross-model validation as a direction. For a paper whose central claim is the existence of difficulty-conditioned compute-optimal scaling behavior, the lack of evidence that this behavior generalizes beyond one model on one benchmark is a significant gap.

---

Sequential Revision Strategies Are Fundamentally Latency-Bound, but the Paper Measures Only Throughput (Generations), Not Wall-Clock Time

The assumption or constraint. The paper measures test-time compute in "generations" — the number of complete solutions sampled — and treats all generations as equivalent units of cost. This is a reasonable proxy for total FLOPs but ignores the fact that sequential operations cannot be parallelized, while parallel operations can. A strategy that allocates 128 generations as 64 sequential revisions × 2 parallel chains takes approximately 64× longer wall-clock time than a strategy that runs 128 fully parallel samples simultaneously, assuming sufficient hardware to execute all parallel samples concurrently.

The consequence. The paper's compute-optimal policies — which favor sequential revisions on easy problems (Section 6.3, Figure 7, right: bin 2 shows fully sequential as optimal) and balanced sequential-to-parallel ratios on medium problems — may be impractical for latency-sensitive applications. In an interactive setting (e.g., a chatbot answering user questions, or a real-time recommendation system), waiting for 64 sequential model calls — each depending on the output of the previous one — introduces latency that is unacceptable regardless of the accuracy improvement. The sequential-to-parallel ratio sweeps in Figure 7 (left) show the x-axis ranging from fully parallel (1 parallel chain of length 1) to fully sequential (1 chain of length up to 256), but the latency implications of moving along this axis are never discussed.

This is particularly relevant for the revision model, where sequential chains of length 16–64 are studied (Figure 6, left). Each step in the chain requires a full forward pass through the revision model (which has the same parameter count as the base LLM), and these passes cannot be batched or parallelized because each step depends on the output of the previous one. At the budget levels where compute-optimal scaling achieves its 4× efficiency gain (e.g., 64 generations matching 256 in Figure 8), the optimal strategy likely involves some sequential component, meaning the wall-clock latency of the compute-optimal strategy at 64 generations could be comparable to or worse than the baseline parallel strategy at 256 generations.

What evidence exists in the paper. None. The paper never reports latency, wall-clock time, or any measure of serial dependency. The x-axes in all scaling plots (Figures 3, 4, 6, 7, 8) are "Number of Generations," not time. The FLOPs accounting in Section 7 uses total inference tokens, which captures computational cost but not temporal cost. There is no discussion of how the sequential nature of revisions affects deployability.

Mitigation status. None. The paper does not acknowledge latency as a concern, does not report any latency measurements, and does not discuss the tradeoff between sequential depth and parallel breadth in terms of wall-clock time. For a paper motivated in part by on-device deployment and self-improvement pipelines (Section 1), where latency constraints are often stringent, this omission is significant. The compute-optimal policies derived in the paper optimize for total FLOPs efficiency, not for FLOPs-latency Pareto optimality, and practitioners deploying in latency-constrained environments would need to re-derive the optimal policies under a joint FLOPs-latency budget that the paper provides no framework for.

7. Implications and Future Directions

How This Work Changes the Landscape

SAIL-Embedding represents a systems-level reframing rather than a single-algorithm breakthrough. The paper's primary contribution is not a new architecture or loss function, but the demonstration that a single omni-modal embedding model — when trained with a carefully staged curriculum that transitions from broad content understanding to recommendation-specific behavioral distillation — can replace multiple purpose-built models across content understanding, cross-modal search, and behavioral recommendation simultaneously. This challenges the prevailing deployment pattern in industrial multimodal platforms, where separate embedding models serve different downstream stages (a CLIP variant for visual similarity, a text embedder for search, a collaborative filtering model for recommendation). The paper shows that unification is possible, but — critically — that it requires explicit architectural and training co-design to resolve the content-collaboration alignment gap that prior work simply ignored.

The magnitude of this contribution is incremental in method but potentially transformative in deployment economics. No individual component of SAIL-Embedding — the Perceiver bottleneck, the CLAP audio encoder, the contrastive loss formulation, the Matryoshka representation learning — is fundamentally novel in isolation. The innovation lies in the integration and the recognition that these components must be orchestrated through a specific training sequence (content-aware progressive training → collaboration-aware recommendation enhancement) to achieve the dual objectives of content fidelity and behavioral alignment. This is a different kind of contribution than a new state-of-the-art on a static benchmark: it's a recipe for building embedding models that work in production recommenders, validated by both offline benchmarks and online A/B tests on a platform serving hundreds of millions of users.

The paper resolves a latent tension in the multimodal embedding literature that has been simmering without explicit acknowledgment. Prior work on MLLM-based embedding models (VLM2Vec, GME, mmE5, MoCa) has focused almost exclusively on content understanding — maximizing retrieval recall on benchmarks where relevance is defined by semantic similarity. The implicit assumption was that better content representations naturally translate to better downstream task performance, including recommendation. SAIL-Embedding's results demonstrate that this assumption is empirically false. The Copair-i2i benchmark (items paired by content representation similarity) and the Gid-i2i benchmark (items paired by ID embedding relevance from the recommender) measure fundamentally different notions of relatedness, and optimizing for one degrades the other (Table 4: CRE improves Gid-i2i Recall@50 by +7.23 percentage points while reducing Copair-i2i Recall@50 by −2.34 percentage points). This is not a matter of "better" or "worse" representations — it's evidence that content similarity and collaborative relevance are orthogonal objectives that must be explicitly balanced, not assumed to converge.

This finding redirects research attention in several ways. First, it makes collaboration-aware training a first-class research problem for embedding models, not an afterthought handled by a separate recommendation model downstream. Future work on multimodal embeddings for platforms with user interaction data should include behavioral distillation objectives by default, and papers that evaluate only on content-similarity benchmarks should be viewed as incomplete. Second, it surfaces verifier/model quality as a bottleneck for embedding generalization — the paper shows that the content-collaboration tradeoff is manageable (the Copair degradation is "tolerable" relative to the Gid gain in the paper's deployment context) but not eliminable, suggesting that better architectures or training objectives might find Pareto improvements that current methods cannot. Third, it establishes multi-dimensional evaluation frameworks (the paper's six-metric protocol in Section 4.2) as necessary for embedding models deployed in recommendation systems — single-metric evaluation (retrieval recall) is insufficient to predict downstream recommendation impact, as the online A/B test results (Table 6) confirm that offline retrieval improvements do not linearly translate to user retention gains.

The paper also shifts the conversation around modality fusion depth. Prior work oscillated between shallow fusion (CLIP-style dual towers, efficient but limited) and deep fusion (MLLM-based, powerful but expensive). SAIL-Embedding demonstrates that deep fusion through an LLM backbone is necessary but not sufficient — the training strategies (mICL loss to prevent text dominance, late fusion to preserve visual fidelity, dataset-driven pattern matching to handle arbitrary modality combinations) are what prevent deep fusion from collapsing into text-only representations. This suggests that future work on multimodal fusion should focus as much on optimization strategies that maintain modality balance as on architectural innovations.

Follow-Up Research This Work Enables

Isolating audio modality contribution through controlled ablation. The paper claims that the audio modality is "important and effective" for multimedia retrieval (Section 4.4.1), but this claim is confounded with other differences between SAIL-Embedding and the VLM baseline (richer text processing, training strategy differences). A clean ablation would train two versions of SAIL-Embedding — one with the full omni-modal pipeline (vision + text + audio) and one with the audio encoder and audio tokens removed, keeping all other architecture, training data, and hyperparameters identical — and measure the performance gap across all 21 i2i and 9 q2i benchmarks. This would directly quantify audio's marginal contribution and identify which task categories (music, live streaming, ASR-heavy content) benefit most. The hypothesis from the paper's results is that audio provides the largest gains on live streaming tasks (Live-q2i, Live-i2i) and music-related tasks, and minimal gains on text-dominated classification tasks.

Measuring the content-collaboration Pareto frontier across CRE training intensity. The paper's CRE training is applied as a binary treatment (with vs. without) and evaluated on only two benchmarks (Gid-i2i and Copair-i2i in Table 4). A more complete characterization would train a series of models with varying CRE intensity — controlled by the weight of the ID-to-item distillation loss relative to the content contrastive loss, or by the proportion of CRE training steps relative to total training — and measure performance on the full set of 21 i2i benchmarks. This would map the Pareto frontier of content vs. collaboration performance, revealing whether there exists a CRE intensity that improves Gid-i2i substantially with minimal Copair-i2i degradation, or whether the tradeoff is strictly monotonic (any collaborative gain costs content fidelity). The paper's current data (Table 4) shows a roughly symmetric tradeoff (~+7 Gid, ~−2 Copair), but the shape of the frontier at intermediate CRE intensities is unknown. This experiment would also test whether different CRE components (sequence-to-item vs. ID-to-item distillation) produce different tradeoff shapes, enabling practitioners to select the distillation strategy that best matches their deployment's content-collaboration priority.

Cross-architecture generalization of recommendation enhancement training. The paper applies CRE to two architectures (the VLM-based model and SAIL-Embedding) and shows consistent Gid-i2i improvements (Table 4). But both architectures share the same LLM-backbone design paradigm. A strong test of generality would apply the same CRE procedure — ID-to-item distillation with auxiliary i2i loss — to a fundamentally different embedding architecture, such as a CLIP-style dual-tower model with separate vision and text encoders, or to a non-LLM fusion architecture. The hypothesis is that CRE's effectiveness depends on the representational capacity to simultaneously encode content and collaborative signals without destructive interference, and that architectures with limited fusion capacity (dual-tower) would show larger Copair degradation at equivalent Gid gains. A negative result — CRE fails to transfer to non-LLM architectures — would establish that the content-collaboration balance is achievable only with deep fusion capacity, which would be an important architectural insight. A positive result — CRE helps regardless of architecture — would suggest the distillation approach is a general technique for injecting collaborative signals into any content-based embedder.

Cheap difficulty estimation for adaptive test-time compute in embedding-based retrieval. The paper's adaptive multi-source data balancing (Section 3.1.3) uses an early version of SAIL-Embedding to compute dataset similarities, requiring 10k samples per dataset. This is acceptable for a one-time training configuration step, but a deployed system that needs to dynamically decide which embedding dimensionality (128 vs. 768 vs. 1536) or which modality combination to use for a given query-item pair would need much cheaper estimation. A concrete follow-up: train a lightweight classifier that takes only the text query and item metadata (title, tags) as input and predicts whether the 128-dimensional embedding will achieve ≥95% of the 1536-dimensional embedding's retrieval recall for that query. This classifier could be trained using the Matryoshka embeddings already produced by SAIL-Embedding — for each query in the training data, compute recall@k at all three dimensionalities, label queries where 128-dim recall is within 95% of 1536-dim recall as "acceptable for low-dim retrieval," and train a small model to predict this label from cheap features. Success would enable adaptive retrieval where the system defaults to fast 128-dim search and only falls back to 1536-dim for queries flagged as needing higher precision, directly reducing inference cost and latency.

Sequence-to-item distillation with learnable sequence weighting. The paper's sequence-to-item distillation uses mean pooling over the 10-item viewing sequence (Section 3.3.5), which treats all historical items as equally predictive of the next watch. This is almost certainly suboptimal — the most recent item in the sequence is typically more predictive than the 10th most recent, and certain item types (long-form content, series episodes) may have different temporal decay patterns. A natural extension: replace mean pooling with a learned attention mechanism over the sequence, where the attention weights are conditioned on both item content (embeddings) and temporal features (time since watch, watch duration, interaction type). Training this attention module jointly with the embedding model (or as a fine-tuning stage after the main CRE phase) would allow the model to learn that, e.g., recently-watched items from the same creator should be weighted more heavily than older items from different creators. The evaluation would use the same Vanilla Seq2item and Filtered Seq2item benchmarks (Table 5), with the hypothesis that learned attention improves Recall@k beyond the mean pooling baseline, particularly on the harder Vanilla Seq2item where the model must identify which of 10 randomly-sampled historical items are actually predictive.

Stress-testing the over-optimization hypothesis in embedding-based retrieval. The analyzed paper (on compute-optimal test-time scaling) documented verifier over-optimization as a central bottleneck: beam search degrades easy-problem performance at high budgets (Figure 3, right). SAIL-Embedding faces an analogous risk: as the embedding model becomes more powerful and the retrieval corpus grows, there may exist items that score highly under the cosine similarity metric but are actually irrelevant to the query — the embedding space equivalent of PRM over-optimized solutions. A diagnostic experiment: for a fixed set of queries, retrieve the top-k items using SAIL-Embedding at increasing levels of retrieval depth (k = 10, 100, 1000, 10000), and for each retrieved item at each depth, collect human relevance judgments or proxy relevance signals (user click-through rate in a separate logging system). Plot relevance@k as a function of k and embedding dimensionality (128 vs. 768 vs. 1536). The hypothesis: higher-dimensional embeddings will show better relevance at small k (more precise top results) but may degrade faster at large k (more over-optimized false positives) because the additional dimensions provide more capacity to fit spurious correlations that score highly under cosine similarity but don't reflect true relevance. A positive result (degradation observed) would establish that embedding dimensionality itself introduces a precision-recall tradeoff mediated by over-optimization, with practical implications for retrieval system design.

Practical Applications and Downstream Use Cases

Cold-start item and user representation in content platforms. New items (videos, live streams, ads) and new users on platforms like Douyin have minimal behavioral history, rendering collaborative filtering models ineffective. SAIL-Embedding provides a direct solution: for a new item, encode its omni-modal content (frames, title, OCR, ASR, audio) into a 1536-dimensional embedding, and use this embedding for similarity-based recall (retrieving existing items with similar embeddings to bootstrap the new item's audience). The paper's online results validate this: in the Coldstart scenario (Table 6), SAIL-Embedding embeddings and Semantic IDs delivered a +0.05% LT30 gain when used for recall. This gain is substantial in absolute terms for a cold-start intervention (where engagement is inherently low and small improvements compound as users are retained). The practical deployment pattern: maintain a real-time embedding service that encodes new items within seconds of upload, indexes them into an approximate nearest neighbor (ANN) system alongside existing items, and uses similarity search to populate the new item's initial candidate pool for the first few thousand impressions, after which behavioral signals dominate and the cold-start embedding can be gradually down-weighted.

Unified search across heterogeneous content types. Platforms that host multiple content formats (short videos, live streams, image posts, text articles) typically maintain separate search indices with format-specific ranking models. SAIL-Embedding's dataset-driven pattern matching — which trains the model to handle text-to-video, text-to-live, and omni-to-omni retrieval in a single training run — enables a unified search architecture: all content types are embedded into the same 1536-dimensional space, a user's text query (for q2i search) or an example item (for i2i "more like this" search) is embedded, and ANN search retrieves the top-k items across all content types simultaneously. The paper's results support this: Short Video-q2i Recall@50 of 86.53% and Live-q2i Recall@50 of 79.08% (Table 3) are achieved by the same model, and Live Summary2i (text summaries matching live content) achieves 84.33% Recall@50, demonstrating cross-format generalization. The practical benefit is operational simplification — one embedding model, one ANN index, one retrieval pipeline — with the potential for improved recall in cases where a user's query is best answered by a content type they didn't explicitly search for (e.g., a text query for "cooking tutorial" might be better served by a short video than an article, and the unified embedding space enables this cross-format retrieval without explicit format routing).

Multi-stage recommender system feature consistency. Modern recommendation pipelines are sequential: recall retrieves thousands of candidates, pre-rank filters to hundreds, rank scores precisely, and re-rank adjusts for context. Each stage typically uses different features and different models, and inconsistencies between stages — an item that ranks highly at the recall stage but is down-weighted at the ranking stage because different features are used — create inefficiencies where good candidates are discarded early. SAIL-Embedding mitigates this by providing the same embedding features (and their discretized Semantic ID tokens) at every stage, ensuring that the item representation is consistent throughout the pipeline. The paper's online results (Table 6) demonstrate this: in the Feed scenario, Semantic IDs contribute +0.01% LT30 at recall, pre-rank, and re-rank stages, with the ranking model additionally benefiting from dense embeddings (+0.1% Finish AUC). The practical deployment: all pipeline stages read from the same embedding service, with recall using low-dimensional (128-dim) embeddings for fast ANN search, pre-rank using mid-dimensional (768-dim) embeddings with a lightweight scorer, rank using full 1536-dimensional embeddings as features in a deep model, and re-rank using Semantic IDs for rule-based adjustments (e.g., "boost items from the same creator cluster as the last watched item"). The consistency gain comes from the fact that an item's relative ordering is preserved across stages — if it's in the top 1% of the embedding space, it will be retrieved at recall, not filtered at pre-rank, and scored appropriately at rank, reducing the chance of good candidates being lost.

Content moderation and policy enforcement via embedding-based clustering. Platforms must enforce content policies (e.g., detecting duplicated content, identifying policy-violating material, grouping items by sensitive topics for targeted review). SAIL-Embedding's omni-modal embeddings enable these tasks without requiring separate models per policy. For duplicate detection: embed all newly uploaded items, perform ANN search against the corpus of existing items, and flag items whose nearest neighbor has cosine similarity above a threshold (e.g., 0.98) for human review. For sensitive content grouping: use the embeddings to cluster items (via k-means on 128-dim embeddings for efficiency), and have human reviewers label entire clusters rather than individual items, dramatically reducing review cost for content types that are visually and acoustically homogeneous (e.g., a specific type of spam video). The paper's clustering consistency metric (NMI in Table 4) supports this use case: SAIL-Embedding with CRE achieves NMI of 0.65, indicating that positive and negative samples are well-separated in the embedding space, which is a prerequisite for reliable cluster-based moderation.

When to Prefer This Method

SAIL-Embedding is designed for a specific deployment context — large-scale content platforms with rich multimodal items and diverse downstream tasks — and the paper's results articulate several conditions under which it should be preferred or avoided relative to alternatives:

Prefer SAIL-Embedding with full omni-modal training when:

• Items carry information across vision, text, and audio that jointly determine user engagement. The paper's strongest results are on tasks where audio plausibly contributes: Live-q2i (Table 3: +5.2 percentage points over VLM baseline at Recall@50), Music Play-i2i and Music Gameplay-i2i (Table 2: competitive with VLM baseline). If items are primarily visual (static images with minimal text) or primarily textual (articles), the audio modality provides minimal marginal benefit and the added architectural complexity may not be justified — a VLM-based image-text model would be simpler and comparably effective.
• The deployment requires both content understanding and collaborative recommendation from a single embedding model. SAIL-Embedding with CRE is the only model in the paper's comparison that achieves strong performance on both axes (Table 4: Gid-i2i Recall@50 of 59.69% with CRE vs. 46.34% for VLM with CRE). If the deployment only needs content similarity (e.g., duplicate detection, visual search), the CRE phase is unnecessary and may degrade content performance (Copair-i2i Recall@50 drops from 69.17% to 66.83% with CRE). If the deployment only needs collaborative filtering, a pure ID-based recommender model may outperform any content-based embedder on collaborative metrics.
• The platform has sufficient training data to construct the diverse pairing strategies described in Table 1. SAIL-Embedding's training relies on behavioral pairing data (Copair-i2i: 2.9B pairs from user consumption; Search-q2i: 1.8B pairs from search clicks) that requires production-level user interaction logs. For platforms with limited behavioral data, the content-only training stages (without CRE) can still be effective, but the full recommendation enhancement training may not be feasible.

Prefer a simpler alternative when:

• Latency constraints preclude LLM-based fusion at inference time. SAIL-Embedding routes all modalities through a full Transformer LLM backbone, which is expensive per query. For real-time applications with strict latency budgets (e.g., <10ms per embedding), a CLIP-style dual-tower model with precomputed item embeddings and lightweight query encoding may be necessary, accepting the performance degradation documented in Table 2 (CLIP-based model achieves 53.47% on Copair-i2i Recall@50 vs. SAIL-Embedding's 69.17%).
• Audio is unavailable or uninformative for the majority of items. The paper's audio processing (CLAP encoding with length-dependent segmentation) adds complexity to the inference pipeline. If the platform's content is predominantly silent or the audio track carries negligible information (e.g., text-heavy slideshows with background music), the VLM-based baseline (image + text only) achieves competitive performance on most tasks without the audio overhead.
• The downstream tasks are exclusively query-to-item text retrieval, and the target items have sparse multimodal content. On classification-oriented q2i tasks where targets contain only text features (Table 3, AUC tasks), the unimodal text embedding baselines (Doubao-Embedding, Qwen3-Embedding-8B) achieve respectable performance (e.g., Longtail-q2i AUC of 85.84% for Doubao-Embedding vs. 91.22% for SAIL-Embedding). The gap is substantial but the unimodal models are simpler, faster, and require no multimodal training infrastructure. For a text-only search application, the 5–6 percentage-point AUC improvement from SAIL-Embedding may not justify the deployment complexity.

For the CRE phase specifically, prefer to include it when:

• The deployment has a production recommender system with high-quality ID embeddings that can serve as distillation teachers. The paper's ID-to-item distillation (Section 3.3.5) transfers knowledge from existing recommender ID embeddings into the multimodal embedder. If no such teacher exists (e.g., a new platform without an established recommender, or a research setting), the ID-to-item component of CRE is inapplicable, and only sequence-to-item distillation (which requires only user viewing sequences, not ID embeddings) can be used.
• The platform has user viewing sequences with sufficient density to construct meaningful sequence-to-item pairs. The paper constructs sequences from 1000-video viewing histories (Section 3.3.5), filtering for positive interaction behaviors. Platforms with sparse user interactions (e.g., new users, infrequent visitors) may not have sequences long enough to provide useful training signal, and the sequence-to-item distillation may add noise rather than behavioral information.
• The deployment can tolerate a small degradation in content-similarity performance in exchange for collaborative filtering gains. The Copair-i2i degradation with CRE (Table 4: −2.34 percentage points at Recall@50) is explicitly acknowledged as a tradeoff. If the primary downstream application is content moderation or visual search where collaborative signals are irrelevant or harmful, CRE should be omitted, and the content-only SAIL-Embedding (pre-CRE) should be deployed.

For adaptive data balancing and dynamic hard negative mining, the paper does not provide sufficient evidence to establish when they should be preferred over simpler alternatives — neither is independently ablated, and their marginal contributions are unknown. A practitioner implementing SAIL-Embedding's training pipeline should consider these components as hypotheses to validate on their own data rather than as proven best practices: run an ablation comparing adaptive balancing vs. uniform sampling and dynamic hard negative mining vs. fixed-threshold mining on a representative subset of the training data before committing to the full pipeline. The paper's value for these components is in providing a concrete, reproducible algorithm rather than in establishing their necessity.