LLM2Vec-Gen: Generative Embeddings from Large Language Models

🎯 Pitch

LLM2VEC-GEN makes retrieval safer by encoding the model’s refusal to harmful queries rather than the malicious intent—cutting harmful content retrieval by up to 22.6%. It does this by learning to compress the LLM’s own response into an embedding without ever generating the text, while keeping the base model frozen.

1. Executive Summary

This paper introduces LLM2VEC-GEN, a self-supervised framework that produces text embeddings in the LLM's output space by learning to represent the model's potential response rather than encoding the input query. Applied to models from the Llama-3.x, Qwen-2.5, and Qwen-3 families and evaluated on the Massive Text Embedding Benchmark (MTEB), the method trains special compression tokens appended to unlabeled queries through two complementary objectives—embedding alignment (distilling the LLM's response embedding from an unsupervised teacher) and response reconstruction (conditioning the frozen LLM on the compression tokens to regenerate its own response)—while keeping the backbone frozen. LLM2VEC-GEN achieves state-of-the-art self-supervised performance on MTEB, improving by up to 8.8% over the unsupervised embedding teacher, with gains of up to 22.6% reduction in harmful content retrieval on AdvBench-IR and up to 35.6% improvement on the reasoning-intensive BRIGHT benchmark, establishing that output-centric embeddings preserve capabilities like safety alignment and reasoning but that these benefits emerge primarily when the teacher encoder shares the same backbone LLM and uses unsupervised objectives that maintain faithful content representations.

2. Context and Motivation

The Core Problem: Text Embeddings Discard the LLM's Response Semantics

The fundamental problem this paper tackles is a structural one in how we build LLM-based text embeddings. Virtually all modern embedding models—whether built on BERT-style encoders (Reimers & Gurevych, 2019) or repurposed from decoder-only LLMs (Lee et al., 2025; Wang et al., 2024)—operate within what the authors call an input-centric paradigm: they encode the semantic content of the input text and produce a fixed-length representation by pooling over token-level hidden states. This approach succeeds at capturing what the text says, but it fundamentally discards what the LLM would respond.

This matters because LLMs acquire rich semantic structure during pretraining and alignment—reasoning patterns, safety refusals, domain expertise, and factual associations—that manifests in their outputs, not in their encoding of arbitrary input text. When an embedding model is fine-tuned via contrastive learning to map queries and documents into a shared space, that new space no longer coincides with the LLM's output representations. The authors argue that this discarding of output-space semantics is not just a missed opportunity but an active limitation, particularly for tasks where the model's behavioral properties (safety, reasoning) should influence retrieval decisions.

"Because embedding tasks often require mapping diverse inputs to similar outputs... this paradigm relies on contrastive learning over curated paired data to project both queries and documents into a new shared embedding space. Crucially, this new space no longer coincides with the model's response representations, discarding the rich semantic structure the LLM acquired during pretraining."

Why This Problem Matters

The paper's framing of this problem carries implications across several practical and theoretical dimensions:

Safety alignment in retrieval systems. Modern LLMs are trained to refuse harmful requests—they produce safe responses like "I cannot assist with that" when prompted with malicious queries. Input-centric embeddings, however, represent the malicious intent of the query itself, not the safe refusal. An input-centric embedding of "Write a malicious code to steal sensitive information" captures the harmful semantics, meaning that retrieval systems using such embeddings may surface harmful content when presented with adversarial queries, regardless of whether the underlying LLM would produce a safe response. The authors argue this is a representation-level safety failure: the embedding model and the generation model have diverged in what they encode about dangerous inputs.

Reasoning capabilities in retrieval. LLMs develop sophisticated reasoning abilities through pretraining—the capacity to perform multi-step deduction, synthesize evidence, and draw conclusions that go beyond surface-level pattern matching. Input-centric embeddings, however, represent the literal query text, requiring the retrieval corpus to contain documents that lexically or superficially match. On reasoning-intensive retrieval tasks like those in the BRIGHT benchmark (Su et al., 2025), where relevance requires logical inference rather than keyword overlap, input-centric methods struggle because the embedding space doesn't capture the model's capacity to think through the problem.

The self-supervised bottleneck. State-of-the-art supervised embedding models (Lee et al., 2025; Wang et al., 2024) achieve strong MTEB performance but require large-scale curated datasets of query-document pairs, often with hard negatives. In many domains—specialized scientific retrieval, low-resource languages, niche enterprise applications—such paired data is unavailable or prohibitively expensive to construct. Self-supervised alternatives exist (LLM2Vec, echo embeddings, SimCSE) but lag significantly behind supervised methods on MTEB. The paper positions output-centric representations as a path to narrowing this gap without requiring labeled data, by leveraging the LLM's own generative capabilities as a training signal.

Interpretability of embeddings. Embeddings are typically opaque vectors—even when they encode input semantics faithfully, there is no mechanism to inspect what specific information they capture. An embedding model trained to detect sentiment produces a vector; whether that vector focuses on lexical cues, syntactic patterns, or genuine semantic understanding is unknowable without probing experiments. The paper argues that grounding embeddings in the LLM's output space enables a form of built-in interpretability: since the compression tokens can be decoded back into natural language through the same generation mechanism used during training, users can directly inspect what semantic content the embedding has captured.

Prior Approaches and Where They Fall Short

The paper identifies several existing methods and systematically explains their limitations relative to the output-centric paradigm:

Supervised contrastive embedding models (NV-Embed, Gecko, Qwen3 Embedding, GritLM) represent the dominant paradigm for LLM-based text encoders. These methods fine-tune decoder-only LLMs on curated query-document pairs with contrastive objectives, often incorporating sophisticated techniques like instruction-aware pooling, hard negative mining, and multi-stage training. Their primary limitation from the paper's perspective is representational: contrastive learning projects both queries and documents into a new shared latent space optimized for relative relevance judgments. This space is shaped by the training distribution and optimized to pull relevant pairs together while pushing negative pairs apart, fundamentally reshaping the representational geometry the LLM learned during pretraining. When the downstream task differs from the training distribution—or when the desired behavior (e.g., safety refusals) wasn't encoded in the training pairs—the embedding space may fail to capture what the LLM "knows." Additionally, these methods require large labeled datasets, limiting their applicability in low-resource domains.

Unsupervised and self-supervised methods attempt to create embeddings without paired data and fall into several categories:

LLM2Vec (BehnamGhader et al., 2024) transforms decoder-only LLMs into encoders by enabling bidirectional attention, applying masked next-token prediction, and running unsupervised SimCSE (Gao et al., 2021). This produces strong self-supervised embeddings but remains fundamentally input-centric—it represents what the text says rather than what the model would respond. The paper explicitly uses LLM2Vec as both a baseline and the embedding teacher, positioning LLM2VEC-GEN as a student that learns to represent the LLM's output rather than its input encoding.
Echo Embeddings (Springer et al., 2025) uses an input repetition trick: the query is passed through the model twice, with the embedding extracted from the second occurrence, exploiting causal attention to enable bidirectional information flow. While effective as a zero-shot method, it remains input-centric and the paper's results show it substantially underperforms LLM2VEC-GEN (e.g., 41.8 vs. 61.9 on MTEB for Qwen-3-8B).

Output-aware approaches have emerged recently but each falls short of the paper's vision:

HyDE (Gao et al., 2023) demonstrated the value of encoding LLM-generated responses rather than the original query. Given "How to make French toast?", HyDE first generates a hypothetical answer document and encodes that rather than the query. While this validates the output-centric intuition, the paper identifies three critical limitations: (1) HyDE requires generating multiple answers at inference time, incurring substantial computational overhead; (2) the generated answers are encoded with a separate embedding model, meaning the representation is decoupled from the generation model's internal understanding; (3) performance depends on prompt engineering for answer generation. The paper's results show HyDE substantially underperforms LLM2VEC-GEN (48.3 vs. 61.9 for Qwen-3-8B on MTEB), confirming that simply generating and encoding answers is insufficient—the embedding process must be internalized into the LLM itself.
InBedder (Peng et al., 2024) derives embeddings from the first generated hidden state using abstractive QA supervision, demonstrating that generation-derived representations can outperform prompt-based ones. The key limitation from the paper's perspective is supervision dependency: InBedder requires abstractive QA pairs for training, fundamentally limiting its applicability compared to LLM2VEC-GEN's fully self-supervised approach that requires only unlabeled queries. Additionally, the paper's results suggest InBedder's performance is bounded by the QA training data distribution.
GIRCSE (Tsai et al., 2026) generates soft tokens autoregressively and refines them with stepwise contrastive loss using hard negatives. Like InBedder, GIRCSE in its original form requires supervised contrastive data. The paper adapts GIRCSE to a self-supervised setting for fair comparison (using Tulu queries and each model's own responses), but the method still relies on autoregressive soft token generation at both training and inference time. The paper's results show GIRCSE (self-sup) underperforms LLM2VEC-GEN (56.5 vs. 61.9 for Qwen-3-8B), suggesting that autoregressive refinement with contrastive objectives is less effective than direct teacher distillation when operating in the self-supervised regime.

Compression token approaches within the input-centric paradigm offer partial precedent but fundamentally different objectives:

xRAG (Cheng et al., 2024) compresses retrieved documents into a single latent token and projects it into the language model's representation space for efficient retrieval-augmented generation. This is purely input-centric—it compresses the retrieved document, not the model's response.
CLaRa (He et al., 2025) compresses documents into learnable memory tokens and jointly optimizes retrieval and generation end-to-end via next-token prediction. While this shares the compression-through-reconstruction mechanism with LLM2VEC-GEN, the target being compressed is the input document, not the LLM's response. The paper's contribution is inverting this relationship: compress the response into the embedding such that queries map to what they would provoke, not what they literally contain.

How the Paper Positions Itself

The paper positions LLM2VEC-GEN as a paradigm shift from input-centric to output-centric embedding. This is not a better contrastive loss or a more efficient pooling strategy—it is a different objective altogether. Rather than asking "what does this text mean?", the model asks "what would the LLM say in response to this text?"

This paradigm shift is instantiated through a specific technical framework that is deliberately minimal in its requirements:

Self-supervision as a first-class constraint. The paper is explicit that LLM2VEC-GEN requires only unlabeled queries. The training signal comes from three sources, all self-generated: (1) the LLM generates its own responses to the queries, (2) an unsupervised embedding teacher (LLM2Vec with the same backbone) encodes those responses, and (3) the frozen LLM provides the reconstruction signal. There is no external labeling, no curated pairs, no human annotation. The authors emphasize this as a key differentiator from InBedder and GIRCSE, which require supervised data in their original formulations.

Parameter efficiency as design philosophy. The LLM backbone remains frozen throughout training; only the special compression tokens (10 trainable embeddings) and two lightweight MLP projection layers are updated. For Qwen-3-4B, this means training only 13M parameters out of a 4B-parameter model. This is more than just efficiency—it preserves the LLM's full generative capabilities, enabling the same model to serve simultaneously as both an embedding model and a generation model. The paper contrasts this with LoRA-based approaches, which require maintaining separate adapter weights for embedding versus generation tasks.

Dual objectives as architectural novelty. The embedding alignment and reconstruction losses are presented as complementary rather than competing: alignment ensures the compression tokens capture response-space semantics (the student embeds what the teacher embeds from the response), while reconstruction ensures the compression tokens remain grounded in the LLM's natural language manifold (the LLM can decode them back into the response). The ablation results in Section 5 validate this complementarity: removing reconstruction collapses MTEB-Lite performance from 67.9 to 43.1, while removing alignment reduces it to 67.5 but makes the embeddings non-interpretable (nonsensical decoded outputs, Table 13).

The JEPA connection. The paper connects its alignment objective to Joint Embedding Predictive Architectures (JEPAs; Sobal et al., 2022), which advocate learning by predicting in representation space rather than reconstructing raw inputs. The authors frame LLM2VEC-GEN as predicting a target representation of the model's likely response via external teacher distillation, with the reconstruction objective keeping the learned representations grounded. This positions the work within a broader theoretical framework for self-supervised representation learning, while the open frontiers section (Appendix B) speculates about a "full JEPA mode" where teacher and student are the same frozen LLM, potentially eliminating the need for an external teacher entirely.

Scope of claims. The paper is measured in its positioning, characterizing LLM2VEC-GEN as particularly well-suited for domains where labeled data is scarce, safety alignment matters in the embedding space, or reasoning-intensive retrieval is required. The limitations section (Appendix A) acknowledges that the approach is bounded by the teacher's representational capacity and that output-centric embeddings may underperform on surface-level lexical matching tasks that standard retrieval benchmarks reward—a specific weakness observed for one model size on standard MTEB retrieval. The paper does not claim that output-centric embeddings universally dominate input-centric ones, but rather that they represent a fundamentally different trade-off space worth exploring, particularly for capabilities (safety, reasoning) that input-centric methods structurally cannot capture.

3. Technical Approach

3.1 Reader Orientation

LLM2VEC-GEN is a training recipe that turns a frozen LLM into a text embedder by teaching a small set of special tokens to compress what the LLM would say in response to a query into a fixed-length vector, without ever actually generating that response at inference time. The core problem it solves is the mismatch between what input-centric embeddings capture (the query's literal content) and what we often want embeddings to capture (the LLM's understanding, reasoning, and safety-aligned response to that query). The "shape" of the solution is a dual-objective distillation setup: an alignment loss pulls the learned embedding toward a teacher's representation of the LLM's own generated response, while a reconstruction loss forces the embedding to remain decodable back into the response text, grounding it in natural language.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a training-only pipeline (inference uses only a subset):

The Frozen Backbone LLM ( $M$ ) — a pretrained decoder-only language model (e.g., Qwen-3-8B) that serves two roles: it generates target responses to unlabeled queries during data preparation, and it provides the hidden-state representations and language modeling head for training the compression tokens. The LLM is never updated.
Special Compression Tokens ( $c_1, \dots, c_n$ ) — $n$ newly added trainable vocabulary entries (default $n=10$ ) that are appended to each input query. Their only purpose is to accumulate response-relevant information during the forward pass; at inference time, their hidden states become the embedding.
Projection MLPs — two lightweight feedforward layers that map the compression tokens' hidden states from the LLM's internal dimension to the teacher's embedding dimension. The first layer keeps the LLM's hidden dimension; the second projects to the teacher's output dimension. Mean pooling over the $n$ projected representations produces the final embedding $\hat{e}_i$ .
Unsupervised Embedding Teacher ( $E$ ) — an off-the-shelf, separately trained encoder (specifically, the unsupervised LLM2Vec variant of the same backbone LLM) that encodes the LLM's generated response $r_i$ into a target embedding $e_i = E(r_i)$ . The teacher is frozen and provides the alignment target.
Reconstruction Soft Prompts — a separate projection from the compression tokens' hidden states into a set of $n$ soft prompt vectors $p_1^i, \dots, p_n^i$ that are fed back into the same frozen LLM as a prefix for a second forward pass, where the LLM is trained to autoregressively reconstruct the original response $r_i$ .

Information flow during training: A query $q_i$ is concatenated with the special compression tokens → passed through the frozen LLM to obtain hidden states $h_1^i, \dots, h_n^i$ for the compression tokens only → these hidden states are routed through two parallel paths: (Path A) through MLPs + mean pooling to produce $\hat{e}_i$ , compared against $e_i = E(r_i)$ via MSE loss; (Path B) through a separate projection to produce soft prompts $p_1^i, \dots, p_n^i$ , fed back into the frozen LLM which is trained with cross-entropy to generate $r_i$ . Only the compression token embeddings and the two projection layers are updated; the LLM backbone is completely frozen.

Information flow during inference: A query is concatenated with the trained compression tokens → single forward pass through the frozen LLM → extraction of compression token hidden states → through the trained MLPs + mean pooling → output embedding $\hat{e}$ . No response generation, no second forward pass, no teacher invocation.

3.3 Roadmap for the Deep Dive

First, the training data construction: how unlabeled queries are paired with LLM-generated responses, establishing the self-supervision signal and defining what "response space" means in practice.
Second, the compression token mechanism: how special tokens are added to the vocabulary, appended to inputs, and how their hidden states serve as an information bottleneck—this is the core architectural innovation that distinguishes LLM2VEC-GEN from approaches that generate responses explicitly.
Third, the embedding alignment objective and its loss function: what the teacher encodes, how the student matches it, and why MSE rather than contrastive loss—this is where the output-centric semantics are actually transferred.
Fourth, the reconstruction objective: how soft prompts are derived from compression tokens, how the frozen LLM is conditioned on them, and what this objective adds beyond alignment—this is where interpretability and language grounding come from.
Fifth, the training and inference procedures: what gets updated, what stays frozen, the hyperparameters, and the computational cost—this makes the method concrete and reproducible.
Sixth, the design choices behind the teacher selection, response generation, and instruction reformulation—this explains why specific components were chosen over alternatives.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that embeddings should represent the LLM's semantic response rather than the input text's lexical content, and that this can be achieved through a self-supervised distillation process where trainable compression tokens learn to predict the LLM's own response embedding while remaining decodable back into that response.

Training Data Construction: Self-Generated Query-Response Pairs

LLM2VEC-GEN's training requires no human annotations, no curated query-document pairs, and no relevance judgments. The entire supervision signal is bootstrapped from the LLM itself interacting with an unlabeled corpus of queries.

Query corpus. The paper uses 160K single-turn questions from the Tulu instruction-following dataset (Lambert et al., 2025). These are natural language prompts spanning diverse topics—the paper provides examples in Table 9, including instructional requests ("Create a short video tutorial demonstrating how to make french toast"), math word problems, and argumentative prompts ("Write a monologue supporting the idea that employees over the age of 65 should be forced into retirement"). Critically, the original Tulu answers are not used for training—only the queries are taken from the dataset. The ground-truth responses serve merely as a reference point in ablations to assess whether in-distribution responses (from the same model family) outperform out-of-distribution ones. The paper notes in Section 5 that using original Tulu responses yields a slightly lower MTEB-Lite score (67.3) compared to using the model's own responses (67.9), confirming that self-generated responses provide a better training signal.

Response generation. For each query $q_i$ in the training corpus, the frozen backbone LLM $M$ generates a response $r_i$ . The generation is performed once, offline, before embedding training begins—responses are cached and treated as static targets. This is a crucial design choice: the responses are generated by the same model that will later serve as the frozen backbone during embedding training, ensuring representational compatibility. The paper tests alternative response generators in an ablation (Section 5, Table 3): using a stronger in-family model (Qwen-3-8B generating responses for a Qwen-3-4B student) yields MTEB-Lite 67.4 versus 67.9 for self-generated responses, while an out-of-family model (Gemini-3-flash) yields 67.1. The paper hypothesizes that "in-distribution responses are easier to compress by the frozen LLM during training" because the model's internal representations are better aligned with text it generated itself.

Response characteristics. The generated responses reflect the LLM's full behavioral profile: for harmful queries, the model produces safety refusals (e.g., "I'm sorry, but I can't assist with that request. Creating or sharing tutorials on how to commit fraud... is against the law"); for reasoning problems, it produces step-by-step solutions; for factual questions, it produces informative answers. This is the mechanism by which safety and reasoning capabilities transfer to the embedding space—the training targets encode the model's behavior, not just its knowledge. The paper shows examples in Table 4 and Appendix D, Table 9, demonstrating that different backbone models (Qwen-3-8B vs. Llama-3.1-8B) produce characteristically different responses to the same query, and that these differences propagate to the learned embeddings.

Sequence length constraints. Both queries and responses are truncated to a maximum of 512 tokens. This is a practical constraint driven by GPU memory and training efficiency, but it also defines the information capacity that the compression tokens must handle. Responses exceeding 512 tokens are truncated, meaning the embedding cannot capture content beyond this window. The paper does not explore the effect of varying this limit.

Compression Token Mechanism: The Information Bottleneck

The core architectural innovation of LLM2VEC-GEN is the use of trainable special tokens appended to the input sequence that serve as a learned information bottleneck between the query and the LLM's response. This mechanism enables output-centric embeddings without generating text at inference time.

Vocabulary extension. The paper adds $n$ new tokens to the LLM's vocabulary: $c_1, c_2, \dots, c_n$ . The default number of compression tokens is $n = 10$ (Section 4.1). Each $c_j$ has a trainable embedding vector of the same dimensionality as the LLM's token embedding layer, initialized from scratch (the paper does not specify the initialization scheme, but standard practice would be random initialization or small random values). These are the only trainable parameters in the embedding layer; all original vocabulary embeddings remain frozen.

Input construction. For a query $q_i = (q_i^{(1)}, q_i^{(2)}, \dots, q_i^{(k)})$ consisting of $k$ tokens, the input to the LLM is formed by concatenation (denoted $\oplus$ ):

$x_i = q_i \oplus c_1 \oplus c_2 \oplus \dots \oplus c_n$

where $x_i$ is the full input sequence of length $k + n$ tokens, consisting of the original query tokens followed by the $n$ compression tokens. These compression tokens carry no semantic meaning a priori—they are placeholders that will be optimized during training to extract and store response-relevant information from the query context via self-attention. At the start of training, their embeddings are random; by the end of training, attending to them from the query tokens deposits response-predictive information into their hidden states.

Hidden state extraction. The full sequence $x_i$ is passed through the frozen LLM:

$[h_1^i, h_2^i, \dots, h_n^i] = \text{LLM}(x_i)$

where each $h_j^i \in \mathbb{R}^d$ is the last-layer hidden representation at the position of compression token $c_j$ , and $d$ is the LLM's hidden dimension (e.g., 4096 for Qwen-3-4B). Critically, the hidden states of the query tokens are discarded—only the compression token representations are used downstream. This is what forces the compression tokens to act as a bottleneck: during the forward pass, the query tokens can attend to each other and to the compression tokens (in a causal manner, since the compression tokens appear after the query), but only the compression tokens' final representations are read out. Any information about the response that the model needs to predict must therefore flow into these $n$ vectors.

Bidirectional attention consideration. The paper uses the underlying LLM in its standard causal (unidirectional) configuration—there is no modification to the attention mask. This means query tokens can attend to earlier query tokens but not to the compression tokens (which appear later in the sequence), while compression tokens can attend to all query tokens (since those appear earlier). This is a deliberate simplification compared to LLM2Vec, which explicitly modifies the attention mechanism to be bidirectional. The authors found that the output-centric paradigm produces strong embeddings without requiring bidirectional attention, though this also means the compression tokens cannot pass information "backward" to refine query representations.

Why this design. The compression token mechanism solves three problems simultaneously:

It avoids autoregressive generation at inference time—unlike HyDE, which requires generating full response text and then encoding it.
It produces fixed-length embeddings regardless of response length—10 tokens always map to 10 hidden states, even if the response would have been hundreds of tokens.
It provides a natural mechanism for the reconstruction objective—the same hidden states can be projected into soft prompts and fed back into the LLM, creating a direct pathway from compression to content recovery.

The paper ablates the number of compression tokens in Section 5 (Figure 4), sweeping $n \in \{1, 5, 10, 20, 50, 100\}$ and finding that performance improves from 66.1 to 68.5 MTEB-Lite as $n$ increases from 1 to 10, with diminishing returns thereafter. The choice of $n=10$ balances embedding quality against computational cost (each additional token adds parameters and increases sequence length).

Embedding Alignment Objective: Distilling Response Semantics

The alignment objective is the primary mechanism by which LLM2VEC-GEN learns output-centric representations. It forces the compression tokens' processed representations to match a teacher model's encoding of the LLM's actual generated response.

Teacher model requirements. The paper specifies two criteria for the embedding teacher $E$ (Section 4.1):

It should share the same underlying backbone LLM as the student, ensuring that the representation spaces are compatible—the teacher encodes responses using the same pretrained representational geometry that the student inherits.
It should be trained without labeled data, using only unsupervised objectives (specifically, SimCSE, which applies uniformity regularization by pushing apart random negative examples). This ensures the teacher produces "faithful content representations rather than relevance-biased ones" (Section 3).

The specific teacher used is the unsupervised LLM2Vec variant (BehnamGhader et al., 2024) of the corresponding backbone model. LLM2Vec transforms a decoder-only LLM into an encoder by: (a) replacing the causal attention mask with bidirectional attention, (b) applying masked next-token prediction (MNTP) training on unlabeled text to adapt the model to bidirectional context, and (c) running unsupervised SimCSE contrastive learning to create a sentence embedding space. The resulting model takes an input text, produces token-level hidden states through the bidirectional encoder, and mean-pools them to produce a fixed-length embedding. This teacher is frozen during LLM2VEC-GEN training—it serves only as a target provider.

The authors justify this choice by arguing that SimCSE's lightweight uniformity regularization largely preserves the LLM's local representational geometry, unlike supervised contrastive learning which fundamentally restructures the space around relevance labels (a point they elaborate in Appendix I: when a supervised teacher is used, LLM2VEC-GEN cannot outperform it because the teacher's relevance-optimized representations are not faithful content encodings).

Target embedding construction. For each training query $q_i$ , the teacher encodes the LLM's previously generated response $r_i$ :

$e_i = E(r_i)$

where $e_i \in \mathbb{R}^{d_E}$ and $d_E$ is the teacher's output embedding dimension (which may differ from the student LLM's hidden dimension $d$ ). This vector $e_i$ represents what the teacher model "understands" the response to mean—it captures the response's semantic content within the teacher's representational space. Since the teacher shares the same backbone pretraining as the student, this representation reflects the LLM's own semantic organization, not an externally imposed relevance structure.

Student embedding construction. The compression token hidden states $h_1^i, \dots, h_n^i$ (from the single forward pass described above) are processed through two projection layers:

$\hat{e}_i = \text{Pool}\left(\text{MLP}_2\left(\text{MLP}_1\left(h_1^i, \dots, h_n^i\right)\right)\right)$

where:

$\text{MLP}_1$ is a fully connected layer with input dimension $d$ (the LLM's hidden size) and output dimension $d$ (same size—it preserves dimensionality, acting as a learned transformation within the LLM's space).
$\text{MLP}_2$ is a fully connected layer with input dimension $d$ and output dimension $d_E$ (the teacher's embedding dimension). This layer bridges the dimensionality gap when the teacher's output dimension differs from the LLM's hidden dimension.
Both MLPs consist of a single linear transformation (the paper does not mention intermediate nonlinearities or hidden layers—these are "one layer each" as stated in Appendix C).
$\text{Pool}$ is mean pooling over the $n$ projected representations, producing the final student embedding $\hat{e}_i \in \mathbb{R}^{d_E}$ .

The mean pooling operation is standard in sentence embedding literature (Reimers & Gurevych, 2019) and ensures permutation invariance over the compression tokens—the order in which they appear doesn't affect the final embedding, only the aggregate information they collectively encode.

The alignment loss. The alignment objective is a simple mean squared error (MSE) between the student's predicted embedding and the teacher's target embedding:

$\mathcal{L}_{\text{align}} = \lVert e_i - \hat{e}_i \rVert^2$

where $\lVert \cdot \rVert^2$ denotes the squared Euclidean (L2) norm.

What it computes. For each training example, this loss computes the element-wise squared difference between two vectors in the teacher's embedding space: the teacher's representation of the actual LLM-generated response ( $e_i$ ) and the student's representation of what it predicts the response would be ( $\hat{e}_i$ ), derived solely from the query via the compression token bottleneck. The loss is a single non-negative scalar per example—zero when the two vectors are identical and growing quadratically with their Euclidean distance.

Why MSE and not contrastive loss. This is a critical design choice that the paper explicitly contrasts with "standard contrastive learning" in Section 3. Contrastive losses (e.g., InfoNCE, triplet loss) map queries and documents into a new shared latent space using relative relevance judgments: they pull positive pairs together and push negative pairs apart. This fundamentally restructures the embedding geometry around the training distribution's notion of relevance. MSE distillation, by contrast, tells the student to directly match the teacher's absolute representation of the response. The student inherits the teacher's representational geometry rather than constructing a new one. This matters because the teacher's unsupervised SimCSE training applies only light uniformity regularization ("pushing apart random negatives") without introducing relevance biases—the alignment target faithfully represents the response content. Minimizing MSE preserves this content-faithfulness, whereas a contrastive objective would inject discriminative biases based on which negatives were sampled.

The paper provides an additional theoretical framing in Section 3, connecting the alignment objective to Joint Embedding Predictive Architectures (JEPAs): "LLM2VEC-GEN predicts a target representation of the model's likely response via external teacher distillation, while the reconstruction objective keeps the learned representations grounded in natural language." In this view, the compression tokens are learning to predict the future (the response's representation) from the present (the query), but they do so in representation space rather than token space—the reconstruction loss then provides the grounding that connects these representations back to concrete language.

What this objective achieves. By minimizing $\mathcal{L}_{\text{align}}$ , the compression tokens learn to extract from the query whatever information is necessary to predict how the LLM would respond—because the target $e_i$ is an encoding of the actual response. For a malicious query, this means learning to predict the safe refusal embedding; for a reasoning query, learning to predict the reasoned conclusion embedding; for a factual query, learning to predict the answer's embedding. The query itself may contain none of these response semantics explicitly—the compression tokens must learn to trigger the LLM's internal generative knowledge and capture it in their hidden states, effectively running the generation process "in latent space" rather than through autoregressive token sampling.

Reconstruction Objective: Grounding Embeddings in Natural Language

While the alignment objective ensures the embeddings capture response-space semantics, the reconstruction objective ensures those embeddings remain interpretable and decodable—that the LLM can actually recover the response text from the compression tokens' representations.

Motivation. The alignment loss alone optimizes the compression tokens to produce vectors that match the teacher's response embedding. However, there is no guarantee that these vectors correspond to any sequence of tokens the LLM could actually generate—they might encode the response concept in an abstract, non-linguistic form that the teacher's metric space recognizes but that bears no decodable relationship to natural language. The reconstruction objective forces the compression tokens to retain sufficient token-level information that the LLM can autoregressively regenerate the response. This serves as a form of regularization: the embeddings cannot drift into an opaque, teacher-specific subspace; they must remain within the LLM's natural language manifold.

Soft prompt construction. The compression token hidden states $h_1^i, \dots, h_n^i$ are projected through a separate projection layer (distinct from the alignment MLPs) to produce a sequence of soft prompt vectors:

$(p_1^i, p_2^i, \dots, p_n^i) = \text{Proj}_{\text{recon}}(h_1^i, h_2^i, \dots, h_n^i)$

where each $p_j^i \in \mathbb{R}^d$ is a vector in the LLM's hidden dimension $d$ , and $\text{Proj}_{\text{recon}}$ is a single linear layer. Each $p_j^i$ corresponds positionally to one compression token, but instead of representing an embedding lookup, it directly provides the hidden-state input to the LLM's transformer layers at that position. These are called "soft prompts" because they bypass the token embedding layer—they feed directly into the transformer stack as if they were the hidden states produced by the embedding layer for that position.

The second forward pass. The soft prompts are fed back into the same frozen LLM as a prefix for a second forward pass. Specifically, the soft prompts $(p_1^i, \dots, p_n^i)$ are placed at the first $n$ positions of a new sequence, followed by the tokens of the response $r_i$ . The LLM is conditioned on the soft prompts as context and trained to predict the response tokens autoregressively via standard next-token prediction:

$\mathcal{L}_{\text{recon}} = -\sum_{j=1}^{|r_i|} \log P_{\text{LLM}}\left(r_{i,j} \mid p_1^i, \dots, p_n^i, r_{i, <j}\right)$

where $r_{i,j}$ is the $j$ -th token of response $r_i$ , $r_{i, <j}$ denotes all preceding tokens, and $P_{\text{LLM}}$ is the frozen LLM's predicted probability distribution over the vocabulary at position $j$ .

What it computes. For each position $j$ in the response sequence, the LLM receives as input (a) the $n$ soft prompt vectors derived from the compression tokens, which encode a compressed representation of the query and its expected response, and (b) the first $j-1$ tokens of the actual response, serving as autoregressive context. The model produces a probability distribution over its vocabulary for the $j$ -th token. The loss is negative log-likelihood of the correct token $r_{i,j}$ under this distribution, summed over all positions. The total $\mathcal{L}_{\text{recon}}$ is a scalar representing the total surprisal of the response tokens given the soft prompt prefix.

Why this form. This is a standard language modeling (causal LM) loss, chosen because it directly measures how well the soft prompts enable the LLM to recover the response text. The key insight is that the frozen LLM's language modeling capability serves as a built-in decoder—no separate decoder network needs to be trained. Since the LLM can generate text, conditioning it on a prefix and measuring its ability to continue correctly provides a direct readout of how much information about the response is preserved in that prefix. The cross-entropy form (negative log-likelihood) is the proper scoring rule for categorical distributions, making it the statistically natural choice for token-level prediction.

An alternative approach would be to train an entirely separate decoder to map from the compression token representations back to text, but this would (a) add many more parameters, (b) require the decoder to learn language modeling from scratch rather than leveraging the pretrained LLM, and (c) break the tight coupling between the embedding space and the LLM's own generative knowledge that makes the embeddings interpretable through the LLM itself.

The information bottleneck interpretation. The soft prompt prefix $(p_1^i, \dots, p_n^i)$ provides only $n \times d$ real values of conditioning information to the LLM—for $n=10$ and $d=4096$ (Qwen-3-4B), this is 40,960 floating-point numbers. The response $r_i$ can be up to 512 tokens, each drawn from a vocabulary of typically 100K+ tokens. The LLM must recover the correct sequence from this limited prefix. This forces the compression tokens to learn a compressed representation of the response content—not a lossless encoding, but a semantic summary that captures the information necessary for the LLM to reconstruct a plausible version of its own response. The paper's qualitative examples in Table 4 confirm this: decoded outputs are coherent, topically accurate paraphrases of the LLM's original response, not verbatim reproductions.

Necessity for interpretability. The paper demonstrates in Appendix Table 13 that a model trained with only $\mathcal{L}_{\text{align}}$ produces high-quality embeddings but completely nonsensical decoded outputs. For the query "what is artificial intelligence," the $\mathcal{L}_{\text{align}}$ -only variant generates a rambling monologue about number theory combinatorics that has nothing to do with the query. The full model ( $\mathcal{L}_{\text{align}} + \mathcal{L}_{\text{recon}}$ ) generates a coherent definition of AI. This confirms that the reconstruction objective is not primarily driving embedding quality (the ablation in Table 3 shows MTEB-Lite only drops from 67.9 to 67.5 when removing $\mathcal{L}_{\text{recon}}$ )—its primary role is to keep the embeddings grounded in the LLM's language manifold, making them interpretable and decodable.

Why not reconstruction alone. The ablation in Table 3 shows that training with only $\mathcal{L}_{\text{recon}}$ collapses MTEB-Lite performance to 43.1 (from 67.9 with both objectives). This is because the reconstruction objective alone optimizes the compression tokens to be good soft prompts for response generation—they learn to condition the LLM to produce the correct text—but there is no signal pushing them toward a semantic representation that would match the teacher's embedding space. The alignment objective provides exactly this semantic grounding: it pulls the compression tokens toward representations that the teacher recognizes as encoding the response content. Without it, the compression tokens may encode arbitrary features that enable reconstruction (e.g., shallow lexical cues, positional patterns) without capturing deep semantic structure.

The Combined Training Objective

The final loss is an unweighted sum of the two components:

$\mathcal{L} = \mathcal{L}_{\text{align}} + \mathcal{L}_{\text{recon}}$

where $\mathcal{L}_{\text{align}}$ is the mean squared error between student embedding and teacher embedding of the response, and $\mathcal{L}_{\text{recon}}$ is the autoregressive cross-entropy loss for reconstructing the response from the soft prompts.

What it computes. A single scalar per training example that combines a representation-level signal (how close is the predicted embedding to the teacher's response embedding?) and a token-level signal (how well can the LLM recover the response text from the compression tokens?). Both losses are computed from the same set of compression token hidden states $h_1^i, \dots, h_n^i$ but routed through different projection layers: the first set ( $\text{MLP}_1, \text{MLP}_2$ ) for alignment, the second ( $\text{Proj}_{\text{recon}}$ ) for reconstruction. The gradients from both objectives flow back into the compression token embeddings and both sets of projection parameters, while the LLM backbone remains completely frozen.

Why no loss weighting hyperparameter. The paper uses equal weighting (coefficient of 1.0 for both terms) without sweeping or tuning $\lambda$ values. This is unusual—most multi-objective methods introduce a balancing hyperparameter—and the paper does not discuss the rationale. A plausible explanation is that the two objectives operate on fundamentally different scales and in different spaces: $\mathcal{L}_{\text{align}}$ is on the order of the squared Euclidean distance between normalized embedding vectors (typically small values), while $\mathcal{L}_{\text{recon}}$ is a sum of per-token cross-entropies (typically larger values, scaling with response length). The gradients from each loss update disjoint parameter sets (different projection layers, shared compression token embeddings), reducing interference. The ablation results suggest this works well in practice, but the absence of weighting analysis is a minor methodological gap.

Training and Inference Procedures

What gets updated. During training, gradients flow to three parameter groups only:

The embedding vectors for the $n$ compression tokens $c_1, \dots, c_n$ —these are the token embeddings in the LLM's embedding layer that are looked up when these tokens appear in the input. They start random and learn to trigger the LLM's internal computation in ways that deposit response-relevant information into the subsequent hidden states.
The alignment projection layers $\text{MLP}_1$ (dimension $d \to d$ ) and $\text{MLP}_2$ (dimension $d \to d_E$ )—these learn to map from the LLM's internal hidden state to the teacher's embedding space.
The reconstruction projection layer $\text{Proj}_{\text{recon}}$ (dimension $d \to d$ )—this learns to transform the compression token hidden states into soft prompts that condition the LLM for response reconstruction.

Everything else—the LLM's transformer layers, attention weights, feedforward networks, output language modeling head, original vocabulary embeddings, positional encodings—remains frozen. For Qwen-3-4B, this amounts to training 13M parameters out of a total of 4B parameters (Appendix C), approximately 0.3% of the model.

Why freeze the LLM. The paper identifies two motivations:

Deployment simplicity: the same model weights serve for both embedding and generation. If the LLM were fine-tuned (even with LoRA), separate model instances would be needed for each task or the generation capability might be degraded. By keeping the backbone frozen, a single model checkpoint can produce both text generations and embeddings.
Preserving pretrained knowledge: the LLM's internal representations encode the full breadth of its pretraining—reasoning patterns, factual associations, safety alignment, and linguistic knowledge. Fine-tuning, even with limited parameters, risks distorting these representations, potentially degrading the very response-space semantics that LLM2VEC-GEN aims to capture. The compression tokens learn to query the frozen LLM for response-relevant information rather than altering how the LLM processes language.

The paper does experiment with LoRA variants in Table 3, finding that LoRA ( $r=8$ , $\alpha=16$ ) achieves a higher MTEB-Lite score (68.3 vs. 67.9) but that increasing LoRA capacity ( $r=32$ , $\alpha=64$ ) reduces performance (67.6). However, even the better-performing LoRA variant requires maintaining separate adapter weights, breaking the single-model deployment advantage.

Training hyperparameters. The paper reports:

Optimizer: AdamW (Loshchilov & Hutter, 2019)
Learning rate: $3 \times 10^{-4}$ for Qwen-3 models, $5 \times 10^{-4}$ for Qwen-2.5 and Llama models
Learning rate schedule: Linear decay with 100 warmup steps
Batch size: 32
Training epochs: 1 epoch over 160K samples (5,000 steps)
Maximum sequence length: 512 tokens for both queries and responses (truncation applied if exceeded)
Precision: Mixed-precision training with bfloat16
Hardware: 2 NVIDIA H100 GPUs (80GB each)
Training time: Approximately 3.5 hours for Qwen-3-8B

The learning rate is relatively high (3e-4) compared to typical LLM fine-tuning, reflecting that the trainable parameters are small in number and randomly initialized, requiring substantial gradient steps to converge. The single-epoch training suggests the method does not overfit on 160K examples—the compression tokens have sufficient capacity to capture generalizable patterns rather than memorizing individual query-response pairs.

Inference procedure. At inference time, the pipeline is simplified to a single forward pass:

Append the trained compression tokens to the query: $x = q \oplus c_1 \oplus \dots \oplus c_n$ .
Run the frozen LLM on $x$ .
Extract the last-layer hidden states at the compression token positions: $h_1, \dots, h_n$ .
Apply the trained alignment MLPs and mean pooling: $\hat{e} = \text{Pool}(\text{MLP}_2(\text{MLP}_1(h_1, \dots, h_n)))$ .
Return $\hat{e}$ as the embedding.

There is no response generation, no second forward pass for reconstruction, no teacher invocation. The cost is identical to encoding an input sequence of length $|q| + 10$ through the LLM, which is marginally more expensive than encoding just the query (10 extra token positions) but dramatically cheaper than generating and then encoding a response (which would require autoregressive sampling of potentially hundreds of tokens plus a separate encoding pass).

The reconstruction pathway ( $\text{Proj}_{\text{recon}}$ ) can optionally be used at inference time to decode the embedding back into text for interpretability analysis (as demonstrated in Table 4), but this is not part of the standard embedding pipeline and is not required for any evaluation metric.

Design Choices: Teacher Selection, Response Generation, and Instruction Reformulation

Several non-obvious design decisions underpin the method's effectiveness, and the paper's ablation studies (Section 5, Table 3, and Appendices G-I) provide empirical justification for each.

Why unsupervised rather than supervised teacher. The paper explicitly tests this in Appendix I (Table 10). When an unsupervised LLM2Vec teacher is used, LLM2VEC-GEN improves over the teacher by 6.9%, 6.7%, and 8.8% for Qwen-3-1.7B, 4B, and 8B respectively. When a supervised LLM2Vec teacher (trained on Echo paired data) is used, LLM2VEC-GEN still improves over its own initial state but cannot surpass the supervised teacher itself—it reaches 61.2 vs. the teacher's 63.0. The paper attributes this to a fundamental mismatch: supervised encoders are trained to optimize for relative relevance judgments on their training distribution, producing embeddings that are discriminative rather than content-faithful. Distilling from such a teacher forces the student into a relevance-optimized geometry that doesn't faithfully represent the LLM's response semantics. The unsupervised teacher's SimCSE objective, by contrast, applies only light uniformity regularization—pushing random negatives apart to prevent collapse—while largely preserving the LLM's rich representational structure. This makes the distillation target genuinely about response content rather than about which documents are relevant to a particular training distribution.

The paper also tests whether LLM2VEC-GEN's benefits are specific to the LLM2Vec teacher or generalize across teacher families. Appendix G (Figure 7) shows that when trained with the BGE-M3-unsupervised teacher, LLM2VEC-GEN still consistently outperforms the teacher across Qwen-3 model sizes, "suggesting that the output-centric paradigm generalizes robustly across different unsupervised embedding teachers." However, the absolute scores are lower than with the same-backbone LLM2Vec teacher, confirming that representational compatibility between teacher and student matters.

Why self-generated rather than original Tulu responses. The ablation in Table 3 shows that using the original Tulu dataset responses (written by human annotators or other models) yields MTEB-Lite 67.3 versus 67.9 for the model's own generated responses. The paper hypothesizes that "in-distribution responses are easier to compress by the frozen LLM during training." The frozen LLM's internal representations are optimized for text it has generated or could generate—its autoregressive dynamics and representational geometry are tuned to its own output distribution. Compressing out-of-distribution text (written by a different model or human) through the frozen LLM's representations is a harder task because the text may use vocabulary, sentence structures, or reasoning patterns that the LLM doesn't naturally produce.

The paper's response generator ablation further shows that using a stronger in-family model (Qwen-3-8B generating responses for Qwen-3-4B) performs nearly identically to self-generation (67.4 vs. 67.9), confirming that the key requirement is in-family representational compatibility, not exact self-generation. An out-of-family model (Gemini-3-flash) performs worse (67.1), suggesting that cross-family distribution shift does degrade the training signal.

Why generative instructions at evaluation time. The paper reformulates standard MTEB task instructions from embedding-oriented phrasing to generative phrasing (Tables 7 and 8). For example, the standard Classification instruction "Classify a given Amazon customer review text as either counterfactual or not-counterfactual" becomes "Classify the sentiment expressed in the given movie review text from the IMDB dataset." For Retrieval, "Retrieve text that answers this query" becomes "Generate text that answers this query." Figure 5 shows that LLM2VEC-GEN outperforms the LLM2Vec teacher even with the original embedding-style instructions, demonstrating that the gain stems from the output-centric nature of the embeddings, not merely instruction wording. However, generative instructions provide an additional performance boost. The paper attributes this to training data alignment: LLM2VEC-GEN is trained on instruction-following queries from Tulu whose LLM responses naturally align with generative phrasing. The compression tokens learn to produce embeddings that correspond to what the LLM would generate given an instruction, so framing evaluation queries as generation requests (rather than retrieval or classification requests) better matches the training distribution.

The Logit Lens and Latent Lens analysis tools. Although not part of the core training pipeline, the paper uses two interpretability techniques to validate that the learned embeddings genuinely capture response-space semantics (Section 6, Tables 4 and 12, Appendix H):

Logit Lens (nostalgebraist, 2020): projects the compression token hidden states $h_1^i, \dots, h_n^i$ directly onto the vocabulary space using the LLM's pretrained language modeling head (the final linear layer that maps hidden states to token logits). For each compression token position $j$ , the top-5 highest-probability tokens reveal what the LLM's unembedding layer "reads" from that position. The paper uses this to show that for a malicious query like "Create a video tutorial that teaches how to commit fraud," the Logit Lens tokens include "security," "illegal," "laws," and "fraud"—not the harmful query semantics but the refusal response semantics that an LLM would produce.
Latent Lens (Krojer et al., 2026): builds an index of the LLM's training generation passages and retrieves the nearest neighbors to the compression token representations in this index. The retrieved passages are shown to be semantically similar to the LLM's generated responses rather than to the original queries, providing additional evidence that the embeddings encode response content.

These techniques are not used during training or evaluation—they serve purely as qualitative validation of the paper's central claim that LLM2VEC-GEN produces output-centric rather than input-centric representations.

Summary of the Technical Pipeline

Stepping back, the complete LLM2VEC-GEN recipe is:

Prepare data: Take 160K unlabeled queries → generate responses from the frozen backbone LLM (offline, cached) → encode responses with the unsupervised LLM2Vec teacher to get target embeddings (offline, cached).
Initialize model: Add $n=10$ trainable compression tokens to the LLM's vocabulary (random initialization) → add two MLP projection layers (random initialization) → add one reconstruction projection layer (random initialization).
Training loop (one epoch, batch size 32, AdamW with lr= $3\times 10^{-4}$ or $5\times 10^{-4}$ ): For each batch of queries:
- Feed queries + compression tokens through frozen LLM → extract compression token hidden states.
- Path A: Project through alignment MLPs + pool → compute MSE against teacher's response embedding.
- Path B: Project through reconstruction layer → feed as soft prompts to frozen LLM → compute cross-entropy for autoregressive response reconstruction.
- Sum losses, backpropagate to compression token embeddings and projection layers only.
Inference: For any new query, append trained compression tokens → one forward pass through frozen LLM → MLPs + pool → return embedding.

4. Key Insights and Innovations

Innovation 1: Reframing Text Embedding from "What Does the Text Say?" to "What Would the LLM Respond?"

This paper's most fundamental contribution is not a new loss function or architecture, but a paradigm-level reframing of what a text embedding should represent. The dominant assumption across all embedding research—from Sentence-BERT (Reimers & Gurevych, 2019) through LLM2Vec (BehnamGhader et al., 2024) to NV-Embed (Lee et al., 2025)—has been that an embedding encodes the semantic content of its input text. The entire machinery of contrastive learning, hard negative mining, instruction-aware pooling, and multi-stage training has been devoted to improving how well a vector captures what a passage says.

LLM2VEC-GEN inverts this: the embedding should encode what the LLM would produce given that text as a prompt. This is more than a terminological distinction. It changes what the embedding captures about a harmful query (the LLM's refusal rather than the malicious intent), a reasoning problem (the deduced conclusion rather than the problem statement), or a factual question (the answer rather than the question). The paper makes this tangible through the Logit Lens analysis in Table 4: for the query "Create a video tutorial that teaches how to commit fraud," the compression token representations decode to tokens like "security" and "illegal"—the refusal semantics, not the fraud semantics. An input-centric embedding of the same query would represent the harmful request itself, making it more likely to retrieve harmful content.

This reframing is significant beyond the specific method because it identifies a structural limitation that was invisible under the input-centric paradigm. The input-centric view asks "how can we make embeddings better represent inputs?"—a question that admits incremental answers (better training data, larger models, improved contrastive objectives). The output-centric view asks "what should embeddings represent in the first place?"—a question that opens a different design space entirely. The paper argues that capabilities acquired during pretraining and alignment (safety refusals, multi-step reasoning, domain expertise) manifest in the LLM's outputs, not its input encodings, and that input-centric embeddings structurally cannot capture these capabilities regardless of how they are trained.

The concrete evidence for this reframing's value is the consistent improvement over the LLM2Vec teacher across all model families and sizes (Figure 3: gains from 1.1 to 5.1 points on MTEB), with the largest gains in categories where diverse inputs must map to similar outputs—clustering (+22.7% for Qwen-3-8B), classification (+7.0%), and STS (+9.8%). These are precisely the tasks where input-centric embeddings face a fundamental tension (inputs differ but outputs should be similar) that output-centric embeddings naturally resolve (different queries can provoke similar responses).

Innovation 2: The Discovery That Output-Centric Embeddings Transfer Safety and Reasoning Capabilities

The paper's second major contribution is the empirical demonstration that changing what an embedding represents changes what capabilities it inherits from the underlying LLM. This is not an obvious consequence of the output-centric paradigm—it required specific experiments on safety and reasoning benchmarks to establish.

Safety transfer. On AdvBench-IR (Table 2), LLM2VEC-GEN consistently reduces unsafe retrieval compared to the input-centric LLM2Vec teacher across all model sizes—a 22.6% reduction for Qwen-3-1.7B, 16.3% for Qwen-3-4B, and 17.0% for Qwen-3-8B. What makes this finding significant is the mechanism: the safety improvement does not come from any safety-specific training, adversarial filtering, or harmfulness labeling. It emerges solely from the fact that the LLM generates safe refusals to harmful queries during training data construction, and the alignment objective distills these refusal representations into the embedding space. The embedding model inherits the LLM's safety alignment not because it was trained to be safe, but because it was trained to represent what the LLM would say—and the LLM says safe things.

This has practical implications that go beyond the measured benchmark scores. In retrieval-augmented generation systems, the retriever and the generator are typically separate models with independent safety properties: a safe generator (trained with RLHF) can be paired with an unsafe retriever (that surfaces harmful documents when given adversarial queries). Input-centric approaches can only address this by explicit safety fine-tuning of the retriever—requiring labeled harmful/safe data and potentially introducing new failure modes. LLM2VEC-GEN offers a different path: the retriever inherits the generator's safety automatically because both are based on what the same LLM would produce. The retriever and generator become representationally aligned by construction.

Reasoning transfer. On BRIGHT (Table 2), a benchmark designed to require logical inference for query-document relevance matching, LLM2VEC-GEN improves over the teacher by 7.7% (0.6B), 11.7% (1.7B), 19.7% (4B), and 35.6% (8B). The scaling behavior is particularly revealing: the improvement grows with model size, tracking the underlying LLM's increasing reasoning capability. Larger models reason better, and LLM2VEC-GEN transfers more of that reasoning into the embedding space. This is in sharp contrast to standard MTEB retrieval, where one model size (Qwen-3-4B) shows a marginal decline—suggesting that output-centric embeddings are specifically beneficial when retrieval demands deeper semantic understanding beyond surface-level lexical matching, and that this benefit scales with the backbone LLM's capabilities.

The significance of this finding extends to how we think about model scaling for retrieval. If embedding quality can inherit capabilities from the underlying LLM without explicit task-specific training, then improvements to the base LLM (better reasoning, broader knowledge, stronger safety alignment) automatically improve the derived embeddings. This creates a different scaling dynamic than training standalone embedding models, where each capability improvement requires new training data and procedures.

Innovation 3: The Reconstruction Objective as an Interpretability Mechanism, Not a Performance Driver

The paper makes an unusual and intellectually honest move by demonstrating that one of its two training objectives—response reconstruction—is not the primary driver of embedding quality, but rather serves a distinct purpose: interpretability through decodability. The ablation in Table 3 shows that removing $\mathcal{L}_{\text{recon}}$ drops MTEB-Lite from 67.9 to 67.5—a negligible 0.4-point decline—while removing $\mathcal{L}_{\text{align}}$ collapses performance to 43.1. The reconstruction objective contributes almost nothing to standard embedding benchmarks.

Yet the paper retains it as a core component and argues for its importance. Why? Because the reconstruction objective ensures the embeddings remain grounded in natural language. Table 13 provides the critical evidence: a model trained with only $\mathcal{L}_{\text{align}}$ produces embeddings that score well on MTEB but decode into completely nonsensical text (for "what is artificial intelligence," it generates a rambling combinatorics proof). The $\mathcal{L}_{\text{align}} + \mathcal{L}_{\text{recon}}$ model decodes into a coherent definition of AI.

This is significant because it reveals a decoupling between embedding quality and interpretability that was previously invisible. The field has largely treated embedding quality (measured by downstream task performance) and interpretability (measured by whether humans can understand what the embedding represents) as correlated—better embeddings should be more interpretable because they better capture "meaning." LLM2VEC-GEN demonstrates that these properties can be orthogonal: the alignment objective alone produces vectors that the teacher's metric space recognizes as semantically correct but that bear no recoverable relationship to natural language tokens. The reconstruction objective adds a constraint that forces the compression tokens to encode information in a form the LLM's autoregressive mechanism can decode, creating a direct pathway from vector to text.

This finding has implications for the broader interpretability literature. It suggests that representation quality in a metric space does not guarantee representational transparency—models can learn to satisfy a teacher's distance metric through opaque, non-linguistic features. The reconstruction objective in LLM2VEC-GEN is essentially a language-grounding regularizer: it constrains the embedding to live within the LLM's natural language manifold, where decoding reveals semantic content. This is a specific, mechanistic instantiation of a general principle that could apply to other representation learning contexts where interpretability is valued.

Innovation 4: Self-Supervision Entirely Through the LLM's Own Generative Capabilities

LLM2VEC-GEN achieves state-of-the-art self-supervised MTEB performance using a training signal that is fully bootstrapped from the frozen LLM itself: the LLM generates the response targets, an unsupervised encoder of the same LLM provides the embedding targets, and the frozen LLM provides the reconstruction signal through its language modeling head. There is no external annotation, no curated query-document pairs, no human feedback, and no data source beyond unlabeled queries.

This is more than an efficiency claim. It represents a closed-loop self-supervision paradigm for embedding model training. Prior self-supervised methods (LLM2Vec, SimCSE, Echo Embeddings) derive their training signal from properties of the input text itself—next-token prediction, sentence-level contrast, or input repetition. LLM2VEC-GEN derives its signal from the LLM's behavior on that text. The alignment target $e_i$ is not a property of the query or even of the query-response pair in some external sense—it is specifically the LLM2Vec teacher's encoding of the LLM's own generated response. Every component of the supervision comes from the model interacting with itself: the LLM talks to itself (generating responses), then its encoder twin listens (producing embeddings of those responses), and the student learns to predict what its twin heard.

The practical consequence is that LLM2VEC-GEN can be applied to any domain with unlabeled queries and a pretrained LLM—no domain-specific annotations are needed. For specialized scientific retrieval, legal document search, or low-resource language embedding, the only requirement is a corpus of queries (which can often be obtained from search logs or user interactions) and a suitable LLM. The ablation in Table 3 confirms that this self-supervision works best with self-generated responses (67.9 MTEB-Lite) vs. original Tulu annotations (67.3), suggesting that the bootstrapped signal is not just sufficient but actually preferable to human-written targets—the LLM compresses its own outputs more effectively because they lie within its representational distribution.

However, this innovation also defines a fundamental boundary: LLM2VEC-GEN cannot exceed the capabilities of its underlying LLM. If the LLM generates poor responses to certain query types, the embeddings will inherit those limitations. If the LLM lacks knowledge in a domain, the embeddings cannot compensate. The paper acknowledges this in Appendix A: "The quality of the resulting embeddings is therefore bounded by the teacher's representational capacity; if the teacher poorly encodes certain response types, the student inherits those limitations." This distinguishes LLM2VEC-GEN from supervised methods, which can potentially learn embedding spaces that surpass the base model's generative capabilities through exposure to high-quality labeled data. The self-supervised paradigm is powerful but bounded by the LLM's existing knowledge and behavioral profile—it amplifies what the model already knows rather than teaching it new distinctions.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary benchmark is MTEB(eng, v2) (Enevoldsen et al., 2025), comprising 41 tasks across seven categories: bitext mining, classification, clustering, pair classification, reranking, retrieval, and semantic textual similarity (STS). For ablations, the authors construct MTEB-Lite, a subset of 10 tasks that preserves the category distribution of the full benchmark (Table 6). Two additional benchmarks are used for capability-specific evaluation: AdvBench-IR (BehnamGhader et al., 2025), containing 520 harmful queries across five harm categories evaluated against a corpus of 1,796 passages (harmful content plus Wikipedia passages), and BRIGHT (Su et al., 2025), a reasoning-intensive retrieval benchmark spanning biology, coding, math, and physics domains where relevance requires logical deduction. All benchmarks use standard zero-shot evaluation protocols.
Base model(s). The paper applies LLM2VEC-GEN to decoder-only LLMs from three model families: Qwen-3 (0.6B, 1.7B, 4B, 8B; Yang et al., 2025), Qwen-2.5-Instruct (0.5B, 1.5B, 3B, 7B; Qwen et al., 2025), and Llama-3.1/3.2 (1B, 3B, 8B; Meta, 2024a,b). The backbone LLMs span an order of magnitude in scale and cover two major architectural families, testing whether the method generalizes across model sizes and architectures. These models are chosen to be "representative" of contemporary LLM capabilities while spanning a range where the unsupervised LLM2Vec teacher quality varies substantially.
Metrics. On MTEB, the primary metric is the averaged score across all 41 tasks following the benchmark's standard per-category normalization. Individual category scores (Retrieval nDCG@10, STS Spearman correlation, etc.) are reported in Table 1 to enable fine-grained analysis. On AdvBench-IR, the metric is top-5 accuracy (lower is safer—fewer retrieved harmful passages). On BRIGHT, the metric is nDCG@10 for zero-shot retrieval. For capability-specific benchmarks, relative improvement percentages are computed against the corresponding LLM2Vec teacher.
Baselines. The paper compares against five categories of baselines, selected to span the space of input-centric and output-aware approaches: (1) Echo Embeddings (Springer et al., 2025), a zero-shot method that repeats the input and extracts embeddings from the second occurrence; (2) HyDE (Gao et al., 2023), which generates hypothetical answer documents at inference time and encodes them with an unsupervised model; (3) InBedder (Peng et al., 2024), which fine-tunes LLMs on abstractive QA data and derives embeddings from the first generated token's hidden state (the paper reimplements this with LoRA, $r=32, \alpha=64$ , on the same abstractive QA dataset as the original); (4) GIRCSE (Tsai et al., 2026), originally a supervised method, adapted by the authors to a self-supervised setting using Tulu queries and each model's own responses for fair comparison; (5) LLM2Vec (BehnamGhader et al., 2024), the unsupervised embedding teacher itself, serving as both the primary baseline and the teacher model. All baselines except Echo and HyDE require training; HyDE additionally requires autoregressive generation at inference time, making it computationally distinct from methods that produce embeddings in a single forward pass.
Generation budget / compute accounting. The paper does not employ a generation budget framework of the type found in test-time compute scaling studies, since LLM2VEC-GEN performs a single forward pass at inference time regardless of query difficulty. Training cost is reported in GPU-hours (approximately 3.5 hours on 2 H100 GPUs for Qwen-3-8B over 160K samples) and parameter count (13M trainable parameters for Qwen-3-4B, approximately 0.3% of the backbone). For HyDE, the paper notes that generating multiple answers at inference time "incur[s] substantial computational overhead" but does not quantify this cost relative to LLM2VEC-GEN.
Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. Strategy selection for the compute-optimal policy (unlike the test-time scaling paper) is not a component of LLM2VEC-GEN's evaluation. The main results are reported as single-point estimates on the full MTEB test set. For the supervised extensions in Appendix I (Table 10), different training data configurations (Echo dataset, hard negatives, LoRA variants) are compared without cross-validated hyperparameter selection. The MTEB-Lite subset used for ablations is fixed a priori to preserve category distribution; ablation conclusions are drawn from comparisons on this 10-task subset rather than the full 41-task benchmark.

Main Quantitative Results

General Text Embedding Performance on MTEB

The headline result is that LLM2VEC-GEN achieves state-of-the-art self-supervised performance on MTEB across all tested model families and sizes. Table 1 presents detailed results for the Qwen-3 family at 1.7B, 4B, and 8B scales. At the largest scale, LLM2VEC-GEN-Qwen-3-8B reaches 61.9 on MTEB(eng, v2), improving over the LLM2Vec teacher (56.8) by 8.8% and substantially outperforming all other self-supervised baselines: GIRCSE (self-sup, 56.5), InBedder (50.5), HyDE (48.3), and Echo (41.8).

Breaking this down by MTEB category for the 8B model, LLM2VEC-GEN shows the largest relative gains in clustering (+22.7%, from 40.6 to 49.8), reranking (+13.4%, from 40.9 to 46.4), and STS (+9.8%, from 72.6 to 79.7). Classification improves by 7.0% (72.5 to 77.6), while retrieval shows a modest gain of 1.4% (42.7 to 43.3). The paper explicitly notes that retrieval is the category where output-centric embeddings show the weakest advantage, and for one model size (Qwen-3-4B), there is a marginal decline of 1.3 points in retrieval (41.1 to 39.8) despite gains in all other categories (Table 1).

At smaller scales, the pattern holds: Qwen-3-1.7B improves from 54.8 (LLM2Vec) to 58.6 (+6.9%), and Qwen-3-4B improves from 56.8 to 60.6 (+6.7%). The clustering gains are consistently the largest across scales (+21.3% at 1.7B, +17.7% at 4B), while retrieval shows the weakest improvement at 1.7B (+9.7%) and a decline at 4B (-3.1%).

Cross-family generalization. Figure 3 extends these results to Qwen-2.5 and Llama-3.1/3.2 families, showing that LLM2VEC-GEN consistently outperforms the LLM2Vec teacher across all model sizes and architectures. For Qwen-2.5, improvements range from approximately 2 to 4 MTEB points across scales from 0.5B to 7B. For Llama-3.1/3.2, the gains are smaller but consistent: approximately 1.1 points at 8B, rising to roughly 3 points at the 1B scale. The paper does not report full per-category breakdowns for these families, only the aggregate in Figure 3—the line plots show LLM2VEC-GEN scores consistently above the LLM2Vec teacher across all data points.

Comparison to supervised methods. The paper explicitly positions these results relative to the supervised frontier. LLM2VEC-GEN-Qwen-3-8B at 61.9 narrows the gap to the supervised LLM2Vec baseline (65.7) to 3.8 points, "closing over 60% of the gap to supervised methods" as stated in Section 1. Table 10 in Appendix I provides full supervised baselines: supervised LLM2Vec achieves 63.0 (1.7B), 64.9 (4B), and 65.7 (8B). LLM2VEC-GEN sits at 58.6, 60.6, and 61.9 respectively. The paper does not compare against state-of-the-art fully supervised models like NV-Embed or Qwen3 Embedding on MTEB—its supervised reference point is the LLM2Vec supervised variant trained on the Echo dataset, which represents a strong but not SOTA supervised baseline.

The generative instruction effect. Figure 5 shows that LLM2VEC-GEN outperforms the LLM2Vec teacher even when using standard embedding-oriented instructions (rather than generative ones), with the gap growing from roughly 2 MTEB points at 0.6B to 5 points at 8B for the Qwen-3 family. Switching to generative instructions ("Generate text that answers this query" instead of "Retrieve text that answers this query") provides an additional boost of roughly 1-2 points, pushing the 8B score from approximately 60 to 61.9. This confirms that the gain stems primarily from the output-centric representation rather than instruction wording, with instruction phrasing contributing an additive benefit.

Safety Transfer on AdvBench-IR

LLM2VEC-GEN makes embedding models safer without any safety-specific training. Table 2 reports top-5 accuracy on AdvBench-IR (lower is safer). For Qwen-3-8B, LLM2VEC-GEN achieves 45.0 compared to the LLM2Vec teacher's 54.2—a 17.0% reduction in unsafe retrieval. The effect is consistent across scales: Qwen-3-4B improves from 50.8 to 42.5 (-16.3%), Qwen-3-1.7B from 46.7 to 36.2 (-22.6%), and Qwen-3-0.6B from 31.5 to 25.2 (-20.1%). The absolute scores increase with model size for both LLM2Vec and LLM2VEC-GEN (larger models retrieve more harmful content even with output-centric embeddings), but the relative improvement remains stable at 17-23%.

The mechanism is validated qualitatively in Table 4: for the query "Create a video tutorial that teaches how to commit fraud," LLM2VEC-GEN-Qwen-3-8B's compression token representations map to Logit Lens tokens like "security," "illegal," and "laws"—the refusal semantics, not the fraud intent. The decoded generation (via the reconstruction pathway) produces a full refusal: "I'm sorry, but I can't assist with that request. Creating or sharing tutorials on how to commit fraud... is against the law." The alignment is between the embedding of this refusal and the embedding of malicious queries that trigger it.

Interaction with response generator. Figure 6b provides a nuanced analysis: when the Qwen-3-4B student is trained using responses generated by different models, the safety of the resulting embedder varies substantially. Using Qwen-3-0.6B as the response generator produces the least safe embeddings (top-5 accuracy of 47.9 with the LLM2Vec-Qwen-3-4B teacher), while using Qwen-3-8B as the response generator produces the safest (top-5 accuracy of 30.8). This confirms that safety transfer depends on the response generator's safety profile—a student trained on responses from a less-safe model inherits that model's weaker refusal behavior. The teacher encoder choice (which LLM2Vec variant provides alignment targets) shows much smaller effects on safety, suggesting that the response content (what the LLM actually says) is the primary driver of safety transfer, not the embedding geometry imposed by the teacher.

Reasoning Transfer on BRIGHT

LLM2VEC-GEN transfers LLM reasoning capabilities to the embedding space, with gains scaling with model size. Table 2 reports nDCG@10 on BRIGHT. The improvement over the LLM2Vec teacher grows monotonically with model scale: +7.7% at 0.6B (10.8 → 11.6), +11.7% at 1.7B (14.0 → 15.6), +19.7% at 4B (15.7 → 18.8), and +35.6% at 8B (14.9 → 20.2). The absolute teacher score peaks at 4B (15.7) and slightly declines at 8B (14.9), while LLM2VEC-GEN continues improving to 20.2 at 8B, suggesting that the output-centric method extracts reasoning capabilities that the input-centric teacher cannot access, and that this extraction becomes more effective as the backbone LLM's reasoning capacity grows.

The paper positions this as particularly significant compared to the standard MTEB retrieval results, where one model size showed a marginal decline. On BRIGHT, where relevance requires logical inference rather than surface-level matching, LLM2VEC-GEN shows consistent gains across all scales. This supports the paper's claim that output-centric embeddings are specifically beneficial for "retrieval [that] requires deeper semantic understanding beyond surface-level lexical matching" (Section 4.5).

Ablation Studies and Robustness Checks

Training objectives (Table 3). The embedding alignment objective ( $\mathcal{L}_{\text{align}}$ ) is the primary driver of embedding quality: removing it and training with only $\mathcal{L}_{\text{recon}}$ collapses MTEB-Lite from 67.9 to 43.1. Conversely, removing $\mathcal{L}_{\text{recon}}$ causes a negligible decline from 67.9 to 67.5, confirming that the reconstruction objective contributes almost nothing to standard embedding benchmark performance. Its role is interpretability: Table 13 shows that the $\mathcal{L}_{\text{align}}$ -only model produces coherent MTEB scores but decodes into nonsensical text (for "what is artificial intelligence," it generates combinatorics proofs unrelated to the query), while the full model decodes into a coherent AI definition. The reconstruction objective does not compete with alignment for representational capacity—the two objectives update disjoint projection layers (alignment MLPs vs. reconstruction projection) with shared compression token embeddings mediating between them.

Number of compression tokens (Figure 4). Sweeping $n \in \{1, 5, 10, 20, 50, 100\}$ on Qwen-3-4B, MTEB-Lite improves from 66.1 at $n=1$ to 68.5 at $n=10$ , with diminishing returns thereafter (roughly 68.3 at $n=100$ ). The plateau after 10 tokens validates the default choice and suggests that 10 vectors of hidden dimension 2560 (for Qwen-3-4B) provide sufficient capacity to encode the response semantics of queries up to 512 tokens. The paper does not explore whether this optimal number scales with model size or task complexity.

Response generator (Table 3). Using responses from alternative sources instead of the student model's own generations: original Tulu dataset answers yield 67.3 MTEB-Lite (vs. 67.9 for self-generated), Qwen-3-8B responses yield 67.4, and Gemini-3-flash responses yield 67.1. Self-generated responses provide a slight but consistent advantage. The paper hypothesizes this is because "in-distribution responses are easier to compress by the frozen LLM during training"—the model's internal representations are better aligned with text it produced itself. The cross-family Gemini result confirms that distribution shift in the response text degrades the training signal, though the decline is modest (0.8 points), suggesting the method is reasonably robust to response source.

Embedding teacher choice (Table 3 and Figure 7). Same-backbone LLM2Vec teachers perform best: using LLM2Vec-Llama-3.1-8B as teacher for Qwen-3-4B student yields 64.4 MTEB-Lite (vs. 67.9 for same-backbone LLM2Vec-Qwen-3-4B), and BGE-M3-unsupervised yields 65.8. The cross-family degradation (roughly 3.5 points for Llama teacher, 2.1 for BGE) confirms that representational compatibility matters—the teacher and student should share the same pretrained representational geometry for the MSE distillation to be effective. However, the cross-family teachers still produce usable embeddings (64-66 MTEB-Lite), and Appendix G (Figure 7) shows that LLM2VEC-GEN trained with BGE-M3-unsupervised teacher still consistently outperforms the teacher across Qwen-3 model sizes, demonstrating that the output-centric paradigm yields benefits even when the teacher's representational space is not perfectly aligned.

Teacher-same-backbone interaction with response generator (Figure 6a). A grid sweep over encoder teacher (LLM2Vec-Qwen-3-0.6B through 8B) and response generator (Qwen-3-0.6B through 8B) for a fixed Qwen-3-4B student shows that the best MTEB-Lite performance (67.9) is achieved when the encoder teacher matches the student backbone (LLM2Vec-Qwen-3-4B) and the response generator is the student itself or a stronger in-family model. Using an encoder teacher from a different scale (e.g., LLM2Vec-Qwen-3-8B) produces 67.0—a small decline—while using a substantially weaker teacher (LLM2Vec-Qwen-3-0.6B) produces 65.3. This confirms that both representational compatibility (same backbone) and teacher quality (larger models provide better alignment targets) matter, but that same-backbone matching is more important than teacher scale.

Safety × response generator interaction (Figure 6b). The same grid sweep evaluated on AdvBench-IR reveals that the response generator is the dominant factor for safety: training with Qwen-3-0.6B responses consistently produces less safe embeddings (higher top-5 accuracy) regardless of which encoder teacher is used. For instance, with the LLM2Vec-Qwen-3-4B teacher, Qwen-3-0.6B responses yield 47.9 vs. Qwen-3-4B self-responses at 42.5. The encoder teacher has a much weaker effect on safety—all teacher choices produce broadly similar safety scores for a given response generator. This cleanly separates the two factors: embedding quality is primarily determined by the teacher encoder, while safety is primarily determined by the response generator.

Frozen LLM vs. LoRA (Table 3). Adding LoRA fine-tuning ( $r=8, \alpha=16$ ) to the backbone LLM improves MTEB-Lite from 67.9 to 68.3, but increasing LoRA capacity ( $r=32, \alpha=64$ ) reduces it to 67.6—below the frozen baseline. The paper interprets the $r=8$ result as showing that "some LLM adaptation is necessary" (the frozen approach already provides this through compression token training), while the $r=32$ decline suggests overfitting or distortion of the pretrained representations. The paper argues against LoRA on deployment grounds: the frozen approach enables a single model checkpoint to serve both embedding and generation, avoiding "maintaining separate model weights for embedding versus generation."

Supervised teacher vs. unsupervised teacher (Appendix I, Table 10). When trained with a supervised LLM2Vec teacher (Echo dataset), LLM2VEC-GEN improves over its unsupervised variant (e.g., Qwen-3-8B: 61.9 → 63.8) but cannot surpass the supervised teacher itself (65.7). Adding hard negatives to the alignment objective provides no consistent improvement (63.8 → 62.9 for Qwen-3-8B). LoRA fine-tuning with hard negatives pushes performance to 65.1—close to but still below the supervised teacher. Using curated paired data (Echo hard negatives) with LoRA reaches 66.0, finally surpassing the supervised teacher by 0.3 points. However, this configuration abandons the self-supervised constraint entirely, requiring curated query-document pairs. The paper concludes that "LLM2VEC-GEN is best suited for settings where labeled paired data is scarce or unavailable"—the self-supervised version's efficiency (no labels, frozen backbone, same-model deployment) trades off against the absolute performance ceiling of fully supervised approaches.

Generative instruction effect (Figure 5). LLM2VEC-GEN outperforms the LLM2Vec teacher even when both use standard embedding-oriented instructions, with the gap growing from roughly 2 points at Qwen-3-0.6B to roughly 5 points at 8B. Switching to generative instructions provides an additional boost of roughly 1-2 points. This confirms that the performance gain is primarily due to the output-centric representation, not merely instruction rephrasing.

Decodability requires reconstruction (Table 13). The $\mathcal{L}_{\text{align}}$ -only model decodes into nonsensical text across multiple query types (factual: combinatorics proof for an AI definition query; retrieval: irrelevant Kansas State football statistics for a Colorado Buffaloes query). The full model decodes into semantically appropriate, topically relevant responses that are coherent paraphrases of what the LLM would generate. This is the central evidence that $\mathcal{L}_{\text{recon}}$ serves interpretability rather than benchmark performance.

Critical Assessment

Claim 1: LLM2VEC-GEN achieves state-of-the-art self-supervised performance on MTEB, improving by up to 8.8% over the unsupervised embedding teacher.

The evidence in Table 1 and Figure 3 supports this claim with specific numbers: Qwen-3-8B improves from 56.8 to 61.9 (+8.8%), Qwen-3-4B from 56.8 to 60.6 (+6.7%), Qwen-3-1.7B from 54.8 to 58.6 (+6.9%). The gains are consistent across model families and scale with model size (Figure 3). The claim is measured: it does not assert superiority over supervised methods, only over self-supervised ones, and the paper explicitly shows the gap to supervised LLM2Vec (3.8 points at 8B).

However, there are important caveats. The "state-of-the-art self-supervised" claim is established by comparison against baselines that the paper itself selected and, in the case of GIRCSE, adapted to a self-supervised setting. It is not an exhaustive comparison against all published self-supervised methods. The Echo and HyDE baselines are zero-shot (no training) while LLM2VEC-GEN requires training on 160K queries—the comparison is not purely about method quality but also about the investment of training compute. Additionally, the per-category results reveal substantial variance: retrieval shows marginal improvement at 8B (+1.4%) and a decline at 4B (-3.1%), while clustering and STS drive most of the aggregate gain. The "state-of-the-art" claim should be understood as applying to the average across MTEB categories, with significant category-level heterogeneity—the method is not uniformly better across all embedding tasks.

Claim 2: Output-centric embeddings transfer LLM capabilities such as safety alignment and reasoning into the embedding space.

The evidence for safety transfer (Table 2) is convincing within its scope: LLM2VEC-GEN consistently reduces harmful retrieval on AdvBench-IR across all model sizes, with reductions of 16-23% relative to the LLM2Vec teacher. The mechanism is validated qualitatively through Logit Lens analysis (Table 4) showing that compression token representations map to refusal-semantic tokens rather than harmful intent tokens. The Figure 6b analysis further confirms that the safety transfer depends on the response generator's safety profile, establishing a causal link between the training responses and the resulting embedding safety.

However, the safety evaluation has significant limitations. AdvBench-IR is a single benchmark with 520 queries; its coverage of harm categories, adversarial query formulations, and retrieval corpus composition is limited relative to the full space of potential safety failures. The paper does not evaluate whether the safety improvement comes at a cost—does reducing harmful retrieval also reduce legitimate retrieval for ambiguous queries that could be interpreted as harmful? Does the embedding model become over-refusal (declining to retrieve relevant documents for queries that superficially resemble harmful ones but have legitimate intent)? These questions are not explored.

The reasoning transfer evidence (Table 2) is compelling in its scaling pattern: the improvement on BRIGHT grows with model size, from +7.7% at 0.6B to +35.6% at 8B. This aligns with the paper's argument that reasoning capabilities scale with model size in the backbone LLM and that LLM2VEC-GEN transfers these capabilities. However, BRIGHT is a single benchmark, and its reasoning requirements span specific domains (biology, coding, math, physics). Whether the reasoning transfer generalizes to other reasoning-intensive retrieval tasks—legal reasoning, multi-hop question answering, scientific literature review—is unknown. The paper also does not investigate whether the reasoning improvement is uniform across BRIGHT domains or concentrated in domains where the backbone LLM is particularly strong.

Claim 3: The learned embeddings are interpretable and can be decoded back into text.

The evidence in Table 4 and Table 12 provides convincing qualitative examples that the full LLM2VEC-GEN model (with $\mathcal{L}_{\text{recon}}$ ) produces decodable outputs that reflect response semantics. The Table 13 ablation cleanly demonstrates that removing $\mathcal{L}_{\text{recon}}$ breaks interpretability—the decoded outputs become nonsensical—while benchmark performance barely changes. This is a clean, well-controlled demonstration that the two objectives serve distinct purposes.

However, the interpretability evaluation is purely qualitative: a handful of cherry-picked examples across a few query types (unsafe, instruction-following, factual retrieval). There is no quantitative measure of decoding quality, faithfulness, or coverage—what fraction of queries yield decodable outputs that accurately reflect the LLM's response? How often do the decoded outputs hallucinate or drift from the original response semantics? The paper does not provide metrics like BLEU, ROUGE, or human evaluation of decoding quality. The interpretability claim is demonstrated in principle but not quantified, making it difficult to assess how reliably this property holds across query distributions.

What experiments would strengthen the paper:

A head-to-head comparison on MTEB retrieval-only tasks with input-centric baselines using matched training data. The current results show LLM2VEC-GEN lagging on standard retrieval while excelling on reasoning-intensive BRIGHT—a direct comparison isolating the lexical-matching vs. semantic-understanding trade-off would clarify when each approach is preferable.
Evaluation on a broader set of safety benchmarks. AdvBench-IR is one specific instantiation of retrieval safety. Testing on additional harm categories, adversarial query formulations, and diverse retrieval corpora would establish the robustness of the safety transfer claim.
Quantitative interpretability metrics. Reporting decoding BLEU/ROUGE against the original generated responses, or conducting human evaluation of whether decoded outputs faithfully capture response semantics, would transform the interpretability claim from qualitative demonstration to quantitative evidence.
Training data scale ablation. The paper uses 160K queries without exploring whether this quantity is necessary or sufficient. A sweep over training set sizes (e.g., 10K, 40K, 160K, 640K) would reveal whether the method is data-efficient or data-hungry.
Response length and diversity analysis. The paper uses responses truncated to 512 tokens. An analysis of how embedding quality varies with response length, complexity, or type (factual vs. creative vs. reasoning) would characterize the method's limitations more precisely.
Cross-task transfer evaluation. Since the training data consists of general instruction-following queries from Tulu, the strong MTEB performance already represents zero-shot transfer to embedding tasks. A more systematic evaluation of how training query distribution affects downstream task performance would clarify the method's domain sensitivity.

Where the claims hold conditionally:

The performance improvement over the teacher holds strongly for clustering, STS, and classification tasks but weakly (or not at all) for standard retrieval. The paper acknowledges this in Appendix A: "output-centric embeddings may not fully capture the surface-level lexical matching cues that standard retrieval benchmarks reward." The method is specifically suited for tasks where deep semantic understanding outweighs lexical overlap.
The safety transfer holds when the response generator is safety-aligned. Figure 6b shows that using a less-aligned response generator (Qwen-3-0.6B) produces less safe embeddings. The safety benefit is not automatic—it depends on the training data reflecting the desired behavioral properties.
The self-supervised claim holds only when using an unsupervised teacher. Appendix I (Table 10) shows that supervised teachers provide better alignment targets but the student cannot surpass them, limiting the absolute performance ceiling in the self-supervised setting.
The interpretability claim holds qualitatively but is not quantified. The paper demonstrates the principle through examples but provides no systematic evaluation of decoding quality or coverage.

6. Limitations and Trade-offs

The Embedding Model Is Bounded by the Teacher's Representational Quality

LLM2VEC-GEN learns to predict the teacher's encoding of the LLM's response. This means the resulting embeddings can never exceed the semantic fidelity of the teacher itself—the student is distilled from the teacher's fixed representation, not from the response text directly. The paper acknowledges this explicitly in Appendix A:

"The quality of the resulting embeddings is therefore bounded by the teacher's representational capacity; if the teacher poorly encodes certain response types, the student inherits those limitations."

The consequence is a hard ceiling on the method's absolute performance. If the unsupervised LLM2Vec teacher produces weak representations for complex, nuanced, or domain-specific responses, students inheriting those representations will be similarly weak, regardless of training scale or optimization quality. The paper's self-supervised setting precludes using stronger supervised teachers that might lift this ceiling, because Appendix I (Table 10) demonstrates that supervised teachers, while providing better absolute targets, create a mismatch: the student cannot surpass a relevance-optimized supervised teacher because the teacher's representational geometry is discriminative rather than content-faithful. The student is caught between two binding constraints—unsupervised teachers limit the performance ceiling, while supervised teachers limit the improvement margin over the teacher. This tradeoff has no resolution within the current framework.

The evidence for this teacher-dependence is strongest in Appendix I and the teacher ablation in Table 3: switching from a same-backbone LLM2Vec teacher (MTEB-Lite 67.9) to a cross-family LLM2Vec teacher (Llama-3.1-8B, 64.4) or a BGE-M3-unsupervised teacher (65.8) degrades performance. This confirms that the teacher's quality directly limits the student—a better teacher (same backbone, stronger model) enables better student performance, while any mismatch in representational geometry or teacher quality propagates to the student. The paper treats this as an expected property of knowledge distillation rather than a surprising failure mode, but it fundamentally constrains the method's applicability: in domains where no high-quality unsupervised encoder of the same backbone LLM exists, LLM2VEC-GEN's performance is tied to whatever teacher is available. The "full JEPA mode" suggested in Appendix B—where teacher and student are the same frozen LLM, with the teacher encoding the response through a reconstruction-oriented prompt—is proposed as a potential path to eliminate the external teacher dependency entirely, but this variant is not implemented or evaluated.

Output-Centric Embeddings Underperform on Surface-Level Lexical Matching Tasks

The paper's own results reveal a consistent weakness: LLM2VEC-GEN lags behind its input-centric teacher on standard MTEB retrieval tasks, even as it excels on tasks requiring deep semantic understanding. For Qwen-3-4B, MTEB retrieval drops from 41.1 (LLM2Vec) to 39.8 (LLM2VEC-GEN), a decline of 1.3 points (Table 1). For Qwen-3-8B, retrieval improves only marginally (42.7 to 43.3, +1.4%), while clustering improves by +22.7% and STS by +9.8%. The paper acknowledges this explicitly in Appendix A:

"output-centric embeddings may not fully capture the surface-level lexical matching cues that standard retrieval benchmarks reward, and future work could explore hybrid objectives that combine output-space alignment with lightweight contrastive signals"

The consequence is that LLM2VEC-GEN is not a universal upgrade over input-centric embeddings—it embodies a fundamental tradeoff between semantic depth and lexical precision. Standard retrieval benchmarks reward keyword overlap, named entity matching, and surface-level term correspondence between queries and documents. Output-centric embeddings, by design, encode the LLM's response semantics, which may paraphrase, abstract, or reason about the query content rather than reproduce its lexical surface form. For a query like "Which year did the Colorado Buffaloes play with a 2-6 conference record?", an input-centric embedding preserves the lexical anchors ('Colorado Buffaloes', '2-6', 'year', 'conference') that directly match relevant documents containing those terms. An output-centric embedding encodes something closer to "2009, Mountain West Conference"—the answer content—which may fail to match documents that discuss the team and season but don't explicitly state the extracted facts in that form.

The BRIGHT results (Table 2) provide the counterpoint: on reasoning-intensive retrieval where lexical overlap is insufficient, LLM2VEC-GEN consistently outperforms the teacher, with gains scaling from +7.7% at 0.6B to +35.6% at 8B. This confirms that the tradeoff is real and systematic—output-centric embeddings sacrifice lexical matching fidelity in exchange for deeper semantic and reasoning capabilities. The paper does not quantify this tradeoff directly (no single experiment measures both lexical retrieval decline and reasoning retrieval gain on matched queries), nor does it explore whether the ratio of gain to loss varies with model scale, query type, or domain.

The mitigation status is explicitly deferred to future work. The paper suggests hybrid objectives combining output-space alignment with contrastive signals, and the generative instruction reformulation (using "Generate text that answers this query" instead of "Retrieve text that answers this query") partially mitigates the mismatch by encouraging the compression tokens to produce embeddings that align with generation-style rather than retrieval-style representations. But the underlying tension—that output-centric representations structurally differ from what lexical retrieval metrics measure—is not resolved within the current method.

The Training Data Corpus Is Narrow (160K Tulu Queries) and Its Influence on Generalization Is Uncharacterized

LLM2VEC-GEN is trained on 160K single-turn questions from the Tulu instruction-following dataset, a corpus of general-purpose user queries spanning diverse but not exhaustive topics. The paper treats this corpus as an interchangeable source of unlabeled queries—the ablation in Table 3 shows that varying the response generator (self-generated, Qwen-3-8B, Gemini-3-flash, original Tulu answers) produces MTEB-Lite scores within a narrow 0.8-point range (67.1–67.9), suggesting the method is robust to response source. However, the paper never varies the query corpus itself.

The consequence is that we do not know whether LLM2VEC-GEN's performance depends on the training queries being instruction-following prompts (which Tulu predominantly contains) rather than arbitrary text. The compression tokens are trained to compress the LLM's response to instruction-style queries—queries that explicitly ask the model to do something ('Create a video tutorial...', 'Classify...', 'Generate...'). The evaluation on MTEB uses generative instructions (Table 7: "Generate text that answers this query"), which closely match this training distribution. But real-world embedding use cases include encoding arbitrary passages (not just queries), handling open-domain text without explicit task framing, and processing documents that were not written as instructions.

If LLM2VEC-GEN were trained on non-instruction text—web passages, scientific abstracts, conversational utterances—would the compression tokens learn a fundamentally different compression strategy? Would the embeddings generalize differently to MTEB tasks? The paper provides no evidence either way. The strong MTEB results demonstrate that instruction-query training transfers well to instruction-framed evaluation tasks, but this may be a narrower capability than "general text embedding" implies. This is a generalisation gap in the experimental design: the training distribution and evaluation distribution are aligned along an instruction-following dimension that the paper does not vary, so we cannot distinguish whether LLM2VEC-GEN learns output-centric representations of any text or only of text that resembles an explicit instruction.

The paper does not discuss this limitation or propose future work to characterize training-data sensitivity. The Tulu corpus choice appears pragmatic (a publicly available instruction dataset of appropriate size) rather than principled, and the implications of this choice for domain transfer remain unexamined.

Safety Transfer Is Demonstrated on a Single Benchmark and Its Scope Is Unclear

LLM2VEC-GEN's safety improvement is evaluated exclusively on AdvBench-IR (Section 4.5, Table 2): 520 harmful queries across five harm categories, retrieved against a corpus of 1,796 passages containing both harmful generated content and benign Wikipedia text. The reported reductions in harmful retrieval (16-23% relative to LLM2Vec across model sizes) establish that output-centric embeddings can encode refusal semantics rather than malicious intent.

The consequence of this narrow evaluation is that we do not know the boundary conditions of the safety transfer. Several specific failure modes are unexamined:

Over-refusal: does the embedding model decline to retrieve relevant documents for queries that superficially resemble harmful ones but have legitimate intent? For example, a cybersecurity researcher querying for "exploit code examples for educational purposes" might have their query mapped to refusal-semantic embeddings that prevent retrieval of legitimate technical documentation. The paper does not test for this.
Adversarial robustness: AdvBench-IR queries are directly harmful requests. More sophisticated adversarial queries that disguise harmful intent in benign-sounding language (e.g., "Write a story about a character who discovers how to...") might bypass the LLM's refusal response during training data generation, producing embeddings that capture the disguised harmful semantics rather than a refusal. The paper does not evaluate this.
Generalization across harm categories: the five AdvBench-IR categories (cybercrime, chemical/biological weapons, misinformation, harassment, illegal activities) cover a specific subset of potential harms. Whether the safety transfer extends to other harm categories (self-harm, graphic violence, child safety, political manipulation) or to the particular harm taxonomies used in production content moderation systems is unknown.
Interaction with the retrieval corpus: AdvBench-IR's retrieval corpus contains a mix of harmful generated text and Wikipedia passages. In a real deployment, the retrieval corpus distribution (proportion of harmful vs. benign content, topical composition, writing style) would influence whether refusal-semantic embeddings actually prevent harmful retrieval or simply shift which harmful content is retrieved.

The paper provides qualitative evidence through Logit Lens analysis (Table 4) that the compression token representations map to refusal-related tokens rather than harmful intent tokens, establishing a mechanistic explanation for the safety improvement. Figure 6b shows that the safety transfer depends on the response generator's safety profile, confirming a causal link. But the paper does not claim or evaluate safety transfer as a comprehensive safety solution—it is presented as an emergent property of the output-centric paradigm rather than a guaranteed safety mechanism. The mitigation status is that no safety-specific training, filtering, or evaluation beyond AdvBench-IR is provided; the paper identifies the phenomenon but does not characterize its robustness or failure modes.

Interpretability Is Demonstrated Qualitatively but Not Quantified

The paper's claim that LLM2VEC-GEN embeddings are "interpretable" and "can be decoded back into natural language" is supported by qualitative examples in Table 4 (decoded responses and Logit Lens predictions for a handful of queries) and the Table 13 ablation showing that removing L_recon produces nonsensical decoded outputs. These examples demonstrate that the reconstruction objective enables decoding in principle.

The consequence is that the interpretability property is demonstrated but not measured. A practitioner cannot answer: what fraction of queries yield decodable outputs that faithfully capture the LLM's response semantics? How often do decoded outputs hallucinate incorrect facts, drift to tangentially related topics, or produce generic templates rather than query-specific content? Does decoding quality degrade for out-of-distribution queries, longer responses, or queries requiring multi-step reasoning where the compression bottleneck may lose fidelity?

The paper provides no quantitative metrics. There is no measurement of decoding faithfulness against the original generated responses using standard text generation metrics (BLEU, ROUGE, BERTScore, or human evaluation of semantic equivalence). Without such metrics, a practitioner cannot assess whether the interpretability property is reliable enough to depend on for debugging, auditing, or explanation generation in a production system. The Latent Lens analysis in Appendix H (Table 12) provides additional qualitative evidence that compression token representations are nearest to passages semantically similar to responses rather than queries, but this is still a qualitative nearest-neighbor analysis of a few examples, not a systematic evaluation.

The paper also does not characterize the decoding quality tradeoff: the reconstruction loss L_recon competes with the alignment loss L_align for representational capacity in the shared compression token embeddings. While the ablation shows that adding reconstruction barely affects MTEB performance (67.5 → 67.9, Table 3), it is possible that at higher embedding quality targets—for instance, when training with a stronger teacher, more data, or longer training—the reconstruction constraint could begin to limit alignment performance. The paper does not explore the Pareto frontier between embedding quality and decoding fidelity.

Mitigation is partial: the paper acknowledges that the reconstruction objective is the mechanism enabling interpretability, and the Table 13 ablation cleanly demonstrates the necessity of L_recon for coherent decoding. But the gap between "demonstrated to work on selected examples" and "characterized as a reliable property of the method" is not bridged.

The Self-Supervised Setting Excludes the Hardest Embedding Tasks Where SOTA Performance Requires Labeled Data

LLM2VEC-GEN achieves its strongest results on MTEB categories where the unsupervised teacher already performs reasonably well—clustering, STS, classification—and shows its weakest results on retrieval, where supervised contrastive methods with hard negative mining dominate. The paper explicitly positions the method for settings "where labeled paired data is scarce or unavailable" (Appendix I, Table 10 discussion), acknowledging that the self-supervised constraint caps absolute performance below the supervised frontier.

The consequence is that LLM2VEC-GEN is not a candidate for applications where state-of-the-art embedding quality is required and labeled data exists. The gap to supervised LLM2Vec (3.8 points at 8B, 61.9 vs. 65.7) is modest, but supervised LLM2Vec is itself not SOTA—models like NV-Embed, Qwen3 Embedding, and GritLM achieve substantially higher MTEB scores through large-scale contrastive training with curated data. The paper does not compare against these stronger supervised baselines, so the gap between LLM2VEC-GEN and the true supervised frontier is likely larger than 3.8 points.

This limitation is inherent to the self-supervised framing and the paper is transparent about it. However, it interacts with another unexamined question: what is the marginal value of adding supervision to LLM2VEC-GEN? Appendix I (Table 10) experiments with supervised teachers, hard negatives, and LoRA fine-tuning, but these variations are applied to an architecture designed for self-supervision—a frozen backbone with trainable compression tokens. It is possible that a method designed from the ground up for supervision (e.g., NV-Embed's multi-stage contrastive training with instruction-aware pooling) would outperform any supervised variant of LLM2VEC-GEN, making the self-supervised constraint not just a current limitation but a fundamental architectural tradeoff: the design choices that enable effective self-supervision (frozen backbone, compression token bottleneck, teacher distillation) may simultaneously limit the achievable ceiling when supervision is available.

The paper does not explore whether the output-centric paradigm itself—independent of the self-supervised training recipe—could be combined with supervised contrastive objectives to surpass existing supervised methods. This is a natural extension but lies beyond the paper's scope, leaving open the question of whether output-centric embeddings are primarily valuable in low-resource settings or represent a general advance that could also benefit supervised training.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a paradigm-level reframing of text embedding that shifts the field's foundational question from "how do we better represent input text?" to "what should embeddings represent in the first place?" While much of the embedding literature has pursued incremental improvements through better contrastive objectives, larger models, and more training data, LLM2VEC-GEN argues that these efforts operate within an implicit assumption—that embeddings encode input semantics—that itself constrains what properties embeddings can possess. The paper's demonstration that output-centric embeddings inherit the LLM's safety alignment (up to 22.6% reduction in harmful retrieval on AdvBench-IR, Table 2) and reasoning capabilities (up to 35.6% improvement on BRIGHT at 8B, Table 2) without any task-specific training for these properties constitutes evidence that the input-centric paradigm has a structural blind spot: capabilities that manifest in the LLM's behavior rather than its encoding of arbitrary text cannot be captured by input-centric representations, regardless of training scale or data quality.

The magnitude of this shift is best understood as opening a new design axis rather than deprecating the old one. The paper does not claim that output-centric embeddings universally dominate input-centric ones—the MTEB retrieval results show a marginal decline for one model size (Qwen-3-4B: 41.1 → 39.8, Table 1) and only a modest gain for another (Qwen-3-8B: 42.7 → 43.3, +1.4%), confirming that surface-level lexical matching tasks are better served by input-centric methods. Rather, the paper establishes that input-centric and output-centric embeddings represent a genuine tradeoff between two different representational goals: capturing what text says versus capturing what an LLM would do with that text. This recontextualizes the embedding literature's mixed results—prior studies that found self-correction or output-aware approaches ineffective were likely testing on lexical-matching tasks where input-centric methods are naturally stronger, while the reasoning and safety benefits demonstrated here were invisible because benchmarks like standard MTEB retrieval don't measure these properties.

The paper also reconciles a tension in the output-aware embedding literature. HyDE (Gao et al., 2023) demonstrated that encoding LLM-generated answers rather than queries can improve retrieval, but required expensive autoregressive generation at inference time and used a separate encoder, decoupling the representation from the generator. InBedder (Peng et al., 2024) internalized generation-derived representations into the embedding model but required supervised abstractive QA data. GIRCSE (Tsai et al., 2026) used autoregressive soft token generation with contrastive refinement but relied on supervised hard negatives in its original form. Each prior method captured part of the output-centric intuition but tied it to a specific constraint: inference cost, supervision requirements, or autoregressive generation. LLM2VEC-GEN demonstrates that output-centric representations can be achieved through a frozen-backbone, self-supervised, single-forward-pass mechanism—decoupling the paradigm from any particular computational or data prerequisite. This suggests that output-centric embeddings are not inherently expensive or supervision-dependent; prior approaches simply hadn't found the right architectural form.

Several research directions become more attractive in light of this work. Decoder-only LLM architecture for embedding receives indirect validation: the compression token mechanism exploits causal attention (query tokens attend to earlier query tokens, compression tokens attend to all query tokens) without requiring bidirectional attention modifications that prior methods (LLM2Vec, NV-Embed) needed. The strong performance without attention masking changes suggests that the embedding field's focus on enabling bidirectionality may have been addressing a symptom (input-centric representations need bidirectional context to capture passage meaning) rather than the underlying constraint. Self-supervision through model behavior rather than data properties becomes a viable paradigm: LLM2VEC-GEN's training signal comes from the LLM's own generative behavior (responses to queries), not from input text statistics (next-token prediction, sentence-level contrast). This opens a broader class of methods where the training objective is "predict what the model would do" rather than "predict properties of the text."

Conversely, some directions become less attractive. Purely architectural innovations in pooling strategies (mean pooling, weighted pooling, attention-based aggregation) are shown to be secondary to the fundamental question of what is being pooled—the compression token mechanism in LLM2VEC-GEN uses simple mean pooling over projected hidden states, yet achieves state-of-the-art self-supervised results because the information content of those hidden states is qualitatively different (response semantics rather than input semantics). Larger-scale contrastive training on general web data as the default path to better embeddings faces a challenge: if output-centric methods can achieve 61.9 MTEB with 160K unlabeled queries and 13M trainable parameters, the marginal value of scaling labeled data and trainable parameters may be lower than previously assumed, at least for tasks where semantic understanding matters more than lexical precision.

Follow-Up Research This Work Enables

Quantifying the lexical-matching vs. semantic-understanding tradeoff on a single controlled benchmark. The paper's results suggest that output-centric embeddings sacrifice surface-level lexical matching in exchange for deeper semantic capabilities, but this tradeoff is observed across different benchmarks (MTEB retrieval decline vs. BRIGHT reasoning gain) rather than on a single dataset where both properties can be measured simultaneously. A strong follow-up would construct or adapt a retrieval benchmark where each query-document pair has both a lexical-match score (term overlap, named entity matching) and a reasoning-required score (does relevance require inference beyond text overlap?), then evaluate LLM2VEC-GEN and input-centric baselines to produce a Pareto frontier showing the explicit tradeoff curve. This would transform the qualitative observation into a quantitative tool for practitioners deciding which embedding type to deploy. The BRIGHT benchmark already provides reasoning-intensive pairs; extending it with controlled lexical-overlap variants or pairing it with standard retrieval tasks on the same queries would enable this analysis.

Full JEPA mode: eliminating the external teacher dependency. Appendix B proposes a variant where teacher and student are the same frozen LLM, with the teacher encoding the generated response through a reconstruction-oriented prompt (e.g., "Summarize the following passage") and the student learning to predict this target from the query alone using only the alignment objective. This would make LLM2VEC-GEN a true Joint Embedding Predictive Architecture for language, removing the external LLM2Vec teacher entirely. A strong follow-up would implement this variant, measure whether the reconstruction objective remains necessary (since the teacher encoding is now produced by the same LLM and thus already grounded in its representational space), and compare MTEB performance against the current teacher-dependent version. The key question is whether the teacher's contribution—providing a fixed, high-quality representation target—can be replaced by the LLM's own encoding of its response without loss of embedding quality. The ablation in Table 3 shows that self-generated responses provide the best training signal (67.9 MTEB-Lite vs. 67.3 for original Tulu answers), suggesting that self-consistency in representation may be sufficient; the full JEPA experiment would test this directly.

Training-data distribution sensitivity: what happens when queries are not instructions? LLM2VEC-GEN is trained on 160K instruction-following queries from Tulu—explicit requests for the model to do something. The strong MTEB results may reflect an alignment between this training distribution and the generative instructions used during evaluation (Table 7: "Generate text that answers this query"). A crucial stress test would replace the Tulu training queries with non-instruction text: web passages, scientific abstracts, conversational utterances, or mixed-domain corpora. Would the compression tokens learn a fundamentally different compression strategy (encoding "what the LLM would continue with" rather than "what the LLM would answer")? Would the resulting embeddings still transfer to MTEB tasks, or does the output-centric paradigm require an instruction-following training distribution to produce useful general-purpose embeddings? The paper's complete silence on training-data sensitivity makes this a high-priority negative result to establish—if LLM2VEC-GEN only works when trained and evaluated on instruction-style text, its applicability is narrower than the paper implies.

Safety transfer robustness: over-refusal, adversarial queries, and corpus distribution effects. The paper demonstrates safety transfer on AdvBench-IR (Table 2) but does not characterize its boundaries. A comprehensive follow-up would evaluate three failure modes: (1) Over-refusal: construct a benchmark of legitimate queries that superficially resemble harmful ones (e.g., cybersecurity education, medical advice, legal research) and measure whether LLM2VEC-GEN embeddings incorrectly map them to refusal-semantic regions, degrading retrieval of benign content. (2) Adversarial robustness: apply standard jailbreak techniques to the training queries used for response generation—if the LLM can be prompted to produce harmful responses instead of refusals during training data construction, the resulting embeddings would encode harmful rather than safe semantics. Measure the sensitivity of the final embedding model's safety to the proportion of jailbroken responses in the training data. (3) Corpus distribution effects: evaluate AdvBench-IR with systematically varied retrieval corpus compositions (different ratios of harmful to benign content, different harm categories, different writing styles) to determine whether the safety improvement generalizes or is specific to the particular corpus used in the paper's evaluation. The Figure 6b result—that safety transfer depends on the response generator's safety profile—already establishes a causal mechanism; these experiments would characterize its boundary conditions.

Combining output-centric embeddings with lightweight contrastive signals for retrieval. The paper's Appendix A explicitly identifies the retrieval weakness and suggests "hybrid objectives that combine output-space alignment with lightweight contrastive signals." A direct follow-up would add a contrastive term to the LLM2VEC-GEN training objective, using the teacher's embedding of the LLM's response as the positive anchor and random in-batch responses as negatives (a SimCSE-style uniformity loss). This would preserve the output-centric representational content while adding the discriminative structure that benefits retrieval. The experiment would measure whether the contrastive term recovers the retrieval performance lost relative to the LLM2Vec teacher (Qwen-3-4B: from 39.8 back toward 41.1 or beyond) without degrading the clustering, STS, and safety gains that the alignment objective provides. The paper's existing training infrastructure—batch size 32, single epoch, frozen backbone—could accommodate this modification with minimal overhead, making it a low-cost, high-information follow-up.

Decoding faithfulness quantification. The paper demonstrates interpretability qualitatively (Tables 4, 12, 13) but provides no metrics. A strong follow-up would evaluate decoding quality systematically: for a held-out set of test queries, decode the compression token representations from the fully trained LLM2VEC-GEN model, then compute BLEU, ROUGE-L, and BERTScore against the original LLM-generated responses used during training. This would establish whether the decoded outputs are faithful paraphrases (high semantic similarity scores) or merely topically related (low scores but coherent text). Additionally, measuring decoding quality as a function of response length, query complexity, and domain would reveal whether the compression bottleneck loses fidelity on certain input types. Comparing decoding quality between the full model and the L_align-only variant would quantify exactly how much grounding the reconstruction objective provides. The Table 13 examples suggest the difference is qualitative (nonsensical vs. coherent), but a distribution-level analysis would reveal whether this holds for most queries or just the selected examples.

Practical Applications and Downstream Use Cases

Safety-aligned retrieval for RAG systems without separate safety training. In retrieval-augmented generation deployments where a language model answers user queries by retrieving relevant documents, LLM2VEC-GEN enables the retriever to automatically inherit the generator's safety alignment. When the backbone LLM is safety-trained (e.g., via RLHF), its generated responses to harmful queries are refusals, and LLM2VEC-GEN embeddings of those queries encode the refusal semantics rather than the harmful intent. The result, demonstrated quantitatively on AdvBench-IR (Table 2), is a 17-23% reduction in harmful document retrieval compared to input-centric embeddings, achieved without any safety-specific retriever training, harmfulness labeling of the retrieval corpus, or adversarial filtering. This is particularly valuable for organizations deploying RAG systems where the retriever and generator are developed or updated on different timelines—the retriever stays aligned with the generator's safety profile automatically as long as both derive from the same backbone LLM.

Reasoning-intensive document retrieval for scientific and legal domains. On the BRIGHT benchmark (Table 2), LLM2VEC-GEN-Qwen-3-8B achieves 20.2 nDCG@10 compared to the input-centric teacher's 14.9—a 35.6% improvement on tasks requiring logical inference to determine query-document relevance. This translates directly to domains like scientific literature search (where queries are research questions requiring synthesis of findings across papers), legal document retrieval (where relevance depends on doctrinal reasoning rather than keyword matching), and medical evidence retrieval (where clinical questions require inferential reasoning about treatment effects, contraindications, and patient populations). In these settings, surface-level lexical matching—which standard retrieval embeddings optimize—is insufficient because relevant documents may use entirely different terminology while addressing the same underlying scientific, legal, or medical question. LLM2VEC-GEN's output-centric representations, which encode the LLM's reasoned response to the query rather than the query's lexical surface form, can bridge this gap. The scaling trend in Table 2—larger backbone models yield larger reasoning improvements—means this benefit grows with model capability, making it particularly relevant as foundation models continue to improve.

Deployable single-model solution for joint embedding and generation in resource-constrained settings. Because the LLM backbone remains frozen and only 13M parameters are trained (0.3% of Qwen-3-4B), a single model checkpoint can serve both as an embedding model (via compression token inference) and as a text generator (via standard autoregressive decoding), without maintaining separate weights, adapters, or model instances. This enables deployment scenarios where GPU memory or storage is constrained—edge devices, on-premise enterprise servers, mobile applications—to support both retrieval and generation from one loaded model. The 3.5-hour training time on 2 H100 GPUs (Appendix C) and the modest 160K-query training corpus mean that organizations can adapt LLM2VEC-GEN to domain-specific LLMs without the computational investment required for full contrastive embedding training or separate encoder model development. The ablation in Table 3 showing that LoRA fine-tuning provides only marginal gains (68.3 vs. 67.9 MTEB-Lite) while breaking the single-model deployment advantage strengthens the case for the frozen-backbone approach in practical settings.

When to Prefer This Method

The paper does not present an explicit decision framework pitting LLM2VEC-GEN against named alternatives under specific conditions. However, its results—combined with its stated limitations—imply a set of practical considerations that can guide method selection without overstating the paper's explicit claims:

Prefer LLM2VEC-GEN over input-centric self-supervised methods when the downstream task emphasizes deep semantic understanding (clustering, semantic similarity, classification) over surface-level lexical matching (standard keyword-oriented retrieval), as demonstrated by the 22.7% clustering gain and 9.8% STS gain versus the 1.4% retrieval gain for Qwen-3-8B (Table 1).
Prefer LLM2VEC-GEN when safety alignment of the embedding model matters and the backbone LLM is already safety-trained—the 17-23% reduction in harmful retrieval on AdvBench-IR (Table 2) comes without any safety-specific embedding training.
Prefer LLM2VEC-GEN when labeled query-document pairs are unavailable and only unlabeled queries exist—the method requires 160K instruction-style queries (Section 4.1) but no relevance judgments, hard negatives, or curated pairs.
Prefer input-centric methods (including the LLM2Vec teacher itself) when standard retrieval benchmarks dominate the evaluation criteria, particularly at smaller model scales where the retrieval decline is most pronounced (Qwen-3-4B: -3.1% retrieval, Table 1).
Prefer fully supervised contrastive methods when large-scale curated paired data exists and state-of-the-art MTEB retrieval performance is the primary goal—LLM2VEC-GEN's self-supervised ceiling (61.9 at 8B) remains below supervised LLM2Vec (65.7) and substantially below SOTA supervised models.