ArXiv: 2605.05806

🎯 Pitch

A frozen encoder-decoder model can match or beat specialized external retrievers by repurposing its own cross-attention as a retrieval mechanism over pre-encoded evidence. The model learns to score chunks directly via ‘retrieval tokens’ without ever translating between separate retrieval and generation representation spaces. This blurs the line between RAG and purely generative models, redefining retrieval as an intrinsic, self-consistent capability.

1. Executive Summary

This paper introduces INTRA (INTrinsic Retrieval via Attention), a framework that unifies retrieval and generation within a single frozen pretrained encoder-decoder model, eliminating the external retriever typical of RAG pipelines. On multi-hop question-answering benchmarks—HotPotQA, 2WikiMultihopQA, and MuSiQue—using a T5Gemma2 4B model, INTRA outperforms strong engineered retrieval baselines (including BGE, Qwen3-Embedding-4B with a reranker, and hybrid RAG) on both complete-evidence recall and end-to-end answer quality, reaching 46.4% EM on HotPotQA versus 43.4% for the best RAG baseline. The framework achieves these gains by reusing the decoder's own cross-attention queries as retrieval scores—operationalized through learnable retrieval tokens that attend over pre-encoded chunk representations to score the full corpus—and by amortizing evidence encoding across queries, establishing that attention-based models already possess an internal retrieval mechanism that can be elicited rather than externally supplied.

2. Context and Motivation

The Core Gap: Retrieval and Generation Operate in Separate Representation Spaces

The central problem this paper addresses is architectural rather than methodological: standard RAG pipelines force a representation mismatch between retrieval and generation. In a typical RAG system, an external retriever—often a separately trained dense embedding model like BGE or Qwen3-Embedding—encodes queries and documents into one representation space for similarity scoring. The retrieved text is then fed into a language model that re-encodes it from raw tokens into an entirely different representation space for cross-attention during generation. These two spaces are trained independently, optimized for different objectives (retrieval relevance vs. language modeling), and never share parameters or representations.

The paper frames this as a missed opportunity. If the retriever and generator could share a single representation space, several problems would disappear: the retriever's notion of relevance would naturally align with what the generator needs for producing correct answers, evidence representations could be computed once and reused across queries rather than re-encoded per query, and the system would be architecturally simpler because there would be no external retriever to train, maintain, and integrate.

The question the paper asks is therefore: does a pretrained attention-based encoder-decoder already contain the machinery to perform retrieval internally, using its own representations? This is a capability question more than an engineering question. The hypothesis is that cross-attention—which already scores decoder query states against encoder key states to decide which evidence to attend to—is fundamentally a retrieval mechanism, and that this mechanism can be repurposed for explicit evidence selection at the chunk level.

Why This Matters: Practical and Conceptual Stakes

The practical motivation is straightforward. RAG systems are composed of multiple moving parts—retriever, reranker, generator, each with its own training data, hyperparameters, and failure modes—and these parts are typically developed by separate teams using separate codebases. The paper notes (Section 1.1) that this modularity is "often helpful, but it can obscure an important fact: attention is already a query-conditioned mechanism for selecting and weighting relevant information." If that mechanism can be surfaced and exploited directly, the entire retrieval subsystem could be collapsed into the generator, reducing system complexity and eliminating a class of integration failures where the retriever and generator are misaligned.

There is also a computational motivation that the paper develops carefully. In standard RAG, even though the corpus can be encoded offline into an index, once chunks are retrieved for a specific query, the generator must re-encode them from raw text during the prefill phase. For a query of length LqL_q and kk retrieved chunks of length LcL_c, this prefill costs O((Lq+kLc)2)O((L_q + kL_c)^2) under dense self-attention—a cost that grows quadratically with the amount of retrieved evidence. In INTRA, because the model retrieves pre-encoded chunk representations rather than raw text, the prefill cost drops to O(Lq(Lq+kLc))O(L_q(L_q + kL_c)) (Table 3 of the paper). The encoder states are computed once offline, stored, and fed directly into the decoder's cross-attention as key-value memory. This makes evidence encoding an amortized, one-time cost rather than a per-query cost.

The paper frames this as an amortized-encoding regime that emerges when static evidence (like a Wikipedia corpus) is encoded once and reused across many queries. Section 5.3 quantifies the practical benefit: for k=500k=500 retrieved chunks, INTRA's time-to-first-token (TTFT) is approximately 66ms versus 1.25s for standard RAG—a roughly 19×19\times speedup in the generator-side prefill (Figure 4). This is the regime in which the approach is most compelling.

Conceptually, the paper is making a broader argument about what attention-based models "know" how to do. The title—"Retrieval from Within"—is deliberately provocative: it suggests that retrieval is not something that needs to be added to these models through external systems, but rather something that can be elicited from what is already inside them. This reframes the relationship between LLMs and retrieval from "LLMs need retrieval" to "LLMs are already retrievers in a meaningful sense, and we need to surface that capability."

Prior Approaches: What Exists and Where It Falls Short

The paper situates itself against several lines of prior work, each with specific limitations that INTRA addresses:

1. Modular RAG pipelines (Lewis et al., 2020; Karpukhin et al., 2020). The standard architecture couples a dense retriever (e.g., DPR) with a sequence-to-sequence generator (e.g., T5). Retrieval produces text chunks; the generator re-encodes them. This is effective and widely deployed, but the paper identifies three specific failure modes:

  • Representation mismatch: The retriever's embedding space is optimized for retrieval relevance—roughly, "does this passage contain information relevant to the query?"—while the generator's cross-attention queries express a different need: "given what I've generated so far, what information would be most useful to attend to next?" These are correlated but not identical objectives. A passage that scores highly under a cosine-similarity retriever may not contain the specific detail the decoder needs at a particular generation step.

  • No feedback loop: Because the retriever is external and runs once before generation begins, it cannot adapt to what the generator has already produced. If the generator needs supplementary evidence halfway through generating an answer, the standard RAG pipeline has no mechanism to retrieve it. This is why agentic RAG systems (Asai et al., 2024; Li et al., 2025) add iterative retrieval-while-generating loops—but these add further complexity on top of an already modular architecture.

  • Repeated re-encoding: As discussed above, the generator re-encodes retrieved text from raw tokens for every query, even when the same evidence chunks are retrieved repeatedly across different queries.

2. Late-interaction retrieval (Khattab and Zaharia, 2020; Santhanam et al., 2022). ColBERT and its variants score query-document pairs via token-level MaxSim operations—the same matching primitive INTRA uses—and achieve strong retrieval performance by comparing every query token against the best-matching document token. However, ColBERT-style systems are dedicated retrievers: they produce a ranked list of documents, and a separate generator must then consume those documents. The late-interaction matching signal, which is expensive to compute and rich in information, is discarded after retrieval rather than being reused during generation. INTRA inherits the MaxSim scoring mechanism but embeds it within the generator itself, so the same representations used to score chunks are also used to condition generation.

The paper makes this contrast explicit in Section 6: "Whereas late-interaction systems rely on a dedicated retriever to score query-document matches, INTRA lets the decoder's own cross-attention perform this matching and then consume the matched representations during generation." This is the key architectural difference: retrieval and generation are not two systems that communicate through text; they are two passes through the same model operating on the same representations.

3. Jointly trained retrieval-generation models (Guu et al., 2020; Izacard et al., 2023; Lin et al., 2024). Several prior works have trained retrievers and generators together rather than treating them as independent modules. REALM (Guu et al., 2020) jointly pretrains a retriever with a masked language model by backpropagating through the retrieval step. Atlas (Izacard et al., 2023) jointly pretrains a retriever with an encoder-decoder for few-shot learning. RA-DIT (Lin et al., 2024) instruction-tunes both components. These approaches reduce the representation mismatch because the retriever and generator are trained with a shared objective. However, they still maintain a separate retriever architecture with its own parameters, its own forward pass, and its own embedding space—they reduce the mismatch but do not eliminate it.

The paper's relationship to CLaRa (He et al., 2026) is particularly instructive. CLaRa jointly optimizes reranking and generation over latent representations, making it the closest prior work to INTRA. The paper explicitly distinguishes INTRA along three axes (Section 6): INTRA (i) reuses the model's native representation space without training a compression model, (ii) performs full-corpus scoring rather than reranking an initial candidate set, and (iii) operates with a frozen encoder-decoder rather than requiring end-to-end training. These differences are substantive: in CLaRa, the retriever-generator integration requires training a compression model to map documents into a latent space and jointly fine-tuning reranking and generation. INTRA achieves similar integration with a frozen model by reparameterizing the attention computation (the Reverse-QWK transformation, Section 3.1 and Appendix A) so that the decoder's existing cross-attention machinery can operate directly on stored encoder states.

4. Attention as memory (Sukhbaatar et al., 2015; Borgeaud et al., 2022; Behrouz et al., 2025). A separate line of work conceptualizes attention as a content-based memory access mechanism. Memory Networks (Sukhbaatar et al., 2015) frame QA as differentiable lookup over stored memories. RETRO (Borgeaud et al., 2022) injects retrieved chunks into a decoder's cross-attention layers, combining external retrieval with internal processing. Titans (Behrouz et al., 2025) argues that long-context modeling requires explicit memory mechanisms beyond larger attention windows.

These works share the high-level intuition that attention and memory retrieval are deeply related, but they differ from INTRA in where the memory comes from. Memory Networks and Titans learn a memory representation during training. RETRO retrieves chunks into the model's computation but still uses a separate frozen retriever to identify those chunks. INTRA occupies a distinct position: it neither learns a memory representation nor relies on an external retriever; instead, it reuses the model's own encoder activations over an evidence pool as the memory, and uses the decoder's own attention queries as the retrieval mechanism. The memory already exists inside the model; INTRA provides the interface for accessing it.

5. Long-context modeling (Beltagy et al., 2020; Dao and Gu, 2024). The paper acknowledges that long-context models could theoretically subsume retrieval by packing all evidence into the prompt. However, it cites evidence (Yen et al., 2024; Modarressi et al., 2025) that such models "remain brittle when the relevant evidence is sparse and distributed"—the classic needle-in-a-haystack problem. Moreover, even linear-complexity architectures like Mamba (Gu and Dao, 2024) that make long-context processing feasible do not address the fundamental issue that the model must process all evidence for every query, with no amortization across queries. The paper's position (Section 6) is not that INTRA replaces long-context modeling, but that it addresses a complementary problem: "how to identify and use evidence reliably when relevant information is sparse relative to the full available corpus."

How This Paper Positions Itself

The paper's framing is more ambitious than "here is a better retriever." It positions INTRA as evidence for a capability claim: that pretrained attention-based models possess an intrinsic retrieval mechanism that can be surfaced without architectural modification or large-scale retraining. The word "intrinsic" in the title is deliberately chosen—it implies that retrieval is not something the model learns to do during INTRA's lightweight training (which only updates ~164K retrieval token embeddings and 272 aggregation weights, keeping the 4B-parameter backbone frozen), but rather something the model already knew how to do and simply needed the right interface to express.

This claim is operationalized through three design choices that together distinguish INTRA from prior work:

  1. Shared representation space (Section 2.2). The same encoded chunk states ki=Enc(ti)\mathbf{k}_i = \text{Enc}(t_i) are used for both evidence scoring (via MaxSim against decoder queries) and answer generation (as cross-attention key-value memory). There is no translation between a "retrieval embedding" and a "generator embedding"—they are the same thing. This is made technically possible by the Reverse-QWK transformation (Section 3.1, Appendix A), which reparameterizes the decoder's cross-attention so that all layers operate against a single head-agnostic encoder representation Kˉ\bar{\mathbf{K}} rather than requiring layer-specific key projections.

  2. Decoder-side retrieval queries (Section 2.2). Rather than training a separate query encoder, INTRA augments the decoder's input with RR trainable retrieval tokens ρ∈RR×d\rho \in \mathbb{R}^{R \times d}. During a retrieval forward pass, the decoder's cross-attention query states qℓ\mathbf{q}_\ell at these token positions become retrieval queries—they naturally incorporate the query text (via self-attention with the input tokens preceding them) and the initial context (via cross-attention with K(S0)\mathbf{K}(\mathcal{S}_0)). The retrieval tokens act as a learned interface that asks "what does the decoder need?" in a form that can be scored against the chunk pool.

  3. Full-corpus scoring, not reranking (Section 2.3, Figure 3). Many prior systems that integrate retrieval and generation (like CLaRa) use a cheap initial retriever to narrow the corpus to a candidate set, then apply expensive cross-attention-based scoring to rerank that set. INTRA's design supports full-corpus MaxSim scoring—the retrieval scores sis_i are computed against all MM chunks, not just an initial candidate set S0\mathcal{S}_0. The initial set S0\mathcal{S}_0 serves only to provide the decoder with cross-attention context during the retrieval pass; it does not constrain which chunks can be scored. Figure 3 shows that this matters: reranking S0\mathcal{S}_0 helps, but full-corpus INTRA scoring yields the largest gains by recovering evidence that the initial retrieval missed entirely.

The paper's empirical positioning is calibrated: it does not claim INTRA is universally superior to all RAG baselines on all metrics. On Natural Questions—a predominantly single-hop benchmark—INTRA's retrieval recall (29.1% R@5) slightly lags behind Qwen3-Embedding-4B (30.3%) and BGE (29.6%), and its end-to-end EM (51.2%) trails Qwen3-Embedding-4B (54.5%) and the Qwen + reranker combination (55.1%). The paper is transparent about this (Table 1, Table 4) and frames it as expected: single-hop retrieval is a setting where dedicated embedding models—trained on large-scale retrieval corpora that include NQ supervision—have a natural advantage, while INTRA's strength emerges in the multi-hop setting where evidence must be assembled from multiple chunks (e.g., HotPotQA: 59.9% R@5 vs. 54.8% for BGE, 2Wiki: 40.7% vs. 35.4% for Qwen + reranker).

This difficulty-dependent pattern reinforces the paper's conceptual argument: decoder attention queries are a strong signal for what evidence the generator needs because they directly express the information requirements of the answer-generation process. For single-hop questions, this signal adds less value over simple query-document similarity because the retrieval problem is easier. For multi-hop questions requiring evidence assembly, the decoder's queries provide a richer signal than static query-chunk similarity because they implicitly encode which pieces of evidence need to be combined. This is the paper's central explanatory hypothesis, and the empirical pattern supports it.

The paper also positions INTRA as a baseline system rather than a production-ready solution. Section 7 explicitly limits the scope: "We discuss the practicality of a billion-token corpus in App. A.3, but do not position INTRA as a replacement for RAG in open-web retrieval or web-scale settings." The contribution is primarily conceptual—demonstrating what is possible with a frozen pretrained model—rather than engineering a system that beats all baselines on all metrics at scale. This intellectual honesty is important for understanding the paper's contribution: it is not "INTRA is better than RAG," but rather "retrieval is an intrinsic capability that attention-based models possess, and exposing it yields a simpler architecture that performs competitively with or better than modular RAG on multi-hop reasoning tasks."

3. Technical Approach

3.1 Reader Orientation

The paper builds INTRA, a framework that lets a single frozen pretrained encoder-decoder model — specifically T5Gemma2 4B — perform both evidence retrieval and answer generation using its own internal representations, without any external retriever. The core problem it solves is the representation mismatch in standard RAG pipelines where a dedicated retriever operates in one embedding space while the generator re-encodes evidence from raw text into a different space; INTRA's solution shape is to reuse the encoder's pre-computed chunk representations for both scoring relevance (via the decoder's cross-attention queries) and conditioning generation (as cross-attention key-value memory), collapsing two systems into one shared representation space.

3.2 Big-Picture Architecture (Diagram in Words)

INTRA has five major components connected through two sequential forward passes through the same frozen decoder:

1. Pre-Encoding Engine (offline, once per corpus). The pretrained T5Gemma2 encoder maps every text chunk $t_i$ in the knowledge corpus to token-level representations $\mathbf{k}_i = \text{Enc}(t_i) \in \mathbb{R}^{L_c \times d}$. These representations are then normalized via RMSNorm and stored as a single head-agnostic pool $\bar{\mathbf{K}} = \{\bar{\mathbf{k}}_i\}_{i=1}^M$ that all decoder layers share (enabled by the Reverse-QWK reparameterization). This is a one-time cost amortized across all future queries.

2. Initial Coarse Retriever (per query). For a new question $x$, the encoder produces a query representation $\mathbf{k}_x = \text{Enc}(x)$. The system computes MaxSim similarity between $\mathbf{k}_x$ and every chunk in the pool to produce initial scores $s_i^{(0)}$, then selects the top-$n_0$ chunks as $\mathcal{S}_0$ (where $n_0 = 20$ in all experiments). This initial set provides cross-attention context for the retrieval pass but does not constrain which chunks can ultimately be retrieved.

3. Retrieval Pass (per query, first decoder forward pass). The decoder processes the question augmented with $R$ trainable retrieval tokens $\rho \in \mathbb{R}^{R \times d}$ appended to the input: $x_{\text{retrieval}} = [x_1, \dots, x_{L_q}, \rho_1, \dots, \rho_R]$. The retrieval tokens attend to the question tokens via self-attention and to the initial evidence $\mathbf{K}(\mathcal{S}_0)$ via cross-attention. The cross-attention query states $\mathbf{q}_\ell$ at the retrieval-token positions — which now encode both the question and the initial evidence context — are extracted and transformed via Reverse-QWK into $\tilde{\mathbf{q}}_\ell$. These transformed queries are scored against every chunk in the full corpus pool $\bar{\mathbf{K}}$ using MaxSim, and scores are aggregated across layers with learned weights $\alpha_\ell$ to produce a single relevance score $s_i$ per chunk.

4. Chunk Selection (per query, between passes). The system ranks all $M$ chunks by their retrieval scores $s_i$ and selects the top-$n$ as $\mathcal{S}_{\text{INTRA}}$. At evaluation, $n = 5$ total chunks are used for generation: the top 4 from $\mathcal{S}_{\text{INTRA}}$ plus the top 1 from $\mathcal{S}_0$. This selection is a full-corpus ranking operation — chunks from outside the initial set $\mathcal{S}_0$ can be and are selected.

5. Generation Pass (per query, second decoder forward pass). The decoder processes the original question $x$ (without retrieval tokens) with cross-attention over the selected evidence $\mathbf{K}(\mathcal{S}_{\text{INTRA}})$. Because these encoder states were pre-computed, the decoder never re-encodes raw text — it directly attends to the stored representations. The answer $y$ is produced via standard autoregressive decoding from the final decoder state: $y = \text{Dec}(x, \mathbf{K}(\mathcal{S}_{\text{INTRA}}))$.

The information flow is: question enters → encoder produces query vector → MaxSim against chunk pool produces initial candidate set $\mathcal{S}_0$ → retrieval-augmented decoder forward pass produces relevance scores for all chunks → top chunks selected as $\mathcal{S}_{\text{INTRA}}$ → second decoder forward pass attends to selected evidence and generates answer. Crucially, the same frozen model performs both retrieval and generation; the only trained parameters are the retrieval token embeddings (~164K parameters) and the layer aggregation weights (272 parameters).

3.3 Roadmap for the Deep Dive

  • First, the formal framework that defines the retrieval-and-generation setting and the decoder's computation graph, establishing notation and the interface between encoder outputs and decoder attention. This is necessary to understand what problem INTRA is solving mechanistically.

  • Second, how attention-based retrieval works: the retrieval token augmentation, the MaxSim scoring mechanism, and the conversion of token-level attention queries into chunk-level retrieval scores. This is the core retrieval mechanism that makes INTRA work.

  • Third, the initial context selection mechanism and why it matters that INTRA performs full-corpus scoring rather than merely reranking an initial candidate set. This distinguishes INTRA from reranking-based approaches.

  • Fourth, the Reverse-QWK transformation — the implementation-level reparameterization that makes shared representations computationally feasible by avoiding layer-specific key projections. This is the engineering insight that enables the conceptual design.

  • Fifth, the retrieval training objective and what is being optimized during the lightweight training phase. This explains how the retrieval tokens learn to elicit useful retrieval behavior from a frozen model.

  • Sixth, the pooled-chunk approximation for efficient MaxSim scoring, addressing the computational bottleneck of token-level matching.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and capability-demonstration paper whose core idea is that the cross-attention mechanism inside a pretrained encoder-decoder already implements a retrieval operation — scoring decoder queries against encoded evidence — and that this operation can be surfaced for explicit chunk-level evidence selection without changing the model architecture or fine-tuning its backbone parameters.

3.4.1 Formal Framework: Defining the Retrieval-and-Generation Setting

The paper sets up a structured retrieval-and-generation problem. There is a fixed corpus of text chunks $\mathcal{T} = \{t_i\}_{i=1}^M$, a query (question) $x$, and a desired answer $y$. The task has two sub-problems: select a small set of relevant chunks $\mathcal{S} \subseteq \{1, \dots, M\}$ from the corpus, and then generate the answer conditioned on those chunks.

In standard RAG, the selected set comes from an external retrieval function $\mathcal{S} = \text{retrieve}(x, \mathcal{T})$, and the generator is typically a separate language model that receives the retrieved text concatenated with the query: $y = \text{Dec}(x, \mathcal{T}(\mathcal{S}))$, where $\mathcal{T}(\mathcal{S})$ is the concatenated raw text of the selected chunks.

INTRA changes this in one crucial way: rather than feeding raw text to the decoder, it feeds pre-encoded representations. Let $\text{Enc}$ be the pretrained encoder that maps text to token-level representations:

ki=Enc(ti)∈RLc×d\mathbf{k}_i = \text{Enc}(t_i) \in \mathbb{R}^{L_c \times d}

where $L_c$ is the number of tokens in the chunk and $d$ is the model's hidden dimension (2560 for the T5Gemma2 4B model used in experiments). The full pre-encoded corpus is $\mathbf{K} = \{\mathbf{k}_i\}_{i=1}^M$. The encoded context for a selected set $\mathcal{S}$ is simply the concatenation along the token dimension:

K(S)=[ki:i∈S]\mathbf{K}(\mathcal{S}) = [\mathbf{k}_i : i \in \mathcal{S}]

What this computes: the encoder runs once per chunk offline, producing a $L_c \times d$ matrix of token representations that captures the chunk's semantic content in the model's own representation space. These matrices are stored for later reuse.

Why this form: encoding once and reusing shifts the computational burden from per-query (standard RAG must re-encode retrieved text every time) to one-time offline. For a static corpus queried many times, this amortization is the key efficiency argument. The representations live in the model's native space — there is no separate embedding model, no compression, no translation between spaces.

The decoder then generates the answer by attending over these pre-encoded representations:

y=Dec(x,K(S))y = \text{Dec}(x, \mathbf{K}(\mathcal{S}))

To make the decoder's computation explicit, the paper isolates the cross-attention step. Let $\mathbf{h}^0 = x$ be the initial decoder state (the input token embeddings), and let $\mathbf{q}^\ell$ denote the query-side state that enters cross-attention at decoder layer $\ell$. The internal recurrence for each layer is:

qℓ=Ψℓ(hℓ−1)\mathbf{q}^\ell = \Psi^\ell(\mathbf{h}^{\ell-1})

zℓ=Attention(qℓ,K(S),K(S))\mathbf{z}^\ell = \text{Attention}(\mathbf{q}^\ell, \mathbf{K}(\mathcal{S}), \mathbf{K}(\mathcal{S}))

hℓ=Φℓ(hℓ−1,zℓ)\mathbf{h}^\ell = \Phi^\ell(\mathbf{h}^{\ell-1}, \mathbf{z}^\ell)

for $\ell = 1, \dots, L$, where $L = 34$ for T5Gemma2 4B.

What this computes: $\Psi^\ell$ is the layer-specific transformation that produces the query state — this includes self-attention over the decoder's previous states, feed-forward transformations, and residual connections up to the point just before cross-attention. $\text{Attention}(\mathbf{q}^\ell, \mathbf{K}(\mathcal{S}), \mathbf{K}(\mathcal{S}))$ is standard cross-attention where the decoder queries attend to the encoder states as both keys and values. $\Phi^\ell$ is the post-cross-attention processing. The final output is $y = \text{Out}(\mathbf{h}^L)$ where $\text{Out}$ is the text generation head (producing logits over the vocabulary).

Why this decomposition matters: it isolates $\mathbf{q}^\ell$ — the cross-attention query state — as the critical interface between the decoder and the encoded evidence. These query states encode what the decoder "wants to know" at layer $\ell$ given what it has generated so far, and the attention scores $\text{softmax}(\mathbf{q}^\ell \mathbf{K}(\mathcal{S})^\top / \sqrt{d})$ determine which evidence tokens it attends to. The paper's central insight is that these same query states can be used for retrieval — they already constitute a relevance signal, and the framework provides a way to extract and use that signal at the chunk level rather than the token level.

The paper also defines a notation for exposing these query states: $\mathbf{q}^\ell = g\text{Dec}^\ell(x, \mathbf{K}(\mathcal{S}))$, meaning "run the decoder forward pass up to the point where layer $\ell$'s cross-attention query is computed, given input $x$ and context $\mathbf{K}(\mathcal{S})$, and return that query state."

3.4.2 Attention-Based Retrieval: Converting Cross-Attention Queries into Chunk Scores

This is the core retrieval mechanism. The goal is to produce a single relevance score $s_i$ for each chunk $i$ in the corpus using the decoder's own cross-attention machinery.

Step 1: Augment the input with retrieval tokens.

The input to the retrieval pass is not just the question $x$, but $x$ augmented with $R$ trainable retrieval tokens:

xretrieval=[x1,…,xLq,ρ1,…,ρR]x_{\text{retrieval}} = [x_1, \dots, x_{L_q}, \rho_1, \dots, \rho_R]

where each $\rho_r \in \mathbb{R}^d$ is a learnable embedding vector. In the experiments, $R = 64$ retrieval tokens are used (though ablations in Table 7 test 16 and 1 token). These tokens have no semantic meaning initially — they are randomly initialized — but during retrieval training they learn to elicit cross-attention queries that are maximally informative for evidence selection.

What this computes: the retrieval tokens are appended to the question's token embeddings to form a single sequence. During the forward pass, self-attention allows the retrieval tokens to attend to the question tokens, and cross-attention allows them to attend to the initial evidence context $\mathbf{K}(\mathcal{S}_0)$. This means the query states $\mathbf{q}^\ell$ at the retrieval-token positions encode information about both the question and the initial evidence — they are not just static query embeddings but context-conditioned queries.

Why this form: appending retrieval tokens to the input means the retrieval mechanism requires no architectural changes to the decoder. The retrieval tokens participate in the same self-attention and cross-attention computations as the question tokens. This is a lightweight, decoder-side mechanism that can be added to any pretrained encoder-decoder. The number of retrieval tokens $R$ controls the capacity of the retrieval signal: more tokens means the decoder can express more nuanced information needs, but also increases the cost of the retrieval forward pass and the MaxSim computation (since scores are aggregated over $R$ token positions).

Step 2: Compute retrieval scores via MaxSim.

The retrieval pass runs the decoder forward: $g\text{Dec}(x_{\text{retrieval}}, \mathbf{K}(\mathcal{S}_0))$ exposes the cross-attention query states $\mathbf{q}^\ell$ at each layer. Only the states at the $R$ retrieval-token positions are used for scoring — the states at the question-token positions are discarded for retrieval purposes.

The paper uses a scaled ColBERT-style late-interaction score MaxSim to convert token-level query states into chunk-level scores. For a query representation $\mathbf{u} \in \mathbb{R}^{L_u \times d}$ (the retrieval-token query states from one layer) and a chunk representation $\mathbf{v} \in \mathbb{R}^{L_v \times d}$:

MaxSim(u,v)=∑a=1Lumax⁡1≤b≤Lv(uv⊤d)a,b\text{MaxSim}(\mathbf{u}, \mathbf{v}) = \sum_{a=1}^{L_u} \max_{1 \leq b \leq L_v} \left(\frac{\mathbf{u} \mathbf{v}^\top}{\sqrt{d}}\right)_{a,b}

What this computes: For each query token $a$ (one of the $L_u = R$ retrieval-token positions), find the single chunk token $b$ with the highest scaled dot-product similarity, and sum these maximum similarities over all query tokens. The result is a single scalar measuring how well the chunk's tokens match the decoder's query states, with each query token getting exactly one "vote" (its best match).

Why this form: MaxSim uses the same $1/\sqrt{d}$ scaling as attention — the authors note this factor has no effect on ranking (since it's a constant multiplier), but including it makes the relationship to attention scores explicit. The key property of MaxSim is that it performs a late interaction: query and chunk representations are computed independently, and the matching happens at the very end via token-level aggregation. This is more expressive than single-vector similarity (which would collapse each chunk to one embedding and compute a single dot product) because it preserves the token-level structure, allowing the query to match different aspects against different chunk tokens. Unlike attention, MaxSim takes a max over chunk tokens (not a softmax), which means it is not a weighted average — it selects the single best match per query token, capturing whether the chunk contains the specific information each query token is looking for.

Step 3: Aggregate across layers and select chunks.

The retrieval query states are extracted at every decoder layer using the notation $\mathbf{q}^\ell = g\text{Dec}^\ell(x_{\text{retrieval}}, \mathbf{K}(\mathcal{S}_0))$. Then they are transformed via Reverse-QWK (explained in Section 3.4.3) into $\tilde{\mathbf{q}}^\ell$ (the query-side equivalent that operates against the shared normalized encoder pool). The per-layer scores are aggregated with learned scalar weights $\alpha_\ell$:

si=∑ℓαℓ MaxSim(q~ℓ,kˉi)s_i = \sum_{\ell} \alpha_\ell \, \text{MaxSim}(\tilde{\mathbf{q}}^\ell, \bar{\mathbf{k}}_i)

where $\bar{\mathbf{k}}_i = \text{RMSNorm}(\mathbf{k}_i)$ is the normalized encoder representation of chunk $i$ stored in the shared pool.

What this computes: For each decoder layer $\ell$, compute a MaxSim score between that layer's retrieval-token query states $\tilde{\mathbf{q}}^\ell$ and the normalized chunk representation $\bar{\mathbf{k}}_i$. Then take a weighted sum over layers with learned weights $\alpha_\ell$. The result $s_i$ is a single relevance score for chunk $i$.

Why this form: different decoder layers may express different information needs (lower layers might attend to local syntactic patterns, higher layers to semantic content), and the learned weights $\alpha_\ell$ allow the system to discover which layers produce the most useful retrieval signals. The weights are initialized and trained alongside the retrieval tokens during the lightweight training phase. The paper found through ablation (Table 7) that using 16 retrieval tokens instead of 64 drops complete-evidence recall@5 by 4.4 points on HotPotQA, indicating that retrieval capacity matters.

The final selection is a top-$n$ operation over the full corpus:

SINTRA={i∈{1,…,M}:si is among the top-n scores}\mathcal{S}_{\text{INTRA}} = \{i \in \{1, \dots, M\} : s_i \text{ is among the top-}n \text{ scores}\}

At evaluation, $n = 4$ chunks are taken from $\mathcal{S}_{\text{INTRA}}$ plus the top chunk from $\mathcal{S}_0$, for a total of 5 chunks used as generation context.

3.4.3 The Reverse-QWK Transformation: Enabling Shared Representations Across Layers

In a standard Transformer encoder-decoder like T5Gemma2, cross-attention does not operate directly on the stored encoder states $\mathbf{K}(\mathcal{S})$. Instead, layer-specific key projections are applied. The keys at layer $\ell$ are computed as:

kℓ=(kˉ⊙γK,ℓ)WK,ℓ\mathbf{k}_\ell = (\bar{\mathbf{k}} \odot \gamma_{K,\ell}) \mathbf{W}_{K,\ell}

where $\bar{\mathbf{k}} = \text{RMSNorm}(\mathbf{K}(\mathcal{S}))$ is the RMSNorm-normalized encoder representation, $\gamma_{K,\ell} \in \mathbb{R}^{d_h}$ is a learned per-head-dimension scale parameter, and $\mathbf{W}_{K,\ell} \in \mathbb{R}^{d \times n_{kv} d_h}$ is the key projection matrix (with $d_h$ the per-head dimension and $n_{kv}$ the number of key-value heads under Group-Query Attention). This means different layers have different key representations — the attention logits at layer $\ell$ are computed against $\mathbf{k}_\ell$, not against $\bar{\mathbf{k}}$.

The problem this creates for INTRA: If each layer needs its own $\mathbf{k}_\ell$ to compute MaxSim scores, the system would need to store $L$ separate per-layer key representations for every chunk, blowing up storage by a factor of $L$ (34× for T5Gemma2 4B) and preventing the use of a single approximate nearest neighbor (ANN) index shared across layers. This would make the shared-representation design computationally infeasible.

The Reverse-QWK solution: Move the key projection and learned scale from the encoder (key) side to the decoder (query) side. Define a transformed query:

q~ℓ=(qℓWK,ℓ⊤)⊙γK,ℓ\tilde{\mathbf{q}}^\ell = (\mathbf{q}^\ell \mathbf{W}_{K,\ell}^\top) \odot \gamma_{K,\ell}

What this computes: Instead of applying $\mathbf{W}_{K,\ell}$ and $\gamma_{K,\ell}$ to the encoder states $\bar{\mathbf{k}}$ to produce layer-specific keys, apply them to the decoder query states $\mathbf{q}^\ell$ to produce transformed queries $\tilde{\mathbf{q}}^\ell$. The transformed query lives in the same $d$-dimensional space as the original encoder representation $\bar{\mathbf{k}}$.

Why this is correct (the derivation):

\begin{align} \mathbf{q}^\ell \mathbf{k}_\ell^\top &= \mathbf{q}^\ell (\bar{\mathbf{k}} \odot \gamma_{K,\ell} \, \mathbf{W}_{K,\ell})^\top \\ &= \mathbf{q}^\ell \mathbf{W}_{K,\ell}^\top \, \text{diag}(\gamma_{K,\ell}) \, \bar{\mathbf{k}}^\top \\ &= ((\mathbf{q}^\ell \mathbf{W}_{K,\ell}^\top) \odot \gamma_{K,\ell}) \, \bar{\mathbf{k}}^\top \\ &= \tilde{\mathbf{q}}^\ell \, \bar{\mathbf{k}}^\top \end{align}

The step from line 2 to 3 uses the identity that for vectors $\mathbf{a}, \mathbf{b}$, we have $\mathbf{a} \, \text{diag}(\mathbf{b}) \, \bar{\mathbf{k}}^\top = (\mathbf{a} \odot \mathbf{b}) \, \bar{\mathbf{k}}^\top$ (the diagonal matrix multiplication is equivalent to element-wise multiplication of the projected query with the scale). The result is that the attention logits $\tilde{\mathbf{q}}^\ell \bar{\mathbf{k}}^\top / \sqrt{d_h}$ are mathematically identical to the standard computation $\mathbf{q}^\ell \mathbf{k}_\ell^\top / \sqrt{d_h}$, but now they use a single shared encoder representation $\bar{\mathbf{k}}$ against transformed queries.

What this enables: Because $\bar{\mathbf{k}}$ is the same for all layers, the system stores only one representation per chunk (the RMSNorm-normalized encoder output). A single ANN index can be built over this pool and shared across all layers. The per-layer computation is pushed entirely to the query side — for each layer, the query $\mathbf{q}^\ell$ is transformed by $\mathbf{W}_{K,\ell}^\top$ and $\gamma_{K,\ell}$, both of which are small operations on the query (which has $L_q + R$ tokens, typically a few hundred) rather than on the chunk (which may have hundreds of thousands or millions of tokens). This is the algorithmic insight that makes shared representations practical.

Group-Query Attention (GQA) handling: T5Gemma2 uses Group-Query Attention where $n_{kv} \leq n_h$ (number of key-value heads is less than or equal to number of query heads). Under GQA, each KV head is shared by $n_{\text{rep}} = n_h / n_{kv}$ query heads. The per-head Reverse-QWK transformation is:

q~(h)ℓ=(q(h)ℓ⊙γK,ℓ)(WK,ℓ(g(h)))⊤∈Rd\tilde{\mathbf{q}}^\ell_{(h)} = (\mathbf{q}^\ell_{(h)} \odot \gamma_{K,\ell}) (\mathbf{W}^{(g(h))}_{K,\ell})^\top \in \mathbb{R}^d

where $g(h) = \lfloor h / n_{\text{rep}} \rfloor$ maps Q-head $h$ to its corresponding KV-group, and $\mathbf{W}^{(g)}_{K,\ell} \in \mathbb{R}^{d \times d_h}$ is the key projection for that group.

What this computes: Each query head $h$ uses the key projection $\mathbf{W}^{(g(h))}_{K,\ell}$ of the KV group it belongs to. The per-head dimension $d_h$ query is element-wise multiplied by $\gamma_{K,\ell}$ (which is also per-head-dimension), then projected into the full $d$-dimensional encoder space by $\mathbf{W}^{(g(h))}_{K,\ell}$. The dot product $\tilde{\mathbf{q}}^\ell_{(h)} \bar{\mathbf{k}}^\top$ is then computed and scaled by $1/\sqrt{d_h}$ (the standard per-head attention scale in T5Gemma2).

Why this matters for storage: In standard GQA cross-attention, the key representations must be expanded to $n_h$ heads (replicating each KV head's keys $n_{\text{rep}}$ times), costing $O(N n_h d_h)$ memory where $N$ is the number of encoder tokens. Under Reverse-QWK, only the head-agnostic pool $\bar{\mathbf{k}} \in \mathbb{R}^{N \times d}$ is materialized, costing $O(N d)$ regardless of $n_h$. The GQA replication happens entirely on the (small) query side. For the T5Gemma2 4B model with $L = 34$ layers and $n_{kv} d_h \approx d / 2.5$, the compression ratio for cached encoder states is approximately $2L n_{kv} d_h / d \approx 30\times$ (Appendix A.3). This is the practical enabler for storing and searching over a corpus of hundreds of thousands of chunks.

Position encoding handling: In T5Gemma2, Rotary Position Embeddings (RoPE) are applied to decoder queries but not to encoder representations during cross-attention (this is enforced by skipping the encoder prefix when applying RoPE to the merged KV stream). Reverse-QWK preserves this: RoPE is applied to $\mathbf{q}^\ell$ before the Reverse-QWK transformation, while $\bar{\mathbf{k}}$ remains positionally invariant. This means $\bar{\mathbf{k}}$ can be precomputed once without position information, and the ANN index built over it does not need to account for positional structure.

Implementation detail (Appendix A.2): The paper provides PyTorch-style pseudocode showing that the structural changes are minimal: $\mathbf{W}_k$ and $\gamma_k$ are moved from the key-side projection to a query-side transformation, and the key input to attention becomes the head-agnostic pool $\bar{\mathbf{k}}$. The value path ($\mathbf{V} = \bar{\mathbf{k}} \mathbf{W}_V$) and the rest of attention are unchanged.

3.4.4 Initial Context Selection for the Retrieval Pass

The retrieval pass needs some cross-attention context to produce meaningful query states — without any evidence to attend to, the cross-attention is effectively the identity function ($\mathbf{z}^\ell = \mathbf{q}^\ell$ in Eq. 1) and the query states encode only the question, not any interaction with evidence.

The initial selection mechanism (Section 2.3): The system encodes the question $x$ with the same encoder: $\mathbf{k}_x = \text{Enc}(x) \in \mathbb{R}^{L_q \times d}$. It then computes MaxSim between the question representation and every chunk in the pool:

si(0)=MaxSim(kx,ki)s_i^{(0)} = \text{MaxSim}(\mathbf{k}_x, \mathbf{k}_i)

and selects the top-$n_0$ chunks (with $n_0 = 20$ in all experiments) as the initial context set $\mathcal{S}_0$.

What this computes: A coarse, encoder-only relevance score between the question and each chunk. This is similar to standard dense retrieval but uses the model's own encoder rather than a separately trained embedding model, and uses MaxSim (token-level late interaction) rather than single-vector cosine similarity. The result is a small set of chunks that are likely to be relevant and that the decoder uses as cross-attention context during the retrieval pass.

Why not just use $\mathcal{S}_0$ as the final retrieval set? Table 7 (ablation) answers this directly: setting $\mathcal{S}_{\text{INTRA}} = \mathcal{S}_0$ (using only initial retrieval) drops complete-evidence recall@5 from 59.9% to 37.1% on HotPotQA (-22.8 points) and from 40.7% to 17.7% on 2Wiki (-23.0 points). The initial selection is a useful starting point but is substantially weaker than the full INTRA retrieval pipeline.

Why not use an empty initial context $\mathcal{S}_0 = \emptyset$? Table 7 also tests this: performance drops to 26.9% CE-recall@5 on HotPotQA (-33.0 points). Without cross-attention context, the retrieval tokens cannot interact with evidence during the retrieval forward pass, so their query states encode only the question — they lose the ability to condition on and compare against actual evidence representations. This demonstrates that the retrieval pass benefits from having some evidence to attend to, even if that evidence is imperfect.

Relationship to reranking (Figure 3): The paper explicitly distinguishes full-corpus INTRA scoring from reranking. When evaluating $\mathcal{S}_0$ reranked by the decoder scores $s_i$ (i.e., compute $s_i$ only for chunks in $\mathcal{S}_0$ and reorder them), performance improves over $\mathcal{S}_0$ alone — but full-corpus INTRA scoring (evaluating $s_i$ for all $M$ chunks) yields the largest gains. This is because INTRA can recover relevant chunks that the coarse initial selection missed entirely. Figure 3 quantifies this on HotPotQA and 2Wiki, showing that reranking $\mathcal{S}_0$ helps but that the largest jump comes from full-corpus scoring.

Alternative initial similarity metrics (Table 7): The paper ablates using cosine similarity (single-vector) instead of MaxSim for the initial selection: CE-recall@5 drops to 30.1% on HotPotQA (-29.8 points). This confirms that the token-level late interaction in MaxSim provides a better initial selection signal than single-vector similarity, consistent with the ColBERT literature.

3.4.5 Retrieval Training Objective

The only parameters trained in INTRA are the retrieval token embeddings $\rho_r \in \mathbb{R}^d$ (64 tokens × 2560 dimensions ≈ 164K parameters) and the layer aggregation weights $\alpha_\ell$ (272 parameters for 34 layers with GQA heads). The encoder and decoder backbones remain frozen throughout.

Training data: For each question $x$ in the training set, the system has access to oracle evidence chunks $\mathcal{O}(x) \subseteq \{1, \dots, M\}$ — the chunks annotated as containing supporting evidence for the correct answer. These oracle annotations come from the benchmark datasets (HotPotQA, 2Wiki, MuSiQue, NQ).

Loss function: The training objective is a soft cross-entropy loss that encourages the retrieval scores $s_i$ to assign equal probability mass to all oracle chunks:

Lretrieval=−1∣O(x)∣∑j∈O(x)log⁡(softmax(s)j)\mathcal{L}_{\text{retrieval}} = -\frac{1}{|\mathcal{O}(x)|} \sum_{j \in \mathcal{O}(x)} \log(\text{softmax}(\mathbf{s})_j)

where $\mathbf{s} = [s_1, \dots, s_M]$ is the vector of retrieval scores for all $M$ chunks.

What it computes: Apply softmax to the retrieval scores to get a probability distribution over chunks. Then compute the average negative log-probability assigned to the oracle chunks. If there are three oracle chunks, each should ideally receive probability $1/3$ (the loss is minimized when $\text{softmax}(\mathbf{s})_j = 1/|\mathcal{O}(x)|$ for each oracle chunk $j$). The loss is a single non-negative scalar.

Why this form: soft cross-entropy with equal target mass treats all oracle chunks as equally important — the model is not forced to rank one oracle chunk above another. This is appropriate because the downstream generator benefits from having all supporting evidence, not just the single "best" chunk. The alternative of using a hard target (e.g., one-hot on a single oracle chunk) would train the model to retrieve only one piece of evidence, which would be counterproductive for multi-hop QA where multiple chunks must be assembled. The $1/|\mathcal{O}(x)|$ normalization ensures the loss magnitude does not depend on the number of oracle chunks per question.

Training configuration (Appendix B.1): The retrieval training runs for 10,000 optimization steps with AdamW optimizer, learning rate $3 \times 10^{-3}$, 100 warmup steps, and global batch size 256. This is a relatively short, lightweight training phase — the model backbone is frozen, so only ~164K parameters are updated. The high learning rate (3e-3, which is large for language model training) reflects that the trainable parameters are a tiny fraction of the model and are randomly initialized rather than pretrained.

Initialization: The model is initialized from a T5Gemma2 checkpoint whose decoder has been fine-tuned on the CLaRa QA pretraining dataset. This warm-start serves three purposes (Appendix B.1): (1) aligning the decoder with QA-style generation, (2) adapting generation to chunk-based pre-encoded representations rather than re-encoded text, and (3) incorporating the Reverse-QWK reparameterization of cross-attention. The retrieval tokens $\rho_r$ and layer weights $\alpha_\ell$ are then trained on the benchmark-specific training splits (HotPotQA, 2Wiki, MuSiQue, NQ jointly).

What the retrieval tokens learn: During training, gradients flow from the retrieval loss through the retrieval tokens and into the decoder's self-attention and cross-attention computations (since the query states $\mathbf{q}^\ell$ depend on both the retrieval token embeddings and the cross-attention context). The tokens learn to position themselves in embedding space such that, after the decoder processes them through self-attention (with the question tokens) and cross-attention (with $\mathbf{K}(\mathcal{S}_0)$), the resulting query states produce MaxSim scores that are high for oracle chunks and low for distractors. In effect, the retrieval tokens are learning to "ask the right questions" of the evidence pool — questions that, when expressed as cross-attention queries, surface the chunks most relevant to answering the actual question.

Why freeze the backbone: freezing the encoder and decoder preserves their pretrained representations and generation capabilities. If the backbone were fine-tuned, the representations would shift, potentially breaking the alignment between training and inference or degrading generation quality. The paper's claim is that retrieval capability is intrinsic — it already exists in the pretrained model — and the retrieval tokens merely provide an interface for accessing it. Training only the interface parameters (~164K out of 4B) is a strong test of this claim: if retrieval required substantial model retraining, only training the interface would fail.

3.4.6 Pooled Chunk Embeddings for Efficient MaxSim

Computing MaxSim against every token in a chunk is expensive. For a chunk of length $L_c$ and query length $L_q$, the cost is $O(L_q L_c d)$ per chunk-layer pair. With $L_c$ potentially hundreds of tokens (the average chunk length is ~92 words, corresponding to roughly 120–150 tokens), this becomes a bottleneck for full-corpus scoring.

The approximation: Replace each encoded chunk $\mathbf{k}_i \in \mathbb{R}^{L_c \times d}$ with a fixed-length mean-pooled representation $\hat{\mathbf{k}}_i \in \mathbb{R}^{L_p \times d}$ where $L_p \ll L_c$. This reduces the MaxSim cost from $O(L_q L_c d)$ to $O(L_q L_p d)$. In experiments, $L_p \in \{3, 5, 7\}$ is used, with $L_p = 7$ as the default.

What this computes: The chunk's token representations are divided into $L_p$ evenly sized segments, and the representations within each segment are averaged (mean-pooled) to produce a single vector. This yields a compressed representation with $L_p$ pseudotokens that approximates the full token-level representation.

Why this is natural for INTRA (and distinct from latent-compression approaches): The pooled vectors are fixed deterministic averages of the model's own encoder states — they require no additional compressor model, no compression-specific training, and no learned mapping. This is fundamentally different from latent-compression approaches like CLaRa, which train a separate compression model to map long documents into shorter latent representations. In INTRA, the pooling is a simple architectural choice that trades off retrieval granularity for computational efficiency while staying entirely within the model's native representation space.

Ablation (Table 7): Reducing the pooled chunk length from $L_p = 7$ to $L_p = 1$ (i.e., using a single mean-pooled vector per chunk, equivalent to single-vector similarity) drops CE-recall@5 from 59.9% to 48.9% on HotPotQA (-11.0 points) and from 40.7% to 26.6% on 2Wiki (-14.1 points). This quantifies the value of multi-vector (late-interaction) retrieval over single-vector retrieval within INTRA's framework: preserving even a small amount of token-level structure (7 pseudotokens instead of 1) yields substantial gains.

Why $L_p = 7$ is sufficient: The paper does not provide an extensive sweep of $L_p$ values, but the fact that going from 1 to 7 pseudotokens captures most of the benefit of full-token MaxSim (the gap between $L_p = 7$ and no pooling is not reported, but $L_p = 7$ is the default used for all main results) suggests that the model's encoder produces relatively smooth token representations — a chunk's semantic content can be adequately summarized by a small number of representative pseudotokens.

4. Key Insights and Innovations

Innovation 1: Retrieval as an Elicitable Capability, Not a Module to Add

The paper's most fundamental conceptual move is reframing retrieval from an architectural component—something you build and bolt onto a language model—to an intrinsic capability that pretrained attention-based models already possess and that can be surfaced with minimal intervention. This is a category shift in how to think about the relationship between language models and retrieval.

The dominant assumption in the RAG literature, from Lewis et al. (2020) through to modern agentic systems (Asai et al., 2024; Li et al., 2025), is that retrieval is a distinct subsystem with its own architecture, training data, and optimization objective. Even approaches that jointly train retrievers and generators—REALM (Guu et al., 2020), Atlas (Izacard et al., 2023), RA-DIT (Lin et al., 2024)—preserve this architectural separation; they reduce the representation mismatch by training both components toward a shared objective, but they still maintain a separate retriever with its own parameters and forward pass. The field's default mental model is: LLMs need retrieval, therefore we must build retrievers for them.

INTRA challenges this assumption directly. The paper demonstrates that a frozen 4B-parameter encoder-decoder model, with only ~164K additional trainable parameters (the retrieval token embeddings) and 272 layer aggregation weights—roughly 0.004% of the backbone's parameter count—can perform retrieval that rivals or exceeds dedicated embedding models trained on large-scale retrieval corpora. On HotPotQA, INTRA achieves 59.9% complete-evidence recall@5 versus 54.8% for BGE-large, a model explicitly pretrained for retrieval on datasets that include HotPotQA itself as supervision (Thakur, 2023). On 2WikiMultihopQA, INTRA reaches 40.7% versus 35.4% for Qwen3-Embedding-4B with a Jina reranker (Table 4). These are not marginal improvements from fine-tuning a retriever; they are competitive results from a model whose backbone parameters never saw a retrieval-specific training objective.

What makes this more than an incremental training efficiency claim: The paper is not merely saying "you can train a retriever with fewer parameters." It is making a capability claim about the nature of attention-based models. Cross-attention is already a query-conditioned matching operation—scoring decoder states against encoder states to decide what information to incorporate into generation. The paper's insight is that this operation is retrieval, not just something that resembles retrieval. The retrieval tokens are not learning to retrieve from scratch; they are learning to formulate queries that elicit the retrieval behavior already latent in the attention mechanism. This is why freezing the backbone works: the capability is already there, and the interface just needs to be surfaced.

The significance of this reframing extends beyond INTRA's specific architecture. If retrieval is intrinsic to attention-based models, then the entire modular RAG paradigm—separate retrievers, separate indexes, separate embedding spaces—may be an artifact of how we chose to build these systems rather than a fundamental requirement. Future work should investigate what other capabilities are similarly intrinsic, waiting to be elicited rather than added.

Evidence anchor: The core empirical support is the combination of (a) competitive retrieval performance against dedicated baselines with a frozen backbone (Tables 1 and 4), (b) the ablation showing that removing retrieval tokens entirely (using only $\mathcal{S}_0$) causes catastrophic recall drops (Table 7: -22.8 CE-recall@5 on HotPotQA), confirming the retrieval tokens are doing meaningful work, and (c) the parameter count asymmetry (~164K trainable vs. 4B frozen), which isolates the contribution of the interface from the contribution of the backbone.

Innovation 2: The Shared Representation Space as a Unifying Principle

INTRA's architectural contribution is not any single mechanism—learnable retrieval tokens, MaxSim scoring, or Reverse-QWK—but rather the principle that retrieval and generation should operate in one shared representation space, and the demonstration that this principle can be realized with a pretrained model through a specific combination of techniques. This is conceptually distinct from prior work that jointly trained retrievers and generators but kept their representations separate.

To understand why this matters, consider what happens in a standard RAG pipeline. The retriever (say, BGE) encodes the query into a dense vector and searches an index of document vectors for nearest neighbors. These vectors live in BGE's representation space, which is optimized for the contrastive retrieval objective BGE was trained on. The retrieved text is then tokenized and fed into the generator (say, T5Gemma2), which re-encodes it from raw tokens into its own internal representation space through the encoder. These two spaces are incommensurable—there is no guarantee that a chunk which is "close" to the query in BGE space contains the specific information the decoder's cross-attention queries will be looking for. The retriever optimizes for relevance, while the generator's attention optimizes for utility—usefulness for the specific generation step in progress.

INTRA eliminates this mismatch by construction. The same encoder that produces the chunk representations $\mathbf{k}_i$ also produces the generator's cross-attention key-value memory. When the retrieval tokens score a chunk via MaxSim against decoder query states, they are operating in exactly the same space that the generation pass will use for attention. A high retrieval score means "the decoder's queries, conditioned on the question and initial evidence, find strong token-level matches in this chunk's encoder representation"—which is precisely the signal that this chunk will be useful during generation. The retriever and generator are not just trained together; they are the same model evaluating the same representations for the same downstream purpose.

Comparison to prior integration attempts: CLaRa (He et al., 2026) is the closest prior work and the most instructive contrast. CLaRa jointly optimizes reranking and generation over latent representations, which reduces the mismatch, but it does so by training a compression model that maps documents into a latent space and jointly fine-tuning reranking and generation. The representations used for retrieval and generation are still distinct—the compression model produces latent codes for retrieval, while the generator uses its own encoder representations for generation. INTRA achieves a stronger form of unification: there is one encoder, one representation per chunk, and this representation is used identically for scoring and generation. No compression model, no translation step, no separate latent space. The Reverse-QWK transformation is essential here because it ensures that the same stored representation $\bar{\mathbf{k}}$ can serve as the key input to cross-attention across all decoder layers without per-layer reprojection—it is a single, shared, stable representation.

What this means for system design: The shared representation principle suggests that the ideal retriever for a given generator is the generator's own encoder—or more provocatively, that the distinction between "retriever" and "generator" is an artifact of modular system design rather than a natural decomposition of the problem. This has practical implications: deploying INTRA requires maintaining only one model and one index (the encoder states), rather than a retriever model, a reranker model, a generator model, and the indices and data pipelines connecting them. It also has research implications: future work on retrieval-augmented generation might productively focus on better ways to surface models' intrinsic retrieval capabilities rather than on building better external retrievers.

Evidence anchor: The shared-representation benefit is most directly supported by the gap-closure analysis in Table 2. Using the T5Gemma2 INTRA retriever with the same T5Gemma2 decoder as generator closes 59.4% of the gap between random and oracle evidence on average across benchmarks. The next best external generator (Qwen3.5-9B) closes only 54.1% using the same INTRA-retrieved evidence. The 5.3-point gap-closure advantage comes from the fact that INTRA's retrieval scores are computed from the decoder's own cross-attention queries—they naturally surface evidence that aligns with the decoder's specific attention patterns, rather than evidence that aligns with a generic notion of query-document relevance.

Innovation 3: Full-Corpus Scoring with Decoder Queries as a Third Way Between Reranking and Dense Retrieval

INTRA introduces an approach to retrieval that occupies a previously empty point in the design space: full-corpus scoring using decoder cross-attention queries as the scoring signal, without an external retriever for candidate generation. This is neither standard dense retrieval (single- or multi-vector similarity between query and document embeddings) nor reranking (reordering an initially retrieved candidate set), but something distinct that the comparison in Figure 3 makes empirically visible.

Standard RAG pipelines use a two-stage retrieval design: a cheap first-stage retriever (BM25, dense embeddings, or hybrid) narrows the corpus from millions of chunks to a candidate set of dozens or hundreds, then an expensive second-stage reranker (cross-encoder or late-interaction model) reorders the candidates. This architecture is driven by computational constraints—full-corpus cross-attention scoring is infeasible at scale—but it introduces a structural limitation: chunks that the first-stage retriever misses are permanently lost. If the initial retriever's representation space is not well-aligned with the information need (as is often the case for multi-hop questions requiring evidence assembly), no reranker can recover the missing evidence.

INTRA's design escapes this limitation by using a cheap initial retrieval ($\mathcal{S}_0$ via encoder-only MaxSim) not as a constraint on which chunks can be scored, but as cross-attention context for the retrieval pass. The retrieval tokens attend to $\mathbf{K}(\mathcal{S}_0)$ to produce query states $\mathbf{q}^\ell$, but those query states are then scored against every chunk in the full corpus via the MaxSim operation in Equation 3 and Algorithm 4. The initial set $\mathcal{S}_0$ influences the retrieval signal by shaping the query states, but it does not restrict which chunks can be retrieved. This is what allows INTRA to recover evidence that $\mathcal{S}_0$ missed entirely.

The evidence that this matters: Figure 3 quantifies exactly how much value comes from full-corpus scoring versus reranking. On HotPotQA, $\mathcal{S}_0$ alone achieves 37.1% CE-recall@5. Reranking $\mathcal{S}_0$ with the decoder scores $s_i$ improves this, but full-corpus INTRA scoring pushes performance to 59.9%—a jump of 22.8 points over $\mathcal{S}_0$ and a substantial margin over reranking alone. This pattern holds across benchmarks (2Wiki: 17.7% → 40.7%). The gap between reranking and full-corpus scoring represents chunks that the initial encoder-only MaxSim missed, but that the decoder's query states—which incorporate the question text and $\mathcal{S}_0$ context through self-attention and cross-attention—were able to identify.

Why this is more nuanced than "full-corpus is better": The initial context $\mathcal{S}_0$ is not arbitrary. The ablation in Table 7 shows that using $\mathcal{S}_0 = \emptyset$ (no initial context) causes CE-recall@5 to collapse to 26.9% on HotPotQA—33 points below the full INTRA result. The retrieval tokens need some evidence to attend to during the retrieval pass to produce useful query states. But the quality of that initial evidence is less critical than one might expect: even a weak initial set provides useful cross-attention context, and the full-corpus scoring can recover evidence the initial set missed. This suggests a "good enough" initial retrieval is sufficient, with the decoder's queries providing the precision.

Comparison to ColBERT-style retrieval: ColBERT (Khattab and Zaharia, 2020) also uses MaxSim for full-corpus scoring, but with a dedicated query encoder and document encoder—two separate models trained for the retrieval task. INTRA's MaxSim scoring is different in kind: the query representation is not a static embedding but a set of decoder cross-attention query states that are context-conditioned on both the question and the initial evidence $\mathcal{S}_0$. This conditioning makes the retrieval signal adaptive: the same question with different initial evidence contexts could produce different query states and therefore different retrieval rankings. This is desirable because the information the decoder needs depends on what evidence it has already seen—a property that static query embeddings cannot capture.

Evidence anchor: Beyond Figure 3 and the Table 7 ablation, the difficulty-dependent results across benchmarks provide converging evidence. INTRA's largest advantages over baselines appear on multi-hop benchmarks (HotPotQA: +5.1 CE-recall@5 over BGE; 2Wiki: +5.3 over Qwen+Reranker) where evidence assembly requires identifying chunks that may not individually show high similarity to the question. On single-hop NQ, the advantage narrows or disappears (Qwen3-Emb-4B: 30.3% vs. INTRA: 29.1% R@5). This pattern is consistent with the hypothesis that decoder-conditioned queries provide the most value when the retrieval problem requires assembling multiple pieces of evidence whose relevance is not fully captured by question-chunk similarity alone.

Innovation 4: The Reverse-QWK Transformation as a Practical Enabler for Shared Encoder Representations

While the shared representation space (Innovation 2) is the conceptual contribution, the Reverse-QWK reparameterization is the engineering insight that makes it computationally feasible. It is easy to underestimate the importance of this contribution because it appears in Section 3.1 and Appendix A as a technical detail, but without it, the entire INTRA design would be impractical for models with standard per-layer key projections.

The problem Reverse-QWK solves is concrete. In a Transformer encoder-decoder, cross-attention keys are not raw encoder states but layer-specific projections $\mathbf{k}_\ell = (\bar{\mathbf{k}} \odot \gamma_{K,\ell}) \mathbf{W}_{K,\ell}$. With 34 decoder layers (T5Gemma2 4B), storing per-layer key representations for every chunk in a corpus of 759K chunks would require roughly 34× more storage than storing a single shared representation—pushing the storage requirement from ~2.5 TB to ~85 TB for a 1B-token corpus (using the numbers from Appendix A.3). This would make the approach economically impractical even at modest corpus sizes, and would prevent the use of a single ANN index shared across layers.

The standard solution to this in prior work is to avoid the problem entirely—use a separate retriever with its own embeddings, or compress documents into a learned latent space (as in CLaRa). Reverse-QWK takes the opposite approach: it accepts that the model has per-layer key projections and finds a way to invert them mathematically from the key side to the query side, preserving exact mathematical equivalence while enabling shared storage. The derivation in Equation 10 shows that this is not an approximation or a learned transformation—it is an algebraic identity:

qℓkℓ⊤=((qℓWK,ℓ⊤)⊙γK,ℓ) kˉ⊤=q~ℓkˉ⊤\mathbf{q}^\ell \mathbf{k}_\ell^\top = ((\mathbf{q}^\ell \mathbf{W}_{K,\ell}^\top) \odot \gamma_{K,\ell}) \, \bar{\mathbf{k}}^\top = \tilde{\mathbf{q}}^\ell \bar{\mathbf{k}}^\top

Why this is more than an implementation trick: Reverse-QWK enables a qualitative change in system architecture. Without it, INTRA would be forced to choose between (a) storing per-layer representations (impractical), (b) using the initial encoder-only MaxSim as the final retrieval mechanism (which Table 7 shows loses 22.8 CE-recall@5 on HotPotQA), or (c) training a separate compression model (abandoning the shared-representation principle). With Reverse-QWK, the system gets the retrieval quality of full decoder-conditioned scoring while maintaining the storage efficiency of a single shared encoder representation.

The transformation also handles Group-Query Attention (GQA) cleanly. Under GQA, each KV head is shared by multiple Q-heads, and in standard cross-attention, the KV representations must be replicated across heads on the encoder side. Reverse-QWK moves this replication to the query side, where the per-query cost is negligible. The storage savings are quantified in Appendix A.3: the compression ratio is approximately $2L n_{kv} d_h / d \approx 30\times$ for the T5Gemma2 4B configuration. This is what makes it practical to store the entire encoded corpus and serve it as cross-attention memory during generation.

Connection to broader implications: The Reverse-QWK idea—moving a per-layer, per-head transformation from a large stored representation to a small query—is not specific to retrieval. Any application that needs to serve precomputed encoder states to a multi-layer decoder with cross-attention (multi-turn dialogue with shared context, batch inference over long documents, knowledge-grounded generation with a fixed knowledge base) could benefit from this reparameterization. The paper frames it modestly as an "implementation device," but it is effectively a contribution to the engineering of efficient encoder-decoder inference that could be adopted independently of INTRA's retrieval framework.

Evidence anchor: The practical impact is demonstrated in the TTFT benchmarks. Figure 4 shows that for $k = 500$ retrieved chunks, INTRA's prefill time is ~66ms versus ~1.25s for standard RAG—a ~19× speedup. This speedup has multiple sources (amortized encoding, no re-encoding of evidence), but Reverse-QWK is the enabler that allows the retrieval pass to operate against a single shared pool and the generation pass to use the same pool as cross-attention memory without per-layer key computation. The absence of a Reverse-QWK ablation (it is a mathematical identity, not a hyperparameter) is appropriate—the correctness is proven algebraically—but the empirical evidence that the system works at scale validates that the transformation is correctly implemented and numerically stable.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on four Wikipedia-based QA benchmarks: HotPotQA (Yang et al., 2018), 2WikiMultihopQA (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), and Natural Questions (Kwiatkowski et al., 2019). These span bridge and comparison reasoning (HotPotQA), cleaner two-hop evidence chains (2Wiki), compositionally harder multi-hop questions (MuSiQue), and single-hop open-domain QA (NQ). The authors use the deduplicated end-to-end QA dataset derived from CLaRa (He et al., 2026), with training and evaluation splits for each benchmark. Table 8 in Appendix E shows the split sizes: HotPotQA train has 90,185 examples, 2Wiki train has 167,454, MuSiQue train has 277,577, and NQ train has 53,301. Evaluation splits range from 2,417 (MuSiQue) to 12,576 (2Wiki).

  • Base model(s). All experiments use T5Gemma2 4B-4B (Zhang et al., 2025), a 4-billion-parameter encoder-decoder model. The checkpoint is warm-started on the CLaRa QA pretraining dataset, which serves three purposes: aligning the decoder with QA-style generation, adapting it to chunk-based pre-encoded representations, and incorporating the Reverse-QWK reparameterization (Appendix B.1). For the generator compatibility study (Table 2), the paper additionally tests Mistral-0.3-7B, Phi4-3.8B, Llama3.1-8B, Gemma4-E2B, Qwen3.5-9B, and Qwen3.5-27B as external generators with the frozen T5Gemma2 INTRA retriever.

  • Metrics. The paper reports two primary metric families. For retrieval quality, the central metric is complete-evidence recall@k for k ∈ {5, 10, 20} — the fraction of examples where all annotated supporting chunks are retrieved within the top-k results. This is more stringent than standard recall@k because it requires recovering the full supporting set, not just any single oracle chunk. Standard recall and other retrieval metrics are reported in Appendix C. For end-to-end QA, the paper reports exact match (EM) and token-level F1, both computed using the standard evaluation scripts for each benchmark. Answers are generated via deterministic greedy decoding from the T5Gemma2 decoder conditioned on the top-5 retrieved chunks' pre-encoded representations.

  • Baselines. The paper compares against nine retrieval baselines, described in Appendix B.2:

    • Sparse lexical methods: TF-IDF (Salton and Buckley, 1988) and BM25 (Robertson and Zaragoza, 2009), both operating directly on chunk text.
    • Dense single-vector models: BGE-large (Xiao et al., 2024), Qwen3-Embedding-0.6B, and Qwen3-Embedding-4B (Zhang et al., 2025). For all three, the query and each chunk are encoded independently, and chunks are ranked by cosine similarity against the query embedding.
    • Reranking: Qwen3-Embedding-4B with the Jina reranker v2 base multilingual (Jina AI, 2024). The reranker reorders the top retrieval candidates from Qwen3-Embedding-4B's initial ranking.
    • Hybrid RAG: A combination of BM25, TF-IDF, BGE-large, and Qwen3-Embedding-0.6B using reciprocal rank fusion (RRF) (Cormack et al., 2009) to merge rankings.
    • MaxSim late-interaction: A ColBERT-style baseline (Khattab and Zaharia, 2020) that scores query embeddings against chunk-vector encodings in the T5Gemma encoder output space, using MaxSim without decoder conditioning. This is the closest retrieval primitive to INTRA's scoring mechanism, but operates in the encoder-only space rather than the decoder-conditioned space.

    For end-to-end QA (Tables 1 and 2), the same T5Gemma2 decoder is used as the generator across all retrieval methods, isolating the effect of retrieval quality on answer accuracy.

  • Generation budget / compute accounting. The paper uses a shared retrieval candidate pool of approximately 100M tokens, containing 758,500 chunks after deduplication (Appendix E). The pool construction ensures coverage of all oracle chunks referenced by QA examples and fills the remaining budget with uniformly sampled non-oracle chunks. This fixed pool enables fair comparison across methods — all retrievers search over the same set of chunks. For generation, each method retrieves k = 5 chunks (except where noted in ablations). The computational cost model is analyzed in Table 3 and Appendix D, using standard transformer FLOPs accounting (pre-query encoding, retrieval cost, prefill, and generation), with retrieval cost modeled as O(√M · L_q · L_c) under IVF-based approximate nearest-neighbor search. The TTFT benchmarks in Figures 4–6 measure wall-clock time on NVIDIA H100 GPUs, excluding retrieval time to isolate the generator-side re-encoding cost.

  • Cross-validation / statistical protocol. The paper does not use cross-validation for strategy selection — unlike the reference paper's compute-optimal policy which required two-fold cross-validation to avoid overfitting, INTRA has no adaptive strategy meta-parameter that needs selection per difficulty bin. Retrieval training runs for a fixed 10,000 optimization steps on the training splits of all four benchmarks jointly (Appendix B.1). Evaluation is performed once on the held-out evaluation splits. Confidence intervals (95%) are reported for the end-to-end QA results in Tables 5 and 6 (Appendix C), computed via standard error propagation across the evaluation examples.

Main Quantitative Results

Retrieval Results: Complete-Evidence Recall

The headline retrieval result is Figure 2 and Table 4: INTRA achieves the highest complete-evidence recall across all three multi-hop benchmarks, with the largest margins at k = 5 where the retrieval budget is most constrained.

On HotPotQA at k = 5, INTRA achieves 59.9% complete-evidence recall, compared to the next best baseline of 54.8% (BGE-large) — a margin of +5.1 percentage points. At k = 10, the margin widens to +7.4 points (70.9% vs. 63.5% for BGE), and at k = 20, it is +6.5 points (76.1% vs. 69.6%). The advantage over the strongest combined baseline (Qwen3-Emb-4B + Jina reranker) is larger: +11.1 points at k = 5 (59.9% vs. 48.8%), +11.3 at k = 10, and +10.7 at k = 20. Sparse baselines (BM25: 32.2%, TF-IDF: 18.2% at k = 5) are substantially weaker, as is the MaxSim encoder-only baseline (36.1% at k = 5). The Hybrid RAG combination (48.0% at k = 5) underperforms INTRA by 11.9 points.

On 2WikiMultihopQA at k = 5, INTRA achieves 40.7%, compared to 35.4% for Qwen3-Emb-4B + reranker (+5.3 points) and 30.9% for BGE-large (+9.8 points). At k = 10, the advantage grows to +10.0 points over Qwen3-Emb-4B (50.3% vs. 40.3%) and +14.4 points over BGE. The MaxSim baseline achieves only 16.7% at k = 5, underscoring that decoder conditioning — not just MaxSim scoring — provides the gains.

On MuSiQue at k = 5, INTRA achieves 12.8%, versus 10.1% for Qwen3-Emb-4B + reranker (+2.7 points) and 6.2% for Hybrid RAG (+6.6 points). The absolute numbers are low across all methods because MuSiQue is compositionally harder — questions require assembling 3–4 evidence chunks — and the model's pass@1 for complete evidence is inherently low. At k = 20, INTRA reaches 23.7%, with Qwen3-Emb-4B at 19.7% and Hybrid RAG at 17.1%. The pattern is consistent: INTRA maintains its advantage even in the low-recall regime.

On Natural Questions, the single-hop benchmark, the pattern reverses. INTRA's recall@5 is 29.1%, which lags behind Qwen3-Emb-4B (30.3%, -1.2 points), Qwen3-Emb-4B + reranker (31.9%, -2.8 points), and BGE-large (29.6%, -0.5 points). At k = 10, INTRA's 38.3% trails Qwen3-Emb-4B + reranker's 42.0%. At k = 20, INTRA (45.9%) trails Qwen3-Emb-4B + reranker (50.9%) and Qwen3-Emb-4B (50.5%). This is the paper's key negative result, and the authors interpret it as expected: "NQ's single-hop nature minimizes this benefit" (Figure 2 caption). For single-hop questions, query-document similarity — which dedicated retrieval models like Qwen3-Embedding are explicitly trained to optimize — is a sufficient signal, and the decoder-conditioned queries provide less marginal value.

The R@5 initial vs. reranked vs. full-corpus decomposition (Figure 3): The paper breaks out the contribution of INTRA's full-corpus scoring on HotPotQA and 2Wiki. On HotPotQA, the initial MaxSim retrieval $\mathcal{S}_0$ achieves 37.1% CE-recall@5. Reranking $\mathcal{S}_0$ with decoder scores $s_i$ (but without expanding to the full corpus) improves performance over the initial set alone, but full-corpus INTRA scoring pushes CE-recall@5 to 59.9% — a jump of 22.8 points over $\mathcal{S}_0$. On 2Wiki, the corresponding numbers are 17.7% ($\mathcal{S}_0$) → 40.7% (INTRA), a +23.0 point gain. The gap between reranking and full-corpus scoring represents evidence chunks that the encoder-only initial retrieval missed entirely but the decoder-conditioned queries recovered. This decomposition is central to the paper's claim that INTRA is fundamentally different from reranking approaches.

End-to-End Question-Answering Results

Table 1 is the core end-to-end result, pairing each retrieval method with the same frozen T5Gemma2 decoder for generation. Across all four benchmarks, INTRA achieves the highest average EM (40.2%) and F1 (48.6%), with the strongest results on multi-hop benchmarks.

On HotPotQA, INTRA achieves 46.4% EM and 58.0% F1. The next best baseline is Hybrid RAG (43.4% EM, 54.3% F1), followed by Qwen3-Emb-4B + reranker (41.6% EM, 53.6% F1) and BGE (41.9% EM, 53.0% F1). The margin over Hybrid RAG is +3.0 EM points, and over BGE is +4.5 EM points. Sparse baselines trail substantially: BM25 at 40.5% EM, TF-IDF at 34.2%.

On 2WikiMultihopQA, INTRA achieves 49.2% EM and 53.2% F1, versus 46.8% EM / 50.8% F1 for Qwen3-Emb-4B + reranker (+2.4 EM points) and 46.0% EM / 49.9% F1 for Hybrid RAG (+3.2 EM points). The MaxSim encoder-only baseline reaches only 41.6% EM, confirming decoder conditioning contributes substantially to answer quality beyond what encoder-only MaxSim can achieve.

On MuSiQue, all methods struggle with low absolute scores due to the benchmark's difficulty. INTRA achieves 14.0% EM and 23.0% F1, compared to Qwen3-Emb-4B + reranker at 13.3% EM / 22.5% F1 (+0.7 EM points) and Hybrid RAG at 10.6% EM / 20.1% F1. The margins are small in absolute terms but represent a ~5% relative improvement over the strongest baseline (14.0% vs. 13.3%).

On Natural Questions, the pattern follows the retrieval results: INTRA (51.2% EM, 60.3% F1) lags behind Qwen3-Emb-4B + reranker (55.1% EM, 64.2% F1, -3.9 EM points) and Qwen3-Emb-4B alone (54.5% EM, 63.7% F1, -3.3 EM points). BGE achieves 52.2% EM — 1.0 point above INTRA. INTRA does outperform BM25 (43.4% EM), MaxSim (48.4% EM), and Hybrid RAG (50.5% EM) on NQ, but the strongest dedicated retrieval models maintain an advantage.

The average EM and F1 across benchmarks (final column of Table 1) show INTRA at 40.2% EM / 48.6% F1, versus 39.2% EM / 47.5% F1 for Qwen3-Emb-4B + reranker — a modest +1.0 EM point overall average. This average masks the benchmark-dependent pattern noted above: INTRA's advantage is concentrated in the multi-hop benchmarks (+3.0 HotPotQA, +2.4 2Wiki, +0.7 MuSiQue) and offset by a deficit on NQ (-3.9).

Generator Compatibility: The Shared Decoder Advantage

Table 2 addresses a critical question: is INTRA's end-to-end advantage primarily due to better retrieval, or due to the alignment between retriever and generator that comes from sharing a decoder? The experiment keeps the INTRA retrieval fixed (trained with the T5Gemma2 decoder producing retrieval scores) but varies the generator used for answer production.

The metric is gap closure — the percentage of the EM gap between random-chunk generation and complete-evidence (oracle) generation that INTRA retrieval closes:

GapClosure=100⋅EM(INTRA)−EM(random)EM(complete)−EM(random)\text{GapClosure} = 100 \cdot \frac{\text{EM}(\text{INTRA}) - \text{EM}(\text{random})}{\text{EM}(\text{complete}) - \text{EM}(\text{random})}

Higher gap closure means the INTRA-retrieved evidence is more useful to that specific generator.

The key finding: using the same T5Gemma2 decoder for both retrieval and generation closes the largest average gap (59.4%) across benchmarks. The next best external generators close substantially smaller gaps: Qwen3.5-9B at 54.1% (-5.3 points), Gemma4-E2B at 52.6% (-6.8 points), Llama3.1-8B at 48.7% (-10.7 points), Mistral-0.3-7B at 38.1% (-21.3 points). The per-benchmark pattern holds as well: on HotPotQA, T5Gemma2 closes 66.4% versus Qwen3.5-9B's 64.0%; on 2Wiki, 56.8% versus Qwen3.5-27B's 51.8%; on MuSiQue, 38.5% versus Gemma4-E2B's 25.7%.

This result provides direct evidence for the paper's core claim about representation mismatch. The INTRA retriever scores chunks based on what the T5Gemma2 decoder's cross-attention queries find relevant. When a different generator (say, Llama3.1-8B) consumes those same chunks, the evidence may be less well-aligned with that generator's attention patterns, producing lower gap closure. The 5.3-point gap-closure advantage of the shared decoder over the best external generator demonstrates that the retrieval signal is generator-specific — INTRA retrieves evidence that is particularly useful for its own decoder, and this specificity has measurable impact on end-to-end answer quality.

Caveat: The paper correctly notes (discussion surrounding Table 2) that the T5Gemma2 generator is weaker in absolute terms than some of the external generators. The gap-closure metric controls for absolute generator strength (by normalizing against each generator's oracle performance), but it does not change the fact that a stronger generator with somewhat less well-aligned retrieval might still produce higher absolute EM. The paper frames this as highlighting "the need for stronger INTRA backbones, given that open-source encoder-decoder models are currently scarcer and weaker than decoder-only options."

Efficiency Results: Time-to-First-Token and Throughput

Beyond retrieval and QA quality, the paper demonstrates a computational efficiency advantage from reusing pre-encoded evidence. Figure 4 is the central efficiency result, measuring time-to-first-token (TTFT) after evidence has been selected, as a function of the number of retrieved chunks k.

The measurement setup (Appendix D.4) fixes query length L_q = 128, chunk length L_c = 128, generation length L_g = 128, and total corpus tokens N = 65,536. Retrieval time is excluded from the measurement to isolate the generator-side cost — the question is: once we know which chunks to use, how fast can the generator start producing tokens?

As k increases from 1 to 500:

  • INTRA TTFT grows from 12.8 ms to 65.7 ms — a roughly 5× increase over a 500× increase in evidence volume. This near-constant scaling is possible because INTRA reuses pre-encoded chunk states stored as $\bar{\mathbf{k}}$; the decoder's prefill computes cross-attention directly against these precomputed states without re-encoding.
  • Standard RAG TTFT grows from 23.1 ms to 1.25 seconds — a roughly 54× increase over the same evidence increase. This quadratic growth reflects the $O((L_q + kL_c)^2)$ self-attention cost of re-encoding the query and all k retrieved chunks from raw text.
  • Full-context prompting (processing all N = 65,536 tokens without retrieval) achieves approximately 1.31 seconds TTFT — comparable to RAG at k ≈ 500, but requiring all tokens to be processed rather than just the retrieved subset.

At k = 500, INTRA's TTFT of 65.7 ms represents an approximately 19× speedup over standard RAG (1.25 s). At more typical retrieval depths (k = 5–20), INTRA's TTFT is 12.8–15.2 ms versus RAG's 23.1–40.8 ms — a roughly 2–3× speedup.

Figure 5 (Appendix D.4) extends the analysis to generation throughput, sweeping over k. The throughput gap is smaller because generation cost is dominated by the autoregressive decode over output tokens (O(L_g · kL_c)), which is similar for both methods — both must attend to the same amount of evidence during generation, just from different sources (pre-encoded states vs. re-encoded text). The measurable throughput difference comes from the prefill phase, where INTRA's advantage is concentrated.

Figure 6 (Appendix D.4) measures TTFT while sweeping over chunk length L_c. As chunk length increases, the gap between INTRA and RAG widens, because RAG's quadratic re-encoding cost grows with the square of the chunk length while INTRA's prefill cost is linear in chunk length (since the encoder representations are precomputed). This reinforces that the efficiency advantage is most pronounced when evidence chunks are long — a common scenario in knowledge-intensive applications where each supporting passage may be several hundred tokens.

Table 3 formalizes the computational complexity comparison between full-context prompting, standard RAG, and INTRA. The key term distinguishing INTRA from standard RAG is the prefill cost: $O(L_q(L_q + kL_c))$ for INTRA versus $O((L_q + kL_c)^2)$ for standard RAG under dense self-attention. The pre-query encoding cost $O(N L_c)$ (where N is total corpus tokens) is identical for both RAG and INTRA — both must encode the corpus once offline. The retrieval cost $O(\sqrt{M} L_q L_c)$ using IVF-ANN is also identical. The generation cost $O(L_g(L_q + kL_c + L_g))$ is the same functional form, though INTRA's constant factors may be lower because it uses Reverse-QWK-compressed cross-attention states (Appendix A.3: ~30× compression in KV cache size over standard per-layer K/V states).

Storage analysis (Appendix A.3): For a 1B-token corpus with the T5Gemma2 4B model (d = 2560), storing the shared encoder states $\bar{\mathbf{k}}$ at 8-bit precision requires approximately 1B × 2560 bytes ≈ 2.56 TB. This fits on a single NVMe SSD, and product quantization or further compression could reduce this. The paper notes that values (V = W_V · encoder_kv) are not precomputed — they are generated on-demand only for the (small) top-k selected encoder positions during generation, so their storage cost is negligible.

Ablation Studies and Robustness Checks

Initial context $\mathcal{S}_0 = \emptyset$ (no initial retrieval, Table 7): Removing the initial context entirely causes CE-recall@5 to drop from 59.9% to 26.9% on HotPotQA (-33.0 points) and from 40.7% to 15.4% on 2Wiki (-25.3 points). EM drops correspondingly by 13.2 and 10.9 points. This is the largest single ablation effect, demonstrating that the retrieval pass critically depends on having some evidence to attend to during the retrieval forward pass. The retrieval tokens cannot produce useful cross-attention query states without cross-attention context.

$\mathcal{S}_{\text{INTRA}} = \mathcal{S}_0$ (only initial retrieval, no decoder scoring, Table 7): Using only the initial MaxSim retrieval (encoder-only, no decoder-conditioned scoring) drops CE-recall@5 from 59.9% to 37.1% on HotPotQA (-22.8 points) and from 40.7% to 17.7% on 2Wiki (-23.0 points). EM drops by 5.7 and 7.6 points respectively. This ablation isolates the contribution of decoder-conditioned scoring: the initial encoder-only MaxSim is substantially weaker than full INTRA, confirming that the decoder's queries add retrieval signal beyond what the encoder can provide alone.

Initial retrieval similarity metric: cosine vs. MaxSim (Table 7): Replacing MaxSim with cosine similarity (single-vector) for computing $\mathcal{S}_0$ drops CE-recall@5 from 59.9% to 30.1% on HotPotQA (-29.8 points) and from 40.7% to 15.0% on 2Wiki (-25.7 points). EM drops by 10.8 and 10.0 points. This demonstrates that token-level late interaction (MaxSim) is substantially more effective than single-vector similarity even for the coarse initial retrieval stage, consistent with ColBERT findings.

Pooled chunk length $L_p$ (Table 7): Reducing the pooled chunk representation from $L_p = 7$ to $L_p = 1$ (a single mean-pooled vector, equivalent to single-vector retrieval) drops CE-recall@5 from 59.9% to 48.9% on HotPotQA (-11.0 points) and from 40.7% to 26.6% on 2Wiki (-14.1 points). EM drops by 2.5 and 4.2 points. This quantifies the value of multi-vector (late-interaction) retrieval over single-vector within INTRA's framework — even 7 pseudotokens capture substantially more retrieval signal than a single average vector.

Number of retrieval tokens (Table 7): Reducing from 64 retrieval tokens to 16 drops CE-recall@5 from 59.9% to 55.5% on HotPotQA (-4.4 points) and from 40.7% to 35.8% on 2Wiki (-4.9 points). EM drops by 1.2 and 1.0 points — relatively modest. Reducing to 1 retrieval token drops CE-recall@5 to 44.9% on HotPotQA (-15.0 points) and 24.7% on 2Wiki (-16.0 points), with EM drops of 3.6 and 5.0 points. The non-linear degradation suggests that a small number of retrieval tokens (16) captures most of the benefit, but a single token is insufficient — the query needs some capacity to express its information needs across multiple token positions. The paper uses 64 tokens for main experiments, which appears to be in the saturated regime.

Context construction for generation (Table 7): Using only the top-5 chunks from $\mathcal{S}_{\text{INTRA}}$ (without including the top initial chunk from $\mathcal{S}_0$ as a sixth) drops CE-recall@5 from 59.9% to 51.7% on HotPotQA (-8.2 points) and from 40.7% to 35.0% on 2Wiki (-5.7 points). EM drops by 2.9 and 2.5 points. This shows that the initial retrieval set and the refined INTRA set are somewhat complementary — including one chunk from $\mathcal{S}_0$ provides coverage for evidence that may have been missed in the refined scoring.

Per-benchmark retrieval detail (Table 4): The full retrieval results table confirms that INTRA's advantage is consistent across k ∈ {5, 10, 20} on multi-hop benchmarks. On HotPotQA, INTRA's lead over BGE is +5.1 at k=5, +7.4 at k=10, +6.5 at k=20 — the advantage does not vanish with larger k, suggesting it is not simply a precision-at-low-k effect. On NQ, all methods show larger absolute improvements from increasing k, and the ranking between methods remains relatively stable.

Confidence intervals (Tables 5 and 6, Appendix C): The 95% confidence intervals on EM are approximately ±1.0–1.3 percentage points across benchmarks. INTRA's 46.4% EM on HotPotQA has a CI of ±1.1, while Hybrid RAG's 43.4% has ±1.1 — the 3.0-point gap exceeds the combined margin of error, confirming statistical significance. On MuSiQue, where absolute scores are low (14.0% ±1.3 for INTRA vs. 13.3% ±1.3 for Qwen + reranker), the 0.7-point gap falls within overlapping confidence intervals — the difference is not statistically significant at this sample size. On NQ, Qwen + reranker's 55.1% ±1.2 versus INTRA's 51.2% ±1.2 shows a clear significant gap in the opposite direction. The paper does not report significance tests beyond the CIs.

Critical Assessment

Claim: INTRA outperforms strong RAG baselines on multi-hop QA evidence recall and end-to-end answer quality.

What was tested: The paper evaluates INTRA against nine baselines on four benchmarks, using a fixed 759K-chunk retrieval pool. Complete-evidence recall@k and end-to-end EM/F1 are the primary metrics. The T5Gemma2 generator is held constant across all retrieval methods in Table 1, isolating retrieval quality. INTRA trains only ~164K parameters on the benchmark training splits with the backbone frozen.

What the experiments demonstrate: INTRA convincingly outperforms all baselines on the three multi-hop benchmarks (HotPotQA, 2Wiki, MuSiQue) for both retrieval recall and end-to-end QA. The margins over the strongest baseline (Qwen3-Emb-4B + reranker) are +11.1 CE-recall@5 on HotPotQA, +5.3 on 2Wiki, +2.7 on MuSiQue (Table 4). End-to-end EM margins are +3.0, +2.4, and +0.7 respectively (Table 1). These are meaningful gains, particularly given that the baselines are dedicated retrieval models trained on large-scale retrieval corpora (BGE and Qwen3-Embedding's training data include HotPotQA and NQ as supervision, as noted in Section 5.2 and Appendix B.2), while INTRA's backbone is frozen and never trained for retrieval.

What the experiments do NOT demonstrate:

  • Generality beyond the T5Gemma2 family. All retrieval experiments use a single model architecture (encoder-decoder with cross-attention) at a single scale (4B). The paper does not test whether the intrinsic retrieval capability scales with model size, transfers to other encoder-decoder architectures (e.g., vanilla T5, BART), or appears in decoder-only architectures (which lack explicit cross-attention and would require a different mechanism). The paper acknowledges this limitation in Section 7: "INTRA's reliance on encoder-decoder cross-attention also excludes decoder-only models." This is a substantive constraint given that decoder-only models dominate the current LLM landscape.
  • Competitiveness on single-hop QA at the highest performance tier. On Natural Questions, INTRA lags behind Qwen3-Embedding-4B + reranker by -3.9 EM points (Table 1) and -2.8 CE-recall@5 (Table 4). While the paper frames this as expected ("NQ's single-hop nature minimizes this benefit"), a user wanting a single retrieval system for a mixed workload would need to accept sub-single-hop performance as the price of superior multi-hop performance. The paper does not explore whether the retrieval tokens could be trained to match dedicated retrievers on single-hop while maintaining multi-hop advantages — the current results represent a specific point in this tradeoff space.
  • Scalability to web-scale corpora. The experiments use a ~100M-token, 759K-chunk pool. The paper explicitly states it does "not position INTRA as a replacement for RAG in open-web retrieval or web-scale settings" (Section 7). The IVF-ANN retrieval complexity of O(√M · L_q · L_c) is standard and should scale, but the practical bottlenecks — building and updating the ANN index, storing encoder states for billions of chunks, handling dynamic corpora where chunks change — are not evaluated.

Claim: INTRA demonstrates that attention-based models possess an intrinsic retrieval capability that can be elicited rather than added as an external module.

What was tested: The retrieval tokens (~164K parameters) and layer aggregation weights (272 parameters) are trained with the 4B-parameter backbone frozen. If retrieval required substantial model retraining, this setup would fail. The ablation in Table 7 confirms the retrieval tokens are necessary — removing them collapses performance.

What the experiments demonstrate: The frozen-backbone design is a strong test of the "intrinsic capability" claim and the results largely support it. The fact that ~0.004% of total parameters, trained for 10K steps, can produce competitive retrieval is evidence that the cross-attention mechanism already encodes retrieval-relevant signal — the interface just needs to surface it. The ablation showing $\mathcal{S}_0 = \emptyset$ causes a 33-point CE-recall@5 drop on HotPotQA confirms that the retrieval signal comes from the decoder's interaction with evidence representations, not from the retrieval tokens operating in isolation.

Important nuance the paper does not fully explore:

  • The CLaRa warm-start. The T5Gemma2 checkpoint is first fine-tuned on the CLaRa QA pretraining dataset before retrieval training (Appendix B.1). The paper describes this as serving to "align the decoder with QA-style generation" and "adapt generation to chunk-based pre-encoded representations." While the backbone is frozen during retrieval training, it has been fine-tuned on QA data before that step. An "intrinsic" capability claim would be more strongly supported by starting from the base T5Gemma2 checkpoint with no QA-specific fine-tuning. The current setup leaves open the possibility that the CLaRa fine-tuning, not the original pretraining, is the source of the retrieval capability. This is not a fatal flaw — the paper is transparent about the warm-start — but it qualifies the strength of "intrinsic."
  • The retrieval tokens are trained with oracle supervision. The retrieval tokens learn to score oracle chunks highly by training against annotated evidence sets (Equation in Section 3.2). This is a form of retrieval-specific training, even if the backbone is frozen. A stronger demonstration of "intrinsic" capability would show that the untrained decoder's cross-attention queries already rank oracle chunks above distractors without any retrieval training — i.e., zero-shot retrieval. The paper does not report zero-shot retrieval performance. The current results show that a small amount of interface training elicits strong retrieval, which is interesting but subtly different from showing retrieval is already present without any training signal.

Claim: The shared representation space eliminates the retriever-generator mismatch and this yields measurable end-to-end gains.

What was tested: The gap-closure analysis in Table 2 compares using the same T5Gemma2 decoder for retrieval and generation versus using INTRA retrieval with external decoders. The gap-closure metric controls for absolute generator strength.

What the experiments demonstrate: The results support the claim with the caveat that the effect size varies by generator. The T5Gemma2 shared-decoder configuration closes 59.4% of the gap to oracle evidence on average, versus 54.1% for Qwen3.5-9B — a 5.3-point advantage attributable to representation alignment. The per-benchmark pattern is consistent (T5Gemma2 closes the largest gap on 3 of 4 benchmarks; on NQ, Qwen3.5-9B achieves 78.5% vs. T5Gemma2's 75.9%). This is a well-designed experiment that isolates the alignment effect from absolute generator quality.

What could strengthen this claim:

  • A control using the T5Gemma2 encoder as a standalone retriever. The gap-closure analysis compares INTRA retrieval (decoder-conditioned) with INTRA generation (same decoder) versus INTRA retrieval with different generators. But it does not compare against using the T5Gemma2 encoder as a dense retriever (encoder-only MaxSim, $\mathcal{S}_0$) with the T5Gemma2 decoder as generator. This control would distinguish the value of decoder conditioning from the value of merely sharing the same encoder between retriever and generator. If T5Gemma2 encoder-only retrieval + T5Gemma2 generator also achieves high gap closure, the benefit is from encoder sharing (still a form of representation alignment) rather than specifically from decoder-conditioned queries. This control is not reported.
  • Analysis of which retrieved chunks differ between T5Gemma2 and external generators. The gap-closure metric shows that decoder alignment matters, but not why. Do the external generators fail because INTRA-retrieved chunks are missing something they need, or because the chunks contain distracting information that the T5Gemma2 decoder naturally ignores but external decoders attend to? A qualitative or quantitative analysis of retrieval differences would strengthen the mechanistic interpretation.

Claim: INTRA provides computational efficiency through reusable pre-encoded evidence.

What was tested: TTFT and throughput benchmarks (Figures 4–6) measuring generator-side cost after evidence selection, using fixed synthetic parameters (L_q = L_c = L_g = 128, N = 65,536). The cost model in Table 3 formalizes the asymptotic comparison.

What the experiments demonstrate: The TTFT advantage is real and substantial — 19× at k = 500, 2–3× at typical k values. This comes from avoiding quadratic re-encoding of evidence during prefill. The throughput advantage is smaller (Figure 5) because generation cost dominates at typical output lengths, and the evidence attention cost is similar for both methods. The experiments correctly exclude retrieval time to isolate the encoding-reuse benefit.

Important caveats for the efficiency claim:

  • Retrieval time is excluded from the benchmark. Figure 4 "excludes retrieval time to isolate the cost of re-encoding versus reuse." In a real deployment, the total latency is retrieval time + prefill + generation. While retrieval cost is identical for INTRA and standard RAG in their asymptotic model (both use O(√M · L_q · L_c) IVF-ANN), the constant factors may differ because INTRA's retrieval pass requires an additional decoder forward pass (the retrieval forward pass $g\text{Dec}(x_{\text{retrieval}}, \mathbf{K}(\mathcal{S}_0))$). This cost is not captured in the benchmark, which starts after evidence selection. The paper acknowledges this in Section 7: "Deployment cost also depends on indexing, storage format, and data movement."
  • The synthetic benchmark parameters may not reflect real-world evidence lengths. The experiment uses L_c = 128 tokens per chunk, but the actual average chunk length in the paper's corpus is ~92 words (Appendix E), which likely corresponds to ~120–150 tokens — the benchmark is representative. However, the retrieval pool size N = 65,536 tokens is much smaller than the experimental corpus (~100M tokens). The quadratic self-attention cost in standard RAG grows with evidence length, so the speedup at longer chunk lengths (Figure 6) may be even larger in practice than the reported 19×.
  • Storage cost is non-trivial. The 2.56 TB figure for a 1B-token corpus (Appendix A.3) is for the shared encoder representation only. A production system would additionally need the ANN index structure (which has its own memory overhead), the raw text to serve to users or for debugging, and storage for any metadata. While 2.5 TB of SSD storage is economically feasible, it is substantially larger than a traditional retrieval index based on compressed embeddings (which might be 10–100× smaller for the same corpus). This storage-performance tradeoff is not analyzed.

Missing experiments that would strengthen the paper

  1. Scale analysis. All experiments use T5Gemma2 4B. How does INTRA's retrieval performance change with model scale? Does a 1B encoder-decoder also exhibit intrinsic retrieval? Does an 8B or larger model show even stronger capability? This is critical for the "intrinsic capability" claim because retrieval quality should, if the capability is indeed intrinsic and scales with model quality, improve with model scale.

  2. Zero-shot retrieval performance. The untrained retrieval tokens (random initialization, no retrieval training) would provide a baseline for how much retrieval capability exists in the pretrained model without any retrieval-specific signal. If random retrieval tokens already achieve non-trivial recall, the "intrinsic" claim is strengthened. If they perform at chance, the capability emerges entirely from the 10K-step training, which would soften the intrinsic capability interpretation.

  3. Dynamic corpus evaluation. All experiments use a static, pre-encoded corpus. Real-world deployments require handling corpora that change over time (new documents, updated information). INTRA's architecture requires re-encoding the entire corpus or the changed chunks, rebuilding the ANN index, and potentially retraining the retrieval tokens if the distribution of evidence shifts. The paper does not evaluate the cost or performance impact of corpus updates.

  4. Encoder-only retrieval with the same generator as a control for Table 2. As noted above, this would isolate whether the gap-closure benefit comes from sharing the encoder (encoder-only MaxSim retrieval + same decoder) or specifically from decoder-conditioned queries. This would sharpen the interpretation of which aspect of shared representations drives the alignment benefit.

  5. Retrieval token analysis. What do the trained retrieval tokens attend to during the retrieval pass? Do different retrieval tokens specialize in different aspects of evidence (e.g., one token consistently retrieves date-related evidence, another retrieves entity-related evidence)? Such analysis would provide mechanistic insight into how the retrieval interface works, beyond the black-box performance numbers.

  6. Sensitivity to $\mathcal{S}_0$ size and quality. The default $n_0 = 20$ is used throughout. How does performance change with smaller or larger $\mathcal{S}_0$? How does it interact with the number of retrieval tokens? The ablation shows $\mathcal{S}_0 = \emptyset$ is catastrophic (33-point drop) and $\mathcal{S}_{\text{INTRA}} = \mathcal{S}_0$ is also poor (22.8-point drop), but the sensitivity to intermediate values is unexplored. This matters for practical deployment because larger $\mathcal{S}_0$ increases the retrieval pass cost (more cross-attention context to process).

6. Limitations and Trade-offs

The Difficulty Estimation Overhead Is Not Accounted for in Efficiency Claims

The assumption or constraint. The entire INTRA framework depends on pre-encoding every chunk in the corpus and storing the resulting encoder states as a shared pool $\bar{\mathbf{K}}$. While this pre-encoding is amortized across queries, the initial retrieval pass that produces $\mathcal{S}_0$ must score the query against every chunk in the corpus — an $O(M L_q L_c)$ operation in the exact case, reduced to $O(\sqrt{M} L_q L_c)$ under IVF-ANN. The paper acknowledges that the TTFT benchmarks "exclude retrieval time to isolate the cost of re-encoding versus reuse" (Section 5.3, Figure 4 caption) and explicitly states in Section 7: "Deployment cost also depends on indexing, storage format, and data movement."

The consequence. The headline 19× TTFT speedup over standard RAG at k = 500 (Figure 4) is a generator-only measurement that starts after evidence has been selected. In a real deployment, the total end-to-end latency is retrieval time + prefill + generation. INTRA requires two decoder forward passes per query — one for retrieval (producing the query states $\mathbf{q}^\ell$ used to score chunks) and one for generation — while standard RAG requires only an embedding lookup (one encoder forward pass for the retriever) followed by one decoder forward pass for generation. The retrieval decoder pass in INTRA involves a full forward pass through 34 decoder layers with cross-attention over $\mathcal{S}_0$ (20 chunks, each of length $L_c$), which is substantially more expensive than computing a single embedding vector and performing ANN search against a pre-built index. The paper never measures or reports the cost of this retrieval forward pass, making the total end-to-end latency comparison incomplete.

What evidence exists in the paper. The paper's computational cost model (Table 3) assigns the same asymptotic retrieval cost $O(\sqrt{M} L_q L_c)$ to both INTRA and standard RAG, citing IVF-ANN as providing this complexity for both. However, this model ignores the constant factors and the decoder forward pass cost that INTRA incurs before the MaxSim scoring step. For INTRA, retrieval involves: (1) an encoder forward pass to get $\mathbf{k}_x = \text{Enc}(x)$, (2) a MaxSim-based ANN search to build $\mathcal{S}_0$, (3) a full decoder forward pass $g\text{Dec}(x_{\text{retrieval}}, \mathbf{K}(\mathcal{S}_0))$ to produce query states, and (4) MaxSim scoring of those query states against all chunks. Standard RAG's retrieval involves: (1) an encoder forward pass through the retriever model to produce a query embedding, and (2) ANN search against the index. The extra decoder forward pass — processing 34 layers with cross-attention over 20 × $L_c$ tokens — is a substantial per-query cost that is entirely unaccounted for in the efficiency analysis. The paper's Appendix D.5 lists training compute but does not report inference latency including the retrieval pass.

Mitigation status. The paper partially acknowledges this gap. Section 7 notes that "Deployment cost also depends on indexing, storage format, and data movement, and token-level memories can be substantially larger than compressed retrieval indices, so end-to-end trade-offs may differ from those analyzed here." The caption for Figure 4 states upfront that retrieval time is excluded. However, the paper does not report end-to-end latency measurements, does not quantify the cost of the retrieval decoder forward pass, and does not provide an apples-to-apples latency comparison against standard RAG including all stages. This is flagged as deferred work: "extending these findings across scales, modalities, dynamic corpora, and architectures is left for future work" (Section 7). A practitioner evaluating INTRA for deployment would need to measure the full end-to-end pipeline to determine whether the prefill speedup outweighs the additional retrieval cost for their specific workload.

The "Intrinsic Capability" Claim Is Qualified by the CLaRa Warm-Start and Oracle Retrieval Training

The assumption or constraint. The paper's central conceptual claim is that retrieval is "an intrinsic capability of attention-based models" (title and abstract) that "can be elicited, rather than added as an external module." The experimental setup used to support this claim trains retrieval tokens on benchmark-specific data with oracle evidence supervision, initialized from a checkpoint that has already been fine-tuned on the CLaRa QA pretraining dataset. The paper describes the CLaRa warm-start as serving three purposes: "aligning the decoder with QA-style generation, adapting generation to chunk-based pre-encoded representations rather than re-encoded text, and incorporating the Reverse-QWK reparameterization of cross-attention" (Appendix B.1).

The consequence. The claim of "intrinsic" capability is ambiguous about what "intrinsic" means. If the capability exists in the pretrained model but is surfaced only after (a) fine-tuning on a QA pretraining dataset and (b) training retrieval-specific interface parameters with oracle supervision, then the capability is better characterized as latent but requiring task-specific elicitation rather than immediately accessible without modification. This distinction matters for two reasons:

First, a practitioner wanting to deploy INTRA on a new domain would need to replicate at minimum the retrieval token training (requiring oracle evidence annotations, which may be expensive to obtain) and possibly the CLaRa-style warm-start (requiring a QA pretraining dataset). The paper does not establish whether the warm-start is necessary — the ablation studies in Table 7 test variations of the retrieval mechanism (removing $\mathcal{S}_0$, changing $L_p$, varying retrieval token count) but never test starting from the base T5Gemma2 checkpoint without CLaRa fine-tuning. A configuration that starts from the pretrained checkpoint and trains only the retrieval tokens would directly test the "intrinsic capability" claim. Its absence means the claim rests on a model that has already been adapted for QA with pre-encoded representations.

Second, the retrieval tokens are trained with explicit supervision — the soft cross-entropy loss pushes retrieval scores toward oracle evidence chunks. This is a form of retrieval-specific training, and the fact that only ~164K parameters are updated does not change the fact that the system has been optimized for retrieval with labeled data. The paper does not report zero-shot retrieval performance — what recall would untrained, randomly initialized retrieval tokens achieve? If random retrieval tokens already produce non-trivial complete-evidence recall, the "intrinsic" claim is strongly supported; if they perform near chance, the capability emerges from the 10K-step supervised training and the interpretation shifts toward "retrieval can be learned with a remarkably small parameter budget" rather than "retrieval is already present."

What evidence exists in the paper. The paper does not measure zero-shot retrieval performance or ablate the CLaRa warm-start. The only ablation that approaches this question is the $\mathcal{S}_{\text{INTRA}} = \mathcal{S}_0$ condition (Table 7), which removes the trained retrieval tokens and decoder scoring entirely — but this also removes the CLaRa warm-start benefit because it bypasses the decoder entirely. The evidence for "intrinsic" capability is therefore indirect: the fact that only ~164K parameters need training, combined with the fact that the backbone remains frozen during retrieval training. But this confounds two potential sources of retrieval capability — the original pretraining and the CLaRa fine-tuning — and does not separate the contribution of the retrieval-specific training from the pre-existing capability.

Mitigation status. The paper is transparent about the warm-start (Appendix B.1) but does not discuss it as a potential qualification to the intrinsic capability claim. Section 7 lists limitations about "a fixed context pool" and "a single implementation family" but does not mention the warm-start or the absence of zero-shot evaluation. A reader adopting the paper's conceptual framing at face value — "attention-based models already possess a retrieval mechanism" — would not realize that the evidence for this claim comes from a model that has been fine-tuned on QA data and whose retrieval interface has been trained with oracle supervision for 10K steps. The paper would be strengthened by reporting zero-shot retrieval performance (untrained retrieval tokens), by ablating the CLaRa warm-start, or by more precisely scoping the claim to "retrieval can be elicited with minimal parameter training after task-specific warm-starting."

Single Model Architecture and Scale Prevent Generalization Claims

The assumption or constraint. All experiments use a single model family (T5Gemma2), a single architecture class (encoder-decoder with cross-attention), and a single scale (4B parameters). The paper acknowledges this explicitly in Section 7: "we focus on a single implementation family: a T5Gemma2-style encoder-decoder with Reverse-QWK, evaluated on text-QA benchmarks with short answers. INTRA's reliance on encoder-decoder cross-attention also excludes decoder-only models. Extending these findings across scales, modalities, dynamic corpora, and architectures is left for future work."

The consequence. This limitation has two distinct implications:

Architectural exclusivity. INTRA fundamentally depends on cross-attention between a decoder and pre-encoded evidence states. Decoder-only architectures (GPT-style, Llama, Qwen, Mistral, etc.) — which constitute the vast majority of deployed large language models — have no mechanism for this. In a decoder-only model, all attention is self-attention over the concatenated prompt; there is no separate encoder whose outputs can be stored and reused across queries without being incorporated into the decoder's context window. This means INTRA's approach is architecturally inapplicable to the most widely used class of language models. A practitioner using GPT-4, Claude, Llama, or any other decoder-only model cannot adopt INTRA without switching to an encoder-decoder architecture. The paper does not discuss whether an analogous mechanism might exist for decoder-only models (e.g., through prefix caching or key-value reuse), leaving open the question of whether the "intrinsic retrieval" claim generalizes or is specific to architectures with explicit cross-attention.

Unknown scaling behavior. The paper demonstrates that a 4B encoder-decoder exhibits retrieval capability with a small amount of interface training, but provides no evidence about how this capability varies with model scale. Several questions are unanswered: Does a 1B encoder-decoder also exhibit the capability, or is there a threshold scale below which the cross-attention queries are too weak to serve as retrieval signals? Does an 8B or larger model show even stronger retrieval — i.e., does the intrinsic retrieval capability scale with model quality, as one would expect for a genuinely "intrinsic" property? Conversely, could a smaller model augmented with INTRA retrieval match a larger model without retrieval, analogous to the inference-compute vs. pretraining-compute tradeoff studied in other work? The paper provides no data to address these questions. The ablation in Table 7 varies hyperparameters (retrieval token count, pooled chunk length) but always within the 4B model, so the sensitivity of retrieval performance to model scale is unknown.

The single-benchmark-domain limitation (four Wikipedia-based QA benchmarks, all text-based with short answers) compounds this. The paper's findings about multi-hop advantage and single-hop disadvantage might be specific to factoid question answering over an encyclopedia corpus. Whether the decoder-conditioned retrieval signal provides similar benefits for code retrieval, scientific literature search, legal document retrieval, or multi-modal retrieval is untested.

What evidence exists in the paper. The paper provides no scaling experiments, no architecture ablations (beyond the single T5Gemma2 variant), and no domain-transfer experiments. Table 2 tests generator compatibility but varies only the generator while keeping the INTRA retriever fixed — it does not test whether other encoder-decoder architectures could serve as the INTRA backbone. Section 7's limitation statement is honest but does not mitigate the risk that a practitioner might over-interpret the "attention-based models" claim as applying to models without cross-attention.

Mitigation status. The limitation is explicitly acknowledged but not addressed. The paper positions itself as a capability demonstration rather than a comprehensive evaluation, and this is a reasonable scope for a first paper introducing a new framework. However, given the strong conceptual claim about "attention-based models" (which linguistically includes decoder-only transformers), the architectural restriction to encoder-decoder models with cross-attention is a significant qualification that readers should understand. The paper suggests future work on extending to other architectures and scales but provides no preliminary evidence that such extension is likely to succeed.

Storage Requirements Make the Approach Impractical for Large or Dynamic Corpora

The assumption or constraint. INTRA's amortized-encoding efficiency depends on storing pre-computed encoder states for every chunk in the corpus. Appendix A.3 quantifies this: for a 1B-token corpus with the T5Gemma2 4B model (d = 2560), the shared encoder representation $\bar{\mathbf{k}}$ requires 1B × 2560 bytes ≈ 2.56 TB at 8-bit precision. The paper frames this as "practical" because it "fits on a single NVMe SSD" and notes that "product quantization or further compression could shrink this substantially."

The consequence. The storage cost creates two practical barriers:

Absolute scale. While 2.5 TB of SSD storage is economically feasible (a single high-capacity NVMe SSD costs a few hundred dollars), it is 10–100× larger than what a traditional retrieval index over the same corpus would require. A standard dense retrieval index using 768-dimensional embeddings at 8-bit precision would be approximately 1B tokens / (average tokens per chunk) × 768 bytes. If average chunk length is ~128 tokens, this is roughly 7.7M chunks × 768 bytes ≈ 5.9 GB — roughly 430× smaller than INTRA's storage. Even a multi-vector ColBERT-style index storing 16–32 vectors per chunk would be on the order of 100–200 GB — still 10–25× smaller. The paper acknowledges this in Section 7 ("token-level memories can be substantially larger than compressed retrieval indices") but does not discuss the tradeoff explicitly. A practitioner with a 10B-token corpus would need ~25 TB of SSD storage for INTRA's encoder states, which crosses from "fits on one SSD" to "requires a storage array" — a meaningful infrastructure difference.

Corpus mutability. All experiments use a static, pre-encoded corpus. Real-world knowledge bases change over time — Wikipedia is updated continuously, news archives grow daily, code repositories receive commits, and internal document stores are modified. INTRA's architecture requires three operations when the corpus changes: (1) encoding the new or modified chunks through the encoder, (2) inserting their representations into the stored pool $\bar{\mathbf{K}}$, and (3) rebuilding or updating the ANN index over $\bar{\mathbf{K}}$. The paper does not discuss the cost of these operations, how to handle chunk deletions, or whether the retrieval token training must be updated when the corpus distribution shifts. This is in contrast to standard RAG, where updating the retrieval index typically requires only re-embedding the changed documents (which are small relative to the corpus) and updating the ANN index incrementally — no model forward passes beyond the retriever's encoder.

Relatedly, the one-time pre-encoding cost is substantial but amortized: encoding 100M tokens through the T5Gemma2 encoder (a 4B-parameter model) requires a non-trivial amount of compute — the paper reports training used up to 160 H100 GPUs for the QA pretraining stage (Appendix D.5), and while the encoder forward pass is cheaper than training, encoding a billion-token corpus is a meaningful compute expenditure that must be weighed against the per-query savings it enables. The break-even point — how many queries must be served before the pre-encoding cost is recovered through per-query speedups — is not analyzed.

What evidence exists in the paper. Appendix A.3 provides the storage calculation and notes that compression could reduce the footprint. Section 7 acknowledges the storage concern. Appendix D.5 reports GPU-hours for training but not for corpus encoding. The paper does not report: incremental update costs, break-even analysis for encoding amortization, performance with compressed encoder states (product quantization is mentioned but not tested), or comparison to retrieval index sizes for the baselines. The TTFT benchmarks use a corpus of only N = 65,536 tokens (Figure 4 caption), which is 1,500× smaller than the experimental corpus and 15,000× smaller than the 1B-token corpus discussed in the storage analysis, so the benchmarks do not reflect the storage and I/O costs of a realistically sized deployment.

Mitigation status. Partial. The paper acknowledges the storage concern and suggests compression as mitigation, but provides no experimental validation of compressed-storage INTRA performance. The storage comparison to traditional retrieval indices is noted but not quantified. The corpus mutability issue is not discussed. For a practitioner evaluating INTRA for a production system with a dynamic corpus, the paper provides no guidance on update costs, frequency, or performance impact, and the static-corpus experimental setup does not model this aspect of real-world deployment. This is fundamentally a tradeoff the paper accepts but does not resolve — INTRA trades storage cost (and the associated I/O and encoding costs) for per-query speedup, and the tradeoff's favorability depends heavily on corpus size, query volume, and update frequency, none of which are analyzed.

No Mechanism for Multi-Turn or Iterative Retrieval During Generation

The assumption or constraint. INTRA performs retrieval in a single shot before generation begins: the retrieval pass scores all chunks, selects $\mathcal{S}_{\text{INTRA}}$, and the generation pass produces the answer conditioned on that fixed evidence set. There is no mechanism for the decoder to request additional evidence mid-generation if it discovers it needs information not present in $\mathcal{S}_{\text{INTRA}}$. This contrasts with agentic RAG systems (Asai et al., 2024; Li et al., 2025) that interleave retrieval with generation, allowing the model to issue new queries as it generates and discovers gaps in the retrieved evidence.

The consequence. This is a structural limitation for tasks that require iterative evidence gathering. Consider a complex multi-hop question on MuSiQue where the answer requires chaining through 3–4 pieces of evidence: the model might need to read chunk A to identify an entity, then retrieve chunks about that entity, then use those to identify a second entity, and so on. INTRA's single-pass retrieval must identify all relevant evidence chunks based solely on the question and the initial context $\mathcal{S}_0$. If the decoder's query states — which are conditioned on the question and $\mathcal{S}_0$ but not on intermediate reasoning steps — fail to anticipate the full chain of evidence needed, the generation will be missing critical information with no way to recover it.

The paper's results on MuSiQue illustrate this limitation indirectly. Complete-evidence recall@5 on MuSiQue is only 12.8% for INTRA (Table 4), and end-to-end EM is 14.0% (Table 1) — both quite low in absolute terms. While MuSiQue is inherently difficult and all methods struggle, the low scores are consistent with the hypothesis that single-pass retrieval is insufficient for compositional multi-hop questions where the evidence chain is long and the relevance of later chunks depends on information from earlier chunks that isn't available to the retrieval pass. The paper does not discuss this as a limitation of the single-pass design, but the gap between INTRA's multi-hop performance on HotPotQA (59.9% recall@5, typically 2-hop questions) and MuSiQue (12.8%, typically 3–4 hop questions) suggests that retrieval difficulty grows superlinearly with the number of hops — a pattern that iterative retrieval could potentially mitigate.

A related but more subtle limitation: even for two-hop questions, the retrieval signals are static — the same scoring is applied to all chunks regardless of what the retrieval pass has already "seen" in $\mathcal{S}_0$. There is no mechanism for the scoring of chunk $i$ to depend on which other chunks have been scored highly, which matters when evidence chunks are complementary (both needed) or redundant (either suffices). Standard RAG rerankers have the same limitation, but agentic RAG systems address it by conditioning subsequent retrievals on previously retrieved information.

What evidence exists in the paper. The paper does not evaluate iterative or multi-turn retrieval, does not ablate a multi-pass variant of INTRA, and does not compare against agentic RAG baselines. The related work section (Section 6) discusses agentic RAG but positions INTRA as targeting "the single-pass retrieval block that could be used within such pipelines," explicitly deferring the multi-turn aspect to the pipeline level. The per-benchmark pattern of results — strong on HotPotQA (2-hop), weaker on MuSiQue (3–4 hop) — provides circumstantial evidence consistent with the single-pass limitation but is not a controlled experiment testing it.

Mitigation status. The paper acknowledges the scope limitation in Section 6 ("INTRA targets the single-pass retrieval block that could be used within such pipelines, rather than the pipeline-level agentic loop itself") but does not discuss whether INTRA's architecture can support iterative retrieval. Extending INTRA to multi-turn retrieval would require: (1) a mechanism for the generation pass to signal when additional evidence is needed (e.g., special "retrieve" tokens in the output), (2) a way to update the retrieval query states based on partial generation (e.g., running a second retrieval pass conditioned on the partial answer), and (3) efficient combination of the new evidence with the existing cross-attention context. None of these are explored, and the current architecture provides no natural interface for them. This is a consequential gap because many real-world QA and reasoning tasks — particularly those that motivated the RAG paradigm in the first place — benefit from or require iterative evidence gathering. The paper's results should therefore be interpreted as establishing what single-pass retrieval can achieve within a shared representation space, not as demonstrating a replacement for iterative retrieval architectures.

Hard Problems Show Diminishing Returns, and the Retrieval Capability Has an Effective Ceiling

The assumption or constraint. The paper's results show a clear difficulty-dependent pattern: INTRA's advantage is largest on benchmarks where the retrieval problem is moderately hard but not impossible (HotPotQA, 2Wiki), and it diminishes or reverses on both the easiest benchmark (NQ, where dedicated retrievers are competitive) and the hardest benchmark (MuSiQue, where all methods perform poorly). This pattern is observed but not systematically analyzed as a limitation — the paper does not investigate whether there are fundamental reasons INTRA's retrieval capability saturates, or whether the saturation can be overcome with more capacity or different training.

The consequence. This limitation manifests in two ways:

Saturation on hard problems. On MuSiQue, INTRA achieves only 12.8% complete-evidence recall@5 — a gain of only +2.7 points over Qwen3-Emb-4B + reranker and +6.6 points over Hybrid RAG (Table 4). The absolute score is low, and more importantly, it is unclear whether it can be substantially improved. The retrieval tokens are trained with a fixed capacity (64 tokens) on a fixed amount of training data (the MuSiQue training split of 277K examples), and the decoder backbone is frozen. If the cross-attention queries of a 4B model fundamentally lack the capacity to express the complex, multi-step evidence needs of MuSiQue questions, then no amount of retrieval token training will close the gap to oracle retrieval — the intrinsic capability has a ceiling determined by the base model's representational capacity. The paper provides no evidence about where this ceiling lies, whether it can be raised by scaling the model, or whether alternative training objectives (beyond the soft cross-entropy used in Section 3.2) could push retrieval performance higher.

Single-hop performance below dedicated retrievers. On NQ, INTRA's complete-evidence recall@5 (29.1%) lags behind Qwen3-Emb-4B (30.3%), BGE-large (29.6%), and Qwen3-Emb-4B + reranker (31.9%). End-to-end EM on NQ (51.2%) trails Qwen3-Emb-4B + reranker (55.1%) by -3.9 points (Table 1). The paper interprets this as expected: "NQ's single-hop nature minimizes this benefit" (Figure 2 caption). But this framing implies that INTRA offers no advantage and a measurable disadvantage on single-hop retrieval, which constitutes a large fraction of real-world QA and RAG use cases. A practitioner considering INTRA for a general-purpose retrieval system would need to accept degraded performance on single-hop queries as the cost of improved multi-hop performance — there is no evidence that the retrieval tokens can be trained to match dedicated retrievers on single-hop while maintaining multi-hop gains. This may represent a fundamental tradeoff: the decoder-conditioned queries are optimized for evidence assembly (which matters for multi-hop) but are less precise than dedicated query-chunk similarity models for simple relevance matching (which is all that single-hop requires).

What evidence exists in the paper. The difficulty-dependent pattern is visible across Figures 2 and Table 1: INTRA leads on HotPotQA and 2Wiki, roughly ties on MuSiQue (within confidence intervals), and trails on NQ. The paper presents this as a feature of the method (decoder conditioning helps most on multi-hop) but does not analyze the flip side — that the approach introduces a single-hop performance regression relative to dedicated retrieval models trained on large-scale retrieval data. The absence of scaling experiments means there is no evidence about whether a larger INTRA backbone would close the single-hop gap, outperform on MuSiQue, or both. The retrieval token capacity ablation (Table 7) shows that 16 tokens nearly match 64 tokens on CE-recall@5 (-4.4 points on HotPotQA, -4.9 on 2Wiki), suggesting the current configuration may be near the retrieval signal ceiling for this model — but this is not discussed.

Mitigation status. The paper does not explicitly treat this as a limitation to be solved. The difficulty-dependent behavior is presented as a characteristic of the approach (decoder queries are especially useful for multi-hop evidence assembly) rather than as a problem (the approach underperforms on single-hop). Section 7's limitations do not mention the single-hop performance gap or the unknown ceiling on hard problems. A practitioner reading the paper should understand that INTRA's performance is best understood as complementary to dedicated retrieval models, not uniformly superior — it excels in a specific regime (multi-hop reasoning over a moderate-sized static corpus with an encoder-decoder model) and underperforms in another (single-hop retrieval where large-scale trained embedding models have an advantage). The paper would benefit from a more explicit characterization of this operating envelope, including guidance on when a practitioner should prefer INTRA over standard RAG and vice versa.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a conceptual intervention rather than an incremental architectural improvement. The central reframing — that retrieval is an intrinsic capability of attention-based models that can be surfaced rather than added as an external module — challenges the dominant modular decomposition that has organized the RAG field since Lewis et al. (2020). If the claim holds up under broader validation, it shifts retrieval from something you build (a separate system with its own architecture, training data, and optimization objective) to something you elicit (an interface into capabilities already latent in a pretrained generator). This is a category shift in how to think about the relationship between language models and retrieval infrastructure.

The magnitude of this shift should not be overstated. The paper does not demonstrate that INTRA replaces RAG pipelines at scale, nor does it show that the intrinsic capability generalizes beyond encoder-decoder architectures with cross-attention — a class that excludes the decoder-only models dominating deployment today. The experiments are confined to a single model family (T5Gemma2 4B), a single corpus scale (~100M tokens), and a single task domain (Wikipedia QA). What the paper does establish is a proof of existence: under controlled conditions, a frozen pretrained model with ~164K trainable interface parameters can perform retrieval that rivals dedicated embedding models trained on large-scale retrieval corpora, including BGE-large and Qwen3-Embedding-4B with a reranker (Table 4: +11.1 complete-evidence recall@5 over Qwen+Reranker on HotPotQA). The existence proof is sufficiently strong — and the interface training sufficiently lightweight — that the burden of explanation shifts: rather than asking "can models do this?", the question becomes "under what conditions does this capability manifest, and what are its limits?"

Reconciling prior contradictions. The paper provides a resolution to a tension in the late-interaction retrieval literature. ColBERT (Khattab and Zaharia, 2020) and its variants demonstrated that token-level MaxSim scoring between query and document embeddings achieves strong retrieval performance, but these systems are dedicated retrievers whose representations are discarded after ranking — the generator that ultimately consumes the retrieved documents must re-encode them from raw text. INTRA shows that the MaxSim signal can be embedded within the generator itself, so the same representations that score relevance also condition generation. This resolves the apparent inefficiency in ColBERT-style pipelines where expensive token-level comparisons are computed only to be discarded: INTRA's design reuses those comparisons (or, more precisely, the representations that enable them) for both retrieval and generation, collapsing two passes through different models into two passes through the same model.

The paper also implicitly reconciles the observed brittleness of long-context models on sparse evidence tasks (Yen et al., 2024; Modarressi et al., 2025) with the practical success of retrieval-augmented approaches. Long-context models can theoretically attend to all evidence without retrieval, but they fail to reliably surface relevant information when it is sparse and distributed. INTRA suggests a middle ground: the model does attend to all evidence, but through an explicit retrieval mechanism (MaxSim scoring of decoder queries against pre-encoded chunks) rather than through dense self-attention over a long concatenated context. The retrieval step makes the selection explicit and auditable (we know which chunks were scored highly and why), while the generation step benefits from the shared representation space. This is not a new architecture but a new decomposition of an existing architecture's capabilities.

Research directions that become more attractive. The paper's strongest contribution is opening the question of what else is latent in pretrained models' attention mechanisms. If cross-attention queries can be repurposed for chunk-level evidence retrieval with only ~164K interface parameters, what other information-retrieval or reasoning capabilities might be similarly accessible? Specific candidates include:

  • Evidence contradiction detection: Can decoder queries be trained to score chunks that contradict rather than support a hypothesis, enabling fact-verification without a separate natural language inference model?
  • Source attribution: Can the retrieval token scores across layers be decomposed to attribute which parts of a generated answer depend on which evidence chunks, providing a built-in citation mechanism?
  • Uncertainty-aware retrieval: Can the variance of retrieval scores across multiple forward passes (with dropout enabled) serve as a confidence signal for whether relevant evidence exists in the corpus?

These directions are speculative, but the paper makes them newly tractable by demonstrating that the decoder's query states carry task-relevant information that can be extracted with lightweight training.

Research directions that become less attractive. If INTRA's core claim holds up — that a frozen pretrained model can perform competitive retrieval with minimal interface training — then the cost-benefit calculus for building and maintaining separate retrieval models shifts. For deployments where the generator is already an encoder-decoder model, training a dedicated retriever in a separate representation space may become harder to justify when the generator's own encoder can serve as the retrieval backbone. The paper does not establish this for all settings (decoder-only models, web-scale corpora, dynamic corpora), but it establishes a lower bound: at minimum, INTRA performance is a baseline that any separate retriever must beat to justify its additional complexity. This raises the bar for "do we need a separate retriever?" from "yes, by default" to "only if it significantly outperforms what the generator can already do."

Follow-Up Research This Work Enables

Zero-shot retrieval benchmarking to isolate the contribution of pretraining vs. fine-tuning vs. interface training. The paper's claim of "intrinsic" capability is confounded by the CLaRa warm-start and the oracle-supervised retrieval token training. A clean experiment would measure complete-evidence recall@5 on HotPotQA and 2Wiki using: (a) the base T5Gemma2 checkpoint with no QA fine-tuning and no retrieval training (using only the encoder's MaxSim scoring, equivalent to S_0 but with the pretrained encoder), (b) the CLaRa-warm-started model with no retrieval token training (random retrieval tokens), (c) the CLaRa-warm-started model with retrieval tokens trained for varying numbers of steps (100, 1K, 10K), and (d) the full INTRA pipeline. This would decompose the ~60% recall@5 into contributions from pretraining, QA warm-start, and retrieval interface training. If (b) achieves non-trivial recall (say, >20% CE-recall@5), the "intrinsic" claim is strengthened — the untrained decoder queries already carry retrieval-relevant signal. If only (d) works, the capability is better described as "elicitable with minimal task-specific interface training" rather than "already present and merely surfaced." The experiment would also reveal how many training steps are needed to saturate the retrieval signal, informing practitioners about the data requirements for new domains.

Scaling law for intrinsic retrieval capability across model size. All experiments use T5Gemma2 4B. A natural follow-up would measure INTRA retrieval performance on T5Gemma2 variants at 1B, 2B, 4B, and 8B scales (if available) using identical retrieval token training protocol and corpus. The key question: does complete-evidence recall improve with model scale when the backbone is frozen? If retrieval capability scales with model quality — as one would expect for an "intrinsic" property — the scaling curve would inform whether larger encoder-decoder models could close the single-hop gap with dedicated retrievers on NQ (where INTRA currently lags Qwen3-Emb-4B by ~1.2 points recall@5) and whether the hard-problem ceiling on MuSiQue (12.8% recall@5) can be raised. If retrieval capability does not scale — if a 4B model is already near the ceiling — that would bound the approach and suggest that architectural innovations beyond scale are needed. A follow-up would also measure whether larger models require proportionally more retrieval tokens to express their expanded query capacity, or whether 64 tokens saturates at all scales.

INTRA-style retrieval for decoder-only models via key-value cache reuse. The paper explicitly notes that INTRA's cross-attention mechanism excludes decoder-only architectures. An ambitious extension would investigate whether an analogous "intrinsic retrieval" mechanism exists in decoder-only models through a different interface. One candidate: store the key-value cache from processing each chunk in a long-context decoder-only model (since the model must encode all chunks to attend to them), build an ANN index over the query representations from a dedicated set of trainable prefix tokens prepended to the question, and score chunks via MaxSim between these query states and the stored KV cache entries. This would require a Reverse-QWK-style reparameterization to share representations across layers, but decoder-only models have only self-attention (not cross-attention), so the mathematical structure differs — the KV cache entries are per-layer and per-head, and the transformation to a shared representation space is non-trivial. The experiment would test whether the intrinsic capability generalizes across architectural families or is specific to encoder-decoder cross-attention. Success would dramatically expand INTRA's applicability to Llama, Mistral, Qwen, and other widely deployed model families; failure would clarify the boundary conditions of the capability claim and suggest that cross-attention provides a uniquely suitable interface for intrinsic retrieval.

Iterative INTRA with mid-generation re-retrieval for compositional multi-hop reasoning. The paper's single-pass design limits performance on MuSiQue (12.8% recall@5, 14.0% EM), where questions require chaining through 3–4 evidence pieces and the full evidence set is hard to identify from the question alone. A natural extension would introduce a special "retrieve" token into the decoder's vocabulary. When the decoder generates this token during answer generation, the current decoder state is used as a query for a second retrieval pass over the corpus (using the same MaxSim + retrieval token mechanism, but now the retrieval tokens can attend to the partial answer via self-attention), and the newly retrieved evidence is appended to the cross-attention context for subsequent generation steps. This would allow the model to condition retrieval on intermediate reasoning — it could identify an entity from a first chunk, issue a query for that entity, and integrate the result into the ongoing generation. The experiment would measure whether iterative retrieval closes the gap between INTRA's current MuSiQue performance and the oracle upper bound (which Table 2 shows to be substantially higher). The cost would be additional retrieval forward passes per question (one per hop), trading latency for recall. A simpler variant could perform a fixed two-pass retrieval: a first pass identifies "seed" entities from S_0, a second retrieval pass scores chunks conditioned on those entities, and both sets are combined for generation.

Cross-domain transfer of retrieval tokens without retraining. The retrieval tokens are trained jointly on the four benchmark training splits from Wikipedia-based QA datasets. To what extent does this training produce general-purpose retrieval queries, versus queries specialized to the Wikipedia QA domain? A controlled experiment would take the retrieval tokens trained on HotPotQA/2Wiki/MuSiQue/NQ and evaluate them zero-shot on a held-out retrieval benchmark from a different domain — for example, a biomedical QA dataset (BioASQ, PubMedQA) or a legal retrieval task (CaseHOLD, LexGLUE) — using a corpus pre-encoded by the same T5Gemma2 encoder. If the retrieval tokens transfer well (maintaining a significant fraction of their in-domain recall advantage over baselines), the learned queries are capturing domain-general evidence-seeking behavior, supporting the "intrinsic capability" interpretation — the decoder's cross-attention queries express information needs that are largely independent of the specific domain. If they transfer poorly (performance drops to near the encoder-only MaxSim baseline or below), the retrieval tokens have learned domain-specific heuristics rather than a general retrieval skill, and practitioners would need to retrain tokens for each new domain — a significant practical limitation. The experiment would also reveal whether joint training on multiple domains produces more transferable queries than training on a single domain.

Robustness to corpus distribution shift and adversarial distractors. The paper's corpus is de-duplicated and constructed to include all oracle chunks plus uniformly sampled non-oracle distractors (Appendix E). Real-world corpora contain near-duplicates, partially overlapping passages, contradictory information, and adversarially chosen distractors. A stress test would evaluate INTRA on: (a) a corpus where 50% of chunks are near-duplicates of oracle chunks (testing whether MaxSim scoring collapses and assigns high scores to all duplicates, diluting retrieval precision), (b) a corpus with injected "adversarial" chunks that are highly similar to oracle chunks under encoder-only MaxSim but contain factually incorrect information (testing whether the decoder-conditioned queries can distinguish surface similarity from genuine relevance — this would be a direct test of whether decoder conditioning provides robustness beyond what encoder-only similarity offers), and (c) a corpus where oracle chunks are gradually removed (testing the retrieval degradation curve — does INTRA gracefully assign uniformly low scores to all chunks when no oracle exists, or does it hallucinate high scores for spurious partial matches?). These experiments would establish safety boundaries for deployment in uncontrolled corpora and would reveal whether the decoder-conditioned retrieval signal is more or less robust than dedicated embedding models under distribution shift — a critical question for production use.

Practical Applications and Downstream Use Cases

Single-model retrieval-generation for on-device or edge deployment of encoder-decoder models. In settings where deploying multiple models is constrained by memory or power — mobile devices, embedded systems, or edge servers — INTRA's collapse of retriever and generator into a single model eliminates the need to load and run a separate retrieval model. The entire system is one encoder-decoder with its precomputed encoder states stored on device. For a ~100K-chunk knowledge base (e.g., a technical manual, a medical reference, a legal code), the encoder states at the T5Gemma2 4B scale would require approximately 100K × 128 tokens/chunk × 2560 dimensions × 1 byte (8-bit) ≈ 33 GB of storage, feasible on a high-end mobile device or tablet with sufficient SSD capacity. The per-query computation involves two decoder forward passes (retrieval + generation) rather than retriever-forward + generator-forward + re-encoding. This is architecturally simpler than maintaining a separate retriever, reduces the number of model artifacts to distribute and update, and eliminates a class of integration failures where retriever and generator versions become mismatched. The paper's TTFT benchmarks (Figure 4) show 12.8 ms prefill at k = 1 and 15.2 ms at k = 5 for INTRA — if the retrieval forward pass adds comparable latency, interactive response times are achievable. The primary practical barrier is storage: 33 GB is large for current mobile devices, and compression (product quantization, dimensionality reduction) would need to be applied and validated for INTRA's retrieval performance.

Cost-efficient batch inference over static corpora with high query volume. For organizations running batch QA or information extraction over a fixed document collection — regulatory filings, scientific literature, internal documentation — INTRA's amortized-encoding model provides a direct cost reduction. The corpus is encoded once (at a one-time cost proportional to corpus size) and then queried repeatedly. Each query avoids re-encoding retrieved evidence — the prefill cost drops from O((L_q + kL_c)^2) for standard RAG to O(L_q(L_q + kL_c)) for INTRA (Table 3). For a workload of 1M queries over a 100M-token corpus, the initial encoding cost is spread over all queries, and the per-query prefill savings accumulate. The paper's Figure 4 quantifies the per-query benefit: at k = 20 chunks (~2,560 evidence tokens), INTRA's prefill is ~15 ms versus ~41 ms for standard RAG — a ~2.7× speedup. At 1M queries, this saves approximately 7.2 GPU-hours of prefill compute. The tradeoff is the storage cost: ~2.5 TB for a 1B-token corpus at 8-bit precision (Appendix A.3). For server-side deployments with attached NVMe SSDs, this is economically feasible — a high-capacity SSD costs a few hundred dollars, while GPU compute is substantially more expensive. Organizations should weigh this one-time storage cost against the cumulative compute savings for their specific query volume and corpus size.

Multi-hop QA systems where evidence assembly is the primary bottleneck. INTRA's largest empirical gains are on multi-hop benchmarks: +11.1 complete-evidence recall@5 over Qwen3-Emb-4B + reranker on HotPotQA and +5.3 on 2WikiMultihopQA (Table 4). For applications where questions routinely require assembling evidence from multiple documents — legal research (finding all relevant precedents for a case), scientific literature review (identifying all studies meeting specific criteria for a systematic review), customer support (combining information from multiple knowledge base articles that individually provide partial answers) — INTRA provides a retrieval signal specifically optimized for multi-document evidence assembly. The decoder's query states encode the informational requirements of answer generation, not just query-document similarity, which matters when the relevance of evidence chunk B depends on information in evidence chunk A. A deployment in this regime would pre-encode the document collection using the T5Gemma2 encoder, train retrieval tokens on a domain-specific set of annotated multi-hop questions (if available, or use the jointly-trained tokens from the Wikipedia benchmarks if the domain is close enough), and generate answers using the shared decoder. The paper's gap-closure analysis (Table 2) shows that the shared-decoder configuration closes 59.4% of the EM gap to oracle evidence on average across benchmarks — strong evidence that the retrieval signal is aligned with generation needs specifically for evidence assembly. On HotPotQA, the gap closure is 66.4%, meaning INTRA retrieval recovers two-thirds of the benefit that having perfect oracle evidence would provide. The primary practical barrier is obtaining oracle evidence annotations for retrieval token training on the target domain, but the paper's joint training across benchmarks — with tokens trained on four datasets simultaneously and evaluated on each — suggests that retrieval tokens can learn transferable evidence-seeking behavior, reducing the annotation burden for new domains.

When to Prefer This Method

The paper does not position INTRA against named alternatives in a way that yields a crisp decision rule. It explicitly states it does "not position INTRA as a replacement for RAG in open-web retrieval or web-scale settings" (Section 7) and frames the contribution as conceptual rather than as a universally superior system. However, the experimental results and architecture imply a set of conditions under which INTRA is most naturally preferred, and conditions under which standard RAG with dedicated retrievers remains the better choice. We extract these from the empirical patterns rather than from explicit author guidance.

Prefer INTRA when:

  • You are already using an encoder-decoder model (T5, BART, T5Gemma2) as your generator, making the shared representation space immediately available without switching model families.
  • Your retrieval problem involves multi-hop evidence assembly where decoder-conditioned queries provide the largest gains — HotPotQA and 2WikiMultihopQA are the canonical examples, with INTRA showing +3 to +11 point CE-recall@5 advantages over strong baselines (Table 4).
  • Your corpus is static or changes slowly enough that periodic re-encoding and index rebuilding is acceptable — the amortized-encoding advantage requires the encoding cost to be spread across many queries.
  • Your query volume is high enough that per-query prefill savings (2–3× TTFT reduction at typical retrieval depths, Figure 4) outweigh the up-front storage cost of pre-encoded encoder states (~2.5 TB per 1B tokens, Appendix A.3).
  • You want to minimize the number of models in your deployment pipeline (no separate retriever to train, distribute, update, and debug), accepting that this comes at the cost of larger per-model storage.

Prefer standard RAG with a dedicated retriever when:

  • You are using a decoder-only model (Llama, Mistral, Qwen, GPT, Claude) as your generator, where INTRA's cross-attention mechanism is architecturally inapplicable without major modifications.
  • Your retrieval is predominantly single-hop (entity lookup, passage retrieval, factoid QA) and you need the best possible single-hop performance — on NQ, Qwen3-Emb-4B + reranker achieves 55.1% EM versus INTRA's 51.2% (Table 1), a 3.9-point gap that matters for production systems.
  • Your corpus is large (billions of tokens), dynamic (updated frequently), or both — the storage cost of INTRA's encoder states scales linearly with corpus size, and update costs (re-encoding changed chunks, rebuilding the ANN index) are proportional to corpus change rate. Standard dense retrieval indices are 10–100× smaller and support incremental updates.
  • You need retrieval from the open web rather than a fixed corpus, where pre-encoding is impossible because the retrieval target is not known in advance.
  • Latency is critical and the INTRA retrieval decoder forward pass — a full pass through 34 decoder layers with cross-attention over S_0 — adds unacceptable per-query overhead. The paper does not measure this overhead, but it is a structural cost of the two-pass design that standard RAG avoids by using a lightweight encoder for retrieval.