Fusion-in-Decoder: Open-Domain Question Answering with Retrieval-Augmented Generation

ArXiv: 2007.01282

🎯 Pitch

A generative reader doesn’t saturate as you give it more retrieved passages—unlike extractive models, performance keeps climbing from 10 up to 100 passages, yielding massive gains. The key is fusing all passages only in the decoder, letting the model effortlessly synthesize evidence spread across dozens of documents to nail competing answers.

1. Executive Summary

This paper introduces a simple two-step approach to open-domain question answering that retrieves support passages from Wikipedia—using either sparse BM25 or dense DPR representations—and then feeds them, together with the question, to a pretrained sequence-to-sequence model (T5) that generates the answer. The core architectural contribution is Fusion-in-Decoder, a method where each retrieved passage is processed independently by the encoder (making computation scale linearly with passage count rather than quadratically) while the decoder attends over the concatenated representations of all passages jointly, enabling evidence aggregation from many sources. The approach sets new state-of-the-art results on NaturalQuestions (51.4 EM with the large model) and TriviaQA (67.6 EM), and exhibits a striking scaling property: accuracy continues to improve when increasing retrieved passages from 10 to 100—yielding gains of 6 EM points on TriviaQA and 3.5 on NaturalQuestions—without the performance saturation around 10–20 passages that extractive models typically suffer, establishing that generative seq2seq models are effective at combining evidence from large document sets only when passages are fused in the decoder rather than in the encoder.

2. Context and Motivation

The Two Worlds of Open-Domain QA: Retrieval and Generation Were Separate

The fundamental problem this paper addresses is a specific architectural limitation in how open-domain question answering systems handle multiple passages of evidence. By the time this paper was written, the field had developed two largely separate paradigms for answering general-domain questions using Wikipedia as external knowledge. Both had established strengths, but each had a well-known and stubborn weakness that limited their ability to exploit the full set of retrieved documents.

The extractive paradigm (Chen et al., 2017) retrieved support documents and then selected a span from within those documents as the answer. This approach, built on top of powerful pretrained contextualized representations like BERT (Devlin et al., 2019), worked well when the answer appeared verbatim in the retrieved text. However, it faced a serious architectural bottleneck: aggregating evidence from multiple passages is not straightforward in extractive models. When a system retrieves 50 or 100 passages, many of which may contain partial or complementary information, an extractive model must somehow combine signals across these disjoint text segments using only span-prediction mechanisms. The literature had responded with a variety of increasingly complex aggregation techniques—global normalization over all spans corresponding to the answer (Clark and Gardner, 2018; Wang et al., 2019), hard expectation-maximization for noisy supervision (Min et al., 2019a), and confidence-and-coverage scoring for answer re-ranking (Wang et al., 2018b)—but these felt like patches on a fundamentally ill-suited architecture. The empirical result, which the paper cites as a motivating observation, was that extractive models' performance peaked around 10 to 20 passages and then flatlined or degraded: Wang et al. (2019) and Yang et al. (2019) both observed this saturation, suggesting that extractive models cannot effectively use the additional evidence available in larger retrieved sets.

The generative paradigm, most prominently represented by Roberts et al. (2020), took a radically different approach: train a sequence-to-sequence model (T5) to answer questions directly from its parameters, with no retrieval at all—a "closed-book" setting. This was philosophically appealing: the model stored knowledge in its weights during pretraining and produced answers through generation. The catch was scale. Roberts et al. (2020) obtained competitive results on NaturalQuestions and TriviaQA, but only with models containing billions of parameters—their T5-11B model—because all factual knowledge had to be compressed into the model weights. Smaller models simply did not have enough capacity to memorize sufficient knowledge. This made the approach expensive to train, expensive to query, and impractical for many deployment scenarios. The paper quotes this directly: closed-book T5 obtained 36.6% accuracy on NaturalQuestions with 11B parameters, while the authors' retrieval-augmented approach would eventually obtain 44.1% with only 770M parameters plus Wikipedia.

The Gap: Nobody Had Combined Generative Models with Large-Scale Passage Aggregation

The gap the paper identifies is not that retrieval and generation had never been combined—they had, in contemporaneous work by Min et al. (2020) and Lewis et al. (2020)—but that these combinations processed retrieved passages in a way that prevented scaling to large numbers of them. Lewis et al. (2020)'s RAG (Retrieval-Augmented Generation) and Min et al. (2020)'s SpanSeqGen both fed retrieved passages into the generative model's encoder. However, in a standard Transformer encoder, self-attention is computed over all input tokens. If you concatenate 100 passages together and feed them through the encoder, the self-attention cost grows quadratically with the total sequence length—making processing 100 passages computationally prohibitive. This meant that these retrieval-augmented generative models were effectively limited to small numbers of retrieved passages (RAG used 5 or 10), preventing them from exploiting the observation that the authors found most compelling: generative models seem to get better as you give them more passages.

This architectural constraint matters because it created a structural ceiling. Even if you believed (as the authors did) that seq2seq models were naturally good at aggregating evidence—after all, they can read multiple sources and synthesize information through the decoder's cross-attention mechanism—the standard encoder architecture prevented you from giving them enough evidence to demonstrate that capability. The quadratic self-attention cost was the bottleneck, and prior work had not found a way around it.

Why Retrieval Augmentation Matters: Efficiency and Updatability

The paper's motivation goes beyond accuracy numbers on leaderboards. There are genuine practical reasons why retrieval augmentation is preferable to closed-book generation, which the paper gestures at explicitly:

Model size vs. memory tradeoff. The closed-book T5-11B model stores knowledge in 11 billion parameters, which occupy roughly 44 GB in float32. Wikipedia, the retrieval corpus used in this work, occupies a similar order of magnitude in storage. The paper makes the pointed comparison: both methods use "roughly the same amount of memory to store information," but retrieval-based explicit memory in text form turns out to be competitive with implicit memory in weights for knowledge retrieval tasks. This suggests that at a given memory budget, storing knowledge as retrievable text is at least as effective as compressing it into parameters—and it has the additional advantage that you can update the text corpus without retraining the model.

Training cost. Training an 11B-parameter model from scratch is enormously expensive. Fine-tuning a 770M-parameter T5 model on retrieved passages is much cheaper. For deployment scenarios where the question distribution might shift (new domains, new knowledge), being able to swap out the retrieval corpus without retraining the generator is a significant practical advantage.

Interpretability and trust. A closed-book model produces an answer with no provenance—you cannot inspect why it gave that answer or which sources it relied on. A retrieval-augmented model surfaces the passages it read, enabling users to verify claims against cited sources. This matters for applications where answer trustworthiness is critical.

Contradictory Empirical Signals in the Literature

The paper is also motivated by what appears to be a contradiction in the empirical literature at the time, which the authors do not belabor but which shapes their experimental design. On one hand, extractive models showed that retrieval helps open-domain QA substantially—DPR (Karpukhin et al., 2020) achieved 41.5 EM on NaturalQuestions with an extractive reader, a major improvement over prior retrieval methods. On the other hand, the generative closed-book approach by Roberts et al. (2020) showed that you could match or exceed these numbers without any retrieval at all if your model was large enough (60.5 EM on TriviaQA with T5-11B). This created ambiguity: is retrieval actually necessary, or is it just compensating for insufficient model scale? The paper's contribution can be read as a resolution: retrieval plus generation together outperform either approach alone, and they do so at a fraction of the model size. The gap is not simply about having knowledge versus not having it—it is about having the right architecture to combine knowledge from multiple retrieved sources effectively.

The Specific Insight: Decoder-Fusion as the Key

The paper's key architectural insight emerges from diagnosing why prior approaches saturated. Extractive models saturated at 10–20 passages because they lacked a natural mechanism for synthesizing information across passages: you can only extract one span from one passage, and techniques to aggregate across passages (like global normalization) became noisy and computationally expensive as the passage count grew. Retrieval-augmented generative models (RAG, SpanSeqGen) saturated at a small number of passages because their encoder's quadratic self-attention cost made processing many passages infeasible. The paper hypothesized that a generative model—which can compose information from multiple sources into a single generated answer—should scale better with passage count than an extractive model, if the encoder's computational bottleneck could be removed. This leads directly to the Fusion-in-Decoder design: process each passage independently in the encoder (linear cost in passage count) and fuse them only in the decoder's cross-attention (where the model can learn to attend to relevant information across all passages simultaneously). This is not just an engineering optimization—it is a hypothesis about where multi-document synthesis should happen architecturally. The paper positions this as a conceptual contribution: evidence fusion belongs in the decoder, not the encoder.

Positioning Relative to Contemporaneous Work

The paper explicitly names its closest competitors: RAG (Lewis et al., 2020) and SpanSeqGen (Min et al., 2020). Both are retrieval-augmented generative models published in the months before this paper. The paper's positioning is precise:

"Our approach differs from these works by how the generative model processes the retrieved passages. This allows to scale to large numbers of documents, and to benefit from this large amount of evidence."

The distinction is architectural, not conceptual. All three works share the same high-level approach: retrieve, then generate. But RAG processes each passage independently and then marginalizes over them (treating each passage as a separate latent from which the answer could be generated), while SpanSeqGen concatenates passages in the encoder. Fusion-in-Decoder processes passages independently in the encoder but jointly in the decoder—a design that the authors argue combines the computational efficiency of RAG's per-passage processing with the evidence-combining power of cross-attention over all passages. The paper's results bear this out: Fusion-in-Decoder outperforms RAG (44.5 → 48.2 EM on NQ with base models) while using far more passages (100 vs. 5–10).

3. Technical Approach

3.1 Reader Orientation

The system being built is a two-stage open-domain question answering pipeline: a retriever fetches relevant Wikipedia passages given a natural language question, and a generative sequence-to-sequence model (T5) reads those passages and produces a free-form textual answer. The core problem it solves is that prior retrieval-augmented generative models could not effectively scale to many passages—their encoders became quadratically expensive in the number of input tokens or they aggregated evidence in ways that saturated at 5–10 documents—while extractive models flatlined at 10–20 passages because they lacked any natural mechanism for synthesising information across multiple independent text segments. The shape of the solution is an architectural decision about where multi-document fusion happens: process each passage independently in the encoder so that compute cost grows linearly with passage count, then fuse all passage representations jointly in the decoder's cross-attention so that evidence aggregation is the model's central learned behaviour.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a feed-forward pipeline:

1. Passage Corpus — a preprocessed dump of Wikipedia split into non-overlapping 100-word segments, each with an associated title. This is the external knowledge store that replaces implicit parametric memory.

2. Retriever — given a question, ranks and selects the top-$k$ most relevant passages from the corpus. The paper tests two retriever types: BM25 (sparse bag-of-words with TF/IDF weighting) and DPR (dense BERT-based embeddings with dot-product similarity, indexed via FAISS approximate nearest neighbours). For NaturalQuestions and TriviaQA, DPR is used; for SQuAD, BM25 is used. Typically $k = 100$.

3. Fusion-in-Decoder Reader — a T5-based encoder-decoder model pretrained on unsupervised text. Each of the $k$ retrieved passages is prefixed with special tokens and concatenated with the question to form $k$ independent input sequences. The encoder processes each sequence separately, producing $k$ sets of token representations. The decoder then performs cross-attention over the concatenation of all $k$ sets of encoder outputs simultaneously, and autoregressively generates the answer text through greedy decoding.

4. Answer Normalisation and Evaluation — the generated string is lowercased, stripped of articles and punctuation, and compared via exact string match against the list of acceptable ground-truth answers.

Information flow is strictly sequential: question → retriever → $k$ passage chunks → encoder (each chunk independently) → concatenation of all encoder outputs → decoder (cross-attends to all) → answer token sequence.

3.3 Roadmap for the Deep Dive

• First, the retrieval stage, because the passages define the input distribution the reader sees and the choice of retriever determines what evidence the model has access to—this grounds all subsequent design decisions.

• Second, the passage representation format, including how titles, special tokens, and truncation are handled, because this is the interface contract between retriever and reader and small format choices affect whether the model can distinguish sources.

• Third, the Fusion-in-Decoder encoder architecture, because understanding how passages are processed independently is the key to why the method scales to 100 passages while prior work could not—and why the computation is linear rather than quadratic in passage count.

• Fourth, the decoder cross-attention and evidence fusion mechanism, because this is where all passages are combined and where the model's ability to synthesize multi-document evidence lives—it is the architectural claim the paper makes.

• Fifth, the training procedure and hyperparameters, because model behaviour is a product of both architecture and training recipe, and the paper reports specific configurations that achieved the reported numbers.

• Sixth, the inference and evaluation pipeline, including decoding strategy, answer normalisation, and the computational scaling properties at test time.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architectural design paper whose core idea is that evidence from multiple retrieved passages should be combined in the decoder's cross-attention rather than in the encoder's self-attention, and that this choice enables scaling to 100 passages—a regime where both extractive models (due to their single-span prediction limitation) and prior retrieval-augmented generative models (due to quadratic encoder cost) previously failed to extract additional benefit.

The Retrieval Stage: BM25 and DPR

Before the generative model sees anything, the system must select a subset of Wikipedia passages likely to contain answer-relevant information. The paper uses two retrieval methods, chosen per-dataset based on which performs better empirically (following Karpukhin et al., 2020): DPR for NaturalQuestions and TriviaQA, BM25 for SQuAD.

BM25 (sparse retrieval). BM25 (Robertson et al., 1995) treats each passage and the question as bags of words. The relevance score of a passage $d$ to a question $q$ is:

$\text{BM25}(q, d) = \sum_{w \in q} \text{IDF}(w) \cdot \frac{f(w, d) \cdot (k_1 + 1)}{f(w, d) + k_1 \cdot \left(1 - b + b \cdot \frac{|d|}{\text{avgdl}}\right)}$

where $w$ iterates over each term in the query, $\text{IDF}(w)$ is the inverse document frequency of term $w$, $f(w, d)$ is the term frequency of $w$ in passage $d$, $|d|$ is the passage length in words, $\text{avgdl}$ is the average passage length across the corpus, and $k_1$ and $b$ are tunable saturation and length-normalisation parameters.

What it computes: for each query term, BM25 adds a contribution proportional to how rare that term is in the corpus ($\text{IDF}(w)$) and how often it appears in the candidate passage, damped by a saturation function that prevents a term repeated many times from dominating. The length normalisation term $b \cdot |d|/\text{avgdl}$ penalises unusually long passages because they are more likely to contain query terms by chance. The output is a scalar relevance score per passage; passages are ranked and the top $k$ are selected.

Why this form: BM25 builds on TF/IDF by adding two critical corrections that pure TF/IDF lacks. The $k_1$ parameter (default 1.2–2.0) controls term frequency saturation: without it, a word appearing 100 times would be scored 100× higher than appearing once, which overweights long passages and repetitions. The $b$ parameter (default 0.75) controls length normalisation: without it, longer passages have an unfair advantage because they contain more terms overall. These corrections are derived from a probabilistic relevance model and are widely considered the "default" strong sparse retrieval baseline. The paper uses the Apache Lucene implementation with default parameters and SpaCy for tokenisation.

DPR (dense retrieval). DPR (Karpukhin et al., 2020) represents both questions and passages as dense fixed-dimensional vectors using two separate BERT networks:

$s(q, p) = \text{EQ}(q)^T \cdot \text{EP}(p)$

where $\text{EQ}(\cdot)$ is the question encoder BERT that produces a vector $\mathbf{v}_q \in \mathbb{R}^d$, $\text{EP}(\cdot)$ is the passage encoder BERT that produces a vector $\mathbf{v}_p \in \mathbb{R}^d$, and $s(q, p)$ is the dot-product relevance score.

What it computes: the question encoder takes the question text (prepended with a [CLS] token in standard BERT fashion), passes it through 12 or 24 Transformer layers, and outputs the [CLS] embedding as a $d$-dimensional dense vector (typically $d = 768$ for BERT-base). The passage encoder does the same for the passage text (title concatenated with body). Their dot product produces a scalar that should be large when the passage contains information relevant to answering the question. At inference time, all passage embeddings are precomputed and indexed using FAISS for approximate nearest neighbour search with inner product as the similarity metric, achieving sub-linear retrieval over millions of passages.

Why this form: dense retrieval captures semantic similarity rather than lexical overlap. A passage that says "Alan Turing was born in London" and a question "Where did Turing enter the world?" share essentially zero lexical overlap but dense embeddings (trained to maximise the dot product for passages containing answers to questions) can align their representations. The dual-encoder architecture allows precomputing passage embeddings offline (since passages don't change per query), making retrieval latency dominated by the question encoding time plus the FAISS search time—typically milliseconds. The paper follows the DPR training recipe exactly, including the use of hard negative mining and in-batch negatives during training. Critically, $\text{EQ}$ and $\text{EP}$ are independent networks, not weight-shared, because questions and passages have fundamentally different linguistic distributions (questions are interrogative and short; passages are declarative and long).

Retrieval count. For all main experiments, $k = 100$ passages are retrieved during both training and evaluation, unless otherwise noted in the passage-count ablation. This number is deliberately high relative to prior work (which used 5–20) to test the hypothesis that generative models scale with passage count when given the right architecture.

Passage Formatting: The Interface Between Retriever and Reader

Once the top $k$ passages are retrieved, each must be converted into a format the T5 encoder can consume. The paper uses a specific template with special tokens and structural elements that serve as an interface contract:

question: {question text} title: {passage title} context: {passage body}

Each passage—including its title and the question text—forms a single input sequence that is processed independently by the encoder. The special tokens question:, title:, and context: are not pretrained semantic tokens; they are simple string prefixes that the model learns to interpret during fine-tuning as delimiters marking different information types.

Design rationale for including the question in every passage input. The question is repeated verbatim in every one of the $k$ input sequences, which might seem redundant. This is necessary because the encoder processes each passage independently—it has no access to other passages or to a shared question representation. The encoder self-attention within each passage sequence needs to be able to relate passage tokens to question tokens (e.g., to identify which parts of the passage are relevant to the question). By prepending the question to each passage, the encoder can build question-conditioned representations of every passage without requiring cross-passage communication.

Design rationale for including titles. Wikipedia article titles provide a compact form of context that helps the model disambiguate and ground the passage content. A passage about "Turing" could come from "Alan Turing" (the computer scientist), "Turing test," or "Turing machine"—the title resolves this ambiguity. The title is treated as part of the passage text, not as a separate special field.

Truncation. Each passage-plus-title-plus-question sequence is truncated to 250 word pieces (T5 uses a SentencePiece tokeniser). This means that for very long passages, only the first approximately 250 subword tokens are retained. The choice of 250 word pieces for 100-word passages means there is generous headroom: a 100-word passage plus its title and the question text will almost never exceed 250 word pieces, so truncation is non-aggressive and primarily serves as a safety mechanism for edge cases.

Encoder Architecture: Independent Per-Passage Processing

This is the component where the paper's key engineering insight—and architectural claim—resides. The encoder is a standard T5 Transformer encoder (a stack of bidirectional self-attention layers), but the critical design choice is how the $k$ passages are fed to it.

Per-passage batching. Instead of concatenating all $k$ passages into a single long sequence and feeding it to the encoder (as prior work like SpanSeqGen did), each of the $k$ formatted passage-question pairs is processed as an entirely independent forward pass through the encoder. The encoder sees one passage at a time, applies self-attention only among the tokens within that single passage's sequence, and produces a matrix of token representations $\mathbf{H}^{(i)} \in \mathbb{R}^{L_i \times d_{\text{model}}}$ where $L_i$ is the tokenised length of the $i$-th passage (including question and title) and $d_{\text{model}}$ is the T5 hidden dimension (768 for base, 1024 for large).

Computational cost. The self-attention cost for a sequence of length $L$ is $\mathcal{O}(L^2)$. If all $k$ passages were concatenated into a single sequence of length $\sum_{i=1}^k L_i \approx k \cdot \bar{L}$, the encoder cost would be $\mathcal{O}((k\bar{L})^2) = \mathcal{O}(k^2\bar{L}^2)$—quadratic in the number of passages. This is why prior retrieval-augmented generative models were limited to small $k$ (RAG used 5–10 passages). By processing each passage independently, the encoder cost becomes $\mathcal{O}(k \cdot \bar{L}^2)$—linear in the number of passages. At $k = 100$, this is the difference between a factor of 100 (linear) and a factor of 10,000 (quadratic) in the passage-count-dependent term.

What the encoder produces. After processing all $k$ passages independently, the system has $k$ separate output tensors $\mathbf{H}^{(1)}, \mathbf{H}^{(2)}, \ldots, \mathbf{H}^{(k)}$, one per passage. Each $\mathbf{H}^{(i)}$ is the full sequence of hidden states for that passage after the final encoder layer—one $d_{\text{model}}$-dimensional vector per input token. These $k$ tensors are then concatenated along the sequence-length dimension to form a single large tensor:

$\mathbf{H}^{\text{all}} = \text{concat}\left(\mathbf{H}^{(1)}, \mathbf{H}^{(2)}, \ldots, \mathbf{H}^{(k)}\right) \in \mathbb{R}^{\left(\sum_{i=1}^k L_i\right) \times d_{\text{model}}}$

What information is and is not shared across passages in the encoder. Because each passage is processed independently, there is no self-attention interaction between tokens from different passages during encoding. The representation of a word in passage 3 is uninfluenced by the content of passage 7. This means the encoder cannot perform cross-passage coreference resolution, cannot compare conflicting claims across passages, and cannot identify that two passages are discussing the same entity. All of these cross-passage interactions are deferred to the decoder. This is simultaneously the architectural limitation of the approach and its computational efficiency enabler—the paper's bet is that cross-passage reasoning is better done in the decoder anyway.

Special token embeddings. The tokens question:, title:, and context: are segmented by the T5 tokeniser into whatever subword units the SentencePiece model produces. There are no special additional embeddings or dedicated vocabulary entries for these delimiters—they are simply natural language strings that the model learns to interpret as section boundaries through fine-tuning. This means the model does not have an architectural inductive bias about where question ends and title begins; it must learn this from the data distribution.

Decoder Architecture and Evidence Fusion

The decoder is a standard T5 autoregressive Transformer decoder with one crucial modification: its cross-attention mechanism attends over $\mathbf{H}^{\text{all}}$, the concatenated encoder outputs from all $k$ passages simultaneously, rather than over a single encoder output sequence.

Decoder autoregressive generation. At each generation step $t$, the decoder produces a hidden state $\mathbf{s}_t \in \mathbb{R}^{d_{\text{model}}}$ based on the previously generated tokens $y_{<t}$ (via causal self-attention) and then computes cross-attention over $\mathbf{H}^{\text{all}}$:

$\alpha_{t,j} = \text{softmax}\left(\frac{\mathbf{s}_t \mathbf{W}_Q \cdot (\mathbf{h}_j^{\text{all}} \mathbf{W}_K)^T}{\sqrt{d_k}}\right)$

$\mathbf{c}_t = \sum_{j=1}^{\sum L_i} \alpha_{t,j} \cdot (\mathbf{h}_j^{\text{all}} \mathbf{W}_V)$

where $\mathbf{h}_j^{\text{all}}$ is the $j$-th token representation in the concatenated encoder output, $\mathbf{W}_Q$, $\mathbf{W}_K$, $\mathbf{W}_V$ are learned projection matrices (standard multi-head attention), $d_k$ is the per-head dimension, $\alpha_{t,j}$ is the attention weight assigned by the decoder to the $j$-th encoder token at generation step $t$, and $\mathbf{c}_t$ is the context vector summarising the attended encoder information.

What it computes: at each generation step, the decoder looks across all tokens from all 100 passages and computes a weighted average of their representations, where the weights are determined by how relevant each token's representation is to what the decoder has generated so far. The context vector $\mathbf{c}_t$ is then concatenated with the decoder's self-attention output and fed through feed-forward layers to produce the next-token logits. This means the decoder can, when generating the word "London," attend simultaneously to a token in passage 23 that says "Maida Vale" and a token in passage 67 that says "the capital of England" and a token in passage 4 that says "Alan Turing was born in"—synthesising information from three independent documents in a single generation step.

Why this architecture enables evidence fusion. The decoder's cross-attention is the only mechanism through which information from different passages can interact. Because the cross-attention weights are computed jointly over all passages (the softmax in the attention formula normalises over all encoder positions across all passages), the model can learn to compare, contrast, and combine evidence. For example, if passage 5 claims Turing was born in 1912 and passage 8 claims 1911, the decoder can learn to attend to both, detect the conflict, and either generate the majority answer or produce the correct one based on other contextual cues. This is fundamentally different from RAG, which marginalises over passages by generating a separate answer distribution per passage and then combining them—RAG cannot compare two passages' content during generation because each passage is processed in a separate decoder forward pass.

Cross-attention cost. The cross-attention cost per decoder layer is $\mathcal{O}(L_{\text{target}} \cdot \sum L_i)$ where $L_{\text{target}}$ is the target answer length and $\sum L_i$ is the total number of encoder tokens across all passages. With 100 passages at 250 word pieces each, $\sum L_i$ is approximately 25,000 tokens. This is large but computationally feasible for a single cross-attention operation (which is $\mathcal{O}(n)$ in the source length, not $\mathcal{O}(n^2)$ like self-attention). The paper does not explicitly state the wall-clock time for a 100-passage forward pass, but the linear scaling in passage count makes it tractable.

Greedy decoding. At inference, the decoder generates tokens one at a time by selecting the token with the highest predicted probability (argmax) at each step—pure greedy decoding without beam search, temperature scaling, or nucleus sampling. This is a deliberate simplicity choice: greedy decoding is deterministic, fast, and the authors found it sufficient for strong results. The generation continues until an end-of-sequence token is produced or a maximum length is reached.

Training Procedure and Hyperparameters

The generative reader is not trained from scratch; it is initialised from the publicly available T5 pretrained checkpoints and fine-tuned on question-answer pairs from each dataset independently.

Model initialisation. Two sizes of T5 are used, both from the HuggingFace Transformers library:

• T5-base: approximately 220 million parameters. Encoder has 12 layers, decoder has 12 layers, hidden dimension $d_{\text{model}} = 768$, feed-forward dimension 3072, 12 attention heads.
• T5-large: approximately 770 million parameters. Encoder has 24 layers, decoder has 24 layers, hidden dimension $d_{\text{model}} = 1024$, feed-forward dimension 4096, 16 attention heads.

Both models were pretrained by Raffel et al. (2019) on the C4 corpus using a denoising objective (span corruption), giving them strong general-purpose text generation capabilities before any QA-specific fine-tuning.

Training objective. The model is trained with standard teacher-forced maximum likelihood estimation. Given a question, a set of retrieved passages, and a ground-truth answer $y^* = (y^*_1, \ldots, y^*_T)$, the loss for a single example is:

$\mathcal{L} = -\sum_{t=1}^{T} \log P(y^*_t \mid y^*_{<t}, q, p_1, \ldots, p_k)$

where $P(y^*_t \mid y^*_{<t}, q, p_1, \ldots, p_k)$ is the probability the model assigns to the correct token $y^*_t$ at position $t$, conditioned on the question $q$, all $k$ retrieved passages $p_1, \ldots, p_k$, and the preceding ground-truth answer tokens $y^*_{<t}$.

What it computes: the standard cross-entropy loss for sequence-to-sequence models. At each position in the answer, the model outputs a probability distribution over the entire vocabulary; the loss is the negative log probability it assigns to the actual correct token at that position. Summing over all positions gives the total loss for the example, which is averaged over a minibatch before backpropagation.

Why this form: teacher forcing trains the model to produce the correct answer token by token, using the ground-truth previous tokens as context. This provides a strong, stable learning signal because the model always conditions on correct history during training, avoiding the error-compounding problem that would arise if the model conditioned on its own (potentially incorrect) previously generated tokens. The cross-entropy objective is the standard maximum-likelihood approach for text generation and is well-calibrated for the token-level prediction task.

Answer target handling. For datasets where multiple answer strings are considered correct for the same question, the paper uses different strategies:

• NaturalQuestions and SQuAD: during training, a single target answer is randomly sampled from the list of acceptable answers for each question. This acts as a form of data augmentation and prevents the model from overfitting to one specific phrasing.
• TriviaQA: the unique human-generated answer is used as the target, since TriviaQA provides only one answer per question (the dataset was constructed from trivia questions that have unambiguous answers).

TriviaQA answer normalisation during training. TriviaQA answers that appear in all-uppercase are normalised by converting to title case (lowercase except first letter of each word) using Python's .title() string method. This is a preprocessing step applied to the training targets to make them more consistent with typical generated text formatting.

Optimiser and schedule. The model is trained with Adam (Kingma and Ba, 2014) using:

• Constant learning rate: $10^{-4}$ (0.0001), with no learning rate decay, warmup, or scheduling. This is an unusual choice—most Transformer fine-tuning uses a linear warmup followed by linear decay—but the paper reports it was effective.
• Dropout rate: 10% (applied to attention weights and feed-forward activations in the standard T5 dropout configuration).
• Batch size: 64 (meaning 64 question-passage-answer triples are processed in parallel per gradient step; each triple includes up to 100 passages, making the effective per-GPU memory load substantial).

Training duration and checkpoint selection. The model is trained for 10,000 gradient steps. Evaluation is performed on the validation set every 500 steps, and the checkpoint with the highest validation Exact Match score is selected as the final model. This is the standard best-on-validation protocol, but it is worth noting that training continues well beyond the point where most metrics plateau (10,000 steps at batch size 64 means processing 640,000 examples; for NQ with ~80k training examples, this is roughly 8 epochs through the training data, which is high for fine-tuning and suggests the model benefits from repeated passes over the data with high dropout).

Hardware and training time. Training uses 64 Tesla V100 32GB GPUs. The paper reports that training on 100 passages for NaturalQuestions takes approximately 425 GPU-hours (Section 5, Table 2 discussion). This is expensive but not prohibitive for an academic research lab. The training cost is directly proportional to the number of passages because each passage requires a separate encoder forward pass.

A note on what is NOT tuned. The paper does not mention hyperparameter sweeps over learning rate, batch size, dropout, or training steps for the main results. The stated hyperparameters appear to be a single configuration applied uniformly across all three datasets and both model sizes. This suggests either that the authors found a robust default through preliminary experimentation or that they simply adopted reasonable values from prior work without exhaustive tuning—the methodology section does not specify.

Training with Fewer Passages: The Finetuning Strategy

A practical challenge with training on 100 passages is the computational cost—425 GPU-hours for NaturalQuestions. The paper explores whether models can be trained on fewer passages and then evaluated on 100 passages, which would reduce training cost significantly.

Direct evaluation with mismatched passage counts. When a model trained on $k_{\text{train}}$ passages is evaluated on $k_{\text{test}} = 100$ passages, the encoder encounters a different total number of input sequences than it saw during training. This is an out-of-distribution condition: the decoder's cross-attention must handle 100 passages' worth of encoder outputs when it was only trained to attend over $k_{\text{train}}$ passages' worth. The results in Table 2 show significant degradation: a model trained on 5 passages and tested on 100 achieves 37.8 EM on NQ, compared to 46.5 EM when trained on 100 passages.

The two-phase finetuning strategy. To mitigate this while reducing total compute, the paper proposes a hybrid approach:

1. Phase 1: Train the model from T5 pretrained weights using $k_{\text{train}}$ passages for the full 10,000 steps (or most of them). This phase accounts for the majority of the training cost.
2. Phase 2: Continue training (finetune) the model from the Phase 1 checkpoint using $k = 100$ passages for only 1,000 additional steps.

Results of the finetuning strategy (Table 2). The strategy is effective:

• A model trained on 5 passages achieves 37.8 EM on NQ when evaluated on 100 passages without further training.
• After 1,000 steps of finetuning with 100 passages, the same model reaches 45.0 EM—a gain of 7.2 points, nearly closing the gap to the fully-100-passage-trained model's 46.5 EM.
• The total GPU-hours for this approach (5 passages for ~9,000 steps + 100 passages for 1,000 steps) is approximately 147 GPU-hours, compared to 425 GPU-hours for full training—a 65% reduction in compute cost with only a 1.5 EM point penalty.

Why the finetuning strategy works. The encoder architecture already processes each passage independently, so the encoder's per-passage computation is identical whether there are 5 or 100 passages. The only architectural point where passage count matters is the decoder's cross-attention: the concatenated encoder output matrix $\mathbf{H}^{\text{all}}$ grows in the sequence-length dimension, meaning the cross-attention softmax is normalised over more positions. The two-phase finetuning allows the decoder's cross-attention weights to adapt to this larger normalisation set while the encoder weights (which are already well-trained from Phase 1) remain largely stable. In essence, the model learns to attend over a larger candidate set in a relatively small number of gradient steps because the encoder representations it is attending to are already high-quality.

Design choice: why not always train on 5 passages and finetune? The fully-100-passage model still outperforms the finetuned model (46.5 vs. 46.0 for NQ), so there is an accuracy-compute tradeoff. The paper presents the finetuning strategy as a practical option for resource-constrained settings, not as a replacement for full training.

Inference and Evaluation Pipeline

Retrieval at inference. For test-time evaluation, the same retrieval pipeline is used as during training: DPR for NaturalQuestions and TriviaQA, BM25 for SQuAD. 100 passages are retrieved per question. The passage embeddings for DPR are precomputed and stored in a FAISS index, so retrieval latency is dominated by encoding the question (one BERT forward pass) plus the FAISS search time.

Decoding strategy. Answers are generated using greedy decoding—at each step, the token with the highest predicted probability is selected, and this token is fed back as input for the next step. No beam search, no nucleus sampling, no temperature. The paper does not justify this choice, but it is consistent with the T5 fine-tuning paradigm from Raffel et al. (2019), which also used greedy decoding for most tasks. The implication is that the model's probability distribution under the maximum-likelihood training is sharp enough that the argmax is reliable, and that the additional cost of beam search (which would multiply inference cost by the beam width) is not worth the potential accuracy gain.

Answer normalisation. Once the model generates a textual answer string, it is normalised before comparison with ground truth. The normalisation pipeline:

1. Lowercase the generated string.
2. Remove articles ("a", "an", "the").
3. Remove punctuation marks.
4. Remove duplicated whitespace (collapse multiple spaces into one).

The normalised string is then compared via exact string match against each entry in the list of acceptable ground-truth answers. If the normalised generated string matches any acceptable answer, the example is counted as correct. This is the standard Exact Match (EM) metric introduced by Rajpurkar et al. (2016) and used throughout the open-domain QA literature.

Why EM rather than F1. For open-domain QA, answers are typically short entities or phrases (person names, dates, locations), so token-level F1 overlap is less informative than for reading comprehension where answers are longer spans. EM provides a strict, interpretable metric: the model got the answer exactly right or it did not.

Computational cost at inference. For each question, inference requires:

• One DPR question encoding forward pass (milliseconds).
• FAISS nearest-neighbour search over millions of embeddings (milliseconds).
• 100 independent T5 encoder forward passes, one per passage (each processing ~250 word pieces).
• One T5 decoder forward pass with cross-attention over all 100 encoder outputs.

The encoder passes dominate the cost but are trivially parallelisable across GPUs—each passage can be processed on a separate device with no communication until the encoder outputs are gathered for the decoder cross-attention. The paper does not report inference latency numbers, but the linear scaling in passage count means that 100-passage inference is approximately 10× more expensive than 10-passage inference, which is the regime prior work operated in.

Summary of Design Choices and Their Justifications

• Independent per-passage encoder processing over concatenation: enables linear rather than quadratic scaling in passage count, removing the architectural ceiling that prevented prior generative models from using more than ~10 passages. The bet is that cross-passage interactions are better handled in the decoder anyway, where the model has a task-specific reason to combine evidence (generating the answer) rather than a generic encoding objective.

• Decoder cross-attention over all passages simultaneously over per-passage decoder processing (RAG-style): allows the model to attend to evidence from passage 47 and passage 3 in the same generation step, enabling compositional answer synthesis that marginalisation methods cannot express. The cross-attention softmax over all encoder positions creates a natural competition among passages for the decoder's attention budget.

• 100-passage retrieval with DPR over smaller k: pushes the architectural capability to its limit to demonstrate that generative models scale with passage count, and that the saturation at 10–20 passages observed in extractive models is an architectural limitation of extraction, not a fundamental ceiling on how much a reader can benefit from additional evidence.

• Greedy decoding over beam search: simplicity and speed; sufficient because answers are short entity strings where the model's probability distribution is peaked.

• Constant learning rate over scheduled decay: empirically effective; the paper reports this without ablation, making it a black-box choice that users replicating the method should verify on their own data.

• Two-phase finetuning for passage-count scaling over full retraining: a practical compute-saving measure that recovers most of the accuracy of full 100-passage training at ~35% of the GPU cost. The finetuning phase is short (1000 steps) and demonstrably effective across datasets.

• Special prefix tokens (question:, title:, context:) as plain text over dedicated embeddings: avoids expanding the vocabulary or modifying the pretrained model architecture, making the approach a drop-in fine-tuning recipe on top of off-the-shelf T5 checkpoints. The model learns to interpret these as delimiters through gradient-based training.

4. Key Insights and Innovations

Innovation 1: Evidence Fusion Belongs in the Decoder, Not the Encoder

The paper's most intellectually distinctive contribution is not the retrieval-augmented generation pipeline itself—that was already established by RAG (Lewis et al., 2020) and SpanSeqGen (Min et al., 2020)—but rather a specific architectural hypothesis about where multi-document synthesis should occur in a Transformer: in the decoder's cross-attention, not the encoder's self-attention. This is a claim about the functional specialisation of encoder and decoder layers that had not been articulated in prior work on open-domain QA, and it matters because it directly determines how a system scales with the number of retrieved documents.

The dominant assumption in prior retrieval-augmented generative models was that passages should be combined before generation, in the encoder. SpanSeqGen concatenated all retrieved passages into a single input sequence fed to the encoder; RAG processed each passage through the encoder separately but then marginalised over per-passage answer distributions in the decoder, never allowing the decoder to attend to two passages simultaneously. Both approaches implicitly assumed that cross-passage interaction was the encoder's job, and the decoder's role was to consume already-combined representations. Fusion-in-Decoder inverts this assumption: the encoder is treated as a per-document feature extractor with no cross-document communication, and all evidence combination is deferred to the decoder, where cross-attention weights over the concatenated encoder outputs can dynamically select and synthesise information from any passage at each generation step.

This inversion has a concrete consequence that the paper demonstrates but also argues as a conceptual principle: the decoder is a more natural location for evidence fusion because it conditions on the task-specific objective (generating the answer) when deciding which evidence to attend to. The encoder's representations are task-agnostic by design—they must capture everything about a passage that could be relevant. The decoder's cross-attention is task-driven: when generating the token "1912," it can look across all 100 passages and pull only the birth-year information, ignoring irrelevant details. The paper's results in Figure 3 provide empirical backing for this principle: the model's accuracy keeps improving as passages increase from 10 to 100, a scaling behaviour that extractive models and prior generative models never exhibited. This is not just a quantitative improvement—it is evidence for the architectural hypothesis that decoder-side fusion avoids the saturation bottleneck that encoder-side aggregation creates.

The distinction from RAG is particularly instructive. RAG's marginalisation approach can be seen as doing evidence combination in probability space (averaging per-passage answer distributions) rather than in representation space (allowing simultaneous attention). The probability-space approach has a fundamental limitation: it cannot synthesise an answer that requires information from two different passages, because each passage's answer distribution is computed independently. A question like "Which British computer scientist born in London cracked the Enigma code?" requires combining evidence from a passage about Turing's birthplace and a passage about his codebreaking work—evidence that may appear in entirely separate Wikipedia articles. RAG cannot combine these; Fusion-in-Decoder can, because the decoder can simultaneously attend to both passages when generating "Alan Turing." The architectural choice is therefore not just a computational convenience—it enables a qualitatively different kind of multi-document reasoning.

Innovation 2: Linear Encoder Scaling as an Enabler, Not Just an Optimisation

The paper reframes what might appear to be an engineering trick—processing each passage independently in the encoder to achieve linear rather than quadratic scaling in passage count—as a capability enabler that unlocks a previously inaccessible regime of evidence quantity. Prior work treated the quadratic encoder cost as an unfortunate but accepted constraint, and the field had converged on using 5–20 passages because that was what the architecture could afford. The paper argues that this constraint was actively preventing the discovery that generative models scale gracefully with far more passages than anyone had tried.

This is a methodological contribution about experimental design as much as an architectural one. By removing the computational ceiling, the paper reveals that the saturation at 10–20 passages seen in extractive models (Wang et al., 2019; Yang et al., 2019) was an artifact of extractive architectures—not a fundamental limit on how much a reader can benefit from additional retrieved evidence. The 6 EM point gain on TriviaQA when going from 10 to 100 passages (Figure 3) would have been invisible to prior systems because they could not physically process that many passages. The paper is essentially arguing that the field had misattributed a property of its tools (extractive models saturate at ~20 passages) to a property of the task (reading more than 20 passages doesn't help). Fusion-in-Decoder serves as an existence proof that the task property is different from what the field believed.

The significance of this reframing extends beyond open-domain QA. It suggests a general principle for retrieval-augmented generation: the optimal number of retrieved passages is not known a priori and may be much larger than what is computationally convenient under standard architectures. Systems should be designed to scale the retriever count upward until they empirically saturate, rather than picking a small number based on encoder cost. This principle has since become influential in the design of retrieval-augmented language models, and the paper's demonstration that 100 passages are useful (and that the curve is still rising at 100 for some datasets) shifted the default retrieval count upward in subsequent work.

Innovation 3: A Diagnostic Finding That Generative Models Are Better Multi-Document Aggregators Than Extractive Ones

The paper does not merely claim that Fusion-in-Decoder achieves state-of-the-art results; it uses those results to establish a comparative diagnostic: generative sequence-to-sequence models are inherently better at aggregating evidence from multiple documents than extractive span-prediction models, and this difference becomes visible only when the number of documents is large enough. This is a claim about model class rather than about a specific architecture, and it has implications for how practitioners should choose between extractive and generative readers.

The diagnostic logic works as follows. Extractive models (BERT-based span predictors) and generative models (T5) both achieve competitive accuracy on open-domain QA when given a small number of passages (5–10). But as the number of passages grows, their trajectories diverge: extractive models peak around 10–20 and then degrade, while the generative Fusion-in-Decoder model continues to improve to at least 100 passages (Figure 3). The divergence is what makes the diagnostic convincing—if the generative model also saturated at 20 passages, you could attribute the saturation to the data or the task, not the architecture. The fact that the curves diverge isolates the architecture as the causal factor.

Why would generative models be better aggregators? The paper does not provide a mechanistic analysis (e.g., attention visualisations), but the implied reasoning is structural: extractive models must select a single span from a single passage as the answer, and techniques for combining evidence across passages (global normalisation, voting, confidence-weighted re-ranking) are post-hoc additions that do not allow the model to compose information during answer formation. A generative model can produce an answer that is not a substring of any single passage—it can combine "Alan Turing" from passage 3 with "cryptanalyst" from passage 27 into "Alan Turing, a cryptanalyst"—which is a more expressive form of aggregation. The Fusion-in-Decoder architecture amplifies this capability by allowing simultaneous attention over all passages during generation.

This diagnostic insight helps explain a pattern in the literature that was previously puzzling: why retrieval-augmented extractive models (DPR + BERT reader) and closed-book generative models (T5-11B) could both achieve similar numbers on leaderboards despite radically different approaches. The extractive models were saturating on the evidence they could digest; the generative models were compensating for having no evidence at all through massive parametric memory. Fusion-in-Decoder shows that giving a generative model access to evidence—and enough of it—produces a qualitatively different scaling curve that leaves both prior paradigms behind. The insight is that generation + retrieval is not just additive but synergistic in a way that extraction + retrieval is not.

Innovation 4: Training-Time Passage Count as a Learnable Hyperparameter

A subtle but practically significant finding is that the number of passages used during training can be treated as a hyperparameter that can be decoupled from the inference-time passage count through a short finetuning phase—and that this provides a compute-accuracy tradeoff knob that prior work had not identified or exploited. This is not an architectural innovation but a training methodology innovation with concrete resource implications.

The standard assumption in retrieval-augmented QA would be that the model must be trained with the same number of passages it will see at test time—otherwise the decoder's cross-attention would face a distributional shift in the number of encoder positions it attends over. The paper tests this assumption and finds it is partially true (direct evaluation with mismatched counts degrades performance, as shown in Table 2) but easily correctable: a model trained on 5 passages can be adapted to 100 passages in only 1,000 additional training steps, recovering most of the accuracy gap while using roughly a third of the total GPU-hours.

The finding matters because it changes the economics of experimenting with retrieval-augmented models at scale. Full training on 100 passages requires 425 GPU-hours for NaturalQuestions; training on 5 passages requires substantially less. If researchers can iterate quickly on 5-passage training and only commit to the expensive 100-passage finetuning when they have a promising configuration, the development cycle is much faster. The paper does not frame this as a methodological contribution in its own right—it appears in a brief subsection of the experiments—but it has outsized practical significance for anyone reproducing or extending the work, and the two-phase strategy has since become a common pattern in the retrieval-augmented generation literature.

The deeper conceptual point is that the encoder representations learned on small passage sets generalise well to large passage sets because the encoder processes each passage independently. The per-passage encoding function does not need to change when more passages are added; only the decoder's attention distribution needs to adapt to the larger normalisation set. This is a specific consequence of the Fusion-in-Decoder architecture that would not hold for encoder-concatenation approaches, where increasing passage count fundamentally changes the encoder's self-attention pattern. The finetuning strategy works because of the architectural choice, not despite it—which makes the finding a validation of the architectural hypothesis rather than an orthogonal trick.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses three standard open-domain QA benchmarks: NaturalQuestions (Kwiatkowski et al., 2019), consisting of real Google search queries with short answers, where the open-domain version discards answers longer than 5 tokens; TriviaQA (Joshi et al., 2017), containing trivia and quiz-league questions with human-generated answers, using the unfiltered version; and SQuAD v1.1 (Rajpurkar et al., 2016), a reading comprehension dataset repurposed for open-domain evaluation by discarding the provided paragraphs and requiring the system to retrieve from Wikipedia. Training sets consist of approximately 80,000, 70,000, and 80,000 examples respectively, with the SQuAD validation set used as the test set following Lee et al. (2019). For validation during training, 10% of the training set is held out for each dataset. The Wikipedia dumps are from Dec. 20, 2018 for NQ and TriviaQA, and from Dec. 21, 2016 for SQuAD, preprocessed into non-overlapping 100-word passages following Chen et al. (2017) and Karpukhin et al. (2020).

• Base model(s). The reader is initialised from pretrained T5 (Raffel et al., 2019) in two sizes: T5-base (220M parameters, 12 encoder layers, 12 decoder layers, $d_{\text{model}} = 768$) and T5-large (770M parameters, 24 encoder layers, 24 decoder layers, $d_{\text{model}} = 1024$). Both are fine-tuned on each dataset independently. T5 was chosen because its encoder-decoder architecture provides a natural separation between encoding retrieved passages (encoder) and generating answers conditioned on them (decoder), which the Fusion-in-Decoder approach exploits by processing passages independently in the encoder and fusing them in the decoder's cross-attention. The paper also implicitly compares against T5-11B from Roberts et al. (2020) for the closed-book vs. retrieval-augmented contrast.

• Metrics. The primary metric is Exact Match (EM), following Rajpurkar et al. (2016): a generated answer is considered correct if its normalised string exactly matches any entry in the list of acceptable ground-truth answers. Normalisation consists of lowercasing, removing articles ("a", "an", "the"), removing punctuation, and collapsing duplicated whitespace. Unlike reading comprehension where F1 is commonly reported alongside EM, only EM is used here because open-domain QA answers are typically short entity strings for which token-level overlap provides limited additional information. For SQuAD, the paper also reports F1 in Table 1 alongside EM, presumably for comparability with prior work that reported both, but EM is the primary metric throughout the remaining analyses.

• Baselines. The paper compares against a diverse set of prior systems, organised by approach type. Extractive baselines: DrQA (Chen et al., 2017), Multi-Passage BERT (Wang et al., 2019), Path Retriever (Asai et al., 2020), Graph Retriever (Min et al., 2019b), Hard EM (Min et al., 2019a), ORQA (Lee et al., 2019), REALM (Guu et al., 2020), and DPR (Karpukhin et al., 2020). Generative baselines: SpanSeqGen (Min et al., 2020), RAG (Lewis et al., 2020), closed-book T5 (Roberts et al., 2020), and GPT-3 few-shot (Brown et al., 2020). Where multiple model sizes or configurations are reported by prior work, the best published number is given in Table 1. RAG is the most directly comparable baseline because it shares the retrieval-then-generate paradigm and uses T5-based generation; the architectural differences in how passages are processed is the paper's primary distinguishing claim.

• Generation budget / compute accounting. The paper does not use a formal compute budget for comparison across methods in the FLOPs-matched sense, because it is not making a training-inference tradeoff argument. Instead, compute is discussed in terms of GPU-hours for training and number of passages for inference scaling. Training cost is reported as 425 GPU-hours for full 100-passage training on NQ and 147 GPU-hours for the two-phase finetuning approach (Section 5, Table 2 discussion). Inference cost scales linearly with passage count due to independent encoder processing, and the paper uses 100 passages for all main results unless explicitly varying the count. The computational efficiency claim—that independent encoder processing is linear rather than quadratic in passage count—is argued architecturally rather than measured with wall-clock benchmarks.

• Cross-validation / statistical protocol. The paper uses standard train-validation-test splits provided by the datasets (with SQuAD's validation set serving as test, following convention). Model selection is performed by evaluating on the held-out validation set every 500 training steps and selecting the checkpoint with the highest validation EM score. No k-fold cross-validation, bootstrap confidence intervals, or statistical significance testing is reported. The test set results in Table 1 are single-point estimates without error bars. For the passage-count ablation (Figure 3), results are reported on the validation set rather than the test set, which is standard for diagnostic experiments. The hidden test set results for TriviaQA (Table 1 right column) are obtained by submitting to the public competition leaderboard, providing an independent evaluation not subject to overfitting on the publicly available validation data.

Main Quantitative Results

State-of-the-Art Comparison (Table 1)

The central quantitative claim is that Fusion-in-Decoder sets new state-of-the-art on NaturalQuestions and TriviaQA. The numbers from Table 1:

On NaturalQuestions, the base model (220M parameters) already surpasses all prior systems at 48.2 EM; the large model (770M) extends this to 51.4, representing a substantial margin over the previous best of 44.5 (RAG). On TriviaQA's open test set, the base model achieves 65.0 and large reaches 67.6, compared to DPR's 57.9. On the TriviaQA hidden test set (evaluated through the competition leaderboard), the base model's 77.1 surpasses all published results, and large reaches 80.1—a notably high score that the paper highlights as state-of-the-art. The one comparative weakness is on SQuAD Open, where Fusion-in-Decoder's base model (53.4 EM) trails Path Retriever (56.5 EM) and the large model (56.7) is only marginally ahead, suggesting the architectural advantages may be less pronounced for this dataset.

Comparing against generative baselines specifically: Fusion-in-Decoder base outperforms RAG by 3.7 EM on NQ (48.2 vs. 44.5) and outperforms closed-book T5-11B by 11.6 EM (48.2 vs. 36.6), despite the T5-11B model having roughly 50× more parameters. The large model achieves 51.4 on NQ versus GPT-3 few-shot's 29.9. These gaps are large and consistent: retrieval augmentation substantially outperforms closed-book generation at comparable or larger model sizes.

Scaling Performance with Number of Retrieved Passages (Figure 3, Table 2)

The paper's most diagnostically important experiment examines how EM on the validation set changes as the number of retrieved passages varies across the set {5, 10, 25, 50, 100}. The results, shown in Figure 3 for the base model:

• NaturalQuestions: 5 passages → approximately 40.0 EM, 10 → ~42.3, 25 → ~44.5, 50 → ~45.5, 100 → 46.5 EM. The improvement from 10 to 100 passages is approximately +4.2 EM points, with no sign of saturation—the curve continues to rise at 100 passages.

• TriviaQA: 5 passages → approximately 55.0 EM, 10 → ~58.0, 25 → ~62.0, 50 → ~63.8, 100 → 64.7 EM. The improvement from 10 to 100 passages is approximately +6.7 EM points, with the steepest gain occurring between 10 and 25 passages. The curve still rises from 50 to 100, though the marginal gain diminishes.

• SQuAD: 5 passages → approximately 34.0 EM, 10 → ~38.0, 25 → ~43.0, 50 → ~46.0, 100 → ~50.0 EM (estimated from Figure 3). The improvement follows a similar monotonic pattern.

The paper explicitly contrasts this scaling behaviour with extractive models:

"the performance of most extractive models seems to peak around 10 to 20 passages (Wang et al., 2019; Yang et al., 2019)"

The key interpretive claim is that this divergence establishes that "sequence-to-sequence models are good at combining informations from multiple passages" in a way that extractive span-prediction models are not. The experimental design supports this interpretation because the same retrieval pipeline (DPR for NQ and TriviaQA, BM25 for SQuAD) was used by prior extractive work, isolating the reader architecture as the causal factor.

An important note on what is measured: the numbers in this paragraph are from the validation set, not the test set. The test set results in Table 1 use 100 passages throughout and cannot be used to infer the scaling curve directly. The validation set performance at 100 passages (46.5 for NQ base) is lower than the test set performance (48.2), which is expected if the test set is somewhat easier or if model selection on the validation set introduces a slight selection bias.

Impact of Training Passage Count and Two-Phase Finetuning (Table 2)

Table 2 reports an experiment measuring how the number of passages used during training affects final performance when evaluating at 100 passages. The key numbers:

NaturalQuestions dev set:

TriviaQA dev set:

The pattern is revealing. Without finetuning, there is a large gap between models trained on few passages and those trained on many: 37.8 vs. 46.5 on NQ when training with 5 vs. 100 passages—a gap of 8.7 EM points. This confirms that the decoder's cross-attention over many encoder outputs is a distributional shift that the model does not handle zero-shot.

The two-phase finetuning strategy largely closes this gap. On NQ, finetuning a 5-passage model for 1000 steps with 100 passages lifts performance from 37.8 to 45.0, within 1.5 points of the fully 100-passage-trained model. On TriviaQA, a 10-passage model finetuned on 100 passages reaches 63.6, versus 64.7 for the fully trained model—a gap of 1.1 points. The paper reports this requires 147 GPU-hours versus 425 GPU-hours for full training, a 65% reduction in compute.

Interestingly, the finetuning benefits are most dramatic for models trained on the fewest passages (5 or 10), and the marginal benefit decreases as the training passage count approaches 100. Models trained on 25 or 50 passages show smaller gains from finetuning (e.g., 45.3 → 46.0 on NQ for the 25-passage model). This suggests that the encoder representations from 25-passage training are already sufficiently rich, and the remaining gap is primarily the decoder adapting its attention normalisation to the larger candidate set.

The paper does not report what happens if the finetuning phase is longer than 1000 steps—whether the 5-passage model would eventually reach 46.5 if given more finetuning steps, or whether there is a residual gap that cannot be closed. This is a missing ablation that would clarify whether the 1.5-point gap is a compute limitation or a fundamental limitation of training on few passages.

Ablation Studies and Robustness Checks

The paper is notably thin on formal ablations compared to modern standards. The main experimental analysis focuses on three axes: passage count scaling, training passage count, and the finetuning strategy. Several aspects that would strengthen the paper are not explored as explicit ablations:

• No direct comparison of Fusion-in-Decoder against an encoder-concatenation baseline at equal passage counts. The paper argues that independent encoder processing is superior because it enables scaling to 100 passages, but it does not train a model that concatenates, say, 10 passages in the encoder and compare it against Fusion-in-Decoder at 10 passages. Such a comparison would isolate whether decoder-fusion provides a benefit independent of the passage-count scaling argument—i.e., whether decoder-side fusion is better even when the encoder can handle the input size. The comparison against RAG at 5–10 passages (Table 1) provides indirect evidence but does not control for the many other architectural differences between Fusion-in-Decoder and RAG.

• No ablation on the special prefix tokens (question:, title:, context:). The paper introduces these as part of the passage formatting but does not test whether removing them degrades performance, whether different delimiter choices matter, or whether the tokens provide any benefit beyond what the model's positional embeddings already encode about sequence structure. Given that the model must learn to interpret these as delimiters through fine-tuning (they are not special vocabulary entries), their value is an empirical question the paper leaves unanswered.

• No ablation on greedy decoding vs. beam search. The paper uses greedy decoding throughout without reporting whether beam search (which is standard for most seq2seq tasks) would improve results, and if so, by how much. This is a meaningful omission because beam search could specifically help when the model needs to choose between multiple plausible answers supported by different passages—exactly the multi-document aggregation scenario the architecture is designed for. If beam search provides no benefit, that would be an informative finding about the model's probability distribution sharpness; if it helps substantially, the reported numbers would be lower bounds.

• Retrieval method choice is dataset-dependent but not ablating. The paper uses DPR for NQ and TriviaQA but BM25 for SQuAD, following "the results of Karpukhin et al. (2020)." It does not report what happens if DPR is used for SQuAD or BM25 for NQ within the Fusion-in-Decoder framework. This means the SQuAD results may partly reflect the retriever choice rather than the reader architecture, and the cross-dataset comparability of the scaling curves is confounded by the different retrieval methods.

• No ablation on passage length or truncation. The paper truncates passages to 250 word pieces but does not explore whether shorter truncation (e.g., 128) hurts and whether longer (e.g., 512) helps. Given that passage length interacts with both retrieval quality (longer passages may contain more context but be less precise) and encoding cost, this is a practically important design choice that is treated as a constant.

• Model size ablation is coarse. The paper tests two model sizes (base and large) but does not test anything smaller (e.g., T5-small at 60M parameters) or between base and large. This matters because one of the paper's motivating arguments is that retrieval augmentation allows smaller models to compete with larger closed-book models, but the smallest model tested is still 220M parameters—not small by the standards of on-device deployment. Whether the Fusion-in-Decoder approach works at genuinely small model scales (where the model's reading comprehension capacity may be insufficient to process 100 passages) is an open question.

Critical Assessment

The paper's experimental design is best understood as a demonstration that a specific architectural choice—processing passages independently in the encoder and fusing them in the decoder—enables scaling to many more retrieved passages than prior systems, and that this scaling yields substantial accuracy improvements. The experiments establish this core point relatively cleanly through the passage-count scaling curves in Figure 3, which are the paper's strongest empirical contribution. However, the paper also makes several broader claims that the experiments only partially substantiate.

On the claim that generative models are better multi-document aggregators than extractive models. This claim is supported indirectly but not tested directly. Figure 3 shows that Fusion-in-Decoder (a generative model) continues to improve to 100 passages, while the paper cites prior work (Wang et al., 2019; Yang et al., 2019) showing that extractive models peak at 10–20 passages. The comparison is cross-study rather than within-study—the extractive model results are from different papers, using different training procedures, different retrieval quality (retrieval methods evolved between 2018 and 2020), and different hardware. A stronger test would train an extractive BERT reader on the same DPR retrieval output used for Fusion-in-Decoder and plot both scaling curves on the same figure. Without this, the attribution of saturation to the extractive architecture specifically (rather than to incidental differences in training or retrieval) is plausible but not proven. The paper could also have included a within-study extractive baseline—perhaps a standard BERT-SQuAD model applied to the same retrieved passages—to make the comparison controlled.

On the claim that decoder-fusion is inherently better than encoder-concatenation. The paper strongly implies this but never tests it head-to-head at equal passage counts. The scalability argument (linear vs. quadratic) is a computational claim, not an accuracy claim—it explains why Fusion-in-Decoder can handle 100 passages, not whether decoder-fusion produces better answers than encoder-fusion at the same passage count. To establish that decoder-fusion is architecturally superior, the paper would need an experiment where both approaches are given the same 10 passages, and the decoder-fusion model outperforms the encoder-concatenation model. The RAG comparison partially fills this gap, but RAG also uses per-passage decoder marginalisation rather than joint cross-attention, so it differs from both encoder-concatenation and decoder-fusion. A clean three-way comparison (encoder-concat vs. decoder-fusion vs. RAG-style marginalisation) at a shared small passage count would clarify which architectural choice drives the gains.

On the state-of-the-art claim. The numbers in Table 1 genuinely exceed prior published results at the time, and the margins on NaturalQuestions and TriviaQA are substantial enough (3–7 points) that they likely reflect a real improvement rather than statistical noise. However, the paper does not provide confidence intervals, bootstrap estimates, or any measure of the variance in model performance across random seeds. Deep learning models fine-tuned on datasets of this size can exhibit non-trivial run-to-run variance due to random weight initialisation, data ordering, and dropout. A 1–2 point gap between methods could plausibly arise from seed variance alone, particularly on the smaller TriviaQA and SQuAD test sets. The larger gaps (6–7 points on NQ, 7–8 on TriviaQA) are likely robust to this concern, but the absence of error reporting is a methodological limitation.

On the claim about computational efficiency. The paper argues that independent encoder processing scales linearly rather than quadratically in passage count, which is a well-founded theoretical claim about the architecture. But no wall-clock timing experiments or FLOPs counts are reported, and no comparison of actual inference latency is made against encoder-concatenation baselines. The linear scaling claim is an architectural property rather than an empirical measurement. For practitioners, what matters is whether 100-passage Fusion-in-Decoder is faster than 10-passage encoder-concatenation at the same accuracy—a tradeoff the paper does not analyse.

On dataset-specific findings. The passage-count scaling curves (Figure 3) show somewhat different shapes across datasets: TriviaQA gains most steeply from 10 to 25 passages and then flattens, while NaturalQuestions and SQuAD show more gradual, continuing improvement. The paper does not analyse why this difference exists. Possible explanations include: TriviaQA answers are more factual and depend on a single passage (so extra passages add redundancy rather than new information), NaturalQuestions includes more compositional questions that genuinely require synthesising information from multiple passages, or DPR retrieval quality is higher for TriviaQA so the first few passages already contain the answer. Without this analysis, the generalisability of the "100 passages help" finding to new datasets is uncertain.

Missing experiments that would strengthen the paper. Several specific experiments are conspicuous by their absence. First, an experiment varying retriever quality (e.g., comparing BM25 vs. DPR within the same dataset, or degrading DPR by retrieving fewer relevant passages) to test whether the decoder-fusion model is robust to retrieval noise would be practically valuable—in real deployments, retrieval quality varies, and the model's ability to aggregate might compensate for noisy retrieval. Second, no analysis of what the decoder's cross-attention actually learns—no attention visualisation, no analysis of whether the model attends to the passage containing the answer, no measurement of whether attention is diffuse or concentrated. Such analysis would substantiate the "evidence fusion" narrative with mechanistic evidence rather than just input-output correlations. Third, no experiment on whether the model can genuinely compose information from two passages that each contain partial evidence—the paper's strongest architectural claim is about cross-passage synthesis, but the evaluation (EM on entity answers) does not distinguish between answers that could be derived from a single passage and those requiring multiple passages. A constructed test set of multi-hop questions would test this directly.

On the validation-vs-test discrepancy. The validation set performance for the base model on NQ at 100 passages (46.5, Table 2) is lower than the test set performance (48.2, Table 1). This could reflect the validation set being intentionally harder (since it is a random 10% of training data, which may contain more challenging examples than the curated test set), or it could reflect mild overfitting to the validation set through model selection. The paper does not discuss this gap, but it suggests that the absolute EM numbers should be interpreted as having a ~2-point uncertainty range depending on the evaluation split, which matters for comparing closely matched systems.

On the two-phase finetuning finding. Table 2 provides a practically useful result, but the interpretation is slightly oversold. The paper presents it as evidence that "we can reach 46.0 EM on NaturalQuestions, using 147 GPU hours, compared to 425 GPU hours when training on 100 passages." This is a 1.2-point gap for a 65% compute saving—a genuine tradeoff, but presented as nearly equivalent. The 1.2-point gap is actually larger than the absolute difference between several competing systems in Table 1 (e.g., SpanSeqGen at 42.5 vs. RAG at 44.5 on NQ is a 2.0-point gap). For a paper whose contribution is state-of-the-art accuracy, whether finetuning can fully match full training matters, and the answer appears to be "no" from the data shown.

6. Limitations and Trade-offs

Fusion-in-Decoder Is Never Directly Compared Against Encoder-Concatenation at Equal Passage Counts

The assumption or constraint. The paper's central architectural claim is that fusing evidence in the decoder (via cross-attention over independently encoded passages) is superior to fusing evidence in the encoder (via self-attention over concatenated passages). This claim is argued from computational principles—linear vs. quadratic scaling in passage count—but it is never tested as a controlled empirical comparison. The paper states:

"Processing passages independently in the encoder allows to scale to large number of contexts, as it only performs self attention over one context at a time. This means that the computation time of the model grows linearly with the number of passages, instead of quadratically."

This is an argument about computational tractability at scale, not about whether decoder-fusion produces better answers than encoder-fusion at the same passage count. The paper compares against RAG (which marginalises per-passage answer distributions rather than concatenating in the encoder) but never against an encoder-concatenation baseline—a T5 model where all passages are concatenated into a single input sequence—trained and evaluated at shared passage counts like 5 or 10.

The consequence. Without this comparison, the paper cannot separate two distinct claims it appears to make simultaneously: (a) independent encoder processing enables scaling to 100 passages (a computational claim), and (b) decoder-side cross-attention produces better evidence aggregation than encoder-side self-attention (an accuracy claim). Claim (a) is well-supported by the passage-scaling curves. Claim (b) is plausible but unverified. It is possible that at 5 or 10 passages—the regime where encoder-concatenation is computationally feasible—a standard concatenation approach would match or exceed Fusion-in-Decoder's accuracy, and the observed gains at 100 passages are purely a function of access to more evidence rather than any architectural advantage in how evidence is combined. If true, this would reframe the paper's contribution from "decoder-fusion is a better way to combine evidence" to "linear encoder scaling lets you afford more passages, and more passages help"—a still-valuable but substantially narrower claim.

The ambiguity matters practically. A practitioner who can afford only 5 or 10 passages at inference time (due to latency constraints, for example) has no guidance from this paper about whether to adopt the Fusion-in-Decoder architecture or simply concatenate those passages in the encoder.

What evidence exists in the paper. No direct evidence. The RAG comparison (Table 1: 48.2 vs. 44.5 EM on NQ base) provides indirect evidence because RAG processes passages independently and then marginalises, but RAG's decoder never attends to two passages simultaneously, making it architecturally different from both encoder-concatenation and decoder-fusion. The passage-count scaling curves (Figure 3) show that Fusion-in-Decoder continues improving to 100 passages, but there is no comparator curve from an encoder-concatenation model in the 5–10 passage range to establish whether the curves would coincide at low passage counts. The paper implicitly acknowledges the gap by not including an encoder-concatenation baseline in Table 1 or Figure 3.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, and it does not suggest future work comparing encoder-fusion against decoder-fusion at matched passage counts. The architectural argument is presented as settled by the scalability demonstration, but the missing controlled comparison means a careful reader should treat the claim about decoder-fusion being inherently better (rather than enabling more passages) as a hypothesis supported by circumstantial evidence rather than a proven finding.

The Break-Even Point Between Training with Few Passages and Full 100-Passage Training Is Not Established

The assumption or constraint. The paper proposes a two-phase finetuning strategy to reduce training cost: train on a small number of passages (e.g., 5 or 10) for most of the training budget, then finetune on 100 passages for 1,000 additional steps. Table 2 reports that this recovers most of the accuracy of full 100-passage training at roughly one-third of the GPU cost, framing the approach as a practical efficiency measure:

"we can reach 46.0 EM on NaturalQuestions, using 147 GPU hours, compared to 425 GPU hours when training on 100 passages."

The paper treats the 1,000-step finetuning duration as fixed and does not explore how the gap between finetuned and fully trained models changes with finetuning budget, nor whether the residual gap (1.5 EM points for the 5-passage model on NQ, 1.1 points for the 10-passage model on TriviaQA) can be closed by extending the finetuning phase.

The consequence. There is an ambiguity about whether the residual gap between finetuned and fully trained models reflects a fundamental limitation of training on few passages or simply an insufficient finetuning budget. If 2,000 or 5,000 finetuning steps would close the gap to within 0.1 EM points, then the two-phase strategy is genuinely equivalent to full training at substantially lower cost—the paper's practical recommendation is robust. If even 20,000 finetuning steps cannot close the gap, then there is something about the decoder's cross-attention that requires exposure to many passages throughout training to reach optimal performance, and the paper's finetuning strategy trades accuracy for compute in a way that may be unacceptable for applications where the last 1–2 EM points matter.

The 1.5 EM point gap on NaturalQuestions is not trivially small. In the context of the results in Table 1, 1.5 points is comparable to the difference between Fusion-in-Decoder base (48.2) and RAG (44.5)—a 3.7-point gap that the paper presents as a substantial advantage. If a practitioner deploys the finetuned model instead of the fully trained model, they are sacrificing a margin of improvement that is roughly 40% of the paper's claimed advance over the previous state-of-the-art. Whether this tradeoff is acceptable depends on the application, but the paper provides no evidence to determine whether the finetuned model's gap is easily closable or structural.

What evidence exists in the paper. Table 2 reports performance at exactly one finetuning duration (1,000 steps) across different training-passage counts. The 25-passage model improves from 45.3 to 46.0 after finetuning; the 50-passage model improves from 45.7 to 46.0. The finetuned numbers for 25, 50, and 100 passages cluster at 46.0–46.5, suggesting that the gap narrows as the base training passage count increases. But the paper does not report whether the 5-passage model at 2,000 or 5,000 finetuning steps would reach the 46.0–46.5 range or would asymptote below it. The finetuning experiment is a single data point per configuration, not a sweep over finetuning budgets.

Mitigation status. Partially acknowledged but not addressed empirically. The paper presents the finetuning strategy as a cost-saving option without characterising the accuracy-compute tradeoff curve, leaving practitioners to guess whether 1,000 steps is optimal or whether additional finetuning would help. The paper does not flag this as an open question or suggest investigating the finetuning budget as a controlled variable.

No Analysis of Whether the Model Actually Performs Cross-Passage Evidence Synthesis

The assumption or constraint. The paper's strongest architectural claim is that Fusion-in-Decoder enables the model to aggregate and combine evidence from multiple passages—to read something in passage 3, something else in passage 12, and synthesise them into a single answer that could not be produced from either passage alone. This capability is central to the paper's narrative:

"We believe that this is evidence that generative models are good at combining information from multiple passages."

"Processing passages jointly in the decoder allows to better aggregate evidence from multiple passages."

The paper provides input-output evidence for this claim (the model's EM improves with more passages), but it provides no mechanistic evidence—no attention visualisation, no analysis of which passages the decoder attends to when generating answers, no measurement of how often the correct answer requires evidence from multiple passages versus being derivable from a single passage, and no constructed test of multi-hop or compositional questions where single-passage extraction would necessarily fail.

The consequence. The paper cannot distinguish between two entirely different explanations for why accuracy improves with more passages. Explanation 1 (the paper's claim): the decoder genuinely combines evidence from multiple passages, attending to passage 7 for the entity name and passage 23 for the date, and composing them. Explanation 2 (a weaker but equally consistent interpretation): adding more passages increases the probability that the correct answer appears verbatim in at least one passage, and the decoder is essentially learning to identify and copy the answer span from whichever passage contains it, without doing any cross-passage composition. Under Explanation 2, the performance improvement is purely a function of increased retrieval recall—a larger $k$ means a higher chance that the answer is somewhere in the retrieved set—and the generative model's aggregation capability is limited to better selection among passages rather than synthesis across them. Both explanations predict the upward-sloping curves in Figure 3, and the paper provides no evidence to favour one over the other.

The distinction matters because it determines what kind of questions the model can answer and where the approach will succeed or fail in practice. If the model is primarily a clever answer-span selector, it will fail on questions where no single passage contains the answer but multiple passages collectively contain the necessary information—exactly the kind of multi-hop reasoning that retrieval-augmented QA promises to enable. If the model genuinely synthesises across passages, it should succeed on such questions. Without evidence about which mechanism is operating, a practitioner deploying this system cannot predict its failure modes.

What evidence exists in the paper. None. There is no attention analysis, no ablation where passages are deliberately split such that the answer requires information from two sources (a constructed multi-hop test), no measurement of how often the model's output is a substring of a single retrieved passage (which would suggest extraction-style behaviour), and no comparison of performance on single-passage-answerable vs. multi-passage-answerable questions. The datasets used (NaturalQuestions, TriviaQA, SQuAD) were not designed to distinguish single-hop from multi-hop reasoning; most questions in these datasets can be answered from a single well-chosen Wikipedia passage, making them incapable of testing the synthesis claim directly.

Mitigation status. Not addressed. The paper does not acknowledge the ambiguity between passage-recall and evidence-synthesis explanations for its scaling results, nor does it suggest future work to disambiguate them with targeted experiments. The "evidence fusion" narrative is presented as the interpretation of the scaling curves without considering alternative explanations, which weakens the paper's strongest conceptual claim.

Single Benchmark Family, Single Model Family, Single Task Format Limits Generalisability Claims

The assumption or constraint. All experiments use three datasets from the same task family (open-domain factoid QA with short entity answers), the same underlying corpus (Wikipedia), and a single model family (T5). The paper does not test on datasets requiring long-form generation (e.g., ELI5, NarrativeQA), on reasoning tasks where answers are not named entities (e.g., HotpotQA, StrategyQA), on non-English corpora, on non-Wikipedia knowledge sources, or with non-T5 sequence-to-sequence architectures such as BART (Lewis et al., 2019). The paper acknowledges this implicitly by titling its contribution "Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering" and restricting all experiments to that setting, but it does not discuss the scope of generalisability.

The consequence. There are at least three axes along which the findings may not transfer, and the paper provides no evidence to assess the risk:

• Task format. The paper's answer format is short entity strings (people, dates, locations, numbers) extracted or composed from Wikipedia text. For long-form answer generation, where answers are multi-sentence explanations, the decoder's cross-attention over 100 passages would need to select and organise substantially more information, and the greedy decoding strategy (which works for short answers) may be insufficient. RAG (Lewis et al., 2020) demonstrated that retrieval-augmented generation works for longer outputs on ELI5 using a different architecture; whether Fusion-in-Decoder would scale similarly to long-form tasks, and whether 100 passages remain beneficial when the answer is itself a paragraph rather than a word, is unknown.

• Model family. T5 has a specific encoder-decoder architecture with pretraining on a span-corruption objective. The paper's findings—particularly the decoder's ability to attend over a very large concatenated encoder output (~25,000 tokens for 100 passages)—may depend on T5's pretraining, which involves denoising corrupted spans and may naturally train the decoder to attend broadly over noisy encoder outputs. BART, which uses a different corruption strategy, or models with different encoder-decoder depth ratios, might behave differently at large passage counts. The paper provides no evidence to assess transferability.

• Knowledge source. The retrieval corpus is Wikipedia, which has high-quality, factually consistent, well-structured text with clear article boundaries and titles. For retrieval from noisier sources (web crawl, forums, social media) or domains where passage boundaries are less semantically meaningful, the 100-passage approach may suffer from higher noise levels and the model's aggregation capability may be less effective. The paper's observation that TriviaQA saturates earlier than NaturalQuestions (Figure 3) hints at dataset-specific effects, but without testing across corpora, the generalisability of the 100-passage finding is uncalibrated.

What evidence exists in the paper. The passage-count scaling curves in Figure 3 show different saturation patterns across datasets—TriviaQA flattens after 25 passages while NaturalQuestions and SQuAD continue improving to 100—which is indirect evidence that the benefit of many passages depends on dataset characteristics. But the paper does not analyse these differences or test across a wider set of conditions, so the evidence is suggestive rather than diagnostic.

Mitigation status. Not addressed. The paper does not discuss generalisability limitations or suggest future work to replicate findings across tasks, models, or corpora. The claims in the abstract and introduction are stated in general terms ("generative models," "open domain question answering") but the experimental support is specific to one model family, one corpus, and one task format.

Training and Inference Costs Are Reported Selectively, Making Deployment Tradeoffs Hard to Assess

The assumption or constraint. The paper discusses computational cost in three distinct and inconsistent ways: (a) the architectural argument that independent encoder processing scales linearly rather than quadratically in passage count (Section 3), which is a theoretical property rather than a measurement; (b) training GPU-hours for specific configurations (425 for full 100-passage training on NQ, 147 for the two-phase strategy) reported in the discussion of Table 2; and (c) the broader claim that retrieval augmentation allows a 770M-parameter model to outperform an 11B-parameter model at comparable memory footprint. However, the paper provides no inference-time latency measurements, no FLOPs counts, and no memory footprint quantification for either training or inference.

Specifically, the paper does not report: how long a single 100-passage inference takes on a given GPU; how much GPU memory is required to process 100 passages simultaneously (the decoder's cross-attention over ~25,000 tokens has a significant memory footprint); how inference latency scales with passage count in wall-clock terms; what the retrieval latency overhead is (DPR question encoding + FAISS search); or how the total end-to-end pipeline latency compares to closed-book T5-11B or to encoder-concatenation baselines at smaller passage counts.

The consequence. A practitioner considering Fusion-in-Decoder faces an underdetermined deployment decision. The paper's architectural argument establishes that independent encoder processing is asymptotically more efficient than concatenation—but asymptotics do not translate directly to real-world latency. The constant factors matter enormously: the decoder's cross-attention over 25,000 encoder tokens is not free, and while it scales linearly with passage count, the per-passage encoder cost (100 forward passes through a 12-layer T5 encoder) may dominate in practice, especially if encoders are not parallelised across many GPUs. Without latency numbers, a practitioner cannot determine whether 100-passage Fusion-in-Decoder is feasible under a 100ms latency budget for a production QA system, whether they need to reduce passages to 20 or 50 to meet latency targets, or what GPU provisioning is required to achieve a given throughput.

The comparison with closed-book T5-11B is similarly underspecified. The paper states that both approaches use "roughly the same amount of memory to store information"—the 11B model's parameters (~44 GB for float32 weights) versus Wikipedia text plus the 770M model's parameters—but this is a storage comparison, not a deployment comparison. At inference time, the 11B model requires one forward pass per question. The 770M Fusion-in-Decoder requires 100 encoder passes, one decoder pass with cross-attention over 25,000 tokens, and a DPR retrieval step. The total FLOPs per query may exceed the large model's, even though the parameter count is 14× smaller. The paper does not compute this.

What evidence exists in the paper. Training cost numbers (425 vs. 147 GPU-hours) are provided in the context of Table 2, and these are useful for planning training runs. But the training cost comparison is between two Fusion-in-Decoder variants (full vs. two-phase), not between Fusion-in-Decoder and competing approaches. The claim that the approach "scales well with the number of retrieved passages" (Section 5 conclusion) and that "computation time grows linearly with the number of passages" (Section 3) are both true architecturally, but they are presented without empirical timing validation, making it unclear whether the linear scaling is practically usable or if other bottlenecks (decoder cross-attention memory, retrieval latency) dominate at 100 passages.

Mitigation status. Not addressed. The paper does not frame the lack of inference cost measurement as a limitation, nor does it suggest that future work should characterise the latency-accuracy tradeoff for deployment. The two-phase finetuning strategy partially addresses training cost but leaves inference cost unexamined. This is a substantial omission given that the paper's motivating argument in Section 1 emphasises that large closed-book models are "expensive to query," implying that the proposed approach should be cheaper to query—but the paper never measures whether this is true in practice.

The Paper Provides No Mechanism for Adapting Passage Count Per Question

The assumption or constraint. All experiments in the paper use a fixed number of retrieved passages applied uniformly to every question. The passage-count scaling experiment (Figure 3) sweeps this fixed number across the entire dataset, showing that 100 passages is better on average than 50, which is better than 25. But the paper does not explore whether different questions benefit from different numbers of passages—it is plausible that some questions are easy and need only 5 passages, while others genuinely require 100, and that a fixed budget of 100 passages per question wastes compute on questions that the model can answer from the first few retrieved documents. The paper's conclusion simply states:

"the performance of our method significantly improves when the number of retrieved passages increases"

This is a statement about the dataset-level average, not about per-question optimal allocation.

The consequence. A deployment using 100 passages for every question pays the maximum inference cost on all queries, including those that could be answered correctly with far fewer passages. If, hypothetically, 60% of NaturalQuestions can be answered correctly with 10 passages and only 5% of questions require more than 50 passages to shift from incorrect to correct, then a policy of always using 100 passages wastes roughly 50% of inference compute. The paper provides no data to estimate what fraction of questions benefit from moving from 10 to 100 passages, making it impossible for a practitioner to evaluate whether a variable-passage-count policy (e.g., retrieve 10 passages, generate an answer, assess confidence, and only retrieve more if uncertain) would yield substantial cost savings with minimal accuracy loss.

This limitation is related to the evidence-synthesis ambiguity discussed earlier. If the model is primarily selecting answers from single passages, then adding passages beyond the first one that contains the answer provides redundancy but not new information—and the optimal passage count per question would be determined by retrieval recall, which varies by question. If the model is synthesising across passages, then hard questions might genuinely require more passages than easy ones, and a fixed budget would under-serve hard questions while over-serving easy ones. The paper's experimental design, which treats passage count as a global hyperparameter rather than a per-question variable, cannot distinguish these scenarios or inform an adaptive policy.

What evidence exists in the paper. Figure 3 shows average performance across the entire validation set at each passage count, but it does not report the distribution of per-question improvement—how many questions get the correct answer at 10 passages that were wrong at 5, how many at 25 that were wrong at 10, and so on. The curves are smooth and concave, which could be consistent with many questions improving gradually (the evidence-synthesis interpretation) or with a subset of questions requiring many passages while most benefit from few (the recall-lottery interpretation). The paper does not attempt to characterise this.

Mitigation status. Not addressed. The paper does not identify adaptive passage-count selection as a direction for future work, does not discuss the cost implications of a fixed-100-passage policy for heterogeneous question distributions, and does not provide the per-question improvement data that would allow others to estimate the potential gains from adaptivity. This is a notable gap given the paper's emphasis on compute efficiency through linear encoder scaling—the architectural efficiency is undercut if the system retrieves more passages than needed for most queries.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper is best understood not as a paradigm shift but as a reframing with practical consequences: it identifies an architectural bottleneck that the field had accepted as a fact about the task—that open-domain QA systems cannot benefit from more than ~20 retrieved passages—and demonstrates that the bottleneck is a property of extractive architectures and encoder-concatenation designs, not of the retrieval-plus-reading paradigm itself. By moving evidence fusion from the encoder's self-attention to the decoder's cross-attention, the paper shifts the ceiling on how many passages a system can usefully ingest from ~20 to at least 100, and in doing so changes what counts as a plausible design for retrieval-augmented generation.

The specific reframing is this: prior to this work, retrieval-augmented QA was implicitly designed around the assumption that the reader's job is to locate the answer in one passage, and retrieving more passages was about increasing the probability that the single right passage appears somewhere in the retrieved set. This is why extractive models plateaued—once retrieval recall saturates, extra passages are just noise that the span-prediction head cannot exploit. The paper's scaling curves in Figure 3 refute this model of the task, at least for generative readers. Accuracy continues to rise monotonically from 10 to 100 passages on all three datasets, with gains of 4–7 EM points. This cannot be explained by retrieval recall alone (DPR recall@20 is already high for these datasets; moving from 20 to 100 passages adds mostly marginally relevant documents), which means the generative model is doing something qualitatively different with those extra passages—either selecting answers more robustly by weighing evidence across multiple sources, or genuinely synthesising information that no single passage contains.

This reframing has several downstream effects on how the field thinks about retrieval-augmented systems:

It makes passage count a first-class design dimension. Before this work, passage count was typically treated as an engineering constraint set by what the encoder could afford (5–10 for concatenation approaches, 10–20 for extractive models). The paper demonstrates that passage count is actually a performance lever that should be pushed as high as architecture and latency budgets allow, because the marginal accuracy benefit of additional passages may still be positive at counts far beyond what prior systems could test. Table 2 adds nuance by showing that training with many passages is expensive but can be approximated via two-phase finetuning, giving practitioners a concrete path to high passage counts without the full training cost. The implication is that future retrieval-augmented systems should routinely test passage counts of 50–200 to find their saturation point, rather than defaulting to 10.

It sharpens the architectural question from "retrieval vs. no retrieval" to "where should evidence be combined?" The paper's comparison against closed-book T5-11B (36.6 EM on NQ vs. 51.4 for Fusion-in-Decoder large with retrieval) and against RAG (44.5 vs. 51.4) brackets the design space: retrieval helps substantially over parametric memory alone, and how you combine retrieved passages matters independently of whether you retrieve them. This moves the conversation past the binary "retrieval augmentation: yes or no" debate toward a more detailed engineering question about encoder vs. decoder fusion, independent vs. joint processing, and marginalisation vs. attention-based aggregation. The RAG comparison is particularly important because both systems retrieve and generate, but Fusion-in-Decoder's joint decoder attention outperforms RAG's per-passage marginalisation by 3.7 EM points on NQ base, suggesting that simultaneous attention over all passages is measurably better than combining per-passage answer distributions.

It partly resolves the tension between closed-book and open-book QA results. Roberts et al. (2020) showed that T5-11B without retrieval could achieve 36.6 EM on NQ and 60.5 on TriviaQA, raising the question of whether retrieval was necessary at all if you could just scale the model. The paper's results provide a clear answer: retrieval augmentation with a 770M model (51.4 NQ, 67.6 TriviaQA) substantially outperforms an 11B closed-book model—with roughly the same total memory footprint—and the gap widens as you add more passages (Figure 3). This suggests that explicit text-based memory is not just competitive with implicit parametric memory but actually more parameter-efficient: a 770M model with access to Wikipedia text outperforms an 11B model that has memorised it. The resolution is that retrieval matters, but it matters most when the reader architecture can actually use the retrieved evidence effectively, which prior generative-plus-retrieval systems (RAG) could not fully demonstrate because they were passage-count-limited.

It introduces a practical metagradient for training cost. The two-phase finetuning result in Table 2—reaching 46.0 EM on NQ with 147 GPU-hours vs. 46.5 with 425 GPU-hours—establishes that the number of training passages and the number of inference passages can be partially decoupled through a short adaptation phase. This is not a deep conceptual finding but it changes the economics of experimentation in this area: researchers can iterate on architectures and hyperparameters using small passage counts (e.g., 5 or 10) and only commit the expensive large-passage-count training when they have a promising configuration. The fact that the encoder representations generalise across passage counts (because the encoder processes each passage independently) means the expensive part of training—learning good per-passage representations—can be done once with a small passage budget, and only the decoder's attention normalisation needs to adapt to large passage counts. This pattern has since become common in retrieval-augmented generation research, making the paper a methodological as well as architectural reference point.

Follow-Up Research This Work Enables

A controlled experiment comparing decoder-fusion against encoder-concatenation at identical passage counts. The paper's central architectural claim—that decoder-side cross-attention is better than encoder-side self-attention for combining evidence—is never tested directly because the paper never trains an encoder-concatenation baseline at passage counts where concatenation is feasible (e.g., 5 or 10). A clean follow-up would train a T5 model that concatenates all passages into a single encoder input sequence, matched in parameter count, training data, and retriever output, and compare against Fusion-in-Decoder at 5, 10, and 20 passages on NaturalQuestions and TriviaQA. If the systems perform identically at these counts, then Fusion-in-Decoder's advantage is purely a scalability enablement—it lets you afford 100 passages, but at equal passage counts, there is no accuracy difference. If Fusion-in-Decoder outperforms even at 10 passages, then the architectural claim is stronger: decoder attention genuinely produces better evidence aggregation regardless of scale. This experiment would resolve the ambiguity between the scalability narrative and the fusion-quality narrative, and it requires no new datasets or models—only a baseline that the paper should logically have included.

Multi-hop QA stress test: can the model actually synthesise across passages? The paper claims the model "combines evidence from multiple passages" but provides no mechanistic evidence that answers are generated by synthesising information from two or more documents rather than by selecting the best single passage and extracting from it. A targeted follow-up would construct a test set of multi-hop questions where the answer requires combining information from two Wikipedia passages that are deliberately retrieved separately—for example, "Which British computer scientist born in Maida Vale cracked the Enigma code?" where the birthplace appears in one passage and the codebreaking achievement in another, and no single passage contains both facts. Comparing Fusion-in-Decoder's performance on these questions against a strong extractive baseline (DPR + BERT reader) and against a RAG-style marginalisation model would reveal whether the architecture actually enables cross-passage composition or whether its gains are primarily from better single-passage answer selection. The paper's existing datasets (NaturalQuestions, TriviaQA) are not designed for this because most questions are answerable from a single well-chosen passage; a constructed multi-hop set would test the architectural claim directly. A negative result—that Fusion-in-Decoder performs no better than an extractive model on genuinely multi-hop questions—would significantly narrow the paper's claims, suggesting that the scaling benefits come from improved answer selection robustness rather than genuine synthesis.

Training a lightweight difficulty estimator for adaptive passage-count selection. The paper uses a fixed 100 passages for every question, which is computationally wasteful if many questions can be answered from the first 5 or 10 passages. A natural extension would train a small classifier—perhaps a lightweight linear probe on top of the DPR question embedding, or a simple heuristic based on the DPR retrieval score distribution—to predict how many passages a given question is likely to need. The classifier could be trained using the paper's own data: for each question in the validation set, you can determine the minimum passage count at which Fusion-in-Decoder produces the correct answer by evaluating the model at 5, 10, 25, 50, and 100 passages (or doing a binary search). The question embedding or retrieval score statistics then become features, and the minimum sufficient passage count becomes the regression target. At inference time, the system retrieves the predicted number of passages rather than always 100. The paper's curves in Figure 3 provide the necessary training signal—for each question, you can label how many passages are needed—and the practical payoff would be substantial latency reductions on easy questions while preserving accuracy on hard ones. A strong follow-up would report the accuracy-compute tradeoff curve of this adaptive policy versus the fixed-100-passage baseline.

Replication and stress-testing on non-T5 architectures and non-Wikipedia corpora. The paper's experiments are restricted to T5 (encoder-decoder, span-corruption pretraining) and Wikipedia (high-quality, well-structured text). Two specific replications would map the generalisability boundary. First, replicate Fusion-in-Decoder using BART (Lewis et al., 2019), which has a different pretraining objective (sentence shuffling and token infilling) and a different encoder-decoder architecture, to test whether the decoder-fusion benefit is tied to T5's particular pretraining or is a general property of encoder-decoder Transformers. Second, replicate on a noisier retrieval corpus—web crawl paragraphs, forum text, or scientific abstracts—where passage boundaries are less semantically clean, retrieval precision is lower, and the model must distinguish relevant from irrelevant content across many passages. If the 100-passage benefit persists on noisy corpora, it would strengthen the claim that decoder-fusion is robust; if it degrades, it would suggest that the approach requires high-precision retrieval and clean passage boundaries to work.

Scaling model size downward: what is the smallest model that benefits from 100 passages? The paper's motivating argument emphasises that retrieval augmentation allows smaller models to compete with larger ones, but the smallest tested model is T5-base at 220M parameters—still a substantial model. A practical extension would sweep model sizes downward (T5-small at 60M, T5-mini, or even distilled architectures like DistilT5) and measure how the passage-count scaling curve changes with model capacity. The hypothesis is that at some small model size, the reading comprehension capacity becomes the bottleneck rather than retrieval coverage, and the benefit of going from 10 to 100 passages diminishes or vanishes. Finding this threshold would give practitioners a concrete minimum model size for deploying the approach and would test whether the decoder's cross-attention over 25,000 tokens remains effective when the model has limited capacity to process that much information. The paper's two-phase finetuning strategy (Table 2) would be particularly useful here, as it would allow testing many model sizes at 100-passage inference without retraining each from scratch.

Practical Applications and Downstream Use Cases

Cost-efficient open-domain QA with moderate-sized models. The paper's headline result—a 770M-parameter model with retrieval outperforming an 11B-parameter model without retrieval on NaturalQuestions (51.4 vs. 36.6 EM) and TriviaQA (67.6 vs. 60.5)—directly enables a deployment architecture where organisations use a smaller, cheaper-to-serve model augmented with a text corpus rather than a massive closed-book model. The concrete tradeoff: T5-large occupies ~3 GB in float16, while T5-11B occupies ~22 GB. With the retrieval corpus requiring comparable storage to the 11B model's parameters, the total memory footprint is similar, but the smaller model's per-token generation cost is roughly 14× lower. For applications like customer support QA, internal knowledge base search, or educational question answering where the knowledge domain is bounded and a corpus exists, this means the paper provides a recipe for achieving state-of-the-art accuracy without requiring datacenter-scale model serving infrastructure. The paper's two-phase finetuning strategy (147 GPU-hours to reach 46.0 EM on NQ) makes this accessible to teams without massive compute budgets.

Document-grounded answer generation with source attribution. A directly deployable use of Fusion-in-Decoder is in systems that need to produce answers with cited evidence. Because the decoder attends over all 100 passages simultaneously, its cross-attention weights can be inspected (though the paper does not do this) to identify which passages contributed to the generated answer. This enables a user-facing system where each answer is accompanied by the specific passages the model relied on, giving users the ability to verify claims against source text. This is valuable in medical QA, legal research, or journalistic fact-checking, where answer provenance is as important as answer accuracy. The paper's architecture is well-suited to this because each passage is independently encoded and has a known index in the concatenated encoder output, making it straightforward to trace the decoder's attention back to specific source documents—a property that closed-book models entirely lack and that encoder-concatenation models obscure because passages are mixed in the encoder's self-attention layers. No architectural modification is needed; the attribution mechanism is a byproduct of the design.

Retrieval-augmented fine-tuning for domain-specific QA without retraining the retriever. The paper establishes that the reader model can be fine-tuned on new QA datasets while keeping the retriever fixed (the DPR index and BM25 index are unchanged across datasets). This means an organisation with a domain-specific text corpus—patent documents, internal technical manuals, clinical guidelines—can build a QA system by (a) indexing their corpus with BM25 or training a DPR retriever on their in-domain question-answer pairs, (b) retrieving passages from the corpus, and (c) fine-tuning T5-base or T5-large on their QA data using the Fusion-in-Decoder architecture. The paper's hyperparameters (constant learning rate 10^-4, 10% dropout, 10k steps with 500-step validation) and the two-phase finetuning strategy provide an off-the-shelf recipe. For a corpus of comparable size to Wikipedia, a typical organisation could go from raw documents to a deployed QA system in under 200 GPU-hours on a single 8-V100 node, producing a system that generated answers grounded in their specific documents rather than in general web text.