Lost in the Middle: How Language Models Use Long Contexts

🎯 Pitch

Language models exhibit a striking 'U-shaped' performance curve when using long inputs—they excel at information at the very beginning or end, but performance plummets when critical details sit in the middle, sometimes dropping below not having the context at all.

1. Executive Summary

This paper empirically studies how current language models use long input contexts by analyzing performance on two tasks—multi-document question answering and synthetic key-value retrieval—across open (MPT-30B-Instruct, LongChat-13B) and closed (GPT-3.5-Turbo, Claude-1.3) models. The central finding is a distinctive U-shaped performance curve: models are often best at using relevant information placed at the very beginning of the input context (primacy bias) or the very end (recency bias), with performance degrading significantly—sometimes dropping below closed-book baselines—when models must access information in the middle. For example, GPT-3.5-Turbo's multi-document QA accuracy drops by more than 20% when the answer-bearing document is moved from the beginning to the middle of a 20-document context, and extended-context models like GPT-3.5-Turbo (16K) perform nearly identically to their shorter-context counterparts on inputs that fit within both windows, establishing that longer context windows do not inherently improve robust information access.

2. Context and Motivation

The Core Problem: We Don't Know How Well Models Actually Use Long Contexts

The fundamental question this paper tackles is deceptively simple: when we give a language model a long input context, does it actually use all of that information effectively? This matters because the dominant paradigm for applying language models to downstream tasks—prompting—involves formatting all relevant task specifications, instructions, and data as a textual input context, then having the model generate a completion. As language models are deployed in increasingly ambitious applications, these input contexts grow substantially longer, yet our understanding of whether models can robustly access information throughout that context remains surprisingly thin.

This gap is significant for several practical reasons that the paper explicitly surfaces (Section 1):

Retrieval-augmented generation (RAG) is now standard practice. Many commercial generative search and question-answering systems (e.g., Bing Chat, and the broader class of retriever-reader architectures) operate by retrieving relevant documents from a search engine or vector database and prepending them to the user's query as context. These systems routinely include 10, 20, or even 50+ retrieved passages. If models cannot effectively use information buried in the middle of long contexts, then simply retrieving more documents may be counterproductive—adding cost and latency without improving (or even degrading) accuracy.
Long-document processing is a primary use case. Language models are increasingly used to reason over long documents such as legal contracts, scientific papers, conversation histories spanning thousands of turns, or codebases with hundreds of files. Any degradation in the model's ability to access information in the middle of such documents would systematically bias outputs toward content at the beginning and end, undermining the reliability of these applications.
Context windows are expanding rapidly. Recent hardware and algorithmic advances have extended maximum context lengths from 512–2048 tokens to 32K, 100K, and beyond. This creates a natural but untested assumption: that models with larger context windows can effectively use all that context. The paper challenges this assumption directly, asking whether extended-context models are actually better at accessing information throughout their input, or merely capable of ingesting longer sequences without improvement in robust utilization.
Architectural choices may matter in non-obvious ways. The Transformer's self-attention mechanism is, in principle, equally capable of attending to any token in the input—there is no inherent architectural reason why a token at position 100 should be harder to "see" than a token at position 1 or position 1,000. If empirical results reveal systematic positional biases, it suggests that other factors—training data distribution, optimization dynamics, or the inductive biases of positional encoding schemes—are shaping model behavior in ways that pure architectural theory would not predict.

The Gap in Prior Understanding

Before this paper, the field's understanding of how language models use long contexts was fragmented and incomplete. Several lines of prior work touched on aspects of this question, but no study had systematically characterized how performance varies with the position of relevant information across different models, tasks, and context lengths.

Prior work on long-range context use in language modeling. The pioneering study by Khandelwal et al. (2018) showed that small LSTM language models make increasingly coarse use of longer-term context—they rely heavily on recent tokens and underutilize information from earlier in the sequence. Sankar et al. (2019) found similar results in dialogue models: model predictions were dominated by the most recent conversation turns. Daniluk et al. (2017) found that attentive LSTM language models tend to mainly use recent history. These studies established that small, pre-Transformer models exhibited a recency bias—they were better at using information near the end of the input—but did not explore whether the same pattern holds for Transformer-based models, nor whether larger-scale models trained on more data might exhibit different behaviors.

Long-context Transformer evaluations focused on perplexity, not information access. Much of the prior work on long-context Transformers evaluated models primarily via perplexity on held-out web text. For example, models like Transformer-XL (Dai et al., 2019), Longformer (Beltagy et al., 2020), Big Bird (Zaheer et al., 2020), and various efficient attention variants were assessed by measuring whether they could predict the next token in long documents. While perplexity is a convenient metric, it does not directly measure whether a model can retrieve and use specific pieces of information embedded at arbitrary positions in the input. Sun et al. (2021) explicitly studied whether long-range language models actually use long-range context and found that longer contexts improved prediction of only a few tokens, with the benefit concentrated near the beginning of the context. However, their analysis was restricted to next-word prediction of contiguous text, not to prompted task settings where a model must locate and apply a specific fact.

Needle-in-a-haystack experiments started to probe positional effects, but coarsely. The closest precursor to this paper's systematic approach is the "needle-in-a-haystack" experiments of Ivgi et al. (2023). They compared question-answering performance when the relevant paragraph was placed either at the beginning of the input or at a random position, finding that encoder-decoder models performed significantly better when relevant information appeared at the start. However, this study only probed two positions (beginning vs. random), leaving the shape of the performance curve unexplored. A U-shaped pattern (with degradation in the middle) could not be detected by only comparing extremes. Qin et al. (2023) analyzed efficient Transformers on long-context NLP tasks and found that models were recency-biased—but again, this characterized only one end of the performance spectrum.

No unified understanding of how model architecture, fine-tuning, and prompting interact with positional biases. Prior work was scattered across different model types (LSTMs vs. Transformers), different evaluation protocols (perplexity vs. task performance), and different task designs (language modeling vs. QA). No study had systematically compared: (1) decoder-only vs. encoder-decoder architectures; (2) instruction-tuned vs. base models; (3) different context lengths; (4) fine-grained document positions; and (5) both naturalistic (multi-document QA) and synthetic (key-value retrieval) tasks—all within a single, controlled experimental framework.

A Counterintuitive Disconnect Between Architecture and Behavior

A key intellectual motivation for this paper is the disconnect between what Transformer architectures should be capable of in principle and what models actually do in practice. The standard self-attention mechanism computes attention weights between every pair of tokens in the input, meaning that the representation at any position can theoretically attend to any other position with equal ease. The serial-position effect is a well-documented phenomenon in human memory—people best recall the first and last items in a list (Murdock Jr, 1962), a behavioral pattern often attributed to the mechanics of short-term and long-term memory consolidation. But there is no analogous theoretical reason why a Transformer should exhibit such an effect. If one does observe a U-shaped performance curve, it reveals something important about how training data, optimization, or positional encoding schemes interact with attention to produce systematic biases that the architecture alone does not predict.

This paper is, in part, an empirical investigation into whether that U-shaped curve exists, and if so, what factors contribute to it. The paper frames the investigation as an evaluation protocol: to claim that a language model can robustly use information within long input contexts, one must show that its performance is minimally affected by the position of the relevant information in the input context. The difference between best-case and worst-case performance across positions becomes a diagnostic for robustness, independent of absolute accuracy.

How This Paper Positions Itself

The paper positions itself not as proposing a new model or training method, but as providing a systematic empirical characterization of a phenomenon that has been hinted at but never comprehensively mapped. The key claims of positioning are:

Controlled manipulation is the distinguishing methodology. Unlike prior work that evaluated models on fixed benchmarks where the position of relevant information was an uncontrolled variable, this paper explicitly makes controlled changes to (1) the input context length (by varying the number of distractor documents or key-value pairs) and (2) the position of relevant information within the context (by reordering documents or JSON entries). This design isolates positional effects from other confounds—the desired output is identical regardless of where the relevant information is placed, so any performance variation must be attributable to the model's ability to access that information at different positions.

Two complementary tasks test different capabilities. The multi-document QA task (Section 2) requires models to reason over documents to find relevant information and use it to answer a question. This mimics real-world retrieval-augmented generation setups and tests the full pipeline of identifying relevance plus applying information. The synthetic key-value retrieval task (Section 3) strips away linguistic complexity (using random UUIDs as keys and values) to test the minimal capability of exact token matching within an input context. If models fail at key-value retrieval—where there is no ambiguity about what information is relevant, no reasoning required, just pattern matching—then the failure cannot be attributed to semantic confusion or reasoning deficits; it must reflect a fundamental limitation in information access.

Multiple models, including both open and closed, allow cross-architecture comparison. By evaluating decoder-only models (GPT-3.5-Turbo, MPT-30B-Instruct, LongChat-13B), encoder-decoder models (Flan-T5-XXL, Flan-UL2), base models (MPT-30B, Llama-2), instruction-tuned models (MPT-30B-Instruct, Llama-2-chat), and models with explicitly extended context windows (GPT-3.5-Turbo-16K, Claude-1.3-100K), the paper systematically probes whether positional biases are universal or attributable to specific architectural choices or training procedures.

The framing is diagnostic, not prescriptive. The paper's primary contribution is a better understanding of current model behavior, not a solution to the observed limitations. The U-shaped performance curve is presented as a finding that should inform how practitioners design prompts (putting critical information at the beginning or end), how system designers think about the tradeoff between retrieving more documents and effective utilization (Section 5), and how future long-context models should be evaluated (requiring position-invariant performance as a robustness criterion). The paper provides new evaluation protocols that fill the gap between perplexity-based long-context evaluation and the uncontrolled position distribution in standard benchmarks.

3. Technical Approach

3.1 Reader Orientation

This paper constructs a controlled experimental framework for measuring how well language models access and use information at different positions within long input contexts. The approach is not to build a new system but to design tasks where the position of relevant information can be systematically varied while holding all other variables constant, revealing how model performance changes as a function of where critical information appears in the input.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental apparatus has four major components that work together to isolate positional effects:

Task instantiation — converts a dataset (NaturalQuestions-Open for multi-document QA; randomly generated UUIDs for key-value retrieval) into a formatted prompt containing a query plus a collection of items (documents or key-value pairs), exactly one of which contains the information needed to answer.
Position modulator — reorders the items within the formatted prompt to place the relevant item at a specific, controlled position (index 0 for the beginning, index $k-1$ for the end, and several intermediate positions), while keeping all other content identical.
Context length modulator — varies the total number of distractor items $k$ to create prompts of different lengths (e.g., 10, 20, or 30 documents; 75, 140, or 300 key-value pairs), enabling separate analysis of how total context length interacts with positional effects.
Language model evaluation — feeds each systematically constructed prompt to one or more language models using greedy decoding, measures whether the correct answer appears in the model's output, and plots accuracy as a function of the relevant item's position within the input context.

Information flows linearly: a query-document pair from the dataset → distractor documents are retrieved and combined with the relevant document → the combined set is ordered to place the relevant item at a target position → the prompt is formatted with task instructions → the model generates an answer → the answer is scored against ground truth → results are aggregated across examples for each position and plotted.

3.3 Roadmap for the Deep Dive

First, the multi-document QA task design — how documents are sourced, how distractors are selected and controlled, how input context length and relevant information position are independently modulated, and how answers are evaluated. This is the primary, ecologically valid task.
Second, the synthetic key-value retrieval task design — why UUIDs were chosen, how the task strips away semantic complexity to isolate pure retrieval ability, and how the same positional modulation is applied in a fully controlled synthetic setting.
Third, the model evaluation protocol — which models are tested, how they are prompted, what decoding strategy is used, and how outputs are scored across all experiments.
Fourth, the diagnostic analyses — how model architecture, query-aware contextualization, and instruction fine-tuning are systematically probed to understand why positional effects occur, through controlled comparisons between model variants.
Fifth, the open-domain QA case study — how the retriever-reader setup extends the controlled findings to a realistic deployment scenario where the number of relevant documents is unknown, connecting the controlled experiments to practical system design.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that language model performance on tasks requiring information access from input contexts exhibits a systematic U-shaped curve as a function of the position of relevant information, and that this pattern reveals fundamental limitations in how current models use their context windows.

Multi-Document Question Answering Task Design

Task definition. The multi-document QA task requires a language model to (1) read a question, (2) examine $k$ provided documents, exactly one of which contains the information needed to answer the question, (3) identify which document is relevant, and (4) produce an answer using the information in that document. The $k-1$ other documents are "distractors" that do not contain the answer. The model receives the question and all documents formatted in a single prompt and must generate the answer as a text completion.

This task is designed to mimic the retrieval-augmented generation (RAG) setup underlying many commercial systems: a retriever fetches potentially relevant documents, all documents are concatenated with the query into an input context, and the model must find and use the relevant information among the retrieved set. By controlling exactly which document contains the answer and where it appears, the experiment isolates the effect of position from the effect of retrieval quality.

Data source. The task is instantiated using the NaturalQuestions-Open dataset (Lee et al., 2019; Kwiatkowski et al., 2019), which contains historical queries issued to the Google search engine paired with human-annotated answers extracted from Wikipedia. From this dataset, the authors select the subset of 2,655 queries where the annotated long answer is a paragraph (as opposed to a list or a table). This filtering ensures that the relevant information is a coherent prose passage that can be treated as a single document.

Relevant document selection. For each of the 2,655 queries, the document that contains the answer is taken directly from the NaturalQuestions annotations: it is the Wikipedia paragraph that human annotators identified as containing the correct answer. This provides a ground-truth relevant document that is naturally written and contains the answer in context.

Distractor document selection. The $k-1$ distractor documents (which must NOT contain the answer) are obtained using a retrieval system: Contriever, fine-tuned on MS-MARCO (Izacard et al., 2021). For each query, the system retrieves the $k-1$ Wikipedia chunks that are most relevant to the query, subject to the constraint that they do not contain any of the NaturalQuestions-annotated answers. These are the "hard negatives" — documents that are semantically related to the query (and therefore plausible as relevant documents) but happen not to contain the specific answer. By default, the distractor documents are presented in order of decreasing relevance (most relevant first) in the input context, which mimics how a real retrieval system would present results. Appendix C explores randomly ordering the distractors and finds the same U-shaped performance pattern, confirming that the effect is not an artifact of relevance ordering.

Addressing ambiguity. NaturalQuestions-Open contains some questions where multiple answers might be reasonable, meaning that a small number of "distractor" passages could incidentally contain a valid answer. To verify that this ambiguity does not drive the results, the authors also run experiments on a subset of unambiguous questions (using the ambiguity annotations from Min et al., 2020; AmbigQA), finding similar results and conclusions (Appendix A, Figure 12). They also test with fully random Wikipedia documents as distractors rather than retrieved ones (Appendix B, Figure 13); while absolute accuracy is higher in this easier setting (since the relevant document often has obvious lexical overlap with the query), the U-shaped performance curve persists, indicating that the positional degradation is not solely due to difficulty in distinguishing relevant from irrelevant documents.

Controlled manipulation of position. To modulate the position of relevant information within the input context, the authors adjust the order of the documents. The document containing the answer is moved to a specific target position while the $k-1$ distractor documents occupy the remaining positions. For example, in a 20-document setting, the relevant document might be placed at position 1 (very beginning), position 5, position 10 (middle), position 15, or position 20 (very end). The distractor documents fill the other slots in their default relevance order. This means that for each query, multiple prompt variants are constructed — one per target position — and each is evaluated independently.

Controlled manipulation of context length. To modulate the input context length independently of position, the number of retrieved distractor documents $k-1$ is varied. The paper experiments with $k = 10$ , $k = 20$ , and $k = 30$ total documents. Increasing $k$ strictly adds more distractor documents, which lengthens the input context but does not change the task: there is still exactly one relevant document containing the answer, and the answer itself does not change. This design separates the effect of total context length from the effect of relevant item position — if models were simply worse at longer contexts regardless of position, the curves would shift downward uniformly; if the positional pattern changes with length, the shape of the curve would differ across the three $k$ values.

Prompt formatting. The input context is formatted as a textual prompt with a task instruction, numbered documents with titles, the question, and a prefix for the answer. Figure 2 shows the exact format:

"Write a high-quality answer for the given question using only the provided search results (some of which might be irrelevant)."

Document [1](Title: Asian Americans in science and technology) Prize in physics for discovery of the subatomic particle J/ψ. Subrahmanyan Chandrasekhar shared...

Document [2](Title: List of Nobel laureates in Physics) The first Nobel Prize in Physics was awarded in 1901 to Wilhelm Conrad Röntgen, of Germany, who received...

...

Question: who got the first nobel prize in physics

Answer:

Each document is preceded by a header indicating its index and Wikipedia title. The question appears after all documents. This ordering — documents first, query last — is the standard format used throughout the main experiments, and it means that decoder-only models process all documents before they ever see the query tokens.

Token counts. The paper documents average and maximum token counts for each experimental setting in Appendix F (Tables 2 and 3). For the multi-document QA task:

10 documents: approximately 1,475–1,750 tokens on average (depending on tokenizer), maximum ~2,500 tokens.
20 documents: approximately 2,946–3,465 tokens on average, maximum ~4,955 tokens.
30 documents: approximately 4,419–5,182 tokens on average, maximum ~7,729 tokens.

These lengths are all well within the maximum context windows of the evaluated models (8K for GPT-3.5-Turbo, 16K for the extended variants, 8K for MPT-30B-Instruct, 16K for LongChat-13B), meaning that the models are not operating near their context limits where truncation artifacts might explain the results.

Evaluation metric. Following Kandpal et al. (2022) and Mallen et al. (2023), the primary metric is accuracy: whether any of the correct answers (from the NaturalQuestions annotations) appear as a substring in the model's generated output. This is a lenient metric — the model does not need to produce exactly the annotated answer string, only to include it somewhere in its response. The authors use the grading function released by the original NaturalQuestions work for answer matching.

Closed-book and oracle baselines. To contextualize multi-document QA performance, the paper reports two baselines for each model (Table 1):

Closed-book: the model receives only the question, with no documents in the input context. This measures how much the model can answer from its parametric memory alone. If a model's multi-document QA performance ever drops below its closed-book performance, it indicates that the presence of the documents is actively interfering with the model's ability to answer — the model would have done better ignoring the context entirely.
Oracle: the model receives the single document that contains the answer (no distractors), and must use it to answer the question. This represents an upper bound on what the model can achieve given the relevant document without any interference from distractor documents.

These baselines frame the multi-document QA performance curve between two extremes: the oracle performance when the model is given only the relevant document, and the closed-book performance when it is given no documents at all. Any accuracy between these two values reflects how well the model can extract the relevant information despite the presence of distractors, and any accuracy below closed-book performance indicates that the distractor documents are actively harmful.

Contriever retrieval details. The retriever used (Contriever, fine-tuned on MS-MARCO) is an unsupervised dense retrieval model based on contrastive learning. It encodes queries and documents into dense vector representations and ranks documents by cosine similarity to the query. The paper uses the standard Wikipedia dump from late 2018 as the retrieval corpus, consistent with prior work on NaturalQuestions-Open (Izacard and Grave, 2021). Each document is a chunk of at most 100 tokens from a Wikipedia article. This retrieval setup ensures that the distractor documents are realistically related to the query — they are not random noise but documents that a real retrieval system would surface as potentially relevant, making the task of identifying the truly relevant document among them non-trivial.

Synthetic Key-Value Retrieval Task Design

Motivation for a synthetic task. The multi-document QA task conflates two capabilities: (1) identifying which document is relevant among distractors, and (2) extracting and using the information from that document to produce an answer. If a model fails at multi-document QA, it is unclear whether the failure is in the identification step, the extraction step, or both. The synthetic key-value retrieval task is designed to strip away all semantic complexity and test only the model's ability to locate and copy an exact string match from its input context. If models fail at this minimal task, the failure cannot be attributed to semantic confusion, reasoning deficits, or ambiguity about what information is relevant — it must reflect a fundamental limitation in information access at certain positions.

Task definition. The model receives (1) a string-serialized JSON object containing $k$ key-value pairs, where all keys and values are unique, randomly-generated 128-bit UUIDs, and (2) a specific key from within that JSON object. The task is to return the value associated with that key. This is essentially an exact-match lookup: the correct answer is uniquely determined and appears verbatim in the input context, and no reasoning, inference, or semantic understanding is required beyond finding the matching key and copying its associated value.

Why UUIDs? The use of random UUIDs (e.g., "9f4a92b9-5f69-4725-ba1e-403f08dea695") eliminates any possibility that the model could answer from its parametric memory or use semantic cues to guess which key-value pair is relevant. Unlike natural language keys and values (e.g., {"country": "France"}), UUIDs have no semantic associations that a model could leverage — the only way to produce the correct value is to locate the exact key in the input context and copy the associated value. This makes the task a pure test of in-context retrieval. The paper explicitly notes that this design choice is motivated by a desire to "remove as much natural language semantics as possible" since "Transformer language models may have varying sensitivity to different linguistic features in their input" (O'Connor and Andreas, 2021) — by using UUIDs, they avoid any confound from variable sensitivity to named entities, common nouns, or other linguistic features.

Controlled manipulation of position. The position of the relevant key-value pair is modulated by inserting it at a specific index within the serialized JSON object. For example, in a 140-pair setting, the relevant pair might be placed at position 1, position 35, position 70, position 105, or position 140. All other key-value pairs are randomly generated distractors — keys and values that do not match the query key. The serialized JSON object is presented as a single block of text, and the query key is specified after the JSON object, mirroring the query-after-documents format used in multi-document QA.

Controlled manipulation of context length. Three settings are tested: 75 key-value pairs (~4K tokens on average), 140 pairs (~8K tokens), and 300 pairs (~16K tokens), with 500 examples generated for each setting. The token counts for each model are reported in Appendix F (Table 4), ranging from ~3,769 tokens (GPT-3.5-Turbo / Claude-1.3 at 75 pairs) to ~21,467 tokens (LongChat-13B at 300 pairs). Note that for the 300-pair setting specifically, LongChat-13B's average token count (21,467) exceeds its stated maximum context length of 16K tokens in some cases; however, the authors note that the model can handle sequences up to its maximum context length, and they filter or truncate as needed, though the exact handling for edge cases is not detailed.

Prompt formatting. Figure 6 shows an example prompt:

Extract the value corresponding to the specified key in the JSON object below.

JSON data: {"2a8d601d-...": "bb3ba2a5-...", "a54e2eed-...": "d1ff29be-...", ...}

Key: "9f4a92b9-5f69-4725-ba1e-403f08dea695"

Corresponding value:

The task instruction, JSON data, and query key are all formatted as text. The model must complete the "Corresponding value:" prefix with the correct UUID value.

Evaluation. Accuracy is measured by whether the correct value appears anywhere in the model's output. Since the values are unique UUIDs, there is no ambiguity in scoring — either the exact UUID string is present or it is not.

Comparison to prior synthetic retrieval tests. The paper acknowledges two related prior efforts: the Little Retrieval Test of Papailiopoulos et al. (2023) and the fine-grained line retrieval task of Li et al. (2023). The key distinction is that this paper uses random UUIDs rather than natural language passages or code lines, which the authors argue removes potential confounders from linguistic feature sensitivity and creates a cleaner signal about raw retrieval capability.

Model Evaluation Protocol

Model selection. The paper evaluates six primary language models spanning open-source and closed-source, standard-context and extended-context variants:

MPT-30B-Instruct: an open-source decoder-only model with an 8,192-token maximum context length. It was pre-trained on 1 trillion tokens using 2,048-token sequences, then underwent an additional sequence length adaptation pre-training phase on 50 billion tokens using 8,192-token sequences. It uses ALiBi (Press et al., 2022) for positional information — an approach that adds a linear bias to attention scores rather than using learned or sinusoidal positional embeddings, which theoretically enables better extrapolation to unseen sequence lengths.
LongChat-13B (16K): an open-source decoder-only model with a 16,384-token maximum context length. It extends the LLaMA-13B (Touvron et al., 2023a) context window from 2,048 to 16,384 tokens by using "condensed rotary positional embeddings" — a technique that adjusts the rotation frequencies of RoPE (Rotary Position Embedding) to compress the effective positional range, enabling longer sequences without retraining the base model from scratch — followed by fine-tuning on 16,384-token sequences (Li et al., 2023).
GPT-3.5-Turbo (gpt-3.5-turbo-0613): a closed-source decoder-only model from OpenAI with a 4,096-token maximum context length. Accessed via the OpenAI API.
GPT-3.5-Turbo (16K) (gpt-3.5-turbo-16k-0613): the extended-context variant of GPT-3.5-Turbo with a maximum context length of 16,384 tokens.
Claude-1.3: a closed-source decoder-only model from Anthropic with an 8K-token maximum context length. Accessed via the Anthropic API.
Claude-1.3 (100K): the extended-context variant of Claude-1.3 with a maximum context length of 100K tokens.

The pairing of standard and extended-context variants (GPT-3.5-Turbo vs. GPT-3.5-Turbo-16K; Claude-1.3 vs. Claude-1.3-100K) is deliberate: when both variants are evaluated on inputs that fit within the shorter variant's context window (e.g., 10 or 20 documents at ~2K–4K tokens, well within GPT-3.5-Turbo's 4K limit), any performance difference between them would indicate that the extended-context training changed something fundamental about how the model uses context. Conversely, identical performance would indicate that the extended-context training does not inherently improve information access within the shared context range.

Additional models for diagnostic analyses. Beyond the six primary models, the paper evaluates additional models for specific diagnostic comparisons (Section 4):

Flan-T5-XXL (Raffel et al., 2020; Chung et al., 2022): an encoder-decoder model trained with 512-token sequences. Uses relative positional embeddings, which theoretically allow extrapolation beyond the training-time sequence length.
Flan-UL2 (Tay et al., 2023): an encoder-decoder model initially trained with 512-token encoder and decoder sequences, then pre-trained for an extra 100K steps with 1,024-token sequences, before instruction fine-tuning on sequences with 2,048 tokens in the encoder and 512 tokens in the decoder. Like Flan-T5-XXL, it uses relative positional embeddings and can (in principle) extrapolate to longer sequences; Shaham et al. (2023) found both models perform well with sequences up to 8K tokens.
MPT-30B: the base pre-trained model (before instruction fine-tuning), used to isolate the effect of instruction tuning on positional biases.
Llama-2 models (Touvron et al., 2023b): at 7B, 13B, and 70B parameter scales, with and without supervised fine-tuning and RLHF ("-chat-hf" vs. "-hf" variants), used to study how model scale and fine-tuning interact with positional biases.
GPT-4 (8K): evaluated on a subset of 500 random multi-document QA examples with 20 documents, used as a point of comparison for the strongest available model.

Decoding strategy. All experiments use greedy decoding — at each generation step, the model selects the single most probable next token. The paper explicitly states this choice and leaves "exploration of other decoding methods to future work." Greedy decoding is deterministic, which eliminates sampling variance as a confound and makes the results reproducible. However, it also means that the reported accuracy is a lower bound on what could be achieved with temperature sampling and best-of-N selection, since greedy decoding cannot recover from an early mistake by exploring alternative paths.

Prompt standardization. Each model uses a standard set of prompts consistent across all experiments within a task. For multi-document QA, the prompt includes a system-like instruction ("Write a high-quality answer for the given question using only the provided search results (some of which might be irrelevant)"), numbered documents with titles, the question, and an "Answer:" prefix. For key-value retrieval, the prompt includes the task instruction, the JSON data, the query key, and a "Corresponding value:" prefix. The specific wording is held constant to avoid prompt-engineering confounds. Different models may receive slightly different formatting to match their expected input structure (e.g., OpenAI's chat format vs. raw text completion), but within each model, the prompt format is fixed across all position and length conditions.

Scoring protocol. Outputs are evaluated using exact substring matching: if any of the ground-truth answer strings appear anywhere in the model's generated text, the example is counted as correct. For NaturalQuestions, the ground-truth answers are the human-annotated answer strings (which may include multiple valid phrasings for the same answer). For key-value retrieval, the ground-truth value is the specific UUID associated with the queried key. This binary accuracy metric is then averaged across all examples at each position to produce the plotted performance curves.

Plotting conventions. Performance curves (Figures 5, 7–10, 12–16) plot accuracy on the y-axis against the position of the relevant item on the x-axis, with lower positions corresponding to the beginning of the input context and higher positions corresponding to the end. Multiple curves on the same plot represent different models or different experimental conditions. The key diagnostic pattern is the shape of each curve — flat curves indicate position-invariant performance (the desired behavior for robust context utilization), while U-shaped curves indicate degradation in the middle relative to the beginning and end.

Diagnostic Analysis Framework

Why diagnostics are needed. After establishing the existence of the U-shaped performance curve in both multi-document QA and key-value retrieval, the paper investigates what causes this pattern. Three hypotheses are tested through controlled comparisons:

Hypothesis 1: Architecture matters. Decoder-only models process input sequentially left-to-right, attending only to prior tokens at each timestep. Encoder-decoder models use a bidirectional encoder that can attend to all input tokens simultaneously. If the U-shaped curve is driven by the causal attention mask in decoder-only models, then encoder-decoder models should not exhibit it — at least on sequences within their training-time maximum length, where the bidirectional encoder has been optimized.

Experimental design for testing Hypothesis 1. Compare Flan-T5-XXL and Flan-UL2 (encoder-decoder) against MPT-30B-Instruct and LongChat-13B (decoder-only) on multi-document QA at 10, 20, and 30 documents. Because Flan-T5-XXL was trained on 512-token sequences and Flan-UL2 on up to 2,048-token encoder sequences, their behavior can be compared both within their training-time context window (10 documents, ~2K tokens) and outside it (20 and 30 documents, ~4K and ~6K tokens). If encoder-decoder models are flat within their training window but U-shaped outside it, this suggests that the effect is partly an extrapolation failure rather than something inherent to the architecture.

Hypothesis 2: Query-aware contextualization matters. In the standard prompt format (documents first, query last), decoder-only models cannot attend to query tokens when processing the documents, because the query has not yet been seen. This means document representations are context-independent with respect to the query — the model has no way to mark certain documents as potentially relevant while encoding them. If this lack of query-document interaction drives the U-shaped curve, then placing the query before the documents as well as after them (so the model sees the query first, then processes documents with query-aware attention, then sees the query again) should significantly improve performance.

Experimental design for testing Hypothesis 2. Run both multi-document QA (20 documents) and key-value retrieval (75, 140, 300 pairs) with query-aware contextualization: the query is placed both before and after the documents (or key-value pairs). In the key-value retrieval task, this means the prompt format becomes: "Key: X. JSON data: {...}. Key: X. Corresponding value:" — so the model sees the query key, then processes the JSON data with that key in its attention context, then sees the key again before generating the answer. If query-aware contextualization eliminates the U-shaped curve (especially in the simpler key-value task), it would indicate that the positional bias is largely driven by the temporal ordering of query vs. context processing in decoder-only models.

Hypothesis 3: Instruction fine-tuning affects positional biases. Instruction fine-tuning data commonly places the task specification at the beginning of the input context, which might inadvertently train models to pay more attention to the start of the context. If this "formatting prior" drives the primacy bias, then base models (without instruction tuning) should show only recency bias (the natural tendency to attend to recent tokens observed in pre-Transformer LMs and early Transformers), without the primacy peak. Conversely, if base models already show a U-shaped curve, then the primacy bias arises from pre-training data patterns (e.g., internet text formatting where important information often appears at the start) rather than from instruction tuning specifically.

Experimental design for testing Hypothesis 3. Compare MPT-30B (base) against MPT-30B-Instruct (instruction-tuned) on multi-document QA with 20 documents. If instruction tuning introduces the primacy bias, the base model should show only recency bias while the instruct model shows both. If both show both, instruction tuning is not the primary cause. Additionally, compare Llama-2-7B, Llama-2-13B, and Llama-2-70B against their instruction-tuned counterparts (Llama-2-7B-chat, etc.) on multi-document QA with 20 documents. This multi-scale comparison tests whether model size interacts with positional biases — smaller models might only have capacity for recency bias, while larger models develop primacy bias as they learn more complex patterns from pre-training data.

Controlled variables across diagnostics. In all diagnostic experiments, the same 2,655 queries are used, the same document sourcing (NaturalQuestions annotations for relevant documents, Contriever retrieval for distractors), and the same evaluation metric (accuracy via substring matching). The only manipulations are the model variant, the prompt format (query-aware vs. standard), or the number of documents, isolating the cause of observed differences.

Open-Domain QA Case Study Design

Motivation. The controlled experiments in Sections 2–4 show that model performance degrades when relevant information is in the middle of the input context. But in a real deployment, the position of relevant information is not controlled — a retriever returns $k$ documents in ranked order, and the relevant document(s) could appear anywhere in that ranking (or not at all). The open-domain QA case study connects the controlled findings to this realistic setting by asking: as we provide a model with more retrieved documents, does performance improve, saturate, or degrade?

Experimental setup. This is a standard retriever-reader architecture:

Retriever: Contriever (fine-tuned on MS-MARCO, same as used for distractor selection in the controlled experiments) takes an input query from NaturalQuestions-Open and retrieves the $k$ most relevant documents from Wikipedia. Unlike the controlled experiments, there is no guarantee that the top- $k$ documents actually contain the answer — zero, one, or multiple documents might contain it.
Reader: The language model receives the query and the $k$ retrieved documents formatted in a single prompt (using the same format as the multi-document QA task) and generates an answer.
Varying $k$ : The number of retrieved documents is swept across $k = 5, 10, 20, 30, 40, 50$ . For each $k$ , both retriever recall (whether the answer-containing document appears anywhere in the top $k$ ) and reader accuracy (whether the model produces the correct answer) are measured.

Metrics. Two metrics are tracked as a function of $k$ :

Retriever recall: the fraction of queries for which at least one document containing the correct answer appears in the top- $k$ retrieved set. This measures the upper bound of what the reader could achieve if it could perfectly identify and use the relevant document(s) — it answers the question "does the answer exist somewhere in the provided context?"
Reader accuracy: the fraction of queries for which the model's generated output contains the correct answer. This measures actual end-to-end performance and can be directly compared to retriever recall to assess how much of the available information the model is actually using.

The key diagnostic pattern. If models could effectively use all provided context, reader accuracy would closely track retriever recall — as more documents are retrieved, recall increases, and accuracy should increase proportionally. If models cannot use additional context effectively, reader accuracy will saturate (stop improving) long before retriever recall saturates. The gap between recall and accuracy at a given $k$ quantifies how much potentially useful information the model is failing to utilize.

The paper further notes that effective re-ranking (pushing relevant documents closer to the start of the context, where models are better at accessing them) or ranked list truncation (retrieving fewer documents when the marginal benefit is low) are natural practical implications of the U-shaped finding: if position matters, then system designers should invest in ordering retrieved documents so that the most likely relevant ones appear at positions where the model can best use them.

4. Key Insights and Innovations

Innovation 1: Position Is a First-Class Variable for Evaluating Context Utilization — The U-Shaped Curve as a Diagnostic

The paper's most fundamental intellectual contribution is not the observation that models struggle with long contexts — prior work had already documented recency bias and disappointing long-range utilization (Khandelwal et al., 2018; Sun et al., 2021). Rather, it is the discovery of a specific, systematic functional form — a U-shaped performance curve — that characterizes how performance degrades as a function of where relevant information appears in the input, and the elevation of this curve into a diagnostic protocol for evaluating long-context models.

What came before. Prior work characterized positional effects coarsely, if at all. Khandelwal et al. (2018) and Daniluk et al. (2017) documented recency bias — models attending mainly to tokens near the end of the context — in small LSTM language models. Sun et al. (2021) found that long-range Transformers benefited from context primarily near the beginning. Ivgi et al. (2023) tested only two positions (beginning vs. random) and found beginning placement was advantageous for encoder-decoder models, which would be consistent with either a monotonic decay or a U-shape — their experimental design could not distinguish between them. Qin et al. (2023) tested efficient Transformers on downstream NLP tasks and found they were "recency-biased," characterizing only the right half of the U.

No prior work had sampled position at sufficient granularity to reveal the full U-shaped pattern — a performance peak at the beginning (primacy bias), a trough in the middle, and a second peak at the end (recency bias). This shape is non-obvious: a naive model that simply cannot handle long contexts would show monotonic decay with distance from some reference point, not a recovery at the end. A model that handled all positions equally would show a flat line. The U-shape reveals something specific about how attention and training dynamics interact with position: the model is not uniformly worse at "far" tokens; it is specifically worse at tokens that are neither first nor last, even when they are physically closer to the query than tokens at the very end.

Why this is intellectually distinctive. The U-shaped curve transforms the question of long-context evaluation from a scalar ("can the model handle X tokens?") to a shape requirement ("does the model's performance curve flatten as a function of position?"). This reframing has immediate practical teeth: a model claiming 100K context support should demonstrate not just non-zero accuracy at position 50K, but accuracy comparable to position 1 and position 100K. The difference between best-case and worst-case performance across positions becomes a single-number robustness metric, independent of absolute accuracy. The paper doesn't just report a finding — it provides the intellectual framework for making that finding actionable as an evaluation protocol that the field can adopt.

The paper explicitly makes this diagnostic move: "to claim that a language model can robustly use information within long input contexts, it is necessary to show that its performance is minimally affected by the position of the relevant information in the input context." This is a new criterion for what counts as robust context utilization, not merely a new benchmark.

Evidence anchor. The U-shape is demonstrated across all six primary models in multi-document QA (Figure 5), with GPT-3.5-Turbo dropping from ~76% at position 1 to ~54% at position 10 in the 20-document setting — a >20 percentage point swing — before recovering to ~63% at position 20. The pattern replicates in the synthetic key-value retrieval task (Figure 7), where GPT-3.5-Turbo (16K) drops from near-perfect at the extremes to ~46% at the middle in the 300-pair setting, despite the task requiring only exact string matching. That the pattern appears in a task stripped of all semantic complexity confirms it is a fundamental information-access limitation, not an artifact of reasoning difficulty.

Fundamental vs. incremental. This is a fundamental conceptual contribution — a new way of thinking about and measuring context utilization — not an incremental improvement in evaluation methodology.

Innovation 2: The Distinction Between Context Window Size and Context Utilization — Larger Windows Do Not Imply Better Use

The paper delivers a sharp empirical refutation of an implicit assumption that had been gaining currency in the field: that extending a model's maximum context length (from 4K to 16K, 8K to 100K, etc.) inherently improves its ability to use information throughout that context. By comparing model pairs where an extended-context variant exists alongside a standard-context variant, the paper shows that context window size and context utilization are largely independent properties — a finding with significant implications for how the field develops and markets long-context models.

What came before. The dominant narrative around long-context models was driven by the technical achievement of scaling context windows. Advancements in efficient attention (Dai et al., 2019; Dao et al., 2022; Poli et al., 2023), positional encoding extrapolation (Press et al., 2022), and sequence length adaptation pre-training created a competitive dynamic where models were announced with increasingly large context windows — 32K, 100K, and beyond. The natural (but untested) implication was that these models could effectively use those longer contexts. Evaluation often defaulted to perplexity on long documents, which does not directly measure whether a model can retrieve and apply specific information from arbitrary positions.

The paper's experimental design creates a clean test: evaluate both the standard and extended-context variants of the same model family on inputs that fit within the shorter model's context window. If the extended-context training meaningfully improved how the model attends across positions, the extended variant should outperform the standard variant even on these short-enough inputs. If it did not — if the curves are "nearly superimposed" (as the paper repeatedly reports) — then the extended-context training increased capacity without improving utilization.

What the evidence shows. For GPT-3.5-Turbo vs. GPT-3.5-Turbo (16K) on multi-document QA with 10 and 20 documents (Figure 5, left and center panels): the performance curves are nearly identical — the solid purple (standard) and dashed brown (16K) lines track each other tightly at every position. The same pattern holds for Claude-1.3 vs. Claude-1.3 (100K) on 10 and 20 documents. Even on 30 documents, which exceeds GPT-3.5-Turbo's 4K window but fits within the 16K variant, the 16K variant's performance on the early and middle positions (where the standard variant can be evaluated) is consistent with what we would expect from extrapolating the standard variant's curve.

Why this matters beyond raw performance. This finding reframes the narrative from "bigger context windows" to "better context utilization" as the relevant axis of progress. It implies that the hard problem is not supporting longer sequences architecturally — which has been largely solved through efficient attention mechanisms and positional encoding adaptations — but rather training models to use the context they already have. A model with an 8K context window that uses all 8K tokens effectively may be more useful in practice than a model with a 100K context window that can only reliably access the first and last few thousand tokens. For practitioners, this finding provides a concrete evaluation criterion: test models on position-controlled setups before trusting their long-context claims, and do not assume that upgrading to an extended-context variant will improve performance on tasks where the context already fits.

Fundamental vs. incremental. This is in part a negative result — the absence of a difference where one was expected — but it is a conceptually significant negative result because it challenges an emerging conventional wisdom and redirects attention from the solved problem (architectural context scaling) to the unsolved one (training for position-invariant access). It is a fundamental empirical insight rather than an incremental contribution, because it changes what the field should optimize for.

Evidence anchor. Figure 5, comparing GPT-3.5-Turbo and GPT-3.5-Turbo (16K) curves in the 10-document and 20-document subplots; Table 1, where closed-book and oracle performance are nearly identical between standard and extended variants; and the identical performance of Claude-1.3 and Claude-1.3 (100K) on the 75-pair key-value retrieval setting (Figure 7, left panel), where Claude-1.3 already achieves near-perfect accuracy, showing that the 100K variant adds nothing when the standard variant already handles the context length.

Innovation 3: The Encoder-Decoder Advantage Within Training Distribution — And Its Fragility Out-of-Distribution

The paper does not merely compare decoder-only models against each other; it systematically investigates whether the bidirectional encoder in encoder-decoder architectures provides robustness to positional variation — and discovers that this advantage is bounded by the training-time sequence length. Within their training distribution, encoder-decoder models achieve near-flat performance curves; outside it, they exhibit the same U-shaped degradation as decoder-only models. This finding is important not because it advocates for encoder-decoder models (most frontier models today are decoder-only) but because it identifies the mechanism of the advantage — bidirectional attention during input processing — and simultaneously reveals its limitations, pointing to out-of-distribution sequence lengths as an underappreciated failure mode.

What came before. Ivgi et al. (2023) had noted that encoder-decoder models performed better when relevant information was at the input start. The natural interpretation might be that encoder-decoders are simply "better at long contexts." The paper's finer-grained analysis complicates this picture substantially. By testing Flan-UL2 at three context lengths that span both within and beyond its 2,048-token training-time encoder window, the paper reveals a qualitative change in behavior at the boundary.

The evidence pattern. In the 10-document setting (~2K tokens, within Flan-UL2's training window), Flan-UL2's performance curve is nearly flat — only a 1.9% absolute difference between best and worst position (Figure 8, left). This is the desired behavior: position-invariant access. Move to 20 documents (~4K tokens, 2× beyond the training window), and a U-shape emerges. Move to 30 documents (~6K tokens), and the U-shape deepens. Flan-T5-XXL, with an even smaller training window (512 tokens), shows a monotonic degradation: the U-shape is visible in all three settings, and it becomes more pronounced as context length increases.

What this tells us about the mechanism. The contrast between the within-distribution flatness and out-of-distribution U-shape strongly implicates bidirectional attention as the protective factor — but only when the positional encoding scheme has been trained on sequences of that length. When encoder-decoder models operate within their trained sequence lengths, the encoder can attend bidirectionally across all tokens, allowing it to contextualize each document in the context of both earlier and later documents. This enables the model to compute relative importance across the entire context, making all positions equally accessible. When the sequence length exceeds the training distribution, even relative positional embeddings (which theoretically support extrapolation) fail to maintain this flatness — suggesting that the issue is not merely architectural but lies in how positional information is represented and generalized.

Why this is a conceptual advance. This finding provides a mechanism-level hypothesis for the U-shaped curve: it is not that attention cannot reach certain positions, but that the representations of tokens at different positions have varying quality due to how positional information interacts with training length. It also identifies a practical vulnerability for encoder-decoder models in long-context settings — they may appear robust in benchmarks that match their training lengths, but degrade unexpectedly when deployed on longer inputs. For the field, this suggests that architectural comparisons between encoder-decoder and decoder-only models must control for sequence length relative to training distribution, not just absolute context window size.

Fundamental vs. incremental. This is a fundamental insight about the interaction between architecture, positional encoding, and training distribution that substantially refines the prior understanding (from Ivgi et al., 2023) that encoder-decoders are simply "better" at using context. The finding that the advantage is conditional — present within training distribution, absent beyond it — is a novel empirical contribution that changes how one should think about model selection for long-context tasks.

Evidence anchor. Figure 8, comparing Flan-UL2's flat curve in the 10-document subplot (left) against its U-shaped curves in the 20-document (center) and 30-document (right) subplots; Flan-T5-XXL's monotonic degradation across all three settings.

Innovation 4: Query-Aware Contextualization as a Targeted Intervention That Partially Decomposes the Problem

The paper's investigation of query-aware contextualization — placing the query both before and after the documents rather than only after — serves a dual intellectual purpose. First, it provides an ablation that isolates one specific mechanism contributing to positional bias in decoder-only models: the inability to attend to query tokens during document processing when the query appears only at the end. Second, it reveals that fixing this mechanism is sufficient to solve the synthetic retrieval task but insufficient for the naturalistic QA task, thereby decomposing the overall U-shaped phenomenon into (at least) two distinct underlying problems.

The mechanistic insight. In the standard prompt format (documents first, query last), decoder-only models process all documents through causal self-attention before ever seeing the query. This means document representations are computed without any query-driven gating or relevance marking — the model cannot, for example, up-weight the representation of a document that mentions the query's key entity, because it does not yet know what that entity is. When the query finally appears, the model can attend back to all document tokens (causal masking applies to generation, not to attention over the input context), but the document representations themselves were formed query-blind. Query-aware contextualization fixes this by placing the query first: the model reads "Question: who got the first nobel prize in physics," then processes all documents with those query tokens in its attention context, then sees the question again. This enables query-document interaction during encoding.

The decompositional finding. In the key-value retrieval task (Figure 7 vs. the reported results in Section 4.2), query-aware contextualization produces near-perfect accuracy for all models at all positions and context lengths — the U-shaped curve collapses to a flat line at ~100%. This makes sense: the query is a specific key (a UUID), and when the model sees that key before processing the JSON data, it can simply look for its match during encoding. Exact match retrieval with query preview is trivial.

In multi-document QA (Figure 9 vs. the standard format in Figure 5, center), query-aware contextualization produces only marginal changes: a slight improvement when the relevant document is at the very beginning, slight degradation at other positions, and the U-shape persists. This tells us that the multi-document QA bottleneck is not primarily a retrieval failure — it is not that the model cannot find the relevant document when it knows what to look for. Rather, even when query-aware encoding should help identify relevance, the model still struggles to use the identified information, especially in the middle positions. The decomposition implies at least two distinct failure modes: (1) a query-blind encoding problem that affects pure retrieval tasks, which query-aware contextualization solves, and (2) a more fundamental information-utilization problem for reasoning tasks, which query-aware contextualization does not solve and which remains positional.

Why this matters. Prior work on prompt engineering for retrieval-augmented generation has explored various ordering strategies, but without a mechanistic understanding of why certain orderings work. The query-aware contextualization experiment provides that mechanistic understanding for one specific strategy and, equally importantly, demonstrates its limits. It tells practitioners: if your task is pure retrieval (find the matching value), put the query first — that solves the problem. If your task requires reasoning over retrieved content, query-first prompting is not a sufficient fix, and you need other strategies (such as positioning the most critical documents at the beginning or end, as the U-shape finding implies).

Fundamental vs. incremental. This is a conceptually important decomposition that advances mechanistic understanding of the positional bias phenomenon. While query-aware contextualization as a prompting strategy is incremental (it is a straightforward prompt engineering intervention), the analysis it enables — separating retrieval failures from utilization failures — is a fundamental contribution to understanding the nature of the limitation.

Evidence anchor. Section 4.2 text reporting near-perfect key-value retrieval performance with query-aware contextualization (e.g., GPT-3.5-Turbo (16K) achieving perfect accuracy on 300 key-value pairs, vs. 45.6% worst-case without it); Figure 9 demonstrating the persistence of the U-shape in multi-document QA with query-aware contextualization.

Innovation 5: The Retrieval Saturation Phenomenon — More Documents Cease to Help Long Before They Cease to Be Relevant

The open-domain QA case study (Section 5, Figure 11) crystallizes the practical consequence of the U-shaped curve for real-world system design: reader accuracy saturates long before retriever recall saturates. As more documents are retrieved, the probability that the answer exists somewhere in the context continues to rise, but the model's ability to actually use that answer plateaus and then barely improves. This finding transforms the tradeoff around retrieval depth from a theoretical concern into an empirically quantified design principle with direct cost and latency implications.

What came before. The standard retriever-reader paradigm (e.g., Izacard and Grave, 2021; Petroni et al., 2020; Shi et al., 2023) had largely treated retrieval depth as a hyperparameter to be tuned for accuracy, with the assumption that retrieving more documents could only help (modulo computational cost). Some prior work had found diminishing returns — readers do not improve linearly with recall — but without a mechanistic explanation linking this to positional biases. The U-shaped curve provides that mechanism: as more documents are added to the context, the relevant document (if present) is increasingly likely to be pushed into the middle of the context, where model access is weakest. Even if the relevant document appears in the top 50, if it sits at position 25, the model may effectively ignore it. Adding more documents increases recall but also increases the probability that any given relevant document occupies a disadvantageous position.

The quantitative evidence. Figure 11 shows that retriever recall improves substantially from ~70% at 5 documents to ~87% at 50 documents — a 17 percentage point gain in answer coverage. Reader accuracy for GPT-3.5-Turbo (16K) improves from ~58% at 5 documents to ~65% at 20 documents, but then crawls to ~66% at 50 documents — a gain of only ~1% for the additional 30 documents. The pattern is similar for Claude-1.3. The gap between recall (~87%) and GPT-3.5-Turbo (16K) accuracy (~66%) at 50 documents is ~21 percentage points — representing information that is literally present in the input context but inaccessible to the model.

Practical implications. This finding reframes retrieval depth as a saturation-limited resource rather than a monotonic accuracy lever, with the saturation point determined not by the retriever but by the reader's positional robustness. It motivates two concrete strategies that the paper explicitly names: (1) effective re-ranking that pushes the most likely relevant documents toward the start of the context (where the primacy peak makes them most usable), and (2) ranked list truncation — retrieving fewer documents when the marginal accuracy gain is near zero, since additional documents add latency, cost, and the risk of pushing relevant information into the middle without compensating accuracy benefits. These strategies are not new inventions (re-ranking and truncation have long histories in information retrieval; Arampatzis et al., 2009), but the paper's contribution is connecting them to a specific, empirically characterized mechanism — the U-shaped positional sensitivity — that explains why they should work and how much they might help.

Fundamental vs. incremental. The saturation finding itself is an incremental contribution — diminishing returns to retrieval depth have been observed before. The intellectual contribution is tying this phenomenon to the U-shaped positional bias identified in the controlled experiments, thereby providing a mechanistic explanation for why saturation occurs (documents in disadvantageous positions) and connecting the controlled and ecologically valid settings in a unified framework. This transforms the finding from an empirical observation about one dataset into a general prediction about any long-context reader system: accuracy will saturate when additional retrieved documents predominantly land in the middle of the context, and systems that can route information to positional "sweet spots" can push the saturation point outward.

Evidence anchor. Figure 11, showing the divergence between Contriever recall (continuing to rise from 20 to 50 documents) and reader accuracy (flattening after ~20 documents); the reported marginal gains of ~1.5% for GPT-3.5-Turbo and ~1% for Claude-1.3 when expanding from 20 to 50 retrieved documents.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary dataset for the multi-document question answering task is NaturalQuestions-Open (Lee et al., 2019; Kwiatkowski et al., 2019), containing historical Google search queries paired with human-annotated answers extracted from Wikipedia. The authors select 2,655 queries where the annotated long answer is a paragraph (not a list or table). For each query, the relevant document is the Wikipedia paragraph identified by annotators as containing the answer. Distractor documents are retrieved from a late-2018 Wikipedia dump using Contriever fine-tuned on MS-MARCO (Izacard et al., 2021), with the constraint that they do not contain any NaturalQuestions-annotated answers. The synthetic key-value retrieval task uses randomly generated 128-bit UUIDs as keys and values — 500 unique test examples are created for each of the 75, 140, and 300 key-value pair settings. For the open-domain QA case study, the same NaturalQuestions-Open subset is used with Contriever retrieving the top $k$ documents from Wikipedia.
Base model(s). The paper evaluates six primary language models spanning open and closed families: MPT-30B-Instruct (8K context, decoder-only, ALiBi positional encoding), LongChat-13B (16K) (16K context, decoder-only, condensed rotary positional embeddings extending LLaMA-13B), GPT-3.5-Turbo-0613 (4K context, decoder-only, OpenAI API), GPT-3.5-Turbo-16k-0613 (16K context), Claude-1.3 (8K context, decoder-only, Anthropic API), and Claude-1.3 (100K) (100K context). Diagnostic analyses additionally evaluate Flan-T5-XXL (encoder-decoder, 512-token training sequences), Flan-UL2 (encoder-decoder, up to 2,048-token training encoder sequences), MPT-30B (base model without instruction tuning), Llama-2 at 7B, 13B, and 70B scales with and without chat fine-tuning, and GPT-4 (8K) on a 500-example subset. The models are chosen to span open vs. closed, decoder-only vs. encoder-decoder, standard vs. extended context windows, and base vs. instruction-tuned variants.
Metrics. The primary metric is accuracy, measured as the fraction of examples where any of the ground-truth answer strings appear as a substring in the model's generated output. For multi-document QA, this uses the original NaturalQuestions grading function; for key-value retrieval, this checks whether the correct UUID value appears anywhere in the output. For the open-domain QA case study, retriever recall (whether the answer-containing document appears in the top- $k$ retrieved set) and reader accuracy (whether the model produces the correct answer) are tracked separately. All models use greedy decoding.
Baselines. The paper defines two baselines for contextualizing multi-document QA performance (Table 1): closed-book (the model receives only the question with no documents, measuring parametric memory alone) and oracle (the model receives the single document containing the answer with no distractors, measuring the upper bound of what the model can extract when relevance is trivial). For the key-value retrieval task, the baseline is implicit — performance at the extremes (first and last positions) serves as a reference against which middle-position performance is compared. For the open-domain QA case study, Contriever recall serves as the upper-bound baseline against which reader accuracy is evaluated.
Generation budget / compute accounting. All experiments use greedy decoding across all models, so compute is not explicitly budgeted or compared — the paper does not vary sampling budget or number of generations. Context length is measured in number of items (documents or key-value pairs) and translated to approximate token counts per model tokenizer (Appendix F, Tables 2–4). For multi-document QA, contexts contain 10, 20, or 30 total documents (~1,475–5,182 tokens on average across models). For key-value retrieval, contexts contain 75, 140, or 300 key-value pairs (~3,769–21,467 tokens). Latency and cost are discussed qualitatively in the open-domain QA case study (retrieving 50 vs. 20 documents increases cost with marginal accuracy gain) but are not formally accounted.
Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. Results are reported as point estimates (accuracy percentages) averaged over 2,655 test examples for multi-document QA and 500 examples for key-value retrieval. The two-fold cross-validation protocol referenced in the reference example's format is absent here — the paper's diagnostic comparisons (e.g., position curves) rely on visual comparison of performance curves across the full test set rather than statistical testing. For the GPT-4 evaluation, a random subset of 500 examples is used rather than the full 2,655, to manage API costs reported as exceeding $6,000 for a full evaluation.

Main Quantitative Results

Multi-Document Question Answering: The U-Shaped Performance Curve

The headline finding across all decoder-only models is a U-shaped performance curve: accuracy is highest when the document containing the answer appears at the very beginning (position 1) or very end (position $k$ ) of the input context, and degrades substantially — in some cases below closed-book performance — when the relevant document is in the middle.

Figure 5 presents performance for all six primary models at 10, 20, and 30 total documents as a function of the answer-document's position. At 20 total documents:

GPT-3.5-Turbo achieves 75.8% accuracy when the relevant document is at position 1 (first), drops to a low of 53.8% at position 10 (middle), and recovers to 63.2% at position 20 (last) (Table 6). The 20+ percentage point swing between best and worst position is the largest among evaluated models. Notably, the middle-position accuracy (53.8%) falls below the model's closed-book accuracy of 56.1% (Table 1) — meaning the model would have performed better if given no documents at all than when the relevant information is buried mid-context.
GPT-3.5-Turbo (16K) shows a nearly identical curve: 75.7% at position 1, 54.1% at position 10, 63.1% at position 20. The extended-context variant's performance at every position is within 0.3 percentage points of the standard variant, despite the fact that both fit well within either model's context window (~4K tokens for 20 documents). This is direct evidence that the extended-context training did not improve information access within the shared context range.
Claude-1.3 and Claude-1.3 (100K) show a shallower but still pronounced U-shape: 59.9% → 55.9% → 60.1% (standard) and 59.8% → 55.9% → 60.0% (100K). The 4-point swing is smaller than GPT-3.5-Turbo's but still systematic. The two variants are again nearly identical at all positions.
MPT-30B-Instruct shows the smallest swing: 53.7% at position 1, a trough of 51.8% at position 5, recovering to 56.3% at position 20. The U-shape is present but compressed compared to other models, with only a ~4.5 percentage point range. The recency peak slightly exceeds the primacy peak.
LongChat-13B (16K) shows a monotonically decreasing pattern rather than a symmetric U-shape: 68.6% at position 1, declining steadily to 52.5% at position 15, with only a marginal recovery to 55.0% at position 20. This is more consistent with a strong primacy bias and weak recency bias, though the final point does tick up slightly.

At 10 total documents (Table 5), the U-shape is less pronounced because the context is shorter (~1.5K–1.7K tokens) and there are fewer intermediate positions. GPT-3.5-Turbo achieves 76.8% at position 1 with a trough of 61.2% at position 5 (~6-point swing), while LongChat-13B drops from 72.1% to 58.5%. At 30 total documents (Table 7), GPT-3.5-Turbo (16K) — the only GPT-3.5 variant that can accommodate 30 documents — shows a deeper U-shape: 73.4% at position 1, declining to 50.5% at position 10 (the lowest accuracy observed for this model), recovering to 63.7% at position 30.

Table 1 contextualizes these results: oracle performance (single relevant document, no distractors) ranges from 76.1% (Claude-1.3) to 88.6% (GPT-3.5-Turbo-16K), while closed-book performance ranges from 31.5% (MPT-30B-Instruct) to 56.1% (GPT-3.5-Turbo). The fact that GPT-3.5-Turbo's middle-position accuracy in 20-document settings (53.8%) dips below its closed-book accuracy (56.1%) is a stark indicator that distractor documents actively interfere with the model's ability to access relevant information — this is not merely a retrieval failure but an interference effect where the presence of additional documents reduces accuracy below what the model achieves from parametric knowledge alone.

Synthetic Key-Value Retrieval: Persistent Positional Effects on a Trivial Task

Figure 7 presents key-value retrieval performance at 75, 140, and 300 key-value pairs. The central finding is that even on a task requiring only exact string matching within the input context, most models exhibit a U-shaped performance curve — the phenomenon is not specific to semantic reasoning tasks.

At 75 key-value pairs (~3.8K–5.4K tokens):

Claude-1.3 and Claude-1.3 (100K) achieve near-perfect accuracy (~100%) at all positions.
GPT-3.5-Turbo and GPT-3.5-Turbo (16K) show a pronounced U-shape: near-perfect at the first and last positions (~95–100%), but accuracy drops to approximately 70–80% at middle positions (around the 25th–50th key-value pairs).
MPT-30B-Instruct shows a similar U-shape with accuracy dropping from ~95% at extremes to approximately 60% in the middle.

At 140 key-value pairs (~7K–10K tokens):

Claude variants remain near-perfect.
GPT-3.5-Turbo (16K) performance at the extremes remains high (~90–95%), but the trough deepens to approximately 60% in the middle. GPT-3.5-Turbo (standard, 4K window) cannot be evaluated at this length.
MPT-30B-Instruct falls to approximately 45–50% accuracy in the middle.
LongChat-13B (16K) exhibits a different pattern: when relevant information is at the start, the model tends to generate code to retrieve the key rather than outputting the value directly (a qualitative observation the authors note), resulting in lower accuracy at the beginning and higher accuracy at later positions.

At 300 key-value pairs (~15K–21K tokens):

Claude-1.3 cannot accommodate this length (8K window limit), but Claude-1.3 (100K) remains near-perfect.
GPT-3.5-Turbo (16K) shows the deepest U-shape: near-perfect at the first position (~95%), dropping precipitously to approximately 45.6% at the middle (around position 150), then recovering to near-perfect at the final position. This is the clearest demonstration of the paper's thesis: on a task where the correct answer is literally a string copy from the input, the model fails to retrieve it more than half the time when it sits in the middle of a long context.
MPT-30B-Instruct shows low accuracy across all positions (~40–50%), with minimal positional variation — the model simply struggles with this very long context regardless of position.

The paper explicitly notes that Claude-1.3 and Claude-1.3 (100K) performing near-perfectly on all settings demonstrates that the task is solvable with current technology — the failure modes of other models are not inherent to the task design.

Extended-Context Models Do Not Improve Within-Shared-Window Performance

A consistent pattern across both tasks is the near-identity of performance curves between standard and extended-context variants when evaluated on inputs that fit within both models' context windows.

In multi-document QA (Figure 5): the purple solid (GPT-3.5-Turbo, 4K) and brown dashed (GPT-3.5-Turbo-16K, 16K) lines are "nearly superimposed" for the 10-document and 20-document settings. The same holds for Claude-1.3 vs. Claude-1.3 (100K) at 10 and 20 documents. In key-value retrieval (Figure 7), Claude-1.3 and Claude-1.3 (100K) are identical at 75 pairs; GPT-3.5-Turbo and GPT-3.5-Turbo (16K) are nearly identical at 75 pairs.

Table 1 reinforces this: closed-book performance for GPT-3.5-Turbo vs. GPT-3.5-Turbo (16K) differs by 0.1 percentage points (56.1% vs. 56.0%), and oracle performance differs by 0.3 points (88.3% vs. 88.6%). Claude-1.3 vs. Claude-1.3 (100K) shows identical closed-book (48.3% vs. 48.2%) and near-identical oracle (76.1% vs. 76.4%) performance. This confirms that the extended-context variants are not fundamentally different models — the extension procedure (whether condensed rotary embeddings, sequence length adaptation pre-training, or whatever Anthropic uses for Claude) did not meaningfully alter the model's behavior on inputs it could already handle.

Open-Domain QA Case Study: Reader Accuracy Saturates Long Before Retriever Recall

Figure 11 presents retriever recall and reader accuracy as a function of the number of retrieved documents $k$ for NaturalQuestions-Open. The key finding is a growing divergence between what the retriever provides and what the reader can use.

Retriever recall (measured by Contriever fine-tuned on MS-MARCO) improves substantially with $k$ : from approximately 60% at 5 documents to approximately 80% at 20 documents, and further to approximately 87% at 50 documents. In contrast, reader accuracy for GPT-3.5-Turbo (16K) improves from approximately 58% at 5 documents to approximately 65% at 20 documents, but then stalls: accuracy at 50 documents is approximately 66%, a gain of only ~1.5% for the 30 additional documents beyond 20. Claude-1.3's accuracy similarly saturates, gaining only ~1% from 20 to 50 documents. By 50 documents, the gap between retriever recall (~87%) and GPT-3.5-Turbo (16K) reader accuracy (~66%) is approximately 21 percentage points — this represents information that is physically present in the input context but effectively invisible to the model.

The paper explicitly interprets these results in light of the U-shaped curve: "Using more than 20 retrieved documents only marginally improves reader performance… while significantly increasing the input context length (and thus latency and cost)." The implication is that additional documents push relevant information into the middle of the context where models struggle to access it, so the marginal benefit of deeper retrieval is near zero despite the retriever successfully surfacing more relevant material.

Ablation Studies and Robustness Checks

Ambiguity in distractor documents (Appendix A, Figure 12): Because NaturalQuestions-Open has some temporal mismatch with the Wikipedia retrieval corpus, a small number of distractor passages may incidentally contain a valid answer. The authors replicate multi-document QA experiments on an unambiguous subset (using AmbigQA annotations from Min et al., 2020), finding the same U-shaped performance curves for all models, indicating that the positional degradation is not an artifact of ambiguous distractors.

Random distractors vs. retrieved distractors (Appendix B, Figure 13): Replacing Contriever-retrieved distractors with random Wikipedia documents makes the task easier (absolute accuracy is uniformly higher since the relevant document often has obvious lexical overlap with the query), but the U-shaped curve persists. GPT-3.5-Turbo accuracy still drops from ~78% at position 1 to ~70% in the middle, recovering to ~75% at position 20. This demonstrates that the positional degradation is not solely due to the difficulty of distinguishing relevant documents from hard negatives — even when the relevant document is trivially identifiable by surface features, models still struggle to extract information from it when placed mid-context.

Randomizing distractor order (Appendix C, Figure 14): The default prompt presents distractors in order of decreasing relevance, which might create a prior that early documents are more useful. Testing with randomly shuffled distractors and a prompt explicitly stating "The search results are ordered randomly" shows a persistent but slightly modified U-shape. Randomization slightly decreases performance at the very beginning (since the most relevant distractors no longer cluster near the start) and slightly increases performance in the middle and end positions, but the overall U-shape remains clearly present. This rules out the hypothesis that the primacy effect is purely an artifact of relevance-ordered presentation.

Model architecture: encoder-decoder vs. decoder-only (Section 4.1, Figure 8): Flan-UL2 and Flan-T5-XXL (both encoder-decoder models with relative positional embeddings) are evaluated on multi-document QA. Within their training-time sequence length (10 documents, ~2K tokens, for Flan-UL2), Flan-UL2 shows a remarkably flat performance curve with only 1.9% absolute difference between best and worst position. In contrast, at 20 and 30 documents (beyond its 2,048-token training window), Flan-UL2 develops a clear U-shape, with performance degrading in the middle. Flan-T5-XXL (trained on 512-token sequences) shows U-shaped degradation across all three settings, with the U-shape deepening as context length increases. This ablation demonstrates that bidirectional encoding enables position-invariant access but only within the trained sequence length — the advantage is not an inherent architectural property but depends on the positional encoding's ability to generalize to unseen lengths.

Query-aware contextualization (Section 4.2, Figure 9 for multi-document QA; Section 4.2 text for key-value retrieval): Placing the query both before and after the documents (vs. only after) produces dramatically different effects depending on the task. For key-value retrieval, query-aware contextualization eliminates the U-shaped curve, with GPT-3.5-Turbo (16K) achieving perfect or near-perfect accuracy on all three settings (75, 140, 300 key-value pairs), compared to as low as 45.6% at the middle position without query-aware contextualization. For multi-document QA, query-aware contextualization produces only marginal changes (Figure 9): a slight improvement when the relevant document is at the very beginning, slight degradation at other positions, and the U-shape persists. This differential effect decomposes the positional bias into two distinct failure modes: (1) a query-blind encoding failure in pure retrieval tasks (fully solvable by query-first prompting), and (2) a more fundamental information utilization failure in reasoning tasks (not solvable by query-first prompting).

Instruction fine-tuning (Section 4.3, Figure 10): Comparing MPT-30B (base) against MPT-30B-Instruct on multi-document QA with 20 documents, both models exhibit U-shaped performance curves. The base model shows a ~10 percentage point swing between best and worst position; instruction tuning raises absolute performance uniformly and slightly compresses the swing to ~4 points, but does not eliminate the U-shape. The paper concludes that "the instruction fine-tuning process itself is not necessarily responsible for these performance trends" — the positional bias originates in pre-training, and instruction tuning can somewhat mitigate but not eliminate it.

Model scale and Llama-2 fine-tuning (Appendix E, Figure 16): Evaluating Llama-2 models at 7B, 13B, and 70B parameters (both base and chat variants) on 20-document multi-document QA reveals a scale-dependent emergence of the primacy bias. The 7B models (with and without chat fine-tuning) are solely recency-biased: accuracy steadily increases with position, peaking at the end, with no primacy peak at the beginning. The 13B models show a dramatic U-shape: the base 13B model has a ~20-point accuracy disparity between best and worst positions, and the chat fine-tuned 13B model slightly compresses this to ~10 points. The 70B models (with and without chat fine-tuning) both exhibit clear U-shaped curves, and additional fine-tuning minimally affects the positional bias at this scale. This suggests that primacy bias emerges only above a certain model capacity threshold (between 7B and 13B parameters for Llama-2), with smaller models showing only recency bias — a finding that reconciles the paper's results with prior work (Khandelwal et al., 2018; Sun et al., 2021) that studied sub-1B models and observed only recency effects.

GPT-4 evaluation (Appendix D, Figure 15): Evaluated on a subset of 500 random multi-document QA examples with 20 total documents, GPT-4 achieves higher absolute accuracy than any other model but still exhibits a U-shaped performance curve. Performance peaks at the beginning (above 85%) and end (roughly 82%), with a trough in the middle positions (approximately 72%). This demonstrates that even the most capable available model at the time of writing was not immune to positional degradation.

Critical Assessment

Claim 1: Language model performance degrades significantly when relevant information is in the middle of long input contexts, forming a U-shaped curve. Strongly supported. The U-shape is demonstrated across six primary models in multi-document QA (Figure 5, Tables 5–7), across four models in key-value retrieval (Figure 7), persists across multiple context lengths (10, 20, 30 documents; 75, 140, 300 key-value pairs), survives multiple robustness checks (random distractors, randomized distractor order, unambiguous question subsets), and appears even in GPT-4 (Figure 15). The pattern is not universal — Claude-1.3 achieves near-perfect key-value retrieval at all positions, and LongChat-13B shows a monotonic decline rather than symmetric U-shape in multi-document QA — but it is the dominant pattern across most model-task-context length combinations. The evidence for the U-shape as an empirical phenomenon is the strongest in the paper.

Caveat: The paper demonstrates that most models exhibit a U-shape under these specific prompt formats (task instruction, then items, then query, then answer). The query-aware contextualization experiment shows that prompt format can dramatically affect the shape (collapsing it entirely for key-value retrieval), which raises the question of whether alternative prompt formats for multi-document QA — not explored in the paper — might similarly reduce or eliminate the U-shape. The paper's claim is about how models behave under standard prompting conventions, which is practically important but narrower than a claim about an inherent architectural limitation.

Claim 2: Extended-context models are not necessarily better at using their input context within the shared context range. Strongly supported with the evidence provided, but with an important interpretational subtlety. The near-identical performance of GPT-3.5-Turbo vs. GPT-3.5-Turbo (16K) and Claude-1.3 vs. Claude-1.3 (100K) on inputs that fit within both context windows is convincingly shown (Figure 5, Tables 5–7, Table 1). However, the paper interprets this as evidence that "extended-context models are not necessarily better at using input context." This interpretation is correct for within-shared-window performance, but the paper does not test whether the extended-context models are better at using context beyond the standard model's window — i.e., at 30 documents for GPT-3.5-Turbo-16K vs. what GPT-3.5-Turbo would do if it could handle that length. The claim is true as stated but somewhat narrow: extended-context training enables processing longer inputs (trivially true), and the paper shows it does not improve processing of shorter inputs. Whether it improves processing of inputs at its extended length relative to what a hypothetical extrapolated standard model would do is not tested. A fairer framing might be: extended-context training solves the capacity problem but not the utilization problem, even within the capacity it already had.

Claim 3: Encoder-decoder models are relatively robust to changes in the position of relevant information, but only within their training-time sequence length. Supported with two models but limited generalizability. The evidence from Flan-UL2 and Flan-T5-XXL (Figure 8) clearly shows flat performance within training window and U-shaped degradation beyond it. However, these are both T5-family models fine-tuned on the same instruction dataset (Flan). Whether this pattern generalizes to other encoder-decoder architectures (e.g., BART, PEGASUS) or other training recipes is untested. Additionally, the fact that both studied encoder-decoder models were trained on relatively short sequences (512 and 2,048 tokens) means their "robust" regime is quite narrow — for practical long-context applications, they would almost certainly be operating out-of-distribution, where the paper shows they fail in the same U-shaped pattern as decoder-only models. The claim is well-supported for the models tested but extrapolating it to "encoder-decoders are robust given appropriate training length" would require testing an encoder-decoder model actually trained on long sequences.

Claim 4: Model performance on open-domain QA saturates long before retriever recall. Supported but with a confound that limits mechanistic attribution. Figure 11 convincingly shows reader accuracy flattening after ~20 documents while retriever recall continues to rise. The paper attributes this to the U-shaped positional bias: additional documents push relevant information into disadvantageous middle positions. However, this mechanism is inferred rather than directly demonstrated — the open-domain QA experiment does not track where in the ranked list the answer-containing documents appear for the saturated cases. The saturation could alternatively be explained by: (a) the additional documents beyond 20 being lower-quality (less relevant) even though they technically contain the answer, making extraction harder regardless of position; (b) the answer appearing in multiple documents, with the model already having extracted it from earlier documents; or (c) a subset of questions being fundamentally harder for the reader regardless of retrieval quality. The U-shape provides a plausible mechanism for the saturation, but the open-domain experiment itself does not isolate this mechanism. A stronger design would have tracked accuracy as a function of the position of the answer-containing document within the retrieved set for the saturated regime, directly connecting the controlled and ecological experiments.

Missing experiments that would strengthen the paper:

Direct combination of the U-shape finding with a practical intervention. The paper suggests re-ranking and truncation as natural responses to the U-shape (Section 5), but tests neither. An experiment showing that simple re-ranking (e.g., moving the highest-retrieval-score document to position 1) improves reader accuracy at high $k$ would close the loop from diagnosis to solution and validate the mechanistic attribution.
Confidence intervals or statistical testing. The paper reports point estimates without any measure of variance, despite evaluating on 2,655 questions (which get split across positions). When comparing curves that differ by only a few percentage points (e.g., GPT-3.5-Turbo vs. GPT-3.5-Turbo-16K), it is impossible to know whether small differences are meaningful or noise. The paper's conclusions rely heavily on visual comparison of curve shapes, which is reasonable for large effects (20-point swings) but fragile for the finer comparisons (e.g., whether instruction tuning reduces the U-shape by 4 points or 6 points).
Ablation of prompt format components. The paper tests one alternative prompt format (query-aware contextualization) but does not systematically ablate other prompt components: the effect of the system instruction, the document numbering scheme, the title inclusion, or the instruction position. If the U-shape is partly driven by formatting conventions that cue the model about where important information typically appears, varying these components would help distinguish formatting effects from architectural limitations.
More encoder-decoder models, especially trained on long sequences. The encoder-decoder analysis is limited to two T5-based models with short training sequences. Testing an encoder-decoder model actually pre-trained on long sequences (e.g., LongT5, or a modern equivalent) would address whether the within-training-window robustness generalizes to practically useful context lengths or is a quirk of short-sequence encoder-decoder training.
Latency and cost-aware analysis of the truncation recommendation. The paper recommends ranked list truncation based on the saturation finding, but does not actually compare cost-adjusted performance for different $k$ . A plot of accuracy per dollar or accuracy per millisecond would make the practical recommendation quantitatively grounded.

Genuine weaknesses:

The difficulty estimation for each example is implicit and uncontrolled. The 2,655 questions from NaturalQuestions-Open vary widely in inherent difficulty (some are simple lookup questions, others require complex multi-hop reasoning), and the paper does not analyze whether the U-shape varies by question difficulty. It is possible that the U-shape is most pronounced for easy questions (where models could answer closed-book and distractors interfere most) and less pronounced for hard questions (where models rely heavily on the provided document regardless of position), or vice versa. This difficulty-dependence, shown to be critical in other test-time compute analyses, is unexplored here.

The key-value retrieval task with UUIDs is so synthetic that failure on it is diagnostically useful (showing a fundamental access limitation), but success on it (Claude-1.3's near-perfect performance) provides limited information — it does not tell us whether Claude is "good at long contexts" in any practical sense, only that it can solve a task far simpler than any real use case. The paper appropriately uses the synthetic task as a lower-bound diagnostic, not as evidence of general capability, but readers should be careful not to over-interpret Claude's perfect key-value retrieval as general long-context robustness (Claude-1.3 still shows a U-shape in multi-document QA, just a shallower one).

The single-dataset limitation is substantive. All multi-document QA experiments use NaturalQuestions-Open. While this is a standard benchmark, it has specific properties — factoid questions with short answers, Wikipedia as the knowledge source, English only — that may interact with positional biases in unknown ways. The key-value retrieval task partially addresses domain specificity by removing natural language entirely, but there is no intermediate test (e.g., synthetic retrieval with natural language passages but no reasoning requirement) that could bridge the gap between UUID matching and factoid QA.

Finally, the paper's recommendation to re-rank or truncate — while logically sound and practically useful — is not empirically validated in the paper itself. The paper provides the diagnostic foundation that motivates these strategies but does not demonstrate their effectiveness. This is a reasonable scope limitation for a diagnosis paper, but it means the practical takeaways remain at the level of informed hypothesis rather than demonstrated best practice.

6. Limitations and Trade-offs

Limitation 1: Single Benchmark, Single Task Family — All Naturalistic Results Come from Factoid QA on Wikipedia

The assumption or constraint. The paper's entire naturalistic evaluation — the multi-document QA experiments that constitute its primary evidence for the U-shaped curve in ecologically valid settings — uses a single dataset: NaturalQuestions-Open, consisting of short factoid questions answerable from Wikipedia paragraphs. The key-value retrieval task is synthetic by design and deliberately strips away semantic content. This means all practical-domain evidence comes from one task family (factoid question answering), one knowledge source (Wikipedia), one language (English), and one answer format (short spans extracted from passages). The paper does not test code generation, multi-hop reasoning, long-form generation from multiple sources, dialogue history utilization, or any domain where the "relevant information" is not a discrete fact-bearing passage.

The consequence. A practitioner deciding whether to trust a long-context model for, say, legal document review — where the "relevant information" might be a subtle clause whose importance only becomes clear after reading the entire contract — cannot determine from this paper whether the U-shaped positional degradation applies to their use case. Factoid QA has specific properties that might interact with positional effects in unknown ways: answers are typically named entities or short phrases, the relevant document often contains obvious lexical overlap with the question (making relevance identification easier than in more subtle reasoning tasks), and Wikipedia passages are relatively self-contained. It is plausible that the U-shape is more severe in tasks requiring integration across multiple documents (where mid-context documents must be linked to earlier or later ones) or less severe in tasks where the model can rely on parametric knowledge to fill gaps. The paper provides no evidence either way.

What evidence exists in the paper. All multi-document QA experiments (Figures 5, 8–10, 12–16; Tables 1, 5–7) use the same 2,655 NaturalQuestions-Open queries. The open-domain QA case study (Section 5, Figure 11) uses the same dataset and retriever. The synthetic key-value retrieval task (Figure 7) provides complementary evidence that the position effect persists in a domain-agnostic setting, but this task is designed to be a lower bound on complexity — it tells us the problem exists at the retrieval level but does not characterize how it interacts with domain-specific reasoning demands. The paper acknowledges this limitation implicitly by the scope of its experiments but does not flag it as a limitation or discuss how findings might generalize.

Mitigation status. Not addressed. The paper does not test on any other dataset or task family, nor does it discuss the generalizability of its findings beyond factoid QA. This is a standard scope limitation for an empirical analysis paper, but it means the paper's practical recommendations (put critical information at the beginning or end; truncate retrieval lists; use re-ranking) are, strictly speaking, validated only for Wikipedia-based factoid QA with short answers. Extrapolation to other domains is reasonable but untested.

Limitation 2: The Prompt Format Is Held Constant — Positional Effects Might Be Partly an Artifact of the Specific Prompting Convention

The assumption or constraint. All multi-document QA experiments (except the query-aware contextualization ablation in Section 4.2) use a fixed prompt format: a system-style instruction first, then numbered documents with titles, then the question, then an "Answer:" prefix. This format is reasonable and mimics common RAG implementations, but it is one specific prompting convention among many possible ones. The paper's core claim — that language models exhibit a U-shaped performance curve — is demonstrated under this format, and the paper does not systematically vary formatting elements to determine whether the U-shape is format-specific or format-invariant.

The consequence. If the U-shape is partially driven by how the prompt signals importance to the model — for instance, numbered lists might implicitly communicate that earlier items are more important (a convention from web text where ordered lists are often ranked), or the placement of the question after all documents might create an attentional bottleneck that particularly disadvantages mid-context items — then the paper's findings overstate the architectural nature of the limitation and understate the role of prompt design. A practitioner who uses a different prompt format (e.g., interleaving documents with sub-questions, placing the query at the beginning, using XML-style tags instead of numbered lists) might observe a substantially different positional sensitivity curve. The paper's recommendation to "put important information at the beginning or end" might be format-contingent advice.

What evidence exists in the paper. The paper does test one alternative format: query-aware contextualization (Section 4.2, Figure 9), which places the query both before and after the documents. This shows that prompt format can dramatically change results for key-value retrieval (collapsing the U-shape entirely) but only marginally affects multi-document QA (slight improvement at the beginning, slight degradation elsewhere, U-shape persists). The randomized distractor order experiment (Appendix C, Figure 14) also tests a format variant — shuffling documents and explicitly stating they are randomly ordered — and finds the U-shape persists, though with slightly different magnitudes. However, no other format variations are tested: the effect of removing the system instruction, changing document delimiters (e.g., XML tags vs. numbered lists), placing the question at different positions relative to documents (interleaved, at the beginning only, between documents), or using different answer format expectations (e.g., "Extract the answer from the documents" vs. "Answer the question").

Mitigation status. Partially addressed. The query-aware contextualization experiment and the randomized ordering experiment demonstrate that the U-shape is not purely a formatting artifact — it persists under two non-trivial format changes. However, the space of possible prompt formats is vast, and the paper does not establish whether the U-shape is a lower bound on positional degradation (i.e., other formats might be worse but none are substantially better) or merely one point in a distribution of format-dependent curves. The paper does not discuss this as a limitation.

Limitation 3: The Difficulty Estimation and Position-Aware Allocation Are Entirely Absent — There Is No Adaptive Strategy

The assumption or constraint. The paper diagnoses the U-shaped positional sensitivity as a static property of models averaged across all questions. It does not investigate whether the U-shape varies by question difficulty, question type, or any other characteristic of the input, nor does it propose or test any adaptive strategy that would allocate positioning or retrieval depth based on estimated question properties. The finding is presented as a uniform behavioral regularity rather than a difficulty-conditioned phenomenon.

The consequence. This is a significant gap for practitioners because it prevents selective intervention. If the U-shape is most severe for easy questions (where the model could answer from parametric memory and distractor documents create interference) and less severe for hard questions (where the model relies heavily on the provided document regardless of position), then the fix might be to use closed-book answering for easy questions and reserve document conditioning only for hard ones. Conversely, if the U-shape is most severe for hard questions (where the model needs to carefully integrate document information and mid-context placement disrupts this integration), then the fix might be to re-rank only for hard questions. Without difficulty-conditioned analysis, the paper's recommendations (put everything important at the start or end; truncate retrieval) are applied uniformly, which may be suboptimal — some questions might not need any retrieved context, while others might benefit from deeper retrieval despite the positional penalty.

Furthermore, the paper provides no mechanism for a deployed system to decide how many documents to retrieve or how to order them for a given query. The saturation analysis (Figure 11) shows that aggregate accuracy barely improves beyond 20 documents, but this is an average effect. Some questions might see large gains from additional documents (if the answer only appears in the 30th result), while others might be harmed (if additional documents push the relevant one into a disadvantageous position). Without per-question difficulty estimation and adaptive allocation, the system designer is forced to choose a single $k$ that optimizes average performance, leaving per-question gains on the table.

What evidence exists in the paper. The paper provides no difficulty-conditioned analysis. All performance curves are averages across all 2,655 questions. The paper does not bin questions by any property — not by the model's closed-book accuracy on that question (which would indicate how much the model already knows), not by question length or complexity, not by the number of valid answers, not by the semantic distance between the question and the relevant document. The Llama-2 scale analysis (Appendix E, Figure 16) demonstrates that model size affects the shape of the positional curve, which indirectly suggests that "capability" interacts with positional sensitivity, but this is a between-model comparison, not a within-model, across-question-difficulty analysis.

Mitigation status. Not addressed at all. The paper does not mention difficulty conditioning, adaptive strategies, or per-question allocation as a direction. This is a natural extension — analogous to the difficulty-conditioned test-time compute allocation studied in other work — but the paper provides no foundation for it beyond the aggregate U-shape characterization.

Limitation 4: The Retrieval Saturation Claim Lacks Direct Mechanistic Validation — Position of Relevant Documents Is Not Tracked

The assumption or constraint. The open-domain QA case study (Section 5, Figure 11) demonstrates that reader accuracy saturates while retriever recall continues to rise, and the paper attributes this to the U-shaped positional bias: "additional documents push relevant information into the middle of the context where models struggle to access it." However, this attribution is an inference from the controlled experiments (Sections 2–3) rather than a directly measured mechanism in the open-domain setting. The case study does not track where in the ranked list the answer-containing document actually appears for the examples where accuracy fails to improve.

The consequence. Alternative explanations for the saturation are not ruled out. It could be that the additional documents retrieved beyond rank 20 are genuinely harder to extract answers from — they might contain the answer in a less direct form, buried in longer passages, or with different phrasing that the model fails to map to the question — even if they appear at favorable positions. It could be that the questions that would benefit from deeper retrieval (where the answer first appears at rank 30 or 40) are a small minority, and the aggregate saturation is driven by the majority of questions where the answer already appeared in the top 20. It could be that the model's parametric knowledge already covers the answer for many questions, and deeper retrieval adds redundant information that does not improve output. Without tracking the actual position of the answer-containing document for each query at each $k$ , the connection between the U-shape and the saturation observation remains correlational rather than causal.

What evidence exists in the paper. Figure 11 plots two aggregate curves: retriever recall (did the answer appear anywhere in the top $k$ ?) and reader accuracy (did the model produce the correct answer?). These curves diverge, but the paper does not break down reader accuracy conditioned on where the answer-containing document appeared in the ranking. For example, it would be informative to see: for queries where the answer first appears at rank 1–5, what is reader accuracy? At rank 6–10? At rank 11–20? At rank 21–50? If accuracy is high when the answer appears early (positions 1–5, benefiting from primacy) and low when it appears late (positions 20–50, landing in the middle of a long context), that would provide direct mechanistic evidence for the U-shape driving saturation. The paper does not provide this analysis.

Mitigation status. Not addressed. The paper treats the controlled experiments (where position is explicitly manipulated) and the open-domain case study (where position is an emergent property of the retrieval system) as complementary but does not bridge them with position-conditional accuracy analysis in the open-domain setting. This is a missed opportunity to directly validate the paper's central mechanistic claim in a deployment-like scenario.

Limitation 5: Greedy Decoding Only — Results May Not Generalize to Stochastic Decoding or Best-of-N Selection

The assumption or constraint. All experiments use greedy decoding — at each generation step, the model selects the single most probable next token. The paper states this explicitly (Section 2.2) and "leaves exploration of other decoding methods to future work." Greedy decoding is deterministic, which eliminates sampling variance and makes results reproducible, but it also means the reported accuracy is a lower bound on what could be achieved with temperature sampling, nucleus sampling, or best-of-N selection.

The consequence. This choice matters for the paper's practical claims in two ways. First, absolute accuracy numbers are lower than what a practitioner using stochastic decoding would observe, which means the "below closed-book performance" benchmark — while striking as a demonstration of interference — may not hold under best-of-N sampling. If stochastic sampling occasionally recovers the correct answer from a mid-context document (even if greedy decoding misses it), then the practical severity of the positional degradation might be overestimated. Second, and more subtly, the shape of the positional curve might change under stochastic decoding. If the model's probability mass over correct answers is concentrated differently at different positions — for instance, if mid-context placement makes the correct answer lower-probability but still above the sampling threshold — then temperature sampling might partially recover performance in the trough of the U-shape, while greedy decoding would miss it entirely. Conversely, if mid-context placement introduces confusion that scatters probability mass across many incorrect answers, stochastic sampling could make performance worse than greedy at those positions (by sampling from the confused distribution). The paper provides no evidence about which regime applies.

What evidence exists in the paper. None. The paper does not run any temperature sampling, nucleus sampling, or best-of-N experiments, nor does it analyze the model's output token probabilities to assess whether greedy decoding is representative of the model's underlying confidence at different positions.

Mitigation status. Not addressed beyond the brief acknowledgment that exploration of decoding methods is left to future work. This is a reasonable scoping decision for an initial characterization paper, but it means that practitioners should not interpret the reported accuracy numbers as achievable performance — they represent a particular (and conservative) decoding strategy. Whether the U-shape is a property of the model's internal representations (which would persist under any decoding method) or of greedy decoding specifically (which might amplify positional differences) is an open question the paper does not resolve.

Limitation 6: The Encoder-Decoder Advantage Is Not Practically Actionable — Within-Training-Window Robustness Requires Short Sequences

The assumption or constraint. The paper's finding that encoder-decoder models exhibit flat (position-invariant) performance curves within their training-time sequence length (Section 4.1, Figure 8) is presented as evidence that bidirectional encoding enables robust context utilization. However, both encoder-decoder models evaluated — Flan-T5-XXL and Flan-UL2 — were trained on relatively short sequences: 512 tokens for Flan-T5-XXL, up to 2,048 tokens for Flan-UL2's encoder. The "robust" regime where these models achieve flat performance curves is therefore 2,048 tokens or less — far shorter than the contexts used in modern long-context applications.

The consequence. A practitioner seeking to deploy a model that robustly uses 8K, 16K, or 100K-token contexts cannot look to these encoder-decoder results for guidance, because the demonstrated robustness does not extend to those lengths. The paper's finding is architecturally informative — it suggests that bidirectional attention is the mechanism enabling position invariance — but it does not translate into a practical recommendation, because no existing encoder-decoder model (at the time of writing) was trained on the sequence lengths where robustness would be practically useful. The finding also does not address whether an encoder-decoder model could be trained to achieve flat performance at longer sequences if given sufficient long-sequence training data, or whether some other bottleneck (e.g., positional encoding extrapolation failure) would prevent position invariance from scaling to modern context lengths.

Furthermore, the paper shows that beyond the training-time sequence length, encoder-decoder models exhibit the same U-shaped degradation as decoder-only models (Figure 8, center and right panels). Since virtually all practical long-context applications involve sequences beyond 2,048 tokens, the encoder-decoder "advantage" is not available in practice — the models that are robust are not usefully long-context, and at usefully long contexts, they lose their robustness. The paper's architectural insight is valuable for understanding why position invariance is possible, but it does not provide a deployable solution.

What evidence exists in the paper. Figure 8 shows Flan-UL2 with a flat curve (1.9% best-worst difference) at 10 documents (~2K tokens, within its training window) and pronounced U-shapes at 20 documents (~4K tokens) and 30 documents (~6K tokens). Flan-T5-XXL (512-token training window) shows U-shapes at all tested lengths, deepening with context size. The paper does not evaluate any encoder-decoder model trained on longer sequences (e.g., 8K or 16K tokens), meaning there is no evidence about whether the within-training-window robustness property would hold at practically relevant scales.

Mitigation status. The paper does not discuss this as a limitation. It presents the encoder-decoder finding as evidence that "encoder-decoder models may make better use of their context windows because their bidirectional encoder allows processing each document in the context of future documents, potentially improving relative importance estimation between documents" (Section 4.1). While architecturally informative, this framing does not acknowledge that the advantage is practically unavailable at the context lengths that matter for deployment. A candid discussion would note that the finding motivates future work on long-sequence encoder-decoder training but does not provide an immediately actionable alternative to decoder-only models for long-context applications.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new model architecture, training algorithm, or prompting technique. Instead, it introduces something arguably more consequential for the field's trajectory: a diagnostic protocol that transforms how the research community evaluates and thinks about long-context utilization. Before this work, the dominant framing for long-context models was capacity — can the model process X tokens without running out of memory or producing garbled output? After this work, the relevant question shifts to robustness of access — can the model use information equally well regardless of where it appears within those X tokens?

This is not a paradigm shift in the Kuhnian sense. The underlying architecture (Transformers with self-attention) remains unchallenged, and the paper explicitly notes that attention is "technically equally capable of retrieving any token." But it is a substantive reframing of the evaluation criterion, from a scalar threshold (maximum context length) to a shape requirement (flatness of the accuracy-vs-position curve). The difference between best-case and worst-case accuracy across positions becomes a single diagnostic number for long-context quality, analogous to how worst-case accuracy across slices measures fairness or robustness in other subfields.

This reframing has several downstream effects on how the field allocates research effort:

It redirects attention from architectural scaling to training-for-utilization. The paper's finding that extended-context models (GPT-3.5-Turbo-16K, Claude-1.3-100K) perform identically to their standard-context counterparts on inputs that fit within both windows (Figure 5, Tables 1, 5–7) is a direct challenge to the narrative that longer context windows inherently improve context use. The hard problem, the paper implies, is not supporting longer sequences — FlashAttention, ALiBi, RoPE condensation, and related techniques have largely solved the memory and compute scaling challenges. The hard problem is training models to uniformly access the context they already have. This shifts research priority from efficient attention mechanisms (a mature subfield) to training data curation and optimization procedures that produce position-invariant representations.

It provides a unified explanation for previously contradictory findings about long-context use. Prior work painted an inconsistent picture: Khandelwal et al. (2018) and Sun et al. (2021) found that small or base LMs exhibited recency bias (attending mainly to the end of the context); Qin et al. (2023) found that efficient Transformers were "recency-biased"; Ivgi et al. (2023) found an advantage for relevant information at the beginning for encoder-decoder models. These results appeared to conflict, but the U-shaped curve reconciles them: recency bias is the right half of the U, primacy bias is the left half, and which half dominates depends on model scale and training procedure. The paper's Llama-2 scale analysis (Appendix E, Figure 16) makes this explicit — the 7B models show only recency bias (matching prior work on smaller LMs), the 13B models show both primacy and recency (a U-shape), and the 70B models deepen the U. Prior work that studied sub-1B models (Khandelwal et al., 2018; Sun et al., 2021) captured only the recency tail of a phenomenon that, at larger scales, becomes symmetric. This reconciliation transforms a confusing set of partial results into a coherent, scale-dependent picture.

It identifies a new failure mode: mid-context information can be worse than no information. The finding that GPT-3.5-Turbo's accuracy with the relevant document at position 10 in a 20-document context (53.8%, Table 6) falls below its closed-book accuracy (56.1%, Table 1) is a striking demonstration that longer contexts are not merely unhelpful — they can be actively harmful under certain positional configurations. This has no analog in standard perplexity-based evaluation, where longer context rarely hurts. It implies that retrieval-augmented generation systems must consider not just whether to include a document, but where to place it, and that naïve concatenation of retrieved results can degrade end-to-end performance below what the model would achieve with no retrieval at all.

It makes the serial-position effect in language models a first-class research topic. The paper explicitly connects its empirical findings to the serial-position effect from cognitive psychology (Ebbinghaus, 1913; Murdock Jr, 1962), noting that "observing a serial-position-like effect in language models is perhaps surprising, since the self-attention mechanisms underlying Transformer language models is technically equally capable of retrieving any token." This connection opens a bridge between the machine learning and cognitive science literatures: if Transformers trained on internet text recapitulate a well-documented human memory phenomenon, it suggests that the effect arises not from biological constraints on human memory systems but from statistical properties of the training data (e.g., discourse structure where important information tends to appear at the beginning or end of documents) that shape learned attention patterns. Understanding why transformers develop human-like positional biases despite their architectural capacity for position-invariant attention becomes a fundamental research question the paper surfaces but does not resolve.

Which research directions become more attractive, and which become less so:

More attractive: (1) Training interventions that penalize positional sensitivity — e.g., data augmentation that randomizes the position of key information during pre-training, or auxiliary losses that explicitly reward position-invariant retrieval. (2) Prompt engineering for positional optimization — systematically studying how different prompt templates, delimiters, and instruction placements affect the shape of the positional curve. (3) Re-ranking and list optimization for RAG systems — developing lightweight models that predict, for a given query, whether the model is likely to struggle with mid-context documents and adjust ordering accordingly. (4) Architecture modifications that hard-code position invariance — e.g., attention mechanisms that are explicitly regularized toward uniform attention across the context, or token-level position embeddings that decouple content and position.
Less attractive: (1) Purely architectural approaches to long-context scaling that focus solely on memory and throughput without evaluating positional utilization — the paper demonstrates that scaling context capacity is necessary but far from sufficient, and evaluations that report only perplexity or accuracy on benchmarks without position-controlled analysis miss the critical dimension of robustness. (2) Claims about long-context capability based solely on handling long inputs — the paper provides a concrete, low-cost evaluation protocol (varying the position of a single relevant item and measuring accuracy) that should become a standard complement to any long-context model release.

Follow-Up Research This Work Enables

Position-controlled evaluation as a standard benchmark for all long-context model releases. This paper provides the blueprint for a position-controlled evaluation protocol but applies it to only two tasks (multi-document QA, synthetic key-value retrieval) and a single dataset per task. A natural and immediately actionable follow-up is to create a standardized benchmark suite — call it PositionBench — that applies the position-modulation methodology across a diverse set of tasks: code generation (where the relevant function definition is placed at different positions within a long file), multi-hop reasoning (where documents must be integrated across positions), dialogue state tracking (where the critical utterance is placed at different points in a long conversation history), and long-form summarization (where the key information to include is placed at controlled positions in the source). Each task would report not just aggregate accuracy but the positional-accuracy curve and the best-vs-worst-case delta. This would enable apples-to-apples comparison across model releases and track progress on the utilization dimension over time.

Difficulty-conditioned analysis of the positional U-shape. The paper's aggregate curves average over 2,655 questions of varying inherent difficulty, but it does not analyze whether the U-shape varies by how "hard" a question is for the model. A natural extension — drawing on the approach of the compute-optimal test-time scaling literature — would bin questions by the model's closed-book accuracy on that specific question (i.e., whether the model already knows the answer without any documents). Hypothesis: the U-shape is most severe for easy questions where the model has strong parametric knowledge, because the distractor documents create interference that degrades performance below the closed-book ceiling; for hard questions where the model has no parametric knowledge, the U-shape might be shallower because the model relies entirely on the provided document regardless of its position. This would refine the paper's practical recommendations: for easy questions, retrieve fewer documents or none at all (the model already knows the answer and additional context can only interfere); for hard questions, retrieve more aggressively but use position-optimized ordering (relevant documents at the edges). This requires no new models, only a re-analysis of the existing experimental data with a difficulty split.

Direct validation that the U-shape causes retrieval saturation via position-conditional accuracy. The paper's open-domain QA case study (Figure 11) infers that reader accuracy saturates because additional documents push relevant information into disadvantageous middle positions, but it does not directly measure this mechanism. A targeted follow-up would take the same NaturalQuestions-Open queries and Contriever retrieval pipeline, but for each query and each $k$ , track the rank position of the answer-containing document, then report reader accuracy conditioned on that rank position. The prediction from the U-shape model: accuracy should be high when the answer document appears at ranks 1–5 (benefiting from the primacy peak), dip for ranks 10–15 (in the trough for a 20-document context), and partially recover for ranks 18–20 (recency peak). If this position-conditional pattern holds, it provides direct causal evidence linking the controlled U-shape experiments to the ecologically valid saturation observation. If it does not hold — if accuracy declines monotonically with rank without a recency recovery — then the saturation has a different explanation (e.g., lower-quality documents at deeper ranks, even when they nominally contain the answer), and the connection between the controlled and open-domain experiments is weaker than the paper implies.

Training interventions to reduce positional sensitivity. The paper diagnoses a problem but does not propose solutions. A natural next step is to design and test training procedures that explicitly penalize positional dependence. Concrete candidates include: (1) data augmentation during pre-training that randomizes the position of key sentences or paragraphs within documents, forcing the model to learn position-invariant representations; (2) an auxiliary contrastive loss that penalizes differences in the model's answer distribution when the same relevant document is placed at different positions; (3) fine-tuning on synthetic key-value retrieval data with the relevant pair uniformly sampled across positions. Each candidate could be evaluated using the paper's own evaluation protocol: train a model variant with the intervention, then measure whether the U-shape flattens in multi-document QA and key-value retrieval compared to a baseline trained without the intervention. The paper's Llama-2 scale analysis (Figure 16) suggests that larger models develop the primacy bias through pre-training, which implies that pre-training data distribution matters — interventions at the pre-training stage might be more effective than fine-tuning patches.

Query-aware document encoding in decoder-only models without full query duplication. The paper's query-aware contextualization experiment (Section 4.2) shows that placing the query before the documents dramatically improves key-value retrieval but minimally affects multi-document QA. This suggests an architectural question: is the multi-document QA bottleneck that the model cannot use query information during document encoding, or that even with query-aware encoding, the model still cannot effectively utilize the identified document for reasoning? A targeted experiment would test an intermediate setup: encode the query into a short vector representation, prepend this as a "query summary" token before each document during encoding (so each document is contextualized with query information without duplicating the full query text), and measure whether this reduces the U-shape. If it helps, the bottleneck is encoding-side; if it doesn't, the bottleneck is utilization-side. This would further decompose the positional effect into sub-mechanisms the paper has already begun to separate.

Long-sequence encoder-decoder training at modern context lengths. The paper's finding that encoder-decoder models achieve flat positional curves within their training-time sequence length (Figure 8) is architecturally significant but practically limited — both Flan-T5-XXL and Flan-UL2 were trained on sequences far shorter than modern long-context applications. A direct follow-up would train an encoder-decoder model (e.g., T5-XXL architecture) from scratch or via continued pre-training on 8K or 16K-token sequences, then evaluate it on the paper's multi-document QA protocol at 10, 20, and 30 documents (which would all be within the training window). If the flat positional curve persists at these practically useful lengths, it would establish encoder-decoder training as a deployable solution for position-invariant context utilization (modulo the encoder-decoder vs. decoder-only tradeoffs in generation quality and efficiency). If the flat curve does not persist — if even within-training-window encoder-decoder models show U-shapes at 8K+ contexts — it would suggest a more fundamental limitation in how long-range attention generalizes, even bidirectionally.

Practical Applications and Downstream Use Cases

Retrieval-augmented generation (RAG) system design with position-optimized document ordering. The most direct practical application of this work is in the thousands of production RAG systems that retrieve documents and concatenate them into a prompt for a language model. The paper's findings imply that simply appending documents in relevance-ranked order is suboptimal: the most relevant document — which typically appears first — benefits from the primacy peak, but documents of moderate relevance that appear in the middle of the prompt are in the worst position for model access. A concrete deployment improvement: after retrieval, re-rank documents so that the most likely relevant documents occupy the beginning (positions 1–3, primacy peak) and end (positions $k-2$ to $k$ , recency peak), with lower-confidence documents in the middle. The paper's Figure 5 provides the magnitude of the effect: for GPT-3.5-Turbo with 20 documents, moving the relevant document from position 10 (53.8% accuracy) to position 1 (75.8%) yields a 22-percentage-point gain — a ~40% relative improvement — from a change that costs zero additional FLOPs and requires only a document reordering module that could be implemented in a few lines of code.

Ranked list truncation to reduce cost and latency without accuracy loss. The open-domain QA case study (Figure 11) provides a data-driven threshold for retrieval depth: with Contriever on NaturalQuestions-Open, reader accuracy for GPT-3.5-Turbo (16K) improves from ~58% at 5 documents to ~65% at 20 documents — a 7-point gain — but then crawls to ~66% at 50 documents — only ~1 additional point for 30 more documents. For a production system serving millions of queries daily, this translates directly to infrastructure savings: retrieving and processing 50 documents per query instead of 20 costs 2.5× more in retrieval compute, prompt token count, and generation latency, for a ~1.5% accuracy improvement. A system that caps retrieval at 20 documents — accepting the ~1-point accuracy loss relative to 50-document retrieval — would reduce per-query cost and latency substantially with negligible impact on end-user experience. The paper does not provide latency or dollar figures, but the token count data (Tables 3, 4) and the saturation curve (Figure 11) give system designers the quantitative evidence needed to make this tradeoff.

Prompt engineering for long-document tasks by front-loading or back-loading critical information. For applications where the input is a single long document rather than multiple retrieved passages — legal contract review, scientific paper Q&A, long-form summarization — the U-shape finding translates into a concrete prompt engineering heuristic: place the query or task instruction at both the beginning and end of the prompt, and if possible, restructure the input so that the most critical information appears near the edges. For example, when asking a model to answer questions about a 50-page contract, prepend a summary of the key clauses at the start and append the specific question at the end, rather than placing the question only after the full contract text. The paper's query-aware contextualization results (Section 4.2) show this is not a panacea for reasoning tasks (the U-shape persists in multi-document QA), but for retrieval-like sub-tasks (locating specific clauses, extracting defined terms), it can substantially improve accuracy — analogous to how query-aware contextualization boosted key-value retrieval from 45.6% to near-perfect in the 300-pair setting. Practitioners working with long documents can adopt this pattern immediately, with the paper providing empirical justification.

Model selection and evaluation criteria for long-context applications. The paper's finding that extended-context models perform identically to their standard-context counterparts on inputs within the shared window (Figure 5, Tables 1, 5–7) has direct implications for technology procurement and model selection. If an application's typical context length is ~4K tokens, the paper suggests there is no accuracy benefit to using GPT-3.5-Turbo-16K over the cheaper GPT-3.5-Turbo, or Claude-1.3-100K over Claude-1.3 — the extended-context variants provide additional capacity but not better utilization. System designers should evaluate models using position-controlled benchmarks (e.g., varying the location of a key fact within a prompt of fixed length and measuring accuracy variance) rather than relying on vendor claims about maximum context length. The paper provides the evaluation protocol; organizations can replicate it on their own task-specific data to determine which model genuinely offers robust long-context access for their particular use case.