Reusing Pre-Training Data at Test Time is a Compute Multiplier

🎯 Pitch

Retrieving from a model's own pre-training data at test time delivers accuracy gains equivalent to a ~5× increase in training compute, showing that standard pre-training leaves substantial knowledge on the table. This compute multiplier degrades with scale but can be amplified by additional test-time compute like self-consistency over varied retrieved contexts, yielding a 10.5-point absolute improvement on MMLU for Llama 3.1 8B.

1. Executive Summary

This paper studies how reusing pre-training data as a retrieval corpus at test time quantifies the value left unextracted by standard pre-training, using MMLU, Math-500, and SimpleQA benchmarks with models trained at multiple compute budgets and additionally with a Llama 3.1 8B reader. The core mechanism is retrieval augmented generation on the pre-training dataset itself, which recovers knowledge the model was trained on but failed to absorb—operationalized as retrieving documents from the same corpora (DCLM, FineWeb-edu, Wikipedia, and math sources) and feeding them into the model's context at inference time—combined with parallel test-time compute strategies including self-consistency, reranking, and variance-reduction techniques like MMR and bagging. The headline finding is that retrieval acts as a ~5× compute multiplier on MMLU (matching the accuracy of a model trained with roughly 4.86× more pre-training FLOPs on average), while the full suite of test-time methods yields a 10.5 percentage point absolute improvement on MMLU for Llama 3.1 8B, establishing that pre-training leaves substantial information on the table that test-time retrieval can recover even after decontamination—though the multiplier degrades with scale, dropping from 7.17× at the smallest compute budget to 2.88× at the largest.

2. Context and Motivation

The Core Problem: Pre-Training Leaves Knowledge Unextracted

The fundamental question this paper tackles is: how much usable information remains in pre-training datasets after standard pre-training is complete? We typically evaluate language models by measuring their downstream task performance after training, but those metrics conflate two very different things: (1) whether the necessary knowledge exists in the training data, and (2) whether the pre-training process actually extracted it into usable parametric memory. A model's failure on a task could mean the knowledge was never in the dataset, or it could mean the dataset contained the answer but pre-training didn't absorb it effectively. Prior to this work, we lacked a systematic method for distinguishing these cases.

The paper's central insight is that retrieving from the pre-training corpus at test time creates a natural measurement instrument: if a model can answer a question when given retrieval access to its own training data, then the knowledge was in the dataset all along—pre-training simply failed to encode it. The gap between retrieval-augmented performance and base model performance therefore quantifies the "data value left behind" by pre-training. This framing transforms retrieval from a deployment optimization into a diagnostic tool for pre-training efficiency.

This gap matters for several concrete reasons the authors establish (Section 1):

Scaling returns are diminishing. LLMs exhibit log-linear scaling trends—each additional unit of improvement requires exponentially more compute (Kaplan et al., 2020; Hoffmann et al., 2022). If a substantial fraction of this compute is spent re-extracting knowledge that could have been recovered through better algorithms rather than larger models, then we're spending scaling budget inefficiently. Understanding how much knowledge remains unextracted tells us whether the next best investment is more parameters or better learning algorithms.
Long-tail knowledge remains elusive. Even large models struggle with facts that appear infrequently in training data (Kandpal et al., 2023), and exhibit peculiar failures like the reversal curse—models trained on "A is B" fail to infer "B is A" (Berglund et al., 2023). These failures are puzzling: the information exists in the training corpus, yet the model cannot access it. Retrieval-augmented test-time access provides a clean test of whether the data supports the inference at all.
Dataset investment lacks feedback loops. Researchers dedicate enormous effort to curating pre-training datasets (Li et al., 2024; Penedo et al., 2024), but the standard evaluation pipeline only measures final model quality. It doesn't answer whether the dataset could support much better performance if models were better at learning from it. A dataset that produces a 50% MMLU model might actually contain enough knowledge for 80% MMLU—the dataset is good, the learning algorithm is the bottleneck. Without measuring this gap, dataset improvement efforts operate in the dark.
Test-time compute is an underexploited resource. Recent work demonstrated that allocating compute at inference time—through repeated sampling, self-consistency, or iterative refinement—can improve performance substantially (Brown et al., 2024; Snell et al., 2024). However, these approaches typically don't connect test-time compute back to the pre-training data. Retrieval provides a natural vehicle for this connection: test-time compute can be spent accessing and reasoning over the original training documents.

Prior Approaches and Their Limitations

The paper situates itself at the intersection of three research threads, each with specific gaps it aims to address.

Pre-training scaling laws don't account for data utilization efficiency. Classical scaling work (Hestness et al., 2017; Kaplan et al., 2020; Hoffmann et al., 2022) established how loss scales with compute, parameters, and data quantity. More recent work incorporated inference costs into the scaling picture (Sardana et al., 2024), recognizing that deployment economics matter alongside training economics. But these scaling laws treat data as consumed and converted into parameters—they don't model the residual value that remains in the data after training. The paper explicitly positions itself as extending the scaling laws tradition (Section 2):

"we first take inspiration from the classical pre-training scaling law studies, and begin to characterize the joint scaling of pre-training and simple retrieval given a fixed and identical data corpus"

The key novelty is characterizing not just how performance scales with pre-training compute, but how the gap between retrieval-augmented and base performance scales—something no prior scaling law study has examined.

Retrieval-augmented generation treats external knowledge as complementary to pre-training, not as a test of pre-training. Retrieval-augmented language models have a rich history—REALM (Guu et al., 2020) and RAG (Lewis et al., 2020) demonstrated that non-parametric memory externalizes knowledge access, while Atlas and later work (Ram et al., 2023) showed that retrieval can improve on knowledge-intensive tasks. Shao et al. (2024) demonstrated that scaling the datastore to a trillion tokens continues to improve performance. Most recently, Lyu et al. (2025) showed that retrieving from a compact subset of pre-training data with a minimal setup improves reasoning benchmarks.

The paper builds directly on these retrieval methods but uses them for a fundamentally different purpose. Prior work asked: "Can retrieval provide knowledge the model doesn't have?" This paper asks: "Can retrieval reveal what pre-training should have taught the model?" The distinction matters because it changes the interpretation of retrieval gains: they're not just a deployment trick, they're a measurement of pre-training inefficiency. The paper's choice to retrieve from the identical dataset used for pre-training (Section 3.1), rather than a separate knowledge source, is what makes this diagnostic possible.

A practical gap the paper identifies in prior retrieval work is the connection to dataset quality. Retrieval systems typically use dedicated retrieval corpora (Wikipedia, domain-specific databases) that are curated independently of pre-training datasets. The paper's approach—retrieving from the same webcrawl sources used for pre-training—reveals something about those sources that wasn't previously obvious: they contain more usable knowledge than pre-training extracts.

Test-time compute scaling has not been combined with retrieval from pre-training data. Recent work by Brown et al. (2024) and Snell et al. (2024) showed that scaling inference compute—either through parallel repeated sampling (self-consistency, Wang et al., 2022; Chen et al., 2023) or sequential refinement—can efficiently improve LLM performance. The paper acknowledges this and notes (Section 2) that "retrieval is naturally suited to both as it can be parallelized across retrieved documents, while sequentially iterating over the search query to improve the retrieved documents."

However, prior test-time compute studies treated the model's knowledge as fixed—they optimized how the model processes what it already knows, not how it accesses additional information during inference. This is a significant gap because test-time compute offers two very different value propositions: it can improve reasoning over existing knowledge (self-consistency), or it can recover unabsorbed knowledge (retrieval). The paper investigates both, finding (Section 5, Table 5) that their effects are additive—suggesting they address complementary limitations.

Large-scale commercial systems like "Deep Research" tools from Google, OpenAI, and Perplexity "likely apply all of the above" (Section 2), but their proprietary nature makes systematic study impossible. The paper's contribution is a controlled, reproducible analysis using fully open datasets and models.

The decontamination question remains unresolved. A persistent concern with retrieval-augmented evaluation is that gains might reflect test-set contamination rather than genuine knowledge recovery—retrieved documents might contain exact problem statements or answers. The paper addresses this head-on (Section 4, Figure 1) by performing n-gram decontamination on retrieved documents and showing that the gains persist (red vs. dark blue lines in Figure 1). Critically, the paper also quantifies the contamination rate: 14.1% of MMLU and 32.0% of Math-500 test items exist in commonly used open-source pre-training datasets. This finding serves double duty: it validates that retrieval gains are not purely contamination-driven, while also providing a sharp warning about benchmark integrity for pre-training research—a point the paper emphasizes explicitly:

"highlighting the importance of strictly decontaminated (or held out) evaluation sets for pre-training science" (Section 4)

How This Paper Positions Itself

The paper makes a deliberate methodological choice that sets it apart: the pre-training corpus and the retrieval corpus are identical (Section 3.1). This is not a convenience—it's the experimental design's central feature. If retrieval used a different corpus, any performance gain could be attributed to novel information the model never saw during training. By using the same corpus, the paper isolates the effect of access method: the model saw these exact documents during pre-training, but storing that knowledge parametrically and accessing it through retrieval are fundamentally different operations. The performance gap between base and retrieval-augmented evaluation therefore cleanly measures the parametric memory bottleneck.

The paper explicitly frames this as a data utilization measurement (Section 1):

"we use retrieval augmented generation along with test-time compute as a way to quantify how much dataset value was left behind by the process of pre-training"

This positions the work not as proposing a new model architecture or training method, but as providing an evaluation methodology for the pre-training process itself. The practical recommendations (improve crawling, improve extraction, develop better learning algorithms) follow from the measurements, not from a new technique.

A second positioning choice is the paper's emphasis on open-source, reproducible science. The datasets are public (DCLM, FineWeb-edu, Wikipedia, math corpora listed in Section 3.1 and Table 1). The test-time methods are simple and well-documented: Qwen3 embeddings and rerankers, FAISS retrieval, self-consistency voting, MMR and bagging for diversity. The reader model for the test-time experiments is a publicly available Llama 3.1 8B. Even the pre-training is described in sufficient detail (Appendix A) to be approximately reproducible. This contrasts with commercial systems where the retrieval corpus, model architecture, and exact methods are unknown, making claims about data utilization unverifiable.

The work also carves out a nuanced position on the relationship between pre-training and inference compute. The finding that retrieval acts as a ~5× compute multiplier on MMLU (Section 4, Table 2) suggests that, at a given scale, dollars spent on inference-time retrieval can substitute for dollars spent on larger models—but the multiplier degrades with scale (7.17× → 2.88× from smallest to largest model). This is not a universal claim that retrieval beats pre-training; it's a characterization of when and how much. The paper is careful to note that both pre-training and retrieval leave room for improvement:

"our work suggests that there is room for improving both pre-training datasets and learning methods using these datasets" (Section 1)

The connection the paper draws to Mixture-of-Experts models (Section 7, citing Clark et al., 2022) is revealing. MoE models achieve efficiency gains by conditionally activating different parameter subsets—effectively doing a form of learned retrieval over their own weights. The paper suggests that retrieval from the training data and MoE routing might be addressing the same underlying problem: the difficulty of compressing the full training distribution into a fixed parameter set. Exploring this connection further is flagged as future work.

3. Technical Approach

3.1 Reader Orientation

This is primarily an empirical analysis paper whose core idea is that pre-training leaves a measurable amount of usable knowledge unexploited in the training data, and that retrieval-augmented generation (RAG)—combined with test-time compute strategies like self-consistency and reranking—can serve as a diagnostic instrument to quantify this residue: the performance gap between a base model and the same model augmented with retrieval from its own pre-training corpus directly measures how much the model failed to absorb. The system being analyzed is not a single novel model but rather a pipeline that stages retrieval over multiple pre-training datasets, augments the reader model's context with retrieved documents, and optionally applies parallel test-time compute (majority voting, document diversity mechanisms) to maximize the probability of extracting correct answers from information the model was already exposed to during training. Importantly, this pipeline is deployed in two distinct modes: (1) as a measurement apparatus trained on controlled pre-training runs (Section 4), where the base model, retrieval corpus, and training data are identical, enabling clean scaling-law analysis; and (2) as a practical system built on an off-the-shelf Llama 3.1 8B reader (Section 5), where the pre-training corpus and retrieval corpus differ but the techniques generalize.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected by a sequential pipeline:

The Retrieval Datastore — the same corpora used for pre-training (DCLM-baseline, FineWeb-edu, Wikipedia, arXiv, peS2o, PubMed Central, Stack Exchange, and seven math-specific sources), indexed for similarity search using FAISS with flat inner-product scoring. This is the "external memory" that holds documents the model already saw during training.
The Embedding and Reranking Stack — Qwen3 Embedding 0.6B converts the user's query into a dense vector; FAISS returns the top-100 most similar documents per dataset; optionally, Qwen3 Reranker 0.6B re-scores these 100 candidates to produce a final relevance ordering. This stack transforms a natural-language question into a ranked list of evidence passages.
The Reader Model — generates the final answer. In Section 4, this is a series of five decoder-only models pre-trained by the authors at different compute budgets (6.4B to 77.8B parameters) on the same datasets used for retrieval. In Section 5, this is an off-the-shelf Llama 3.1 8B instruct model. The reader receives the query plus the top-k retrieved documents concatenated into its context window and produces a chain-of-thought answer.
The Test-Time Compute Layer — optional parallelization and aggregation mechanisms applied on top of retrieval: self-consistency (majority voting across multiple generation trials, each with different retrieved documents), reranking (reordering documents by relevance before feeding to the reader), and variance-reduction techniques (MMR to increase document diversity across trials, bagging to randomize over document subsets). This layer trades additional inference FLOPs for higher answer accuracy.
The Decontamination Filter — an n-gram overlap detector (16-grams for MMLU, 26-grams for Math-500, in token space) that removes retrieved documents containing exact overlaps with test-set questions. This is applied only in the diagnostic mode (Section 4) to verify that retrieval gains are not driven by test-set leakage.

Information flow for a single evaluation query:

The user question enters the system → the embedding model encodes it → FAISS retrieves top-100 documents from each pre-training dataset shard → results are merged and sorted by similarity score → (optional) the reranker re-scores the merged top-100 to produce a final ranking → the top-k documents are concatenated with the question → the reader model generates an answer with chain-of-thought reasoning → (optional, in test-time compute mode) this process repeats N times with different document subsets or orderings, and answers are aggregated via majority voting → the final answer is compared against the ground truth.

What changes between Section 4 and Section 5:

In Section 4 (the diagnostic mode), the reader models are pre-trained by the authors on exactly the same data used for retrieval, and the retrieval corpus is the full, unrepeated pre-training dataset. This creates a closed system where any retrieval gain must reflect pre-training inefficiency. In Section 5 (the practical mode), the reader model is Llama 3.1 8B, which was pre-trained on Meta's proprietary data mixture—not the retrieval corpora—meaning retrieval may provide genuinely novel information in addition to recovering unabsorbed knowledge. The test-time compute experiments in Section 5 also add self-consistency, reranking, and variance reduction, which are not applied in Section 4's scaling-law analysis.

3.3 Roadmap for the Deep Dive

First, the pre-training pipeline (custom model series, dataset mixture, compute budgets, and the muP parameterization strategy), because the compute-multiplier analysis in Section 4 is anchored to these controlled training runs—understanding what "5× compute multiplier" means requires knowing exactly how the base models were trained.
Second, the retrieval pipeline (embedding, FAISS indexing, cross-dataset merging, and reranking), because every result depends on how documents are selected—the mechanism that transforms a question into retrieved context.
Third, the decontamination procedure and its dual role: validating that retrieval gains survive contamination removal while simultaneously quantifying benchmark leakage rates.
Fourth, the test-time compute mechanisms applied in Section 5 (self-consistency with retrieval, the reranker's role as a test-time upgrade to the retrieval tool, MMR and bagging as diversity/variance techniques, and inter-document consistency as an analytical reranker), since these form the practical deployment stack and produce the headline 10.5 percentage point MMLU gain.
Fifth, the compute-multiplier methodology (bounded sigmoid fitting to base-model performance, how the multiplier is computed, and what the category-level breakdowns reveal about retrieval's domain-specific effectiveness), since this is the paper's primary analytical contribution.
Finally, the custom Wikipedia extraction pipeline described in Section 6 and Appendix C, which serves as a concrete case study in how dataset engineering choices propagate to retrieval quality.

3.4 Detailed, Sentence-Based Technical Breakdown

The Pre-Training Pipeline (Controlled Model Series)

The paper pre-trains a series of five dense decoder-only transformer models at increasing compute budgets to serve as controlled baselines for the retrieval-as-compute-multiplier analysis. The pre-training recipe follows the "Honeycrisp" model series from the Axlearn training framework (Lee et al., 2025), which is publicly documented and approximately reproducible. The key design choice is that the pre-training data and the retrieval data are identical—the same corpora, the same tokenization, the same mixing ratios (detailed in Table 1). This makes the retrieval gains directly attributable to the access method rather than to novel information.

Model architecture and scaling. All models use a LLaMA-style architecture with SwiGLU feed-forward networks, Rotary Position Embedding (RoPE), and Grouped Query Attention (GQA) with a key/value-to-query ratio of 1:8. The five models span roughly an order of magnitude in both parameters and FLOPs, following a fixed tokens-per-parameter ratio of approximately 20:

Compute Budget (FLOPs)	Parameters	Training Tokens
5.64 × 10²¹	6.4B	147B
1.90 × 10²²	12.6B	252B
7.04 × 10²²	23.3B	503B
1.74 × 10²³	36.8B	786B
7.34 × 10²³	77.8B	1573B

The 20 tokens-per-parameter ratio is lower than the Chinchilla-optimal ratio (roughly 20 tokens per parameter for the compute-optimal point), placing these models in an "undertrained" regime relative to Chinchilla scaling laws. This is deliberate: training further would consume more compute and shift the analysis toward diminishing returns, making it harder to distinguish retrieval effects from pre-training saturation. The paper's focus is on characterizing how retrieval effectiveness changes with pre-training scale, not on achieving the best possible absolute performance.

Learning rate and optimization. Training uses a cosine-with-linear-warmup schedule with a peak learning rate of 1 × 10⁻², decaying to 0.01 of the peak (i.e., a final learning rate of 1 × 10⁻⁴). The paper uses muP (maximal update parameterization) to achieve learning rate transfer across model scales—a technique where hyperparameters tuned on a small proxy model transfer directly to larger models without re-tuning. This is critical for the scaling analysis because it ensures that differences in final performance across model sizes reflect genuine scaling effects rather than suboptimal hyperparameter choices at particular scales.

Dataset composition. The pre-training corpus (described in Section 3.1 and Table 1) is a mixture of web-crawl-based datasets and specialized sources. The web-crawl component comprises DCLM-baseline (deduplicated, 764.9B tokens) and FineWeb-edu (deduplicated, 197.6B tokens), for a total of 962.5B web tokens. The specialized sources add coverage in academic literature (arXiv: 28.7B tokens; peS2o: 71.9B; PubMed Central: 22.5B), community knowledge (Stack Exchange: 11.8B; Wikipedia: 21.7B), and mathematics (seven datasets totaling 77.6B tokens across AlgebraicStack, AutoMathText, FineMath-3+, FineMath-4+, OpenWebMath, and StackMathQA). The total corpus is approximately 1.2 trillion tokens.

Epoch counts and data repetition. Table 1 reports epochs for the largest pre-training run (77.8B parameters, 1573B tokens). Because all models train on a fixed 20 tokens-per-parameter ratio, smaller models train on proportionally fewer tokens. The epochs reported in the table are relative to the largest run—a dataset with 2.72 epochs for the 77.8B model would be seen approximately 2.72 × (1573B / dataset_size) times. This varying repetition rate matters for the retrieval analysis because it affects how thoroughly the model was exposed to each dataset's content. For instance, StackMathQA at 0.7B tokens is seen 558.67 epochs by the largest model (meaning each token is repeated over 500 times), while DCLM-baseline at 764.9B tokens is seen only 0.88 epochs (meaning only 88% of its tokens are seen even once). A model that has seen a document multiple times might have encoded its content more thoroughly into parameters, potentially reducing the marginal benefit of retrieving that same document at test time.

A critical experimental detail. The paper distinguishes between the full retrieval datastore (all tokens from all datasets, indexed and available at retrieval time) and the subset retrieval datastore (a random subset matching the size of what the base model actually saw during pre-training). In Figure 1, the dark blue "Full Retrieval" line gives the model access to more data than it saw during training—a document that appeared zero times in pre-training (because the model's token budget didn't cover it) could still be retrieved and used. The light blue "Subset Retrieval" line restricts retrieval to a random subset of each dataset exactly matching the token count consumed during pre-training. The fact that these two lines nearly overlap for MMLU and Math-500 (Figure 1) is a key finding: giving the model access to additional documents from the same distribution provides minimal extra benefit, suggesting that what matters is re-accessing documents the model was already exposed to, not accessing new ones from similar sources. The exception is SimpleQA (right panel of Figure 1), where the smaller two models show a piecewise function because they haven't seen a full epoch of Wikipedia—for them, the subset retrieval line is substantially below the full retrieval line because the full datastore provides Wikipedia documents they never encountered.

The Retrieval Pipeline

The retrieval system transforms a natural-language question into a ranked list of documents drawn from the pre-training corpus. The pipeline has four stages: embedding, approximate nearest-neighbor search, cross-dataset merging, and optional reranking.

Embedding model. Queries are encoded into dense vectors using Qwen3 Embedding 0.6B (Zhang et al., 2025a), a relatively small embedding model that produces fixed-dimensional representations. The choice of a 0.6B-parameter embedder reflects a practical tradeoff: larger embedding models might produce better representations but would consume inference compute that could otherwise be spent on the reader model. The embedder is frozen and applied to the query text directly, without any task-specific fine-tuning.

Indexing and search. Each pre-training dataset is indexed separately using FAISS FlatIP (Johnson et al., 2019)—"FlatIP" means flat (exact) inner product search, as opposed to approximate nearest-neighbor methods like IVF or HNSW. The inner product between the query embedding and each document embedding serves as the similarity score. For each dataset (or shard of a dataset, for very large corpora), FAISS retrieves the top-100 most similar documents.

The use of flat (exact) search rather than approximate methods is deliberate and reflects the paper's focus on measurement precision over deployment efficiency. Approximate methods introduce noise in the retrieval ranking, which would confound the analysis of how retrieval quality affects downstream accuracy. For the diagnostic purposes of this paper, exact search eliminates one source of variance, though it would be impractical at production scale.

Cross-dataset merging. After retrieving top-100 documents from each dataset, the system combines all results into a single candidate pool and sorts by the original similarity score. This is equivalent to having all datasets in a single FAISS index and retrieving the global top-100, but is described as "more practical due to compute constraints" (Section 3.2). The merge step is critical because different datasets contain different types of knowledge—a MMLU question about physics might find relevant passages in arXiv, Wikipedia, and Stack Exchange simultaneously. The flat merge by similarity score makes no attempt to balance source diversity at this stage; any diversity optimization happens later through MMR (Section 5).

Reranker. If reranking is enabled, the top-100 merged documents are passed through Qwen3 Reranker 0.6B (Zhang et al., 2025a), which assigns a new relevance score to each document given the query. The reranker is a cross-encoder—it processes the query and document jointly, rather than encoding them separately like the bi-encoder embedder—which enables more nuanced relevance judgments but is computationally more expensive per document. The reranker produces the final document ordering.

The paper demonstrates (Section 5, Tables 5–6) that reranking provides a consistent improvement over raw retrieval across nearly all tasks, typically adding 1–3 percentage points of accuracy. This is notable because it means that the raw embedding similarity alone doesn't fully capture what makes a document useful for the reader model—the reranker learns to prioritize documents whose content is more likely to lead to correct answers, even when the embedding similarity is comparable.

Top-k selection. The final step is selecting the top-k documents from the (optionally reranked) ordering and concatenating them with the query into the reader model's context window. The paper uses different values of k across experiments: the main retrieval results appear to use k=10 ("k=10" mentioned in Table 9 caption), while the LiveCodeBench results (Appendix G, Table 17) use k=3. The paper does not systematically sweep k, which is a limitation—the choice of how many documents to include trades off context saturation (too many documents may dilute attention) against information coverage (too few may miss the relevant passage).

The Decontamination Procedure

Decontamination serves two purposes in this paper: validating retrieval gains and quantifying benchmark leakage. The procedure (detailed in Appendix B) is an n-gram overlap filter applied in token space.

Mechanism. For MMLU, the system computes all 16-grams (consecutive sequences of 16 tokens) from each test-set question. For Math-500, it uses 26-grams. Any retrieved document that shares at least one n-gram with a test question is discarded entirely. The choice of n-gram length was determined through visual inspection: shorter n-grams produced too many false positives—the authors note that their tokenizer "splits numbers into single digits" (Appendix B), which means even short numeric sequences generate many spurious collisions with unrelated documents. The 16-gram and 26-gram thresholds were chosen empirically as the shortest lengths that eliminated obviously spurious matches while removing documents containing genuine test-question text.

Why token-space rather than character-space. Using the model's own tokenizer for decontamination is important because it captures the exact units the model processes. A character-level 16-gram might span very different semantic content than a token-level 16-gram, since the tokenizer compresses common words into single tokens. This choice means that the decontamination is conservative—it may miss some contamination where test content is restated with different token boundaries, but it avoids the false-positive problem of character-level matching.

What the decontamination reveals. The paper reports (Section 4) that 14.1% of MMLU questions and 32.0% of Math-500 questions appear in the commonly used open-source pre-training datasets. This is a finding in itself: it means that roughly one in seven MMLU questions and one in three Math-500 questions exist somewhere in the standard web-crawl and curated corpora used for pre-training. The higher rate for Math-500 likely reflects that math competition problems are often posted on forums, included in Stack Exchange, or archived in math-specific web sources that the pre-training crawl captures.

Decontaminated retrieval performance. In Figure 1, the red "Decontaminated Retrieval" line shows retrieval performance after removing contaminated documents. For MMLU, the decontaminated line is very close to the full retrieval line (dark blue), confirming that the gains are not driven by exact test-set overlap. For Math-500, the decontaminated line shows a larger drop from full retrieval—indicating that some Math-500 gains do come from contaminated documents—but the decontaminated line still substantially outperforms the base model. This is one of the paper's most important validity checks: it demonstrates that retrieval provides genuine knowledge recovery, not just cheating.

The paper explicitly declines to decontaminate for SimpleQA because "the nature of the evaluation task" (Section 4) makes it unworkable—SimpleQA questions are short factual queries like "Who wrote The Great Gatsby?", and removing all Wikipedia documents that share an n-gram with such a query would eliminate essentially all relevant retrieval results, since the answer document necessarily contains the query terms. This is a reasonable but important caveat: SimpleQA results in the paper are not decontaminated, so some fraction of the gains could reflect exact-match retrieval of the answer.

The Test-Time Compute Mechanisms (Section 5)

Section 5 moves from diagnostic measurement to practical system building, using Llama 3.1 8B instruct as the reader model and layering test-time compute techniques on top of retrieval. The core idea is that retrieval provides a vehicle for spending additional inference FLOPs in a data-driven way—each FLOP buys access to or processing over a specific document.

Self-consistency with retrieval (majority voting across trials). Standard self-consistency (Wang et al., 2022) generates N independent answers from the same prompt and selects the most common answer via majority voting. The paper's variant generates N answers, each from a different subset of retrieved documents, and then votes. This means that each trial conditions on a different "view" of the evidence—different documents, different orderings, different subsets—producing answer diversity that reflects the document diversity rather than just the model's stochastic sampling noise.

The key design decision is how to allocate the total generation budget across trials. The paper uses a fixed number of trials (the exact value is not explicitly stated for all experiments, but the total compute budget is presumably held constant when comparing self-consistency variants). Each trial retrieves k documents, generates one chain-of-thought answer, and contributes one vote. The final answer is the one receiving the most votes.

Tables 5 and 6 show that self-consistency provides additive benefits on top of retrieval. On MMLU All, retrieval alone achieves 76.6% while retrieval + self-consistency achieves 81.0% (+4.4 percentage points). On Math-500, retrieval alone achieves 56.7% while retrieval + self-consistency achieves 64.3% (+7.6 percentage points). This additivity is important because it suggests that self-consistency and retrieval address different failure modes: retrieval provides knowledge the model hasn't absorbed, while self-consistency provides robustness against reasoning errors in individual trials. If they addressed the same failure mode, we'd expect sub-additive gains (diminishing returns); the additivity indicates complementarity.

The reranker as a test-time compute upgrade. In the paper's framework, the reranker represents spending additional compute to improve the retrieval tool itself, rather than spending compute to run more trials with the same tool. This is a conceptual distinction the paper draws (Section 5):

"If we view retrieval as a tool for the LLM, then our methods use test-time compute to improve the tool itself. This contrasts with self-consistency by itself, which parallelizes the model without additional enhancement."

The reranker's cross-encoder architecture means it does more computation per candidate document than the bi-encoder embedder—it processes the query and document jointly through the full transformer, enabling richer relevance judgments. Tables 5–6 demonstrate that this additional per-document compute consistently improves results: on MMLU All, reranker adds 1.1 percentage points over raw retrieval (77.7% vs. 76.6%); on SimpleQA, the gain is substantial (74.0% vs. 65.7%, an 8.3 percentage point gain), likely because SimpleQA's factual nature makes precise document relevance critical.

MMR and Bagging (variance-reduction techniques). The paper applies two classic techniques to further improve the test-time compute stack:

MMR (Maximal Marginal Relevance) (Carbonell & Goldstein, 1998): When selecting the top-k documents to present to the reader, MMR balances relevance against diversity. Rather than taking the top-k by pure relevance score, MMR iteratively selects documents that are both relevant to the query and dissimilar to already-selected documents. In the paper's setting, this means that across multiple trials (when running self-consistency), different trials receive different document sets—each one covering a different aspect of the query. The goal is to increase the diversity of evidence the model sees, which should increase the diversity of generated answers and make majority voting more robust.

The paper doesn't specify the exact MMR tradeoff parameter λ (which controls the relevance-diversity balance), but the standard formulation is:

$\text{MMR} = \arg\max_{d \in D \setminus S} \left[ \lambda \cdot \text{sim}(d, q) - (1-\lambda) \cdot \max_{s \in S} \text{sim}(d, s) \right]$

where $D$ is the candidate document set, $S$ is the set of already-selected documents, $q$ is the query, and $\text{sim}$ is a similarity function (here, the embedding inner product). The first term $\lambda \cdot \text{sim}(d, q)$ rewards documents relevant to the query; the second term $-(1-\lambda) \cdot \max_{s \in S} \text{sim}(d, s)$ penalizes documents similar to those already selected.

Bagging (bootstrap aggregating) (Breiman, 1996): When generating N trials for self-consistency, each trial retrieves from a random subset of the available documents rather than from the full ranked list. This is a variance-reduction technique: by exposing each trial to a different sample of the evidence, the ensemble averages over retrieval noise. If a particular document is spuriously highly-ranked (perhaps due to embedding noise), it won't dominate all trials—only the fraction of trials that happened to sample it.

Together, MMR and bagging provide what the paper calls "VR" (variance reduction) in Tables 5–7. The effect on MMLU All is +1.1 percentage points (82.1% vs. 81.0% without VR), a modest but consistent gain that suggests retrieval quality variance is a real but not dominant limitation.

Inter-document consistency as an analytical tool. Section 5.2 and Appendix H describe an alternative approach to reranking that the paper uses diagnostically rather than as a practical method. The idea is: for each retrieved document, run self-consistency N times using only that document as context. The document whose self-consistency majority answer matches the final aggregated answer most often is the "best" document. In other words, inter-document consistency measures how reliably a single document leads the model to the correct answer via majority voting.

Table 8 shows that inter-document consistency selects better top-1 documents than the Qwen3 Reranker: 77.6% MMLU accuracy using inter-document consistency for top-1 selection vs. 73.7% using the reranker. This means that the reranker's relevance judgments don't perfectly capture what makes a document useful for generating correct answers—a document can look relevant but contain misleading information, or look less relevant but contain the key fact needed to answer correctly. The consistency-based ranking implicitly captures answer-utility rather than surface relevance.

However, the compute cost is prohibitive: running N self-consistency trials per document, for all candidate documents, multiplies the total generation budget by the number of documents being evaluated. The paper explicitly notes this inefficiency and suggests distilling the consistency signal into a more efficient reranker as future work:

"We leave to future work ways to distill self-consistency into a more efficient reranker." (Section 5.2)

The Compute-Multiplier Methodology

The paper's central analytical contribution is the compute-multiplier framework: quantifying retrieval's benefit in terms of how much additional pre-training compute would be needed to match the retrieval-augmented performance without retrieval. This is not a simple ratio of accuracies—it requires modeling the full accuracy-vs-FLOPs curve for the base model.

The sigmoid fit. The base model's MMLU accuracy as a function of pre-training FLOPs is modeled using a bounded sigmoid:

$y = 0.25 + \frac{0.6907}{1 + \exp\left(-0.7968 \cdot (\log_{10}(x) - \log_{10}(2.48 \times 10^{22}))\right)}$

where $y$ is the predicted MMLU accuracy, $x$ is the pre-training FLOPs, 0.25 is the random baseline (25% for 4-option multiple choice), and 0.9407 is the maximum achievable accuracy ceiling (from Gema et al., 2024, who estimated the upper bound of MMLU performance after accounting for irreducible ambiguities and errors in the benchmark itself).

What this equation computes: Given a pre-training FLOPs value $x$ , the equation outputs an estimated MMLU accuracy $y$ . The inner term $\log_{10}(x) - \log_{10}(2.48 \times 10^{22})$ measures how far the FLOPs are from the sigmoid's midpoint (2.48 × 10²² FLOPs, where accuracy is steepest). The sigmoid compression $1/(1 + \exp(-\text{gap}))$ maps this log-distance to a value between 0 and 1, which is then scaled by 0.6907 (the total accuracy range above random guessing) and shifted by 0.25 (the random baseline). As $x$ → 0, the sigmoid approaches 0, so $y$ → 0.25. As $x$ → ∞, the sigmoid approaches 1, so $y$ → 0.25 + 0.6907 = 0.9407.

Why this form: A sigmoid in log-FLOPs space is the standard functional form for scaling laws because it captures three empirical regularities simultaneously: (1) initial rapid improvement (the steep region around the midpoint), (2) diminishing returns at large scale (the saturation toward the ceiling), and (3) a non-zero lower bound (random performance). The bounded form explicitly acknowledges that MMLU accuracy cannot exceed 100% (and empirically saturates below 100%, at 94.07% per Gema et al.). An unbounded power law would wrongly predict that accuracy approaches 100% or diverges, which is physically impossible. The specific parameters (-0.7968 slope, 2.48 × 10²² midpoint) are fitted to the five empirical data points from the authors' pre-training runs.

The same fitting procedure is applied separately to each MMLU subject category (STEM, Humanities, Social Sciences, Other), with category-specific ceilings reported in Appendix E: STEM 0.9544, Humanities 0.9377, Social Sciences 0.9575, Other 0.9114.

Computing the multiplier. For each base model at its specific compute budget, the procedure is:

Measure the retrieval-augmented accuracy (the "Retrieval MMLU" column in Table 2).
Solve the fitted sigmoid equation for the FLOPs value $x$ that would give the base model that same accuracy: $x = \text{sigmoid}^{-1}(\text{retrieval\_accuracy})$ .
Divide this required FLOPs by the actual FLOPs of the base model to get the compute ratio.

For example, at the smallest budget (5.64 × 10²¹ FLOPs), the base model achieves 0.4873 MMLU and retrieval achieves 0.6063 MMLU. The sigmoid says the base model would need 2.98 × 10²² FLOPs to reach 0.6063. The ratio is 2.98 × 10²² / 5.64 × 10²¹ = 5.28×.

The degradation with scale. Table 2 reports compute ratios decreasing from 5.28× at the smallest scale to 2.88× at the largest, with an arithmetic mean of 4.86× and geometric mean of 4.66×. This degradation is a central finding: it means retrieval is most valuable when the base model is weakest (smallest compute budget), and its relative advantage shrinks as the model becomes more capable. The paper interprets this as the model gradually absorbing more of the available knowledge, leaving less for retrieval to recover. However, there's a nuance: the absolute accuracy gain from retrieval also varies non-monotonically (it's not a simple decline—the 1.90 × 10²² budget actually shows the largest absolute gain at +9.2 percentage points). The ratio declines because the base model's accuracy is growing faster than the retrieval gain is shrinking.

Category-level multipliers (Table 3). The breakdown by subject area reveals stark heterogeneity. Retrieval is a dramatically better compute multiplier for STEM (6.16× average) than for Humanities (2.52×). The "Other" category shows the largest multiplier at 9.27×. This pattern is surprising because retrieval involves externalizing factual knowledge, so one might expect it to help most on knowledge-intensive subjects like history and law (humanities). The paper investigates this counterintuitive finding in Section 4.1 and finds no clear correlation—subjects like "Medical genetics" (+21.1% from retrieval) and "Philosophy" (+17.9%) both benefit strongly, while "US foreign policy" (+17.7%) and "College physics" (+16.8%) are similarly strong. The mix of apparently fact-based and reasoning-based subjects in the top categories suggests that retrieval provides both factual knowledge recovery and a form of additional processing—seeing relevant documents may scaffold reasoning even when the facts themselves are already in parametric memory.

The Custom Wikipedia Extraction (Case Study in Dataset Engineering)

Section 6.2 and Appendix C present a concrete example of how dataset creation choices affect retrieval quality, using SimpleQA as the testbed. SimpleQA is a factuality benchmark where over 70% of answers can be found on Wikipedia.

The problem with standard extractions. Popular Wikipedia extractions on HuggingFace—Wikimedia (Nov. 2023) and OLM (June 2025)—use extraction pipelines that often fail on structured elements: bullet points, tables, and infoboxes. These elements frequently contain the most direct answers to factual queries (e.g., a table of Nobel Prize winners, an infobox with birth dates). When the extraction drops them, the retrieval system never sees the relevant text, regardless of how good the embedding or reranking is.

The custom extraction pipeline. The authors implement a two-stage extraction:

Lightweight pre-processing on raw HTML: remove script tags, style tags, unmatched meta tags, HTML comments, links, and images. This strips non-textual content without attempting to parse structure.
ReaderLM-v2 (Wang et al., 2025), a small model specialized in HTML-to-markdown conversion, processes the simplified HTML into structured plain text. ReaderLM-v2 is designed to preserve table structures, list formatting, and other layout information that naive extraction often loses.

The pipeline is applied to all pages from the Wikipedia domain found in the authors' general web crawl, resulting in a "Custom June 2025" Wikipedia dataset.

Quantitative impact. Table 10 shows the downstream effect on SimpleQA accuracy (using Llama 3.1 8B with reranker and k=6):

Wikipedia Source	SimpleQA Accuracy
Wikimedia Nov. 2023	55.4%
OLM June 2025	59.1%
Custom June 2025	69.0%
Custom + All Sources	73.7%
Custom + Golden Links	85.2%

The 13.6 percentage point gap between Wikimedia and the custom extraction (69.0 − 55.4) is entirely attributable to better extraction quality—the underlying Wikipedia content is the same, just better preserved. The further gains from adding "All Sources" (+4.7 points) and "Golden Links" (+11.5 points) show the impact of expanding the retrieval corpus beyond Wikipedia. The 11.5-point gap between "All Sources" and "Golden Links" indicates that non-Wikipedia sources in the standard pre-training crawl (CommonCrawl) miss many relevant documents, which the authors interpret as evidence that "open-source datasets could be further improved at the web crawling stage" (Section 6.2). Only a small fraction of the SimpleQA non-Wikipedia golden links are found in CommonCrawl, meaning the current web-crawl misses content that exists on the web but isn't included in the crawl.

Scaling retrieval data without degradation. Section 6.3 and Figure 3 investigate whether adding large amounts of potentially irrelevant data degrades the retrieval system. Starting from the custom Wikipedia baseline (22B tokens), the authors add increasing amounts of DCLM data (up to 765B tokens) and measure SimpleQA accuracy. The result: accuracy stays close to the baseline, showing only "a small distracting effect" (Section 6.3). This is robust behavior—a retrieval system that degrades substantially when irrelevant data is added would be brittle in practice. The paper notes that this robustness finding is specific to SimpleQA (factual knowledge), and the scaling effect might differ for reasoning tasks.

The dataset quality disconnect. Table 9 provides a notable finding: FineWeb-edu is substantially worse than DCLM as a pre-training dataset (42.9% vs. 53.4% MMLU for 8B models trained on 1T tokens, per Su et al., 2024), but is equally good or slightly better as a retrieval dataset (75.2% vs. 74.5% MMLU for retrieval on Llama 3.1 8B, rising to 76.6% vs. 76.4% with reranker). This means that the qualities that make a dataset good for pre-training (where the model must compress information into parameters through gradient updates) are not the same as the qualities that make a dataset good for retrieval (where the model reads relevant passages in context). The paper leaves the characterization of these differing qualities to future work, but the implication is significant: efforts to improve pre-training datasets may be optimizing for properties that are less relevant—or even counterproductive—for retrieval-based use.

Summary of Design Choices and Their Justifications

Identical pre-training and retrieval corpora for the diagnostic experiments (Section 4): This is the central experimental design choice. It isolates the access-method effect—any performance gain must come from the model failing to encode information it was exposed to, since the retrieval corpus contains no new information.
Subset retrieval to match pre-training token exposure: By restricting retrieval to a random subset matching what the model actually saw during training, the paper controls for the fact that the full retrieval corpus contains documents the model never encountered. The near-overlap between subset and full retrieval (Figure 1) shows that additional in-distribution documents provide minimal benefit—the information is already in the model, retrieval just helps access it.
Flat (exact) FAISS search over approximate methods: Eliminates retrieval noise as a confounding variable in the scaling-law analysis. In production, approximate search would be necessary, but the paper prioritizes measurement precision.
Cross-dataset top-100 + merge rather than global top-100: A practical compromise that ensures representation from each dataset while keeping the final candidate pool manageable (100 × number_of_datasets documents before merging).
N-gram decontamination in token space with empirically chosen n-gram lengths: The 16-gram and 26-gram thresholds were tuned by visual inspection to balance false positives (shorter n-grams) against false negatives (longer n-grams that miss paraphrased contamination). The paper doesn't claim this is optimal, but it's a reasonable heuristic.
Bounded sigmoid for compute-multiplier fitting: The upper bound (0.9407 for MMLU) prevents the sigmoid from extrapolating to impossible accuracy values, which an unbounded power law would do. The lower bound (0.25, random chance) reflects the multiple-choice format. The log-FLOPs input captures the log-linear scaling empirically observed in prior work.
Separate sigmoid fits per MMLU category: Different subject areas have different ceilings and different scaling rates, so a single fit would misrepresent the category-level multipliers.
Llama 3.1 8B as the reader for Section 5: Chosen because it is publicly available, relatively capable for its size, and small enough that test-time compute experiments are computationally feasible. The choice also tests whether the paper's findings generalize to models not pre-trained on the retrieval corpus.
Qwen3 embedding and reranker models: Both 0.6B parameters, chosen as a practical balance between retrieval quality and computational cost. The paper doesn't systematically compare embedding models, so these choices represent one reasonable configuration rather than an optimized setup.

4. Key Insights and Innovations

Innovation 1: Retrieval from the Pre-Training Corpus as a Diagnostic for Data Utilization Efficiency

The paper's most fundamental intellectual contribution is not retrieval-augmented generation itself—that technique has existed for years (Guu et al., 2020; Lewis et al., 2020)—but the repurposing of retrieval as a measurement instrument for pre-training efficiency. Prior work on retrieval treated external knowledge as complementary to pre-training: you retrieve from Wikipedia because the model wasn't trained on Wikipedia, or you retrieve from a domain-specific corpus because the model lacks specialized knowledge. The retrieval corpus and the pre-training corpus were distinct by design—the former providing information the latter didn't contain.

This paper inverts that framing entirely. By making the retrieval corpus identical to the pre-training corpus, it transforms retrieval from a knowledge supplement into a knowledge recovery probe. If a model trained on dataset D can answer a question when given retrieval access to D but cannot answer it from parametric memory alone, then the information existed in D all along—pre-training simply failed to encode it in a way the model can access. The performance gap between retrieval-augmented and base evaluation becomes a direct measurement of the "data value left behind" by the pre-training algorithm.

This is a conceptual shift analogous to the difference between asking "does the model know this?" and asking "could the model know this, given what it was trained on?" The paper operationalizes the latter question through a clean experimental design: identical corpora for training and retrieval, subset controls to account for unseen documents, and decontamination to rule out benchmark leakage. None of these individual elements is novel, but their combination into a systematic framework for auditing pre-training data utilization is.

Comparison to prior work: Classical scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) characterized how performance scales with compute, but treated data as a consumed input—they modeled loss as a function of tokens seen, not as a function of tokens that could have been learned from. The Chinchilla laws tell you how many tokens to train on for compute-optimal loss, but they tell you nothing about whether those tokens were well-utilized. Subsequent work on data-constrained scaling (Muennighoff et al., 2023b) studied what happens when you repeat data, but again focused on parametric performance, not on the residual information in the repeated data. This paper introduces a new axis to the scaling-law framework: utilization efficiency, measured not by loss but by the recoverable knowledge gap.

Significance beyond raw performance: The 5× compute multiplier on MMLU (Table 2) is the headline number, but the deeper significance of this innovation is that it provides a feedback loop for dataset and algorithm development. If retrieval from dataset D yields 80% on a benchmark while the base model achieves only 50%, the dataset is not the bottleneck—the learning algorithm is. Conversely, if retrieval from D yields only 55% while a different dataset D' yields 80%, the learning algorithm may be fine but the dataset is information-poor. Prior to this work, these two failure modes were indistinguishable from final model performance alone. The paper's framework separates them.

The finding that the compute multiplier degrades with scale (7.17× → 2.88× from smallest to largest model, Table 2) is also conceptually significant. It means that larger models do absorb more of the available knowledge, but the absorption is incomplete even at 77.8B parameters—retrieval still provides a 2.88× multiplier. This suggests that the parametric memory bottleneck is not trivially solved by scaling, at least within the scale range studied, and that inference-time access to training data continues to provide value even for models with substantial capacity.

Evidence anchor: Figure 1 (MMLU panel) showing the gap between base model and subset retrieval persisting across all five compute budgets, and Table 2 quantifying the degradation, are the primary evidence. The decontaminated retrieval line (red) tracking the full retrieval line (dark blue) in Figure 1 validates that the gap is not contamination-driven.

Innovation 2: Pre-Training Dataset Quality and Retrieval Dataset Quality Are Distinct Properties

A finding that runs counter to natural intuition: better pre-training datasets are not necessarily better retrieval datasets. Table 9 shows that DCLM substantially outperforms FineWeb-edu for pre-training (53.4% vs. 42.9% MMLU on 8B models trained for 1T tokens), but the two datasets are nearly equivalent for retrieval (74.5% vs. 75.2% MMLU on Llama 3.1 8B, with FineWeb-edu slightly ahead), and FineWeb-edu edges ahead with reranking (76.6% vs. 76.4%).

This finding challenges an implicit assumption in the data curation literature: that datasets which produce better pre-trained models are "better" in some universal sense. The paper demonstrates that dataset quality is use-dependent—the properties that make a document useful for gradient-based learning (where the model must compress information into weight updates through stochastic optimization) differ from the properties that make it useful for in-context retrieval (where the model reads and reasons over the document directly). A document with clear structure, explicit statements, and low noise might contribute little to pre-training if its information is redundant with what the model already knows from other documents, yet might be precisely what retrieval needs to provide decisive evidence at test time. Conversely, a document that is valuable for pre-training—perhaps because it contains rare tokens or unusual constructions that improve generalization—might be poorly suited as retrieval evidence if its key information is buried in complex prose.

Comparison to prior work: The data curation literature (Li et al., 2024; Penedo et al., 2024; Allal et al., 2025) has focused almost exclusively on pre-training quality metrics: perplexity, downstream task performance after pre-training, and text quality heuristics. Retrieval evaluation, when considered at all, typically uses dedicated retrieval benchmarks (BEIR, MS MARCO) that measure embedding and reranking quality independently of the pre-training pipeline. No prior work has systematically compared the same dataset's performance in both pre-training and retrieval regimes. This paper provides the first direct evidence that the ranking of datasets differs across these two uses, which has immediate practical implications: organizations building both pre-training and retrieval infrastructure cannot assume that optimizing their datasets for one use case optimizes them for the other.

Significance beyond raw performance: This is a negative result with constructive implications. It tells the community that the substantial investment in pre-training dataset curation—filtering, deduplication, quality scoring—may not translate to retrieval quality, and may even be counterproductive for retrieval if it removes documents that are structurally useful (tables, infoboxes, lists) but textually noisy. The practical implication is that dataset development should target the intended use case: if a dataset will serve both pre-training and retrieval roles, quality metrics should be evaluated separately for each.

Evidence anchor: Table 9 provides the direct comparison. The paper acknowledges that DCLM contains more tokens than FineWeb-edu but cites prior work showing this size difference does not explain the pre-training performance gap, while noting that size should not disadvantage DCLM for retrieval either. The result is presented without a complete mechanistic explanation—the paper doesn't identify which properties of FineWeb-edu make it better for retrieval despite being worse for pre-training—and this is appropriately flagged as future work.

Innovation 3: The Extraction and Crawling Pipeline Dominates Retrieval Quality for Factual Tasks

Through a careful ablation on SimpleQA (Section 6.2, Table 10), the paper demonstrates that seemingly mundane engineering choices in the data pipeline—how HTML is parsed, whether infoboxes survive extraction, which Wikipedia dump date is used—produce larger performance swings than sophisticated retrieval techniques like reranking. Changing the Wikipedia extraction method from the standard Wikimedia dump to a custom pipeline that preserves structured elements yields a 13.6 percentage point improvement (69.0% vs. 55.4%). Adding reranking on top of standard retrieval typically yields ~1-8 percentage points (Tables 5-6). Expanding the retrieval corpus from Wikipedia-only to all available sources adds another 4.7 points, and having access to the ground-truth golden links (the actual Wikipedia pages containing the answers) adds 11.5 points on top of that.

The conceptual punchline is that retrieval quality is bottlenecked earlier in the pipeline than most research attention has been directed. The retrieval literature emphasizes embedding models, reranking architectures, and indexing strategies—the components that come after the text is extracted. This paper shows that for factual tasks, the largest gains come from what happens before the retrieval system ever sees the text: how web pages are crawled, which elements are extracted, and whether tables and lists survive the conversion to plain text. The finding that only a "small fraction" of SimpleQA's non-Wikipedia golden links appear in CommonCrawl (Section 6.2, discussion around Table 10) further indicts the crawling stage: relevant documents exist on the web but are missed by standard crawls.

Comparison to prior work: The retrieval-augmented generation literature has primarily focused on improving the retrieval architecture—better dense retrievers (Shao et al., 2024), better rerankers (Zhang et al., 2025a), better fusion methods—while treating the underlying corpus as fixed. Dataset curation work for pre-training has focused on filtering and quality scoring. Neither research thread has systematically studied how HTML extraction and web crawling choices propagate to downstream retrieval performance. This paper provides a concrete, quantified case study that demonstrates these choices dominate in at least one important setting (factual QA).

Significance beyond raw performance: This is a re-prioritization finding. It suggests that the marginal research dollar in retrieval system improvement might be better spent on better extraction tools and more comprehensive web crawls than on another point of reranker accuracy. The 13.6-point gap from better extraction dwarfs the 1.1-point gain from reranking on MMLU or the ~3-point gain on MMLU from variance-reduction techniques. The magnitudes matter: they tell practitioners where to invest engineering effort for maximum return, and they tell researchers that extraction and crawling are underexamined variables in retrieval experiments.

The paper also demonstrates that this extraction quality difference is not captured by standard dataset metadata. The Wikimedia dump and the custom extraction both claim to be "Wikipedia"—they share the same URL, the same page count, the same provenance. A researcher comparing retrieval systems using the former might conclude their retriever is the bottleneck, when in fact their extraction pipeline silently discarded the most informative content. This has implications for reproducibility and benchmarking in the retrieval literature.

Evidence anchor: Table 10, supported by the qualitative observation in Section 6.2 that standard extractions "often fail to extract elements like bullet points, tables, and info boxes." The paper provides enough methodological detail in Appendix C (ReaderLM-v2 for conversion, lightweight HTML pre-processing) to make the extraction differences concrete, though it does not provide examples of text produced by each extraction method—a minor gap that would have strengthened the argument.

Innovation 4: Self-Consistency and Retrieval Address Complementary Failure Modes Yielding Additive Gains

The paper's test-time compute experiments in Section 5 reveal a pattern that is empirically straightforward but conceptually significant: self-consistency and retrieval provide additive gains across diverse tasks, suggesting they address fundamentally different limitations of the base model. On MMLU (Table 5), retrieval alone adds +5.0 percentage points over baseline (76.6% vs. 71.6%), self-consistency alone adds +3.7 points (75.3% vs. 71.6%), and the combination adds +9.4 points (81.0% vs. 71.6%)—approximately the sum of the individual gains. The same additive pattern holds on Math-500 (Table 6): retrieval alone +8.1 points, self-consistency alone +7.2 points, combined +15.7 points.

Additivity implies complementarity, and complementarity implies that the two techniques address non-overlapping failure modes. This is not obvious a priori. One could imagine that retrieval provides the model with correct information, and self-consistency then helps the model reason correctly over that information—in which case the gains would be multiplicative or synergistic, with self-consistency providing larger benefits when the information is correct. Alternatively, one could imagine that retrieval corrects the same errors that self-consistency would catch—if the model's primary failure mode is lacking factual knowledge, then providing that knowledge through retrieval would leave little for self-consistency to fix—in which case the gains would be sub-additive. The observed additivity suggests a cleaner picture: retrieval addresses knowledge failures (the model knowing the right fact), while self-consistency addresses reasoning failures (the model drawing the right conclusion from facts it possesses), and these failures are largely independent in practice.

The finding that retrieval helps on STEM tasks more than on humanities tasks (Table 3, 6.16× vs. 2.52× compute multiplier) reinforces this interpretation. STEM questions often require recalling specific formulas, constants, or domain-specific facts that retrieval surfaces directly. Humanities questions often require nuanced interpretation of ambiguous text—retrieval still helps, but the bottleneck is more likely to be reasoning than factual recall. The paper's analysis of top MMLU subjects by retrieval impact (Table 4) shows a mix of fact-heavy subjects (Medical Genetics: +21.1%) and reasoning-heavy subjects (Philosophy: +17.9%), suggesting the knowledge/reasoning distinction is not clean, but the aggregate STEM-vs-humanities pattern supports a knowledge-recovery role for retrieval.

Comparison to prior work: Snell et al. (2024) and Brown et al. (2024) studied test-time compute scaling in isolation, finding that self-consistency and sequential refinement improve performance. The RAG literature studied retrieval in isolation, finding that external knowledge improves performance. No prior work systematically tested whether these techniques are additive or redundant when combined. The paper provides the first clear evidence that they stack, and therefore that future systems should deploy both rather than choosing between them.

Significance beyond raw performance: This finding has direct implications for how practitioners should allocate inference budgets. If retrieval and self-consistency were redundant (sub-additive gains), a fixed compute budget would be better spent on whichever technique is more efficient. The additivity finding means that a budget split across both techniques outperforms either alone—each inference FLOP spent on retrieval buys a different kind of improvement than each FLOP spent on self-consistency, so diversifying the allocation is optimal. This is a practical design principle for test-time compute systems.

The paper also introduces a conceptual distinction between using test-time compute to improve the tool (reranking, MMR for better document selection) versus using it to run the tool more times (self-consistency across trials). This distinction is explicit in Section 5: "our methods use test-time compute to improve the tool itself. This contrasts with self-consistency by itself, which parallelizes the model without additional enhancement, as well as with deep research, which in addition to parallelizing also uses test-time compute to use the tool for longer rather than to upgrade it." This framing—test-time compute as tool improvement vs. tool usage—is a useful taxonomy for thinking about inference-time resource allocation.

Evidence anchor: Tables 5 and 6 across all task categories demonstrate the additivity. Table 3 shows the domain-specific compute multipliers. The paper does not itself frame the result as "additive gains implying complementarity"—this interpretation is supported by the data but is drawn out more explicitly here than in the paper's own discussion.

Innovation 5: The 14% / 32% Benchmark Contamination Quantification

The paper's decontamination analysis (Section 4, Appendix B) produces a finding that is methodological rather than architectural: 14.1% of MMLU test questions and 32.0% of Math-500 test questions appear in commonly used open-source pre-training datasets. This is a measurement, not a technique, but it has significant implications for how the field evaluates pre-training research.

The finding serves as both a validity check and a warning. As a validity check, it demonstrates that the retrieval gains reported in the paper are not purely contamination-driven—the decontaminated retrieval line (red, Figure 1) closely tracks the full retrieval line (dark blue), so even after removing any document containing test-set n-grams, retrieval still substantially improves performance. As a warning, it tells the research community that nearly one-third of Math-500 and one-seventh of MMLU exist in standard pre-training corpora, meaning that any model trained on these corpora has been exposed to test content during training. This doesn't necessarily mean the models memorized the answers—exposure is not the same as memorization—but it violates the principle of clean evaluation and creates an unknown confound in any benchmark comparison across models trained on similar data.

The paper's use of the decontamination measurement is dual: it validates its own claims while simultaneously flagging a broader methodological issue for the field. The explicit statement that these findings "highlight the importance of strictly decontaminated (or held out) evaluation sets for pre-training science" (Section 4) is a call to action, not just a footnote.

Comparison to prior work: Individual papers sometimes perform decontamination on their own training sets, but the systematic quantification of contamination rates in widely-used open-source datasets is rare. The 14.1% and 32.0% numbers provide concrete, actionable targets for benchmark creators: if you want your benchmark to be clean for evaluations using standard web-crawl-based pre-training data, you need to remove these overlapping items or release a held-out subset. The paper's choice of n-gram lengths (16-grams for MMLU, 26-grams for Math-500, in token space) was tuned by visual inspection rather than through a principled optimization, so these numbers are approximate—but they're the best available estimates for these specific dataset-benchmark pairs.

Significance beyond raw performance: This is primarily a methodological contribution to evaluation science. The fact that contamination rates differ dramatically across benchmarks (32% for Math-500 vs. 14% for MMLU) suggests that some benchmarks are more vulnerable than others, likely because math competition problems are more frequently posted online in verbatim form than multiple-choice exam questions. Researchers comparing models across benchmarks should account for differential contamination risk. The paper also implicitly demonstrates a methodology for estimating contamination rates: retrieve from the pre-training corpus, check for n-gram overlap, and report the fraction of test items with matches. This could be adopted as a standard reporting practice in model evaluation papers.

Evidence anchor: Section 4 reports the 14.1% and 32.0% figures directly. Figure 1 shows the decontaminated retrieval lines. Appendix B describes the n-gram procedure and the empirical tuning of n-gram length. The paper does not provide per-subject contamination rates for MMLU, which would have allowed analysis of whether certain subjects are more contaminated than others—this is a minor gap.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on four benchmarks: MMLU (Hendrycks et al., 2020), a 57-subject multiple-choice benchmark testing knowledge across STEM, humanities, social sciences, and other domains; Math-500 (Lightman et al., 2023), a 500-question subset of the MATH benchmark requiring open-ended mathematical reasoning; SimpleQA (Wei et al., 2024), a factuality benchmark where over 70% of answers are found on Wikipedia; and GPQA (Rein et al., 2024), a graduate-level QA benchmark across biology, chemistry, and physics. For the controlled pre-training experiments in Section 4, only MMLU, Math-500, and SimpleQA are used. For the test-time compute experiments in Section 5, GPQA is added. The pre-training models are evaluated in a few-shot setting (5-shot for MMLU, 4-shot chain-of-thought for Math-500). The Llama 3.1 8B experiments use zero-shot chain-of-thought prompting. All evaluations use the standard test splits from each benchmark. An additional code generation evaluation on LiveCodeBench (Jain et al., 2024) appears in Appendix G as a preliminary result.
Base model(s). The paper uses two distinct model families serving different analytical purposes. For the controlled scaling analysis (Section 4), the authors pre-train five dense decoder-only transformers at compute budgets from 5.64 × 10²¹ to 7.34 × 10²³ FLOPs, with parameters scaling from 6.4B to 77.8B and a fixed ~20 tokens-per-parameter ratio. These models use a LLaMA-style architecture with SwiGLU FFNs, RoPE positional encodings, and Grouped Query Attention (1:8 key/value-to-query ratio), trained with the muP parameterization for learning rate transfer across scales on exactly the same data mixture used for retrieval. For the test-time compute experiments (Section 5), the paper uses an off-the-shelf Llama 3.1 8B instruct model (Grattafiori et al., 2024) chosen because it is publicly available, relatively capable for its size, and computationally feasible for extensive test-time compute experiments—and also because its pre-training data differs from the retrieval corpus, testing generalization of the approach.
Metrics. Primary metrics are task-specific accuracy scores: MMLU reports macro-averaged accuracy across all 57 subjects (and broken down into STEM, Humanities, Social Sciences, and Other categories), Math-500 reports exact match accuracy on final answers (using universal self-consistency with the reader model as the checker for open-ended answers, compared against GPT-4.1 mini as an alternative checker in Appendix F), SimpleQA reports accuracy on short-form factual answers, and GPQA reports accuracy on multiple-choice graduate-level questions. The paper's central analytical metric is the compute multiplier: for each base model at a given FLOPs budget, the multiplier is the ratio of (1) the pre-training FLOPs the base model would need to match retrieval-augmented accuracy, to (2) the base model's actual FLOPs. This is computed by fitting a bounded sigmoid function to base model accuracy as a function of log-FLOPs, then inverting to find the FLOPs corresponding to the retrieval accuracy. Compute multipliers are reported for MMLU overall (Table 2) and per subject category (Table 3, Appendix E Tables 12-15). The test-time compute experiments (Section 5) also use compute multipliers relative to the Llama 3.1 8B baseline's assumed pre-training compute (Table 7), and additionally report category-level MMLU breakdowns for each method variant (Table 5). Statistical reliability is reported for GPQA and Math-500 results as means over 10 trials (Table 6).
Baselines. The paper employs multiple baselines organized by experimental section. In Section 4 (controlled pre-training): the base model evaluated without any retrieval serves as the primary baseline, with retrieval variants compared against it. The subset retrieval condition (random subset matching the pre-training token exposure) serves as a controlled baseline that accounts for the fact that the full retrieval corpus contains documents the model never saw during training. The decontaminated retrieval condition removes documents with n-gram overlap with test questions, serving as a baseline against the concern that retrieval gains are contamination-driven. In Section 5 (test-time compute on Llama 3.1 8B): the reader model baseline is the Llama 3.1 8B instruct model evaluated zero-shot with chain-of-thought reasoning and no retrieval. The self-consistency baseline (Wang et al., 2022) generates multiple independent answers from the same prompt and selects the majority answer, without retrieval. The retrieval baseline uses the full retrieval pipeline (Qwen3 embedding, FAISS top-100, cross-dataset merge) without reranking, self-consistency, or variance reduction. Each subsequent method (adding reranker, adding self-consistency, adding variance reduction) is compared against the immediately simpler variant, creating a cumulative ablation chain. For SimpleQA (Table 10), the baselines include standard Wikipedia extractions: Wikimedia (Nov. 2023) and OLM (June 2025) (Thrush et al., 2022), compared against the custom extraction. For the dataset comparison in Table 9, DCLM and FineWeb-edu are compared as both pre-training corpora (using 8B models from Su et al., 2024) and retrieval corpora (using the paper's own retrieval pipeline on Llama 3.1 8B). No external retrieval-augmented baselines from prior work (e.g., REALM, Atlas, RAG with dedicated retrieval corpora) are directly compared, since the paper's contribution is the diagnostic methodology rather than state-of-the-art retrieval performance.
Generation budget / compute accounting. Two distinct notions of compute appear in the paper. For the controlled pre-training experiments (Section 4), compute is measured in total pre-training FLOPs, using the standard approximation from scaling laws: approximately 6ND FLOPs for training, where N is parameters and D is training tokens. The five models' FLOPs budgets are computed and reported directly (5.64 × 10²¹ through 7.34 × 10²³). For the test-time compute experiments (Section 5), the generation budget is implicitly measured in terms of inference FLOPs consumed by the reader model, but the paper does not explicitly report how many trials are used for self-consistency or what the total inference FLOPs cost is for each configuration. The compute-multiplier calculation in Section 5 (Table 7) maps Llama 3.1 8B accuracy back to pre-training FLOPs using the sigmoid fit from Section 4, implicitly assuming that Llama 3.1 8B lies on the same accuracy-vs-FLOPs curve as the authors' pre-trained models—an approximation the paper acknowledges as reasonable but approximate since Llama 3.1 8B is trained at a much higher tokens-per-parameter ratio. The retrieval pipeline's own compute cost (embedding model inference, FAISS search, reranker inference) is not included in any compute budget calculation. The inter-document consistency experiment (Section 5.2) explicitly notes that it "requires calling the reader model many more times" and is "not compute efficient when compared to self-consistency on all documents at once," but no explicit FLOPs accounting is provided. For the retrieval datastore scaling experiment (Section 6.3, Figure 3), the datastore size is measured in tokens (48B to 765B tokens of DCLM data), and retrieval system performance is measured as a function of datastore size, not compute cost. The custom extraction pipeline's computational cost (ReaderLM-v2 inference) is not quantified.
Cross-validation / statistical protocol. The paper does not employ cross-validation for model selection or hyperparameter tuning. The sigmoid fits for the compute-multiplier analysis (Section 4) are fitted to five data points (the five pre-training runs) using a bounded functional form with fixed lower bound (0.25) and fixed upper bound (from Gema et al., 2024: 0.9407 for MMLU overall, category-specific ceilings in Appendix E). The fits are parametric (3-parameter sigmoid: slope, midpoint, and scale), and goodness-of-fit is not quantitatively reported—the paper shows the fitted curve implicitly through the compute-multiplier calculations but does not provide R² values or confidence intervals. For the test-time compute experiments (Section 5), GPQA and Math-500 results are reported as means over 10 trials (Table 6 caption), which provides some estimate of variance due to sampling stochasticity, but confidence intervals or standard deviations are not reported. MMLU results in Tables 5 and 7 are presented as point estimates without variance information. The inter-document consistency experiment (Table 8) reports point estimates without trial counts. The contamination analysis (Section 4, Figure 1) applies a deterministic n-gram filter and reports the fraction of test items with matches as point estimates (14.1% for MMLU, 32.0% for Math-500), without sensitivity analysis of the n-gram length choice. The decontaminated retrieval evaluation is a single run per condition. Overall, the paper prioritizes measurement precision (exact FAISS search to eliminate retrieval noise, controlled pre-training runs) over statistical rigor (confidence intervals, significance tests), which is appropriate for an exploratory diagnostic study but limits the strength of quantitative comparisons, especially when differences between methods are small (e.g., the 0.4 percentage point gap between retrieval and retrieval + reranker on MMLU STEM in Table 5).

Main Quantitative Results

4.1: Retrieval on Pre-Training Data as a Compute Multiplier

The foundational measurement of the paper appears in Figure 1 and Table 2. Across all five pre-training compute budgets, retrieving from the full pre-training corpus substantially improves MMLU accuracy over the base model. At the smallest budget (5.64 × 10²¹ FLOPs, 6.4B parameters), retrieval lifts MMLU from 0.4873 to 0.6063, a +11.9 percentage point absolute gain. At the largest budget (7.34 × 10²³ FLOPs, 77.8B parameters), retrieval lifts MMLU from 0.7633 to 0.8186, a +5.5 percentage point gain. The absolute gain shrinks as base model capability improves, but retrieval remains meaningful even at 77.8B parameters.

Subset vs. full retrieval: The light blue "Subset Retrieval" line in Figure 1 shows retrieval restricted to a random subset matching the exact token count the base model consumed during pre-training. On MMLU and Math-500, this line closely tracks the dark blue "Full Retrieval" line—meaning that giving the model access to additional documents from the same distribution provides minimal extra benefit. On SimpleQA, the two smaller models show a gap between subset and full retrieval because they haven't seen a full epoch of Wikipedia—the full retrieval corpus provides Wikipedia documents they never encountered during pre-training. At the three largest budgets, where the models have seen at least one full epoch of Wikipedia, the subset and full lines converge for SimpleQA as well.

Decontamination validity: The red "Decontaminated Retrieval" line in Figure 1 removes all retrieved documents sharing a 16-gram (MMLU) or 26-gram (Math-500) with test questions. For MMLU, the decontaminated line nearly overlaps with the full retrieval line, confirming that gains are not contamination-driven. For Math-500, the decontaminated line drops below full retrieval but remains substantially above the base model, indicating that some Math-500 gains come from contaminated documents but the majority do not. The paper explicitly reports contamination rates: 14.1% of MMLU questions and 32.0% of Math-500 questions appear in the open-source pre-training datasets.

Compute multipliers (Table 2): Fitting a bounded sigmoid to base model MMLU accuracy as a function of log-FLOPs yields the equation: y = 0.25 + 0.6907 / (1 + exp(-0.7968 · (log₁₀(x) - log₁₀(2.48 × 10²²)))), with lower bound 0.25 (random chance) and upper bound 0.9407 (from Gema et al., 2024). Solving this equation for the FLOPs needed to match retrieval accuracy at each compute budget produces the multipliers:

Compute Budget	Base MMLU	Retrieval MMLU	Compute for Base to Match	Multiplier
5.64 × 10²¹	0.4873	0.6063	2.98 × 10²²	5.28×
1.90 × 10²²	0.6021	0.6943	1.36 × 10²³	7.17×
7.04 × 10²²	0.6623	0.7410	3.34 × 10²³	4.74×
1.74 × 10²³	0.7107	0.7775	7.35 × 10²³	4.23×
7.34 × 10²³	0.7633	0.8186	2.11 × 10²⁴	2.88×

The arithmetic mean multiplier is 4.86×, the geometric mean is 4.66×, and the median is 4.74×. The multiplier follows a non-monotonic pattern: it increases from 5.28× to 7.17× between the first two budgets before declining to 2.88× at the largest scale. The paper notes this initial increase but does not deeply analyze it—it may reflect the smallest base model being too weak to effectively use retrieval (the 6.4B model might struggle even to read and reason over retrieved documents), with retrieval efficiency peaking when the base model is capable enough to process retrieved context but not yet capable enough to have absorbed most of the knowledge parametrically.

Category-level multipliers (Table 3 and Appendix E): The breakdown by MMLU subject area reveals substantial heterogeneity. Retrieval acts as a 6.16× average compute multiplier for STEM, but only 2.52× for Humanities, 3.52× for Social Sciences, and 9.27× for Other. The "Other" category's high multiplier is driven partly by its lower sigmoid ceiling (0.9114 vs. 0.9544 for STEM), which means smaller absolute accuracy differences translate to larger FLOPs ratios in the saturation regime. However, the ranking is consistent across compute budgets and across mean, geometric mean, and median aggregations.

Top MMLU subjects by retrieval impact (Table 4): For the 6.4B model, the ten subjects gaining most from full retrieval are Medical Genetics (+21.1 percentage points), Miscellaneous (+19.2), World Religions (+18.5), Philosophy (+17.9), US Foreign Policy (+17.7), International Law (+17.0), High School Physics (+16.9), Virology (+16.9), College Physics (+16.8), and College Medicine (+16.6). The paper notes the mix of apparently fact-based and reasoning-based subjects—neither a pure "retrieval helps factual recall" nor a pure "retrieval helps reasoning" pattern emerges. The subset retrieval column (Table 4) shows similar patterns, confirming that the gains come from re-accessing documents the model saw during training, not from novel documents.

Answer change analysis (Section 4.1): The paper compares the 6.4B and 12.6B models to determine whether retrieval and model scaling affect different types of problems. Increasing model size changes answers for 39.7% of MMLU problems; adding retrieval changes answers for 28.1% of problems. Problems where retrieval corrected an answer that scaling did not include professional law, professional psychology, high-school macroeconomics, philosophy, and high-school mathematics. Problems where scaling corrected an answer that retrieval did not include professional law, professional psychology, moral scenarios, elementary mathematics, and high school statistics. The substantial overlap in subjects suggests "there is not a strong bias for what kinds of problem retrieval can help with compared to increasing model size" (Section 4.1).

Math-500 and SimpleQA scaling (Figure 1, middle and right panels): On Math-500, retrieval provides gains at all compute budgets, with the base model achieving 0.20–0.55 accuracy and retrieval achieving 0.30–0.60 depending on scale. The gap is roughly 5–10 percentage points. On SimpleQA, the three largest models (which have seen at least one full epoch of Wikipedia) show retrieval gains that narrow from roughly 0.55 → 0.65 at the 7.04 × 10²² budget to roughly 0.65 → 0.70 at the 7.34 × 10²³ budget. The two smallest models show a piecewise pattern with a large gap between subset and full retrieval caused by incomplete Wikipedia coverage.

4.2: Test-Time Compute on Llama 3.1 8B (Section 5)

The test-time compute experiments use Llama 3.1 8B instruct as the reader model and layer retrieval, reranking, self-consistency, and variance-reduction techniques. Tables 5 and 6 report the primary results across MMLU, SimpleQA, Math-500, and GPQA.

MMLU (Table 5): The baseline Llama 3.1 8B achieves 71.6% MMLU (macro average). Self-consistency alone lifts this to 75.3% (+3.7 percentage points). Retrieval alone (without reranker) achieves 76.6% (+5.0 points). Adding the reranker yields 77.7% (+1.1 points over retrieval alone). Adding self-consistency on top of reranked retrieval yields 81.0% (+3.3 points over reranked retrieval alone). Adding variance reduction (MMR + bagging) yields 82.1% (+1.1 points over reranker + self-consistency). The cumulative gain from baseline to the full pipeline is +10.5 percentage points.

The gains vary by MMLU category. STEM shows the largest absolute lift: baseline 67.3% → full pipeline 80.2% (+12.9 points). Humanities shows a more modest lift: 71.5% → 79.5% (+8.0 points). Social Sciences: 76.6% → 87.4% (+10.8 points). Other: 73.0% → 82.3% (+9.3 points). The self-consistency gain is largest on STEM (+5.4 points over reranked retrieval: 80.2 − 74.8, from rows in Table 5, though note the baseline + self-consistency on STEM is 72.3% vs. 67.3% baseline, so +5.0 points from self-consistency alone). The reranker gain is largest on Social Sciences and Humanities (+1.6 and +1.7 points respectively over retrieval alone), while on STEM the reranker adds only +0.1 points (73.7% vs. 73.6%).

SimpleQA (Table 6): Baseline accuracy is 1.5%—Llama 3.1 8B essentially cannot answer SimpleQA questions from parametric memory alone. Retrieval alone achieves 65.7%. Adding the reranker yields 74.0% (+8.3 points). Self-consistency is marked "N/A" for SimpleQA because it is a factuality benchmark—the correct answer is a single fact, so majority voting across trials does not meaningfully aggregate evidence. The 74.0% with reranked retrieval represents a +72.5 percentage point gain over baseline, demonstrating that for pure factual knowledge, nearly all the necessary information exists in the retrieval corpus and the bottleneck is access, not reasoning.

Math-500 (Table 6, left section): Baseline accuracy is 48.7%. Self-consistency alone achieves 55.9% (+7.2 points). Retrieval alone achieves 56.7% (+8.0 points). Reranked retrieval achieves 56.8% (+0.1 points over retrieval alone). Adding self-consistency to reranked retrieval achieves 64.3% (+7.5 points over reranked retrieval alone, and +15.6 points over baseline). Variance reduction adds essentially nothing on Math-500: 64.4% (+0.1 points). The paper reports results using Llama 3.1 8B as the universal self-consistency checker; Appendix F (Table 16) compares against GPT-4.1 mini as the checker, which yields higher absolute scores (e.g., 71.8% for the full pipeline) but preserves the same relative pattern. The alternative checker is presented as a robustness check rather than the primary result.

GPQA (Table 6, right section): Baseline GPQA accuracy is 30.6% (averaged across biology, chemistry, and physics). Self-consistency alone achieves 31.4% (+0.8 points), a much smaller gain than on MMLU or Math-500. Retrieval alone achieves 33.2% (+2.6 points). Reranked retrieval achieves 34.8% (+1.6 points). Adding self-consistency yields 36.1% (+1.3 points). Adding variance reduction yields 36.8% (+0.7 points). The cumulative gain is +6.2 percentage points, substantially smaller than on MMLU (+10.5) or Math-500 (+15.7). By subject: Biology shows the largest retrieval gain (45.1% → 49.7% from baseline to full pipeline, +4.6 points), Chemistry shows a moderate gain (27.3% → 29.6%, +2.3 points with an odd dip for retrieval alone at 27.3% matching baseline), and Physics shows the largest absolute and relative gain (28.3% → 38.3%, +10.0 points).

Compute multipliers for test-time methods (Table 7): Using the sigmoid fit from Section 4 (despite Llama 3.1 8B being a different model family trained at different tokens-per-parameter ratios), the paper estimates how many times more pre-training compute the baseline model would need to match each method's performance. Self-consistency alone: 2.10× multiplier. Retrieval alone: 2.78×. Reranked retrieval: 3.56×. Reranked retrieval + self-consistency: 8.14×. Full pipeline with variance reduction: 11.10×. The category-level breakdown shows STEM benefiting most from the full pipeline (15.72×) and Humanities benefiting least (5.68×). This pattern—test-time compute being a larger multiplier for STEM than Humanities—mirrors the pre-training retrieval multiplier pattern in Table 3.

Inter-document consistency (Table 8, Section 5.2): Using inter-document consistency as a reranker—running self-consistency on each retrieved document individually and selecting the document with the highest self-consistency rate—achieves 77.6% MMLU (top-1 document) compared to 73.7% for the Qwen3 Reranker (top-1). The gains appear across all MMLU categories, with the largest margin on STEM (75.4% vs. 71.2%). The paper explicitly notes this is "not compute efficient"—it requires running N self-consistency trials per document, for all candidate documents—but demonstrates that the reranker's relevance judgments are suboptimal for downstream task accuracy compared to a task-specific metric.

4.3: Dataset Quality and Pipeline Analysis (Section 6)

Pre-training quality ≠ retrieval quality (Table 9): FineWeb-edu produces substantially worse pre-trained models than DCLM (42.9% vs. 53.4% MMLU for 8B models trained on 1T tokens, from Su et al., 2024), yet is equally good or slightly better as a retrieval corpus. With Llama 3.1 8B as reader and k=10, retrieval on DCLM achieves 74.5% MMLU; on FineWeb-edu it achieves 75.2%. With reranking, the gap is 76.6% (FineWeb-edu) vs. 76.4% (DCLM). The paper does not investigate why—identifying the properties that make a dataset good for retrieval versus pre-training is left to future work—but the result demonstrates that dataset optimization for pre-training does not automatically optimize for retrieval.

Wikipedia extraction quality (Table 10): Using the custom extraction pipeline (ReaderLM-v2 for HTML-to-markdown, preserving tables and structured elements) yields 69.0% SimpleQA accuracy with reranked retrieval (k=6, Llama 3.1 8B reader). The standard Wikimedia extraction (Nov. 2023) achieves only 55.4%—a 13.6 percentage point gap attributable entirely to extraction quality. The OLM extraction (June 2025) achieves 59.1%, intermediate between the two. Adding non-Wikipedia sources to the custom Wikipedia baseline yields 73.7% (+4.7 points). Adding SimpleQA's golden links (the actual Wikipedia pages containing answers) yields 85.2% (+11.5 points over all sources, +16.2 points over custom Wikipedia alone). The paper notes that "only a small fraction of the non-Wikipedia golden links are present in CommonCrawl," indicting the web crawling stage as well as the extraction stage.

Robustness to datastore scaling (Figure 3): Starting from a custom Wikipedia baseline of 22B tokens, adding increasing amounts of DCLM data (48B → 765B tokens) produces only small changes in SimpleQA accuracy. The "Custom Wikipedia + DCLM" line stays near the ~72–74% range across all DCLM sizes. When also starting with golden links, adding DCLM causes a small decline (from ~86% to ~83% at 765B DCLM). The paper interprets this as demonstrating that "our retrieval system is fairly robust to scaling the retrieval datastore, even if the new data does not contain useful information," but cautions that "SimpleQA measures factual knowledge, and the scaling effect may be different for reasoning tasks."

Detokenized subset vs. random subset (Appendix D, Table 11): At the smallest compute budget (5.64 × 10²¹ FLOPs), the paper compares retrieval from a random subset (matching the token count of what the model saw during pre-training) against retrieval from the exact detokenized pre-training data. For MMLU, the random subset yields 59.0% and the exact subset yields 58.6%. For Math-500, random yields 32.0% and exact yields 26.8%—a 5.2 percentage point gap that the paper attributes to "randomness or contamination" (since the exact pre-training data might contain test-set contamination that a random subset would miss, but the model was actually trained on the exact data, so any contamination existed during pre-training). For SimpleQA, random yields 35.5% and exact yields 33.9%. The paper notes that the gap shrinks at larger compute budgets because the random and exact subsets have increasing overlap as the training token count grows relative to the dataset size.

Ablation Studies and Robustness Checks

Subset vs. full retrieval (Figure 1, light blue vs. dark blue lines): Retrieving from a random subset matching the pre-training token exposure yields nearly identical performance to retrieving from the full dataset on MMLU and Math-500, confirming that additional in-distribution documents provide minimal benefit—the value comes from re-accessing documents the model already saw, not from accessing new ones from the same distribution. The exception is SimpleQA at the two smallest compute budgets, where models haven't seen full Wikipedia epochs and the full retrieval corpus provides genuinely novel documents.
Decontaminated vs. full retrieval (Figure 1, red vs. dark blue lines): Removing documents with n-gram overlap with test questions produces retrieval performance nearly identical to full retrieval on MMLU, confirming gains are not contamination-driven. On Math-500, decontaminated retrieval drops below full retrieval but remains substantially above baseline, indicating partial contamination impact but confirming genuine knowledge recovery is the dominant effect. Contamination rates are quantified at 14.1% (MMLU) and 32.0% (Math-500).
Reranker vs. raw retrieval (Tables 5 and 6, "w/ retrieval" vs. "w/ reranker" rows): The reranker provides consistent but modest gains across tasks: +1.1 points on MMLU All (77.7% vs. 76.6%), +0.1 points on Math-500 (56.8% vs. 56.7%), +8.3 points on SimpleQA (74.0% vs. 65.7%), and +1.6 points on GPQA All (34.8% vs. 33.2%). The SimpleQA gain is notably larger, likely because factual queries have a single correct document and the reranker excels at identifying it precisely.
Self-consistency additivity (Tables 5 and 6, comparing individual gains to combined gains): On MMLU All, self-consistency alone adds +3.7 points, retrieval alone adds +5.0 points, combination adds +9.4 points (approximately additive). On Math-500, self-consistency alone adds +7.2 points, retrieval alone adds +8.0 points, combination adds +15.7 points (roughly additive). On GPQA, self-consistency alone adds +0.8 points, retrieval alone adds +2.6 points, combination adds +5.5 points (slightly super-additive, though the gain may be within noise given the small 10-trial sample). The additivity indicates that self-consistency and retrieval address complementary failure modes.
Variance reduction (MMR + bagging) (Tables 5 and 6, "+ VR" rows): Adding MMR and bagging on top of reranked retrieval + self-consistency yields: +1.1 points on MMLU All (82.1% vs. 81.0%), +0.1 points on Math-500 (64.4% vs. 64.3%), +0.7 points on GPQA All (36.8% vs. 36.1%). The gains are small and unsystematic, suggesting retrieval variance is not a dominant failure mode at these accuracy levels with Llama 3.1 8B. The MMLU gain is concentrated in STEM (+1.5 points) and Social Sciences (+1.6 points).
Inter-document consistency as reranker (Table 8): This alternative reranking method (running self-consistency per document) outperforms the Qwen3 Reranker for top-1 document selection (77.6% vs. 73.7% MMLU All), demonstrating that surface relevance does not perfectly capture answer utility. However, the method is computationally prohibitive for practical use.
Detokenized subset vs. random subset (Table 11): At the smallest compute budget, retrieving from the exact pre-training data (detokenized) vs. a random subset of equivalent size yields: MMLU 58.6% vs. 59.0% (similar), Math-500 26.8% vs. 32.0% (the larger gap is attributed to randomness or contamination), SimpleQA 33.9% vs. 35.5% (similar with slight advantage to random). The gap shrinks at larger budgets, validating that random subset is a reasonable approximation for subset retrieval.
GPT-4.1 mini checker for Math-500 (Table 16): Using a stronger model as the universal self-consistency checker for Math-500 raises absolute scores (e.g., 71.8% vs. 64.4% for the full pipeline) but preserves the relative ordering of methods, confirming that the pattern of gains is not an artifact of the checker model.
LiveCodeBench code generation (Table 17): As a preliminary result for domain generalization, retrieval from the Stack v2 and CommitPack (Python portions) with k=3 improves GPT-4o LiveCodeBench Code Generation from 37.93% to 42.76% (+4.83 points), suggesting the approach extends to code. This is a single data point with a proprietary model, so it is appropriately presented as an initial finding rather than a robust result.
MMLU subject-level retrieval impact (Table 4): The breakdown shows which subjects gain most from retrieval (e.g., Medical Genetics +21.1%, Miscellaneous +19.2%) and confirms that the subset retrieval pattern mirrors the full retrieval pattern, reinforcing that gains come from re-accessing seen documents rather than accessing novel ones.

Critical Assessment

The paper's central empirical claim is that retrieving from the pre-training corpus acts as a ~5× compute multiplier on MMLU, and that the full test-time compute stack provides an ~11× multiplier relative to the Llama 3.1 8B baseline. How well do the experiments support these claims, and what are the limitations?

The ~5× compute multiplier claim rests on a sigmoid fit to five data points. The fitting procedure is mathematically sound—a bounded sigmoid in log-FLOPs space is appropriate for accuracy metrics with known lower and upper bounds—but the small number of data points (five) means the fitted curve is substantially underdetermined. Three parameters (slope, midpoint, and scale—the lower and upper bounds are fixed) are being fitted to five observations, leaving only two degrees of freedom for assessing goodness-of-fit. The paper reports no R², residual analysis, or confidence intervals for the fitted parameters. This matters because small perturbations in the fitted curve can produce large changes in the computed multiplier, especially when interpolating or modestly extrapolating. For instance, at the 1.90 × 10²² budget, the computed multiplier is 7.17×—the highest in the table. The base model achieves 0.6021 MMLU and retrieval achieves 0.6943. The sigmoid maps this retrieval accuracy to 1.36 × 10²³ FLOPs, and the ratio is 7.17×. But if the sigmoid were slightly steeper in this region (perhaps because the 7.04 × 10²² data point were a bit higher or lower), the required FLOPs could change substantially, since the sigmoid is steepest near its midpoint (2.48 × 10²²). The non-monotonic pattern (5.28× → 7.17× → 4.74×) is unusual for a scaling law and may reflect fitting noise rather than a genuine phenomenon. The paper does not discuss this anomaly or provide error estimates.

More fundamentally, the paper does not validate the sigmoid fit against held-out data points. All five pre-training runs are used for fitting, and the same five runs are used for computing multipliers. This is circular in the weak sense: the fit is evaluated on the same data it was trained on. A proper validation would involve leaving out one or two runs, fitting on the remainder, and checking whether the fitted curve predicts the held-out performance. Without this, we cannot distinguish a good fit from an overfitted one.

The ~11× multiplier for test-time compute (Table 7) applies the same sigmoid to Llama 3.1 8B. The paper acknowledges that Llama 3.1 8B "is trained at a much higher tokens per parameter ratio than the models in Section 4" and that "estimates using our previous sigmoid fits would be a lower bound for compute multipliers because of diminishing returns at higher tokens per parameter counts." This is a reasonable argument—overtrained models are further along the saturation curve, so the same absolute accuracy gain corresponds to a larger FLOPs multiplier—but it also means the 11.10× figure is not directly comparable to the 4.86× figure from Section 4. The former represents a lower bound (since Llama 3.1 is overtrained relative to the models used for fitting), while the latter represents a fitted value. The paper also doesn't account for the fact that Llama 3.1 8B's pre-training data is different from the retrieval corpus used in Section 5—the models in Section 4 were trained on the same data used for retrieval, while Llama 3.1 8B was not. This means retrieval in Section 5 may be providing genuinely novel information (documents the model never saw), not just recovering unabsorbed knowledge. The paper's framework doesn't cleanly distinguish these two sources of gain for the Section 5 experiments.

The test-time compute experiments lack explicit compute accounting. Section 4 measures compute in pre-training FLOPs, but Section 5 measures it implicitly through the sigmoid mapping—there is no explicit count of inference FLOPs for each method variant. How many trials does self-consistency use? What is the total generation budget for the full pipeline? What is the inference cost of the reranker? Without this accounting, the claim that test-time compute provides "an 11x compute multiplier over the pre-trained baseline" (Section 5.1) is incomplete—we know the performance gain, but we don't know how much test-time compute was spent to achieve it. The multiplier presumably assumes that the test-time compute cost is small relative to pre-training compute, which may be true for a single evaluation but matters for high-throughput deployments.

The additivity of self-consistency and retrieval is supported but not tested rigorously. Tables 5 and 6 show that the combined gains are approximately the sum of individual gains, but this is a coarse test. A proper additivity test would vary the budget allocated to self-consistency (number of trials) and retrieval (number of documents) independently and check for interactions. A fixed-budget experiment where the budget is split between more trials and more documents in different ratios would test whether the optimal allocation is balanced or skewed. The paper does not perform such an experiment.

The decontamination analysis is a strength but has limitations. The 16-gram and 26-gram thresholds were chosen by visual inspection rather than principled optimization. The paper doesn't report sensitivity to this choice—what if 13-grams were used instead? How many documents are removed at different thresholds? The 14.1% and 32.0% contamination estimates are therefore approximate. Additionally, n-gram decontamination only catches exact overlap—paraphrased versions of test questions or answers would pass through. This is a known limitation of n-gram methods, and the paper doesn't discuss it. The finding that Math-500 shows a larger gap between decontaminated and full retrieval than MMLU does (Figure 1) could indicate either genuine contamination or that the 26-gram threshold is too aggressive for Math-500, removing relevant but non-contaminated documents.

The Wikipedia extraction ablation (Table 10) is compelling but would benefit from more detail. The 13.6 percentage point gap between Wikimedia and custom extraction is large enough to dwarf most retrieval technique improvements, making it the paper's most practically significant finding. However, the paper doesn't provide examples of text produced by each extraction method—readers must trust the qualitative claim about "bullet points, tables, and info boxes" without seeing concrete examples of what is lost. A table showing a specific Wikipedia page as rendered by each extraction method would make the finding far more convincing. The paper also doesn't report whether retrieval with the custom extraction finds documents that the standard extraction misses entirely (recall failures) or finds the same documents but with better-preserved content (precision failures). This distinction matters for understanding whether the fix is better extraction or better crawling.

The dataset quality disconnect (Table 9) is an important finding but limited to two datasets and one model. DCLM and FineWeb-edu differ in many ways—filtering criteria, deduplication method, size—and the paper doesn't attempt to isolate which properties cause the pre-training/retrieval discrepancy. The pre-training numbers come from Su et al. (2024), not from the authors' own pre-training runs, so the training recipe, model architecture, and hyperparameters differ from those in Section 4. The retrieval numbers use Llama 3.1 8B, which was not trained on either DCLM or FineWeb-edu (or at least not on the exact versions used here), so the retrieval evaluation is measuring a mix of knowledge recovery and novel information provision. A cleaner experiment would pre-train models on DCLM and FineWeb-edu separately, then evaluate retrieval on the same corpus, using the same base model for both—this is exactly the Section 4 setup but with a dataset variable.

The absence of confidence intervals or statistical tests throughout the paper is a weakness. Point estimates for accuracy (Tables 5, 6, 8, 10) are presented without error bars, even when differences between methods are small (e.g., 73.6% vs. 73.7% for retrieval vs. reranked retrieval on MMLU STEM). For MMLU with 500 test questions (or the ~100 questions per difficulty bin used in the paper's compute-multiplier analysis), the standard error on a proportion is roughly 2.2 percentage points at 50% accuracy and 1.4 points at 80% accuracy. Many of the reported differences are within or near this margin, meaning they may not be statistically significant. The GPQA and Math-500 results report means over 10 trials but no standard deviations, making it impossible to assess whether, for example, the +0.1 point gain from variance reduction on Math-500 is distinguishable from noise.

What experiments would have strengthened the paper?

A sweep over the number of self-consistency trials and retrieval documents: The paper treats self-consistency and retrieval as binary (on/off) rather than continuous (how many trials? how many documents?). A budget-sweep experiment would reveal whether additivity holds at different budget levels and whether there is an optimal allocation ratio.
A systematic comparison of retrieval corpus quality metrics: The finding that FineWeb-edu outperforms DCLM for retrieval despite being worse for pre-training is tantalizing but unexplained. Measuring properties like average document length, readability, fact density, and structural preservation (table/list presence) for both corpora could identify candidate explanations.
Qualitative examples of extraction differences: For the Wikipedia extraction finding, showing side-by-side examples of text produced by each extraction method for the same page would make the claim concrete and reproducible.
A held-out validation of the sigmoid fit: Leaving out one or more pre-training runs, fitting on the remainder, and predicting the held-out accuracy would quantify the uncertainty in the compute-multiplier estimates.
A test-time compute budget explicitly accounted in FLOPs: Reporting the inference FLOPs for each method variant (embedding + FAISS + reranker + reader model generations × number of trials) would make the "compute multiplier" claim complete—how much inference compute buys how much performance relative to how much pre-training compute.
Evaluation on at least one non-multiple-choice reasoning benchmark with retrieval: The paper evaluates on MMLU (multiple choice), Math-500 and SimpleQA (short answer), and GPQA (multiple choice). A benchmark requiring open-ended reasoning with retrieval (e.g., long-form QA, summarization with source support) would test whether the retrieval benefits extend to generation tasks where the answer format is not constrained.

Summary assessment: The paper's central measurement—that retrieval from the pre-training corpus provides substantial performance gains over base models, and that these gains persist after decontamination—is robustly supported. The compute-multiplier framework provides a useful conceptual tool for comparing retrieval gains to pre-training scaling, but the specific multiplier values (4.86× average, 11.10× for the full pipeline) should be treated as approximate due to limited data points for fitting, lack of validation, and the mismatch between the models used for fitting and the model used in Section 5. The additivity of retrieval and self-consistency is empirically supported but would benefit from a more systematic budget-allocation study. The dataset quality findings (Tables 9 and 10) are the most practically significant and novel contributions, but the extraction finding lacks qualitative examples and the pre-training/retrieval discrepancy finding is limited to two datasets and a single reader model. The paper successfully achieves its stated goal of demonstrating that "pre-training methods do not make full use of the information in existing pre-training datasets" and quantifying this gap, but the precise magnitude of the gap—and how it generalizes across datasets, models, and tasks—remains uncertain and is appropriately flagged as requiring further investigation.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in the Compute-Multiplier Framework

The assumption or constraint. The compute-multiplier methodology in Section 4 measures retrieval's benefit in terms of how many additional pre-training FLOPs would be needed to match retrieval-augmented accuracy. This calculation treats the retrieval pipeline's own cost as negligible relative to pre-training FLOPs, but for practical deployments—and even for the paper's own evaluation runs—the retrieval infrastructure consumes non-trivial compute: embedding every query with Qwen3 Embedding 0.6B, performing FAISS FlatIP search over ~1.2 trillion tokens of indexed documents, optionally reranking the top-100 candidates with Qwen3 Reranker 0.6B, and then running the reader model on augmented context. For the controlled pre-training experiments in Section 4, the retrieval datastore is the full ~1.2T token corpus indexed across multiple shards, and FAISS flat search (not approximate) over this volume is computationally expensive per query. The paper acknowledges only the inter-document consistency cost explicitly (Section 5.2: "it requires calling the reader model many more times and is not compute efficient"), but does not account for the baseline retrieval cost in any compute-multiplier calculation.

The consequence. The headline ~5× compute multiplier on MMLU (Table 2) is computed as pre-training FLOPs needed to match retrieval accuracy divided by actual pre-training FLOPs, with zero FLOPs allocated to retrieval itself. In a realistic deployment, the total compute cost would be: pre-training + (retrieval infrastructure cost × number of queries). For a model that is queried once (e.g., a single benchmark evaluation), the retrieval cost might be small relative to the pre-training savings. For a model queried millions of times, the per-query retrieval cost could dominate. The paper provides no numbers to make this tradeoff concrete—how many queries must be served before the retrieval infrastructure cost exceeds the pre-training savings? At what query volume does the ~5× multiplier become a net loss? These questions are unanswerable from the paper's data. The absence of retrieval cost accounting also means that the "compute-optimal" allocation between pre-training and retrieval (if one were to jointly optimize) cannot be determined from this work.

What evidence exists in the paper. No quantitative evidence. The paper never reports the inference cost of Qwen3 Embedding, FAISS search time, or reranker inference. The pre-training FLOPs are computed using the standard 6ND approximation (implicit in Section 4), but no analogous formula is provided for retrieval pipeline FLOPs. The inter-document consistency discussion (Section 5.2) acknowledges compute cost qualitatively but doesn't quantify it.

Mitigation status. Not addressed. The paper does not mention retrieval infrastructure cost as a limitation, does not provide a FLOPs accounting for retrieval, and does not discuss the query-volume regime where the compute-multiplier framework is valid. This is a significant gap for a paper whose central contribution is a compute-multiplier measurement.

The Compute-Multiplier Sigmoid Fit Rests on Five Data Points Without Validation

The assumption or constraint. The central analytical tool—the bounded sigmoid mapping pre-training FLOPs to MMLU accuracy—is fitted to exactly five data points (the five pre-training runs at 6.4B, 12.6B, 23.3B, 36.8B, and 77.8B parameters). The fit uses three free parameters (slope, midpoint, and scale) with fixed bounds (0.25 random baseline, 0.9407 ceiling from Gema et al., 2024), leaving only two residual degrees of freedom. The paper reports no goodness-of-fit statistics, no confidence intervals on the fitted parameters, no sensitivity analysis of the ceiling value (0.9407 is itself an estimate from a different paper), and no validation against held-out data points. All five pre-training runs are used for fitting, and the same five runs are used to compute the multipliers.

The consequence. The specific multiplier values—5.28×, 7.17×, 4.74×, 4.23×, 2.88× in Table 2—are substantially uncertain. The non-monotonic pattern (increasing from 5.28× to 7.17× before declining) is unusual for a scaling law and may reflect fitting instability rather than a genuine phenomenon: small perturbations in the slope or midpoint parameters could change the computed multiplier by factors of 2× or more, especially when the retrieval accuracy maps to FLOPs in the steep region of the sigmoid (near the 2.48×10²² midpoint). For instance, at the 1.90×10²² budget, retrieval accuracy (0.6943) maps to 1.36×10²³ FLOPs—near the midpoint where the derivative is largest—so a small change in the fitted slope would produce a large change in the multiplier. The paper's category-level multipliers (Table 3) inherit the same uncertainty, and the claim that retrieval is a 6.16× multiplier for STEM vs. 2.52× for Humanities could partially reflect differential fitting quality across categories rather than genuine differences in retrieval effectiveness.

What evidence exists in the paper. The sigmoid equation is reported explicitly in Table 2 with its fitted parameters (-0.7968 slope, 2.48×10²² midpoint, 0.6907 scale). No R², residual standard error, parameter confidence intervals, or leave-one-out validation is reported. The category-level fits (Appendix E, Tables 12-15) use the same procedure with different ceilings but similarly lack diagnostics. The paper does not discuss the non-monotonic pattern in the multipliers or whether it is statistically distinguishable from a monotonic decline.

Mitigation status. Not addressed. The paper treats the sigmoid fit as a deterministic function and the multipliers as precise quantities. The reported arithmetic mean (4.86×), geometric mean (4.66×), and median (4.74×) imply a level of precision that the underlying fit does not support. A validation procedure (leave-one-out, bootstrapping, or reporting prediction intervals) would be standard practice for scaling law work but is absent here.

The Hardest Retrieval Cases (Factual Queries Requiring Precise Document Access) Show a Massive Performance Cliff Without Oracle Retrieval

The assumption or constraint. The paper's SimpleQA experiments (Section 6.2, Table 10) reveal a sharp performance gradient depending on retrieval corpus quality: custom Wikipedia extraction achieves 69.0% (with reranker, k=6, Llama 3.1 8B reader), adding all available non-Wikipedia sources raises this to 73.7%, and adding the ground-truth golden links (the specific Wikipedia pages known to contain the answer) raises it to 85.2%. The gap between "all sources" and "golden links" is 11.5 percentage points—meaning that even with a well-extracted, multi-source retrieval corpus covering ~1.2T tokens, the system still misses the relevant document for more than one in ten SimpleQA questions. The paper attributes this to crawling gaps: "only a small fraction of the non-Wikipedia golden links are present in CommonCrawl" (Section 6.2).

The consequence. For factual QA tasks where the answer depends on retrieving a specific document (rather than synthesizing information distributed across many documents), retrieval performance is fundamentally bounded by corpus coverage. Improving the reranker, the embedding model, or the test-time compute strategy cannot help if the target document is not in the retrieval corpus. The 85.2% ceiling with oracle document access (golden links) establishes an upper bound on what retrieval can achieve on SimpleQA with a perfect retrieval system, and the 73.7% achieved with all available sources represents the real ceiling given current crawl coverage. The 11.5-point gap between these is a coverage problem, not a retrieval quality problem—and the paper provides no evidence that this gap narrows with scale. This has practical implications: deploying the paper's retrieval system on a factual QA task would require either accepting a substantial coverage ceiling or investing in web crawl infrastructure (which is expensive, slow, and not part of the retrieval pipeline the paper evaluates).

What evidence exists in the paper. Table 10 provides the direct evidence. The paper explicitly notes that "only a small fraction of the non-Wikipedia golden links are present in CommonCrawl" and that "open-source datasets could be further improved at the web crawling stage" (Section 6.2). Figure 3 shows that adding large amounts of DCLM data to the retrieval corpus produces minimal improvement on SimpleQA, confirming that the missing documents are not present in the web-crawl data at all—they are absent, not merely hard to retrieve.

Mitigation status. The paper acknowledges the crawling gap as a limitation and suggests it as a direction for dataset improvement (Section 6.2, Section 7: "there is room for improving datasets, at the very least in terms of crawling and extraction"). However, no mitigation is provided—the paper does not propose alternative crawling strategies, evaluate the cost of improved crawls, or estimate what coverage ceiling could be practically achieved. The SimpleQA with golden links result (85.2%) establishes that the pipeline could achieve much higher accuracy if coverage were improved, but the paper cannot tell us how to close this gap.

The Generalization of Findings Beyond the Specific Benchmarks and Model Families Is Unmeasured

The assumption or constraint. The controlled pre-training experiments in Section 4 use a single model architecture (LLaMA-style dense decoder with SwiGLU FFNs, RoPE, GQA), a single data mixture (DCLM + FineWeb-edu + specialized sources, Table 1), and three benchmarks (MMLU, Math-500, SimpleQA). The test-time compute experiments in Section 5 additionally use GPQA and Llama 3.1 8B. The only non-QA evaluation is a single preliminary result on LiveCodeBench (Appendix G, Table 17) using GPT-4o with retrieval from the Stack v2 and CommitPack—a single data point with a proprietary model. The paper's central claims—the ~5× compute multiplier, the additivity of retrieval and self-consistency, the pre-training/retrieval dataset quality disconnect—are demonstrated exclusively on multiple-choice or short-answer benchmarks that have objective correctness criteria and for which the retrieval corpus plausibly contains relevant documents.

The consequence. Several generalization questions are unaddressed. First, do the findings hold for open-ended generation tasks (summarization, creative writing, long-form QA) where there is no single correct answer and retrieval relevance is harder to define? The embedding and reranking pipeline relies on similarity between the query and documents; for tasks where useful documents are not topically similar to the prompt, the entire retrieval stack may fail. Second, do the findings hold for tasks where the necessary knowledge is procedural (how to perform an action) rather than declarative (what is the answer)? Third, do the findings hold for models with substantially different architectures (mixture-of-experts, state-space models) or training objectives (RLHF-tuned, instruction-tuned with different data)? The paper's single controlled architecture (LLaMA-style dense) and the single public model tested in Section 5 (Llama 3.1 8B) are both in the same architectural family, so architecture robustness is untested. Fourth, does the compute multiplier degrade differently for different model families—would a MoE model, which the paper explicitly connects to retrieval (Section 7, citing Clark et al., 2022), show a smaller multiplier because its conditional computation already performs a form of learned retrieval over its parameters?

What evidence exists in the paper. LiveCodeBench (Appendix G, Table 17) provides minimal evidence of generalizability to code generation—a +4.83 percentage point gain for GPT-4o with retrieval (k=3)—but uses a proprietary model and a different retrieval corpus (Stack v2, CommitPack), making it a suggestive but not rigorous extension. The category-level analyses within MMLU (Tables 3, 4, 5) show that retrieval benefits vary substantially across subject areas, which supports the concern that benchmark composition strongly affects the headline multiplier. The paper acknowledges the domain limitation implicitly in Section 6.3: "SimpleQA measures factual knowledge, and the scaling effect may be different for reasoning tasks," but does not extend this concern to the broader generalization question.

Mitigation status. The paper flags generalization as future work (Section 7: "we also believe that these findings apply to even broader domains") and provides the LiveCodeBench result as initial evidence, but does not systematically evaluate on non-QA tasks, on models outside the LLaMA family, or on corpora constructed with different curation philosophies. The finding that FineWeb-edu and DCLM—two datasets with similar provenance and curation goals—show opposite pre-training vs. retrieval quality rankings (Table 9) suggests that dataset composition significantly affects results, and the specific numbers in this paper may not transfer to retrieval corpora built from different sources.

Retrieval Performance on the Hardest Problems Is Near-Zero, Mirroring a Known Test-Time Compute Ceiling

The assumption or constraint. Across all three tasks in the controlled scaling experiments (Figure 1), retrieval provides substantial gains on questions the base model can already partially solve, but the absolute accuracy on the hardest subset of questions is never reported. The paper does not bin questions by difficulty (as, for example, Fang et al. do in the companion paper analyzed in the reference example), so we cannot directly observe whether retrieval helps uniformly or only on problems where the base model has non-trivial capability. However, the SimpleQA result provides indirect evidence: Llama 3.1 8B baseline accuracy is 1.5% (Table 6), meaning the model essentially cannot answer any SimpleQA questions from parametric memory. Retrieval lifts this to 74.0%—a massive gain driven by the fact that SimpleQA answers are directly present in Wikipedia and the model's failure is purely a knowledge access problem, not a reasoning problem. But for tasks where the base model's failure stems from lack of reasoning capability rather than knowledge access, retrieval may provide minimal benefit, and the paper provides no analysis of this regime.

The consequence. The paper's framing—"pre-training methods do not make full use of the information in existing pre-training datasets"—implicitly assumes that the information, if provided at test time, would enable correct answers. But this conflates two failure modes: (1) the model lacked the necessary factual knowledge during pre-training (retrieval can fix this), and (2) the model lacks the reasoning capability to use the knowledge correctly even when it is provided in context (retrieval cannot fix this). The paper's experiments cannot distinguish these cases because they don't separately measure the "knowledge available in the retrieval corpus" and "model can reason correctly over retrieved knowledge" components. The SimpleQA result (1.5% → 74.0%) demonstrates the pure knowledge-access regime; the GPQA result (30.6% → 36.8%, a much smaller relative gain) suggests a regime where reasoning is the bottleneck and retrieval adds less. The MMLU result (71.6% → 82.1%) is intermediate. Without a difficulty breakdown or a reasoning-vs-knowledge decomposition, practitioners cannot predict whether their specific task falls in the SimpleQA regime (massive retrieval gains), the GPQA regime (modest gains), or the hypothetical hard-reasoning regime (negligible gains).

What evidence exists in the paper. Indirect evidence from the varying retrieval impact across benchmarks. SimpleQA: baseline 1.5% → retrieval 74.0% (massive gain, pure knowledge access problem). MMLU: baseline 71.6% → retrieval 76.6% (moderate gain, mixed knowledge and reasoning). Math-500: baseline 48.7% → retrieval 56.7% (moderate gain). GPQA: baseline 30.6% → retrieval 33.2% (small gain). The gradient of retrieval impact (SimpleQA ≫ Math-500 ≈ MMLU > GPQA) is consistent with the hypothesis that retrieval helps most when the primary failure is knowledge access rather than reasoning, but the paper does not analyze this pattern or provide a difficulty-stratified breakdown.

Mitigation status. Not addressed. The paper presents retrieval gains as a uniform property of the pre-training corpus ("how much dataset value was left behind") without decomposing into knowledge-access and reasoning-utilization components. The absence of a difficulty breakdown means the paper cannot characterize which questions retrieval helps with, beyond the coarse category-level analyses (Table 4, Section 4.1) which show a mix of subject types but no systematic difficulty stratification.

The Test-Time Compute Stack Is a Cumulative Ablation Chain Without Budget Control

The assumption or constraint. The test-time compute experiments (Section 5, Tables 5-6) present a cumulative ablation chain: baseline → + self-consistency → + retrieval → + reranker → + self-consistency → + variance reduction. Each step adds a technique and reports the resulting accuracy. The implication is that each technique provides incremental value, and the full stack is the best configuration. However, each technique also consumes additional test-time compute: self-consistency requires N× more reader model generations, retrieval requires embedding + FAISS + document processing, the reranker requires cross-encoding the top-100 documents, and variance reduction (MMR + bagging) requires additional retrieval and document shuffling. The paper does not control for total test-time compute budget—we don't know whether the +9.4 point gain from the full pipeline over baseline (MMLU 82.1% vs. 71.6%) is a fair comparison, because the full pipeline consumes dramatically more inference FLOPs than the baseline. A budget-controlled experiment would ask: given a fixed inference FLOPs budget, how should it be allocated between self-consistency trials and retrieval quality? The paper doesn't answer this.

The consequence. We cannot determine whether the full pipeline's gains come from genuinely complementary mechanisms or simply from spending more compute. For example: does reranked retrieval + self-consistency with 8 trials outperform raw retrieval with 32 trials? The paper cannot answer because it never varies the budget allocation. The additivity claim—that retrieval and self-consistency address complementary failure modes—is plausible but not rigorously tested: if both techniques help because they each increase total compute, the gains would also appear additive when stacking them, even if they address the same underlying limitation (stochastic reasoning noise). A budget-controlled experiment showing that splitting a fixed budget between retrieval quality and self-consistency trials outperforms spending it entirely on either one would be stronger evidence for complementarity.

What evidence exists in the paper. No budget-controlled experiments. Tables 5-6 report accuracy for each cumulative configuration without reporting inference FLOPs, number of self-consistency trials, number of retrieved documents, or total generation budget. The paper reports GPQA and Math-500 results as means over 10 trials (Table 6 caption), which tells us the number of self-consistency trials but not the retrieval budget. The absence of compute accounting for the test-time experiments is particularly notable given that Section 4's central contribution is a compute-multiplier framework—the test-time experiments would be the natural place to extend that framework to inference compute, but the paper instead maps test-time method performance back to pre-training FLOPs via the same sigmoid (Table 7), sidestepping the question of how much inference compute was actually spent.

Mitigation status. Not addressed. The paper's treatment of test-time compute as free—something to be added until accuracy saturates—rather than as a constrained budget to be optimized is a significant gap. The conceptual distinction between "improving the tool" (reranking, MMR) and "running the tool more times" (self-consistency) is useful (Section 5), but the paper doesn't provide guidance on how to allocate a fixed budget between these categories. In the paper's defense, the compute-multiplier framework in Table 7 implicitly assumes test-time compute is negligible compared to pre-training compute (otherwise the mapping from accuracy back to pre-training FLOPs would need to subtract the test-time cost), which may be reasonable for models queried infrequently but breaks down for high-throughput deployments.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new architecture, training algorithm, or retrieval technique. Instead, it introduces a new evaluation methodology—the systematic measurement of residual knowledge in pre-training datasets through retrieval-at-test-time—that shifts how the field should think about data utilization, scaling, and dataset curation. The shift is conceptual rather than architectural, but its implications ripple across multiple research subfields.

From "is the model good?" to "is the dataset good, and did we use it well?" The standard evaluation paradigm for pre-training centers on the model: measure loss, benchmark accuracy, and attribute failures to model capacity or training recipe. This paper's core methodological contribution is inverting that framing. By retrieving from the same corpus used for pre-training, it asks whether the data could support better performance, not whether the model achieved it. The 5× compute multiplier on MMLU (Table 2) is not a statement about how good retrieval is—it's a statement about how inefficient pre-training is. The 1.5% → 74.0% leap on SimpleQA with Llama 3.1 8B (Table 6) tells us the knowledge was in the data all along; the model simply couldn't access it parametrically.

This inversion matters because it provides a feedback loop for dataset development that has been absent. Prior to this work, a team improving a pre-training dataset had to train a model to evaluate whether their changes helped—a cycle that takes weeks to months and conflates dataset quality with training dynamics. The paper's framework enables a faster, cleaner evaluation: retrieve from the dataset using a fixed reader model and measure the accuracy ceiling the data supports. If retrieval accuracy is high but pre-training accuracy is low, the bottleneck is the learning algorithm. If retrieval accuracy is low, the bottleneck is the dataset itself. Prior work could not distinguish these cases from final model quality alone.

The dataset quality findings—FineWeb-edu beating DCLM for retrieval despite losing badly for pre-training (Table 9), and extraction quality producing a 13.6 percentage point swing on SimpleQA (Table 10)—make this concrete. These results demonstrate that the properties making a dataset good for gradient-based learning differ from those making it good for in-context access. A dataset development effort optimized solely for pre-training perplexity might systematically degrade retrieval quality by removing structured elements (tables, lists, infoboxes) that are textually noisy but informationally dense. The paper's framework makes this tradeoff visible for the first time.

A new axis in the scaling-law framework. Classical scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) relate loss to compute, parameters, and data quantity. Data-constrained scaling studies (Muennighoff et al., 2023b) studied repeated data. This paper adds a third axis: data utilization efficiency, measured by the gap between parametric and retrieval-augmented performance. The finding that this gap shrinks but does not close with scale—2.88× multiplier remains at 77.8B parameters (Table 2)—suggests that data utilization efficiency is a property of the training algorithm that scales differently from raw capacity. Characterizing this efficiency as a function of model size, data diversity, and training duration opens a new dimension for scaling-law research.

Reconciling conflicting intuitions about retrieval. There has been a latent tension in the RAG literature between two views: (1) retrieval is a deployment optimization that compensates for model limitations, and (2) retrieval is a fundamental component that all models should use because externalizing knowledge is more efficient than compressing it into parameters. This paper provides evidence that both views are partially correct, but at different scales. The declining compute multiplier (7.17× → 2.88×) shows that larger models do absorb more knowledge parametrically, supporting view (1) that retrieval compensates for capacity limits. But the fact that retrieval still provides a 2.88× multiplier at 77.8B parameters—and that even untrained datasets like FineWeb-edu can outperform trained-on datasets like DCLM for retrieval—supports view (2) that retrieval and pre-training are fundamentally different information-access mechanisms with different optimal data properties, not just a capacity gap that will close with scale.

Shifting research priorities. The paper implicitly reprioritizes several research directions:

Dataset engineering becomes more important, not less. The finding that crawling and extraction quality dominate downstream retrieval accuracy (Table 10) means that the "boring" parts of the pipeline—HTML parsing, web crawl coverage, infobox preservation—deserve serious research attention. A 13.6-point gain from better extraction dwarfs the gains from sophisticated retrieval techniques like reranking (1-3 points) or variance reduction (0.7-1.1 points). The field's focus on embedding architecture and reranker design may be optimizing the wrong component.
Verifier-free test-time compute (self-consistency) is robustly complementary to retrieval, not a competitor. The additivity findings in Tables 5-6 suggest that the two techniques address different failure modes and should be deployed together by default. The paper makes a useful conceptual distinction—test-time compute spent improving the tool (reranking, MMR) vs. running the tool more times (self-consistency)—that provides a taxonomy for thinking about inference budget allocation.
Benchmark hygiene is more critical than many assume. The quantification that 14.1% of MMLU and 32.0% of Math-500 exist in standard open-source pre-training corpora (Section 4) is a warning shot. Researchers comparing models trained on overlapping data may be evaluating contamination resistance rather than genuine capability differences. The paper's decontamination methodology—n-gram matching in token space, with empirically tuned thresholds—provides a template that other evaluation papers could adopt as standard practice.

What this work is not. The paper does not propose that retrieval should replace pre-training, or that all models should use retrieval-augmented evaluation. The compute multiplier degrades with scale, and the paper explicitly notes that scaling effects "may be different for reasoning tasks" (Section 6.3). The contribution is a measurement framework and a set of empirical findings about data utilization, not a prescription for model architecture.

Follow-Up Research This Work Enables

Difficulty-stratified retrieval analysis. The paper reports retrieval gains averaged over entire benchmarks, but does not bin questions by difficulty (as the companion Fang et al. test-time compute paper does with quintile-based difficulty estimation). A natural follow-up would stratify MMLU, Math-500, and SimpleQA questions by the base model's pass@1 rate and measure whether retrieval acts as a larger multiplier on easy, medium, or hard problems. The SimpleQA baseline of 1.5% (Llama 3.1 8B) suggests that for problems where the base model is essentially at chance, retrieval can provide massive gains—but this may only hold when answers are directly present in the retrieval corpus (factual QA). For Math-500, where the baseline is 48.7% and retrieval adds 8.0 points, the gain may concentrate on problems the model could almost solve—retrieval providing the missing formula or theorem that unlocks the solution. A difficulty breakdown would reveal whether retrieval extends the model's capability frontier (solving previously unsolvable problems) or mainly rescues near-misses, which has different implications for deployment and for understanding the nature of parametric memory failures. The experiment would require: (1) estimating per-question difficulty via pass@1 from the base model over many samples, (2) binning into quintiles as in Fang et al., (3) measuring retrieval gain per bin, and (4) comparing the difficulty-gain pattern across factual (SimpleQA), reasoning (Math-500), and mixed (MMLU) benchmarks.

Budget-controlled test-time compute allocation between retrieval and self-consistency. The paper demonstrates that retrieval and self-consistency are additive in a cumulative ablation chain (Tables 5-6), but this does not prove that they are complementary under a fixed inference budget. The critical follow-up experiment varies the total inference FLOPs budget and sweeps the allocation ratio between retrieval quality (more documents, reranking, MMR) and self-consistency (more trials). For a budget of, say, 64 reader model generations, we could allocate 64 trials with k=1 document (all self-consistency, no retrieval quality), 1 trial with k=64 documents (all retrieval quality, no self-consistency), or intermediate splits like 8 trials with k=8 (balanced). The paper's conceptual framework—test-time compute as tool improvement vs. tool usage—predicts that the optimal allocation ratio depends on the task: for SimpleQA (pure knowledge access), the budget should go almost entirely to retrieval quality (finding the right document); for GPQA (heavy reasoning), the budget should tilt toward self-consistency (verifying reasoning chains); for MMLU and Math-500, some intermediate ratio should be optimal. This experiment would transform the paper's qualitative additivity observation into a quantitative resource-allocation principle. It would also reveal whether the optimal ratio changes with total budget (does more total compute favor more retrieval quality or more self-consistency trials?) and with model capability (do stronger models benefit more from retrieval or from self-consistency?).

Isolating the properties that make a dataset good for retrieval versus pre-training. Table 9's finding—FineWeb-edu beats DCLM for retrieval despite losing badly for pre-training—is the paper's most intriguing but least analyzed result. The datasets differ across multiple dimensions: filtering criteria, deduplication method, document length distribution, presence of structured elements, ratio of factual to narrative content. A systematic follow-up would measure these properties for both datasets and correlate them with retrieval quality (measured as downstream accuracy with a fixed reader model) and pre-training quality (measured as model accuracy after training on each dataset). Candidate hypotheses to test: (1) FineWeb-edu preserves more structured text (tables, lists) that benefits retrieval but adds noise that hurts pre-training gradient signals; (2) FineWeb-edu has higher fact density per document, making individual retrieved documents more informative but reducing the diversity of linguistic patterns that pre-training benefits from; (3) DCLM's heavier deduplication removes near-duplicate documents that are redundant for pre-training but provide alternative phrasings that help retrieval match queries. The experiment would pre-train a series of small models on carefully controlled variants of each dataset (e.g., DCLM with and without structured text removal, FineWeb-edu with and without deduplication) and measure both pre-training accuracy and retrieval accuracy on the same benchmarks. This is expensive but tractable at small scale (e.g., 1B parameters) and would provide the first mechanistic understanding of the pre-training/retrieval quality tradeoff.

Extraction quality ablation across retrieval benchmarks. The paper's Wikipedia extraction finding (Table 10: 13.6-point gap) is demonstrated only on SimpleQA, but the mechanism—loss of structured information during HTML-to-text conversion—should affect any task where answers reside in tables, lists, or infoboxes. A natural extension would measure extraction quality impact across a diverse set of tasks: factual QA (SimpleQA, Natural Questions), multi-hop reasoning (HotpotQA), and tasks that mix knowledge and reasoning (MMLU subjects that depend on Wikipedia knowledge). The experiment would compare at least three extraction methods: naive text extraction (strip all HTML tags, concatenate text), a standard pipeline like trafilatura or the OLM extraction, and the custom ReaderLM-v2 pipeline the paper describes. For each task, measure (1) the fraction of golden documents where the extraction method preserves the answer-containing text, (2) the downstream reader model accuracy using each extraction method, and (3) the correlation between preservation rate and accuracy gain. This would reveal whether the extraction bottleneck is universal (all tasks benefit similarly) or task-specific (factual tasks benefit most because answers concentrate in structured elements). The finding would directly inform dataset curation practices: if extraction quality matters predominantly for factual tasks, dataset builders can optimize their pipeline for the intended use case.

Does retrieval from pre-training data improve with longer context windows? The paper uses a fixed k (number of retrieved documents) without sweeping this parameter, and the reader model's context window capacity is not explicitly discussed. As models with longer context windows (128K, 1M tokens) become standard, the retrieval tradeoff changes: we can include more documents, reducing the precision burden on the retriever but potentially diluting the model's attention. A follow-up would sweep k from 1 to the maximum the model's context window allows, measuring accuracy for each configuration on the benchmarks the paper uses. The hypothesis: for tasks where the relevant information is concentrated in one or a few documents (SimpleQA), larger k may hurt by introducing distractors; for tasks where information is distributed across many documents (complex MMLU questions requiring multiple facts), larger k may help. The shape of the accuracy-vs-k curve would inform practical retrieval system design and test whether the paper's default k was near-optimal. Additionally, comparing the accuracy-vs-k curve for models pre-trained on the retrieval corpus (the controlled models from Section 4) versus an off-the-shelf model (Llama 3.1 8B) would reveal whether familiarity with the corpus affects how well the model handles larger retrieved sets.

Joint scaling laws for pre-training and retrieval compute. The paper introduces the concept of retrieval as a compute multiplier and reports multipliers at five discrete pre-training budgets (Table 2), but it does not attempt to fit a joint scaling law that predicts retrieval-augmented accuracy as a function of both pre-training FLOPs and retrieval infrastructure quality (embedding model size, index size, k, reranker presence). A natural extension would train models at more granular compute budgets (e.g., 10 points instead of 5) and at each budget measure accuracy with multiple retrieval configurations (varying k, with and without reranker, with and without self-consistency), then fit a parametric function of the form: accuracy = f(pre-training_FLOPs, retrieval_budget, task). The fitted function would predict, for any given total compute budget split between pre-training and retrieval, the expected accuracy—enabling true compute-optimal allocation between training and inference. This would extend the paper's conceptual framework from a retrospective measurement (how much compute did retrieval save?) to a prospective optimization (how should we allocate our next dollar?). The primary challenge is defining "retrieval budget" in FLOPs commensurate with pre-training FLOPs, which requires careful accounting of embedding model inference, index search, reranker cost, and reader model inference—costs the paper currently ignores.

Practical Applications and Downstream Use Cases

Auditing data utilization during model development. The paper's framework provides a concrete protocol for model development teams to answer: "Should we invest in better data, or in better training?" Before embarking on an expensive data curation effort, pre-train a small proxy model on the current dataset, then evaluate it with and without retrieval from that same dataset. If retrieval provides a large accuracy boost (e.g., +10 points on a target benchmark), the dataset already contains the necessary information—the training algorithm is the bottleneck, and effort should go into better learning methods (curriculum learning, better optimizers, longer training). If retrieval provides only a small boost (e.g., +2 points), the dataset lacks the information, and data quality or coverage should be the investment priority. The paper's subset retrieval control (Figure 1, light blue lines) handles the concern that retrieval might find documents the model never saw—by restricting retrieval to the exact token exposure, the gap cleanly measures parametric memory failure. For a team training a 70B model, running this diagnostic on a 1B proxy model would cost a tiny fraction of the full training run and could redirect millions of dollars of compute toward the higher-impact investment.

Improving factual accuracy in deployed chatbots and QA systems. The SimpleQA result—1.5% baseline to 74.0% with reranked retrieval (Table 6)—demonstrates that for purely factual queries, retrieval from a well-extracted corpus essentially eliminates the knowledge access bottleneck. A deployed system handling user questions about facts (dates, names, measurements, definitions) would benefit from this approach immediately: maintain an indexed corpus of the pre-training data (or a curated subset like Wikipedia), retrieve documents at query time, and present the augmented context to the model. The finding that reranking provides an 8.3-point gain on SimpleQA (74.0% vs. 65.7% without reranker) while being cheap relative to reader model inference makes reranking a high-return investment. The extraction quality finding (Table 10) provides a concrete engineering priority: invest in a custom extraction pipeline that preserves structured elements before investing in more sophisticated retrieval architectures. The 13.6-point gain from better extraction is free in terms of query-time compute—it's a one-time cost paid during index construction.

Retrospective dataset value estimation for research and procurement. When evaluating whether a dataset is worth using or purchasing, the standard approach is to train a model on it and measure performance—an expensive and slow process. The paper's framework offers a faster proxy: retrieve from the dataset using a fixed off-the-shelf model (like Llama 3.1 8B) and measure downstream accuracy on target benchmarks. If retrieval accuracy is high, the dataset contains valuable information regardless of whether a particular training run can extract it. If retrieval accuracy is low, the dataset is information-poor and no training recipe will compensate (the information simply isn't there). This is particularly relevant for domain-specific datasets (legal, medical, scientific) where training a model is expensive but the information value of the dataset is unknown. The finding that FineWeb-edu and DCLM—two widely-used web-crawl datasets—show opposite rankings for pre-training vs. retrieval (Table 9) demonstrates that this evaluation is non-redundant with standard pre-training benchmarks: a dataset that looks bad for pre-training might be excellent for retrieval, and vice versa. An organization building a RAG system should evaluate datasets on retrieval quality, not on pre-training benchmark scores.

Guiding web crawl and extraction infrastructure investment. The paper's most actionable finding for data engineering teams is that crawling and extraction quality dominate downstream retrieval accuracy for factual tasks (Table 10). The 11.5-point gap between "Custom + All Sources" and "Custom + Golden Links" indicates that standard CommonCrawl-based crawls miss many relevant documents; the 13.6-point gap between custom and standard extraction indicates that even when documents are crawled, poor extraction discards the most useful content. For an organization building a retrieval corpus, this suggests a specific investment priority: (1) improve coverage by supplementing CommonCrawl with targeted crawls of high-value domains (Wikipedia, Stack Exchange, arXiv) using direct API access or focused crawling rather than relying on general web crawls to incidentally capture these sources, (2) implement a multi-stage extraction pipeline that uses specialized models (like ReaderLM-v2) to preserve structured content rather than relying on heuristic HTML stripping, and (3) validate extraction quality on a factual benchmark like SimpleQA before deploying, since the downstream accuracy impact is large enough to be measurable with a few hundred queries. These are engineering investments rather than research advances, but the paper quantifies their impact in a way that justifies the engineering cost.

When to Prefer This Method

The paper does not present a clear "prefer X over Y" tradeoff against named alternatives in the sense of arguing that retrieval should replace pre-training or that retrieval-augmented evaluation should replace standard evaluation. Rather, it provides a diagnostic framework whose appropriate use depends on the question being asked. The implicit decision rules from the paper's findings are:

To diagnose whether a pre-training dataset contains sufficient information for a target task, prefer retrieval evaluation with a subset-retrieval control (matching the pre-training token exposure) over training a full model. The retrieval evaluation is faster (hours vs. weeks), cheaper (inference-only vs. full training), and cleanly separates dataset quality from training algorithm efficiency.
To maximize factual accuracy on knowledge-intensive queries at deployment, prefer retrieval from a well-extracted, coverage-rich corpus over relying on parametric memory alone. The SimpleQA baseline of 1.5% vs. 74.0% retrieval accuracy (Table 6) shows that the performance gap is categorical, not marginal—for factual tasks where answers exist in the retrieval corpus, parametric memory fails almost completely at the scales studied.
To maximize performance on mixed knowledge-and-reasoning tasks given a fixed inference budget, prefer combining retrieval and self-consistency over either alone. The additivity finding (Tables 5-6) means that splitting the budget between the two techniques outperforms concentrated allocation, though the optimal split ratio remains to be determined.
When building a retrieval corpus for factual tasks, invest in extraction quality and crawl coverage before investing in better embedding models or rerankers. The 13.6-point gain from extraction (Table 10) and 11.5-point gain from coverage (golden links) dwarf the gains from sophisticated retrieval techniques, and these improvements are one-time costs during index construction rather than per-query compute costs.

The paper does not claim that retrieval should replace pre-training, that retrieval-augmented models should replace standard models for all tasks, or that retrieval from pre-training data is preferable to retrieval from a dedicated, task-optimized corpus. The compute multiplier degrades with scale (2.88× at 77.8B, Table 2), and the paper explicitly notes that findings on factual tasks may not transfer to reasoning tasks (Section 6.3). The contribution is a new way to measure data utilization, not a mandate to deploy retrieval everywhere.