The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

🎯 Pitch

Properly filtered and deduplicated web data alone can train models that outperform those trained on curated corpora like The Pile, matching GPT-3 performance without any 'high-quality' sources. The REFINEDWEB dataset extracts 5 trillion tokens from CommonCrawl using a new pipeline that strips away half the data through aggressive deduplication and filtering, proving that curation is not a bottleneck for scaling.

1. Executive Summary

This paper introduces REFINEDWEB, a five-trillion-token English pretraining dataset built entirely from CommonCrawl web data through a novel pipeline called Macrodata Refinement (MDR) — combining stringent document-wise and line-wise filtering heuristics with aggressive deduplication (both fuzzy MinHash and exact substring removal, achieving ~50% removal rates) — and demonstrates, through zero-shot evaluation of 1B–7.5B parameter autoregressive models on 18-task aggregates, that models trained on properly filtered and deduplicated web data alone can outperform those trained on curated corpora like The Pile, matching the performance of GPT-3 models at equivalent compute budgets (1.3B/7.5B parameter models on 350GT), while additionally showing that applying MDR's deduplication stage to existing datasets like C4 and OSCAR yields consistent zero-shot performance improvements, establishing that curation is not necessary for strong generalization when filtering and deduplication are applied at sufficient scale.

2. Context and Motivation

The Core Problem: The Scalability Crisis of Curated Pretraining Data

The fundamental tension this paper addresses sits at the intersection of two dominant trends in large language model development. On one side, scaling laws — most influentially the Chinchilla framework by Hoffmann et al. (2022) — dictate that both model size and dataset size must grow together to achieve compute-optimal training. Concretely, optimally training a GPT-3 sized model (175B parameters) would require approximately 3,500 billion tokens of text. This is a seismic shift from earlier findings by Kaplan et al. (2020), which had argued that model size should be prioritized while data scaling remained relatively flat. The joint scaling paradigm means that simply building larger architectures is no longer sufficient — you must also find approximately 20 tokens per parameter worth of training data, and this ratio holds as models grow.

On the other side, the prevailing wisdom in the field has held that pretraining data cannot simply come from anywhere. Since GPT-2 and GPT-3 (Radford et al., 2019; Brown et al., 2020), LLM training has relied on a mixture of two kinds of data: massive web crawls (which provide scale) and manually curated "high-quality" corpora (which are believed to provide diversity and depth). These curated sources typically include books, technical papers (arXiv, PubMed), Wikipedia, social media conversations (Reddit, StackOverflow), news articles, and code repositories. The curation process is inherently labor-intensive — each source requires specialized extraction pipelines, quality assurance, and often complex licensing negotiations. Furthermore, many curated sources are finite: there are only so many published books, only so many Wikipedia articles, only so many peer-reviewed papers. As the paper pointedly observes in Section 1:

"Massively scaling-up pretraining data is made even more challenging by the fact LLMs are commonly trained using a mixture of web crawls and so-called 'high-quality' data... Unfortunately, curation is labour intensive: typically, each source requires specialized processing, while yielding a limited amount of data."

This creates a structural bottleneck. If we believe (1) scaling laws demand trillions of tokens, and (2) curated data is essential for quality, then we face a future where pretraining datasets become increasingly difficult and expensive to construct. The authors cite Villalobos et al. (2022) to underscore the concern: some researchers argue that data availability — not compute or algorithmic innovation — may soon become the primary constraint on scaling.

The Web Data Paradox: Abundant but Distrusted

There is no shortage of raw textual data on the internet. CommonCrawl, a public archive of web pages, has been running for over 12 years and has collected petabytes of data across billions of URLs. In principle, web data alone could provide all the tokens anyone would ever need — the authors themselves are able to extract five trillion tokens from CommonCrawl alone, enough to train models well beyond current scales.

However, web data has a reputation problem. As the paper notes, a significant fraction of web pages consists of machine-generated spam, boilerplate text, pornography, keyword-stuffed SEO pages, advertisements, and navigational clutter (Kreutzer et al., 2022; Trinh & Le, 2018). Training a language model on unfiltered web data produces demonstrably worse models (Raffel et al., 2020). The consensus, crystallized by Scao et al. (2022b), has been that web data is fundamentally inferior to curated corpora:

"The increased diversity and quality brought forth by these curated corpora is believed to be a key component of performant models... web data alone is considered insufficient to train powerful large language models (Liu et al., 2019; Scao et al., 2022b)."

This belief has become so entrenched that even carefully processed web-only datasets like C4 (Raffel et al., 2020) — which applies filtering heuristics, NSFW word blocklists, and three-sentence exact deduplication to produce ~360GT of text — and OSCAR (Ortiz Suárez et al., 2019) — which builds on CommonCrawl with language identification and line-level deduplication — are regarded as inferior to aggregated corpora like The Pile (Gao et al., 2020), which combines web data (only ~18% of its content) with books, papers, code, and forum conversations. The prevailing assumption is laid out clearly in the introduction: curation is the differentiator.

The paper identifies a specific irony in this assumption. While the curated components of datasets like The Pile are indeed high quality, they are also small — typically a few tens of billions of tokens each. The bulk of any large pretraining dataset — even supposedly curated ones — is still web data. GPT-3's training corpus was 60% web-crawled content (CommonCrawl plus filtered subsets). MassiveWeb (Rae et al., 2021), used to train Gopher, was 48% web data. The PaLM dataset (Chowdhery et al., 2022) was 27% web data. The question then becomes: rather than mixing limited quantities of curated data into a predominantly web-sourced corpus, can we simply improve the web data itself to the point where the curation premium disappears?

Where Prior Web-Data Pipelines Fall Short

The paper identifies three specific gaps in how existing public web datasets have been constructed (Table 1, Table 12 in Appendix F.3), which collectively explain why web data has underperformed:

1. Insufficient deduplication. Deduplication — removing repeated documents or text spans — has been shown by Lee et al. (2022) to be one of the most impactful preprocessing steps for LLM training. Duplicates cause memorization (Carlini et al., 2021, 2022), degrade generalization, and become increasingly harmful as model scale increases (Hernandez et al., 2022). Yet publicly available web datasets have historically treated deduplication as an afterthought:

C4 applies only exact deduplication on spans of three sentences — a relatively coarse approach that misses any paraphrased or templated content.
OSCAR-22.01 makes deduplication entirely optional, with the main distributed version containing significant duplication (the paper reports a ~60% removal rate when their deduplication is retroactively applied in Table 5).
OSCAR-21.09 applies exact per-line deduplication, which is aggressive for line-level content but blind to cross-document similarity that manifests at the document level (e.g., entire templated pages with only entity names differing).
The Pile applies MinHash-based fuzzy deduplication with only 10 hashes and a similarity threshold of 0.5 (filtering out documents with >50% n-gram overlap), removing about 26% of content. While this is better than nothing, the authors demonstrate that their more aggressive MinHash settings (9,000 hashes, 20 buckets of 450 each) paired with subsequent exact substring deduplication yields substantially larger removal rates (~50% overall) and corresponding performance gains.

The paper positions the absence of rigorous deduplication as a key differentiator. Deduplication at scale is computationally expensive — it requires suffix array construction over the entire corpus for exact matching and pairwise similarity comparisons for fuzzy matching — and prior public datasets simply did not invest the compute necessary to do it thoroughly.

2. Reliance on ML-based content filtering. Multiple prior pipelines (GPT-3's training corpus, MassiveWeb, PaLM) train lightweight classifier models on known "high-quality" sources (e.g., Wikipedia, books) and use these to score and filter web documents, keeping only those that resemble the curated gold standard. While effective at improving average quality, this approach has a documented dark side: it can introduce and amplify biases, particularly against minority perspectives that are underrepresented in the gold-standard training data (Dodge et al., 2021; Welbl et al., 2021). The paper's design philosophy explicitly rejects ML-based filtering for content quality, adhering to what they call "neutral filtering":

"To avoid introducing further undesirable biases into the model, we avoid using ML-based filtering outside of language identification. We stick to simple rules and heuristics, and use only URL filtering for adult content."

This is a deliberate — and potentially controversial — decision. The tradeoff is that rule-based filtering may be less precise than ML-based filtering (letting through some low-quality content that a classifier would catch, or blocking content that a classifier might keep). The paper's implicit argument is that the distributional harm of ML-based filtering (bias amplification) outweighs the precision benefits, and that sufficiently aggressive deduplication can compensate for the lack of a learned quality classifier by removing the most common forms of low-quality text (spam templates, boilerplate, duplicates).

3. Inadequate text extraction. Many pipelines start from CommonCrawl's preprocessed WET files, which strip HTML tags and provide plain text. However, the paper notes — in line with Rae et al. (2021) and Gao et al. (2020) — that WET files contain significant amounts of non-content text: navigation menus, advertisements, social media counters, footer text, and other web chrome. The paper adopts trafilatura (Barbaresi, 2021), a specialized content extraction library that Lopukhin (2019) found to be the best non-commercial option for isolating main article content, and applies it to raw WARC files (the original HTML responses). This choice is more compute-intensive — it requires re-extracting every page rather than using pre-stripped text — but yields cleaner documents where the core content is separated from the surrounding boilerplate.

Additionally, the paper introduces line-wise corrections — a post-extraction step that removes remaining web artifacts (e.g., "3 likes," "Sign in," "Read more...") that trafilatura misses. This is a novel contribution to the pipeline design space: while document-level filtering heuristics (from MassiveWeb) catch egregiously bad pages, line-wise corrections refine salvageable pages by excising specific low-quality spans. If more than 5% of a document is flagged by these corrections, the entire document is discarded — a threshold calibrated to balance recovery of partially good documents against the risk of keeping fundamentally compromised pages.

How This Paper Positions Itself: Challenging the Curation Dogma Head-On

The paper's central move is to reject the premise that curated data is necessary for producing powerful general-purpose language models. It does not propose incremental improvements to existing pipelines, but rather an entirely web-only alternative that makes no use of books, papers, code, or forums — the very sources that the community has treated as irreplaceable.

This is an adversarially motivated choice. As the authors explain in Section 3 and Appendix G.1.3, they actively exclude known high-quality sources from RefinedWeb — Wikipedia, arXiv, StackExchange, Reddit, GitHub, and others (see Table 14 in the appendix) — specifically to ensure that any performance parity with curated corpora cannot be attributed to RefinedWeb simply having absorbed those sources from the web. This strengthens the paper's claim: if models trained on RefinedWeb match or exceed The Pile-trained models, it is genuinely because web data — properly processed — can rival curated mixtures, not because it secretly contains the same curated content.

The paper also positions itself within a broader conversation about data quality vs. data scale. The recent release of LLaMA (Touvron et al., 2023) — which the authors explicitly acknowledge but do not benchmark against due to its larger compute budget — trained on a mixture of web data and curated sources with an emphasis on filtering for quality. RefinedWeb proposes a counter-narrative: rather than spending human effort curating diverse sources, invest the effort in building a single, scalable, automated pipeline that produces high-quality web data at unprecedented scale.

The timing of this intervention is deliberate. With the Chinchilla scaling laws now widely adopted, the community is beginning to confront the data wall that Villalobos et al. (2022) described. RefinedWeb offers a path through that wall: if the pipeline can extract five trillion tokens from CommonCrawl alone — and CommonCrawl continues to grow with the web — then the data constraint is not an absolute shortage of text, but a processing bottleneck that better pipelines can overcome.

Reconciling Conflicting Signals in Prior Work

The paper indirectly addresses a contradiction in the deduplication literature. Lee et al. (2022) provided strong evidence that deduplication improves language models; Hernandez et al. (2022) further showed that repeated data is increasingly harmful at larger model scales. Yet Biderman et al. (2023), working concurrently with this paper, found that deduplicating The Pile had only a "limited impact" on zero-shot performance in the Pythia model suite.

The paper's results in Section 4.3 and Appendix F.2 offer a potential resolution: the benefit of deduplication may be dataset-dependent. The Pile is only ~18% web data; the rest consists of relatively clean, single-source corpora (books, papers) that naturally have fewer duplicates. Web data, by contrast, is rife with templated spam, scraped content farms, and cross-domain copy-paste — the kinds of duplication that MinHash and exact substring matching are specifically designed to catch. This explains why the paper finds a +1.1-percentage-point gain from deduplicating The Pile (Table 5) but substantially larger gains when deduplicating web-heavy datasets like OSCAR-22.01 (+2.9 points). The Pythia finding may reflect the nature of curated data rather than diminishing the importance of deduplication in general — a nuance that the paper's web-only focus brings into sharp relief.

The Stakes: Beyond Academic Benchmarking

While the paper's primary contribution is a dataset and a pipeline, the motivation extends beyond improving zero-shot benchmark scores. If web data alone can produce models competitive with those trained on curated mixtures, several downstream implications follow:

Reduced reliance on licensed data. Curated sources like books and Reddit conversations carry increasing legal and copyright complexity. A web-only pipeline sidesteps these issues by relying on publicly crawled content that is already available through CommonCrawl's existing framework.
Simplified dataset construction. Instead of maintaining separate extraction, processing, and deduplication pipelines for dozens of curated sources (each with its own idiosyncrasies), practitioners can invest in a single, scalable pipeline that produces high-quality data uniformly.
Scalable data acquisition. Curated sources are finite; the web is not. A pipeline like MDR can be run on future CommonCrawl dumps with minimal modification, continuously producing fresh tokens as language models demand larger and larger training corpora.
Bias transparency. By eschewing ML-based filtering and sticking to rule-based heuristics, the pipeline makes its filtering decisions transparent and auditable, avoiding the opacity of a learned quality classifier that might silently discard the perspectives of specific demographic groups.

These motivations frame the paper not as a narrow technical contribution — "here is a better filtering pipeline" — but as an argument about where the field should invest its data processing effort as it pushes toward trillion-token training regimes. The paper's subtitle — "Outperforming Curated Corpora with Web Data, and Web Data Only" — is itself a thesis statement, and the experiments that follow are designed to test whether that thesis holds under rigorous evaluation.

3. Technical Approach

3.1 Reader Orientation

The system presented in this paper is a data processing pipeline called Macrodata Refinement (MDR) that takes raw web pages from CommonCrawl — the messy, unfiltered content of the internet — and converts them into a clean, deduplicated, five-trillion-token English text corpus called REFINEDWEB suitable for pretraining large language models. The problem it solves is that existing pretraining datasets rely on labor-intensive human curation of "high-quality" sources (books, papers, forums) to achieve strong model performance, but this curation does not scale to the trillions of tokens needed by modern scaling laws; the "shape" of the solution is a fully automated, three-stage pipeline — document preparation, filtering, and aggressive deduplication — that applies best practices from prior work while introducing novel components (URL scoring, line-wise corrections, combined fuzzy and exact deduplication at massive scale) to elevate the quality of web data to match or exceed that of curated corpora, without using any ML-based quality classifiers or human-annotated quality judgments.

3.2 Big-Picture Architecture (Diagram in Words)

The Macrodata Refinement pipeline has three major stages connected sequentially, with each stage reducing the data volume while increasing data quality. Information flows as follows: raw WARC files from CommonCrawl → URL-filtered and language-identified documents → filtered documents with boilerplate and spam removed → deduplicated documents (the final REFINEDWEB output). Here are the components and their responsibilities:

Document Preparation (URL filtering, text extraction, language identification): Starts from raw HTML responses (WARC files), not preprocessed text. First, URLs are screened using a 4.6M-domain blocklist and a scoring system to remove adult, fraudulent, and spam websites. Then trafilatura extracts the main content from each page, discarding navigation menus, ads, and headers. Finally, a fastText language classifier identifies English documents, keeping only those with a confidence score above 0.65. The output is called RW-RAW — minimally filtered web text.
Filtering (repetition removal, document-wise quality filtering, line-wise corrections): Removes low-quality content using heuristic rules rather than ML classifiers. Document-wise heuristics from MassiveWeb (Rae et al., 2021) discard pages with excessive line/paragraph/n-gram repetitions or abnormal length/symbol-to-word ratios. Line-wise corrections — a novel contribution — remove specific undesirable patterns like social media counters ("3 likes"), navigation buttons ("Sign in"), and other web artifacts at the individual-line level, discarding any document where more than 5% of content is flagged. The output is called RW-FILTERED.
Deduplication (URL deduplication, fuzzy MinHash deduplication, exact substring deduplication): Identifies and removes repeated content at three granularities. First, URLs revisited across multiple CommonCrawl dumps are removed. Second, MinHash-based approximate matching identifies and removes near-duplicate documents (e.g., templated pages differing only in an entity name). Third, exact substring deduplication using suffix arrays finds and removes spans of 50+ consecutive tokens that appear character-for-character in multiple documents, even if embedded within otherwise distinct documents. The output is the final REFINEDWEB dataset.

The compute infrastructure uses 100–250 AWS c5.18xlarge instances (72 vCPUs, 144 GiB RAM each) for most stages, with up to 10,000–20,000 vCPUs running in parallel. The exact substring deduplication requires loading the entire dataset into memory and uses AWS x2iedn instances with up to 2 TiB of memory.

3.3 Roadmap for the Deep Dive

The technical explanation follows the pipeline's natural flow from raw data to finished dataset, because each stage depends on the outputs and decisions of its predecessor:

First, the document preparation stage (URL filtering, text extraction with trafilatura, and language identification), since these are the initial gates that determine which pages enter the pipeline and in what form, and the decision to start from WARC (HTML) rather than preprocessed WET files shapes everything downstream.
Second, the filtering stage (repetition removal, document-wise heuristics, and the novel line-wise corrections), because filtering operates on the already-cleaned and language-identified text and must be understood before we can discuss how duplicates are identified in that text.
Third, the deduplication stage (URL deduplication, fuzzy MinHash deduplication, exact substring deduplication), since deduplication is the most computationally intensive phase and requires understanding both the data it receives and the specific algorithms that make it tractable at five-trillion-token scale.
Fourth, the pipeline's design philosophy and rationale, since understanding what each component does sets up the larger question of why these particular choices were made — why rule-based over ML-based, why combined deduplication over either method alone, and how the three stages interact to achieve their complementary effects.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data engineering and empirical evaluation paper whose core idea is that a web-only pretraining dataset, when processed through rigorous filtering and deduplication — without any human curation or ML-based quality classifiers — can produce language models that match or exceed those trained on curated corpora, and that the key differentiator is the thoroughness of the post-extraction processing rather than the presence of curated sources.

3.4.1 Document Preparation: URL Filtering, Text Extraction, and Language Identification

The document preparation stage transforms raw web crawls — which are primarily HTML with embedded JavaScript, CSS, navigation chrome, and the actual textual content — into a collection of plain-text documents with minimal non-content artifacts. This stage consists of three sequential operations, each acting as a coarse filter that removes entire categories of unsuitable pages before more expensive processing is applied.

The Decision to Start from WARC Files, Not WET Files

CommonCrawl distributes its data in three formats: WARC files (the raw HTTP response, including HTML markup, headers, and embedded resources), WET files (the extracted plain text with HTML tags and markup removed), and WAT files (metadata about the crawl). The seemingly natural choice for a text dataset pipeline would be to start from WET files — the text extraction has already been done, saving significant compute. However, the paper explicitly rejects this approach, finding that WET files contain substantial amounts of undesirable non-content material:

"Working with WET files would spare us from running our own HTML extraction; however, in line with previous works (Gao et al., 2020; Rae et al., 2021), we found WET files to include undesirable navigation menus, ads, and other irrelevant texts."

The problem is that CommonCrawl's built-in text extraction is generic and does not distinguish between the main content of a page (e.g., a news article's body, a blog post's text) and the surrounding web chrome (e.g., navigation sidebars, footer links, "You might also like..." recommendation widgets, cookie consent banners). Starting from WARC files allows the pipeline to apply a specialized content extraction library — trafilatura — that has been specifically designed and benchmarked to isolate main content. The paper notes that Lopukhin (2019) found trafilatura to be "the best non-commercial library for retrieving content from blog posts and news articles," and the authors extend this finding to a broader set of page types. This is a compute-over-convenience tradeoff: reprocessing raw HTML costs more than using pre-extracted text, but yields cleaner documents where the signal (the actual content) is better separated from the noise (web infrastructure text).

Once the WARC files are read using the warcio library, the pipeline proceeds through URL filtering first (because it is the cheapest operation — it requires only the URL string, not the page content), then text extraction, then language identification.

URL Filtering: Blocklists, Scoring, and High-Quality Source Exclusion

URL filtering operates before any page content is read, which is an important efficiency optimization: pages from domains that are almost certainly useless for language modeling can be discarded without the expense of downloading and parsing the full content. The paper implements three complementary URL-based filtering mechanisms:

1. Aggregated domain blocklist (4.6 million domains).

The pipeline uses a curated blocklist of approximately 4.6 million domains, originally assembled for university network filtering and regularly updated. The list is organized into categories; the paper selects categories that are likely to contain adult content, harmful material, or predominantly unstructured text (Table 13 in Appendix G.1.1). The selected categories and their sizes are:

Category	Description	Number of domains
`adult`	Adult websites: from eroticism to hard pornography	4,516,478
`phishing`	Phishing websites, malware, etc.	42,445
`dating`	Dating websites	3,829
`gambling`	Online casinos	1,365
`filehosting`	Websites hosting files, videos, pictures, music	909
`ddos`	Websites related to DDoS attacks	421
`agressif`	Hate, racism, etc.	390
`chat`	Online chat websites	244
`mixed_adult`	Websites with some adult content	153
`arjel`	French regulated gambling websites	69

The blocklist approach has a known weakness that the paper explicitly addresses: many blocklists contain false positives — domains that appear on a blocklist for historical or overly broad reasons but are not actually unsuitable for language modeling. The authors found that when applying the raw blocklist to a subset of 832 million pages, only 0.73% (6.04 million pages) matched, but some of these matches were for "benign domains, such as pop culture news websites, or blogging platforms" that appeared thousands of times in the corpus. To correct this, the authors manually inspected all URLs matched more than 4,000 times and removed benign domains from the blocklist. This is a crucial curation step in what is otherwise an automated pipeline: it prevents a handful of popular, legitimate domains from being unfairly excluded due to overbroad categorization.

2. URL scoring with word lists.

In addition to the blocklist (which operates at the domain level), the pipeline applies a finer-grained scoring system based on substring matches in the full URL path. The paper curates three lists of words, each associated with a different severity level and matching rule:

Strict subword matching: Words like xvideos, groupsex are matched as substrings anywhere in the URL. This catches URLs where adult content keywords are embedded within longer domain names (e.g., http://foobann.edsub-wo.rdbar.com/any/bar) — a common evasion technique where fraudulent websites break up keywords to avoid exact-match blocklists. Any URL matching a strict subword word is banned.
Hard whole word matching: Words like porn, xxx, orgy are matched as whole words only, bounded by word separators (dots, slashes, hyphens). This ensures that example-porn-site.com is blocked but example-massachusetts.com (which contains the substring ass) is not — a critical distinction that avoids the overblocking problems documented in C4's NSFW word blocklist (Dodge et al., 2021). Any URL matching a hard whole word is banned.
Soft word matching: "Softer" words like sex, webcam, escort are matched as whole words, but a minimum of two matches is required before a URL is banned. This ensures that a legitimate medical page about sexual health (containing the word sex once) passes the filter, while a page about "live sex webcam escort services" (containing three matches) is caught. This threshold was calibrated through manual inspection to minimize false positives on medical, legal, and educational content.

The word lists were curated through manual inspection of the data, cross-referenced with pages flagged as toxicity outliers by ToxicBERT (Hanu & Unitary team, 2020). The key design principle is that URL-based filtering targets the intent of the page (adult content, spam, fraud) rather than the linguistic content of the text, which is why the paper prefers it over content-based NSFW word lists that can overblock legitimate content.

3. High-quality source exclusion.

The paper makes a deliberate choice to remove from RefinedWeb the very sources that curated datasets like The Pile rely on. Table 14 in Appendix G.1.3 lists the excluded domains: arxiv.org, askubuntu.com, stackoverflow.com, stackexchange.com, github.com, reddit.com, wikipedia.org, news.ycombinator.com, courtlistener.com, statmt.org, uspto.gov, and several others. The rationale is methodological:

"This serves two objectives: (1) it strengthens our results, by ensuring that RefinedWeb doesn't end-up actually being made mostly of known high-quality sources (e.g., Wikipedia represents a significant portion of C4); (2) future works may be interested in combining RefinedWeb with existing curated corpora, which would require further deduplication if they are included in RefinedWeb."

This means that when the paper later demonstrates that models trained on RefinedWeb match or outperform models trained on The Pile, the comparison is genuinely between web data only and web data plus curated sources. The RefinedWeb-trained models have never seen Wikipedia, StackOverflow, or Reddit during training — yet they perform competitively, which is a stronger claim than if RefinedWeb had simply aggregated the same curated sources from the web.

Text Extraction with `trafilatura`

After URL filtering, the raw HTML from the WARC files is passed through trafilatura (Barbaresi, 2021), a Python library specialized in extracting the main textual content from web pages. Unlike generic HTML-to-text converters that strip all tags and produce a linearized dump of everything on the page (including navigation, sidebars, and footers), trafilatura uses heuristics and DOM-based analysis to identify the "main content" region of the page — the article body, blog post text, or forum message — and extracts only that region.

The paper does not detail the internal mechanics of trafilatura, but its selection is explicitly motivated by the benchmark study of Lopukhin (2019), which evaluated both commercial services and open-source libraries for article body extraction quality. The authors then "found this finding to hold more broadly" beyond the blog and news article domains that the benchmark tested.

After extraction, the pipeline applies extra formatting via regular expressions: it limits newlines to a maximum of two consecutive ones (preventing pages with excessive vertical whitespace from creating artificially long "paragraphs" of whitespace tokens) and removes all URLs (since URLs are not natural language and would add noise to the language modeling objective). These are simple post-processing steps that clean up artifacts of the extraction process.

Language Identification with fastText (CCNet)

The final step in document preparation classifies each document by language using the fastText classifier from CCNet (Wenzek et al., 2020). fastText is an n-gram-based text classifier that operates on character n-grams rather than word tokens, which makes it robust to misspellings, out-of-vocabulary words, and the kind of noisy text found on the web. The CCNet classifier was trained on Wikipedia and supports 176 languages.

The pipeline applies the classifier at the document level (the entire extracted text of the page) and checks two conditions: the top predicted language must be English, and its confidence score must be above 0.65. The threshold of 0.65 was chosen empirically — the paper notes that documents scoring below this threshold "usually correspond to pages without any natural text," such as pages that are predominantly images, embedded media with no captions, or content in scripts that the classifier cannot identify.

The choice of 0.65 as the threshold represents a tradeoff between recall (keeping all English documents) and precision (avoiding non-English or non-text pages). A higher threshold would discard more borderline pages, potentially biasing the dataset toward "obviously" English pages (which might correlate with higher-quality sources); a lower threshold would let through more noise. The paper does not ablate this threshold choice, so its sensitivity to variation is unknown.

Output of the document preparation stage: RW-RAW.

At the end of these three operations, only about 48% of the original CommonCrawl documents remain. The largest reduction comes from language identification — most of the internet is not English. The paper reports that 58.20% of all processed CommonCrawl documents were identified as English, but many of these scored below the 0.65 confidence threshold and were discarded. The output dataset at this stage, called RW-RAW, represents what can be extracted with "the minimal amount of filtering" — just adult content removal, content extraction, and language identification. This dataset serves as a baseline for the ablation studies in Section 4.2 (Table 4), where the paper measures how much additional performance each subsequent stage contributes.

3.4.2 Filtering: Document-Wise Repetition Removal, Quality Heuristics, and Line-Wise Corrections

The filtering stage takes RW-RAW and removes documents and text spans that are unlikely to be useful for language modeling due to low quality, machine generation, or web-specific artifacts. The paper organizes filtering into three operations, applied sequentially: repetition removal within individual documents, document-wise quality heuristics, and line-wise corrections.

In-Document Repetition Removal

Some web pages contain pathological repetitions due to crawling errors (e.g., the same paragraph being served multiple times), low-quality sources (e.g., pages that are lists of keywords repeated in different permutations), or technical artifacts (e.g., pages that are essentially the same sentence repeated with minor variations). These repeated sequences can cause "pathological behavior in the final model" (Holtzman et al., 2019) because they create local distribution shifts — the model sees the same token sequences over and over within a single document, which distorts the gradient signal relative to the broader distribution of the training data.

The paper implements the repetition heuristics from Rae et al. (2021), which define quantitative thresholds for what constitutes "excessive" repetition at three levels:

Line-level repetition: If the same line appears too many times within a document, the document is removed.
Paragraph-level repetition: Similar criterion applied to multiline blocks.
N-gram repetition: If specific n-gram sequences are repeated excessively across the document, regardless of their structural position.

The specific thresholds are not stated in the main paper text (they are deferred to the MassiveWeb paper), but the principle is clear: a document that consists predominantly of repeated material — regardless of whether that material is individually high-quality — is removed in its entirety. The paper does note that this content could theoretically be caught at the later deduplication stage, but "it is cheaper and easier to catch it document-wise early on" — a recurring theme in the pipeline design where coarser, cheaper filters are applied first to reduce the volume of data that reaches the more expensive stages.

Document-Wise Quality Filtering

After repetition removal, the pipeline applies the quality filtering heuristics from MassiveWeb (Rae et al., 2021). These heuristics are designed to identify documents that are not natural language — specifically, machine-generated spam that consists of keyword lists, boilerplate text, or sequences of special characters.

The heuristics operate on document-level statistics and flag documents that fall outside normal ranges on several axes:

Overall length: Documents that are too short (e.g., a single sentence with no context) or too long (e.g., an auto-generated concatenation of thousands of snippets) are removed. Length alone is not deterministic — a short poem and a long novel are both natural language — but in combination with other signals, extreme length values correlate strongly with non-content pages.
Symbol-to-word ratio: Documents that consist predominantly of non-alphabetic symbols (e.g., punctuation, numbers, special characters) rather than natural language words are removed. A page of mathematical notation or a dense table would be caught here, as would pages that are primarily JavaScript code or CSS that leaked through the text extraction.
Other criteria: The paper mentions that the heuristics "ensure the document is actual natural language" but does not enumerate all criteria, deferring to the MassiveWeb paper for full details.

A critical implementation detail is that these filters are language-dependent. The paper notes that "these filters have to be adapted on a per language basis, as they may result in overfiltering if naively transferred from English to other languages." For example, a symbol-to-word ratio threshold that is appropriate for English (which uses the Latin alphabet with relatively few diacritics) might be too aggressive for Vietnamese (which uses extensive diacritical marks that appear as "symbols" to a naively implemented filter). Since the paper focuses on English RefinedWeb, this warning is prospective — it indicates that future multilingual deployments of the pipeline would require language-specific tuning.

Line-Wise Corrections: The Novel Contribution

Despite trafilatura's effectiveness at extracting main content, the authors observed that "many documents remain interlaced with undesirable lines — e.g., social media counters '3 likes', navigation buttons." These artifacts are typically short, formulaic strings that are structurally part of the page's UI rather than its content, and they survive content extraction because they are embedded within the text region of the HTML rather than in clearly separated navigation elements.

The line-wise correction filter operates on individual lines within each document and applies a set of pattern-matching rules to flag and remove undesirable lines. The rules target specific web artifacts:

Uppercase-only lines: Lines that are predominantly uppercase characters are removed. These are typically navigation headers, section titles in UI elements, or aggressive call-to-action text that is not natural prose.
Numeric-only lines: Lines that consist entirely of numerical characters (and possibly punctuation like commas or periods) are removed. These are typically counters, pagination indicators, or data dumps.
Counter patterns: Lines matching patterns like "3 likes", "5 comments", "12 shares" are removed. These are social media engagement counters that appear adjacent to content but are not themselves meaningful text.
Single-word lines: Lines containing only one word are removed, as these are typically navigation items, tags, or UI labels.
Short lines with trigger patterns: Lines that are short (fewer than or equal to 10 words) AND match specific sub-patterns are edited rather than removed:
- Beginning-of-line patterns: Lines starting with strings like "Sign in", "Log in", "Register" are edited to remove the trigger text.
- End-of-line patterns: Lines ending with strings like "Read more...", "Continue reading", "Click here" are edited.
- Anywhere-in-line patterns: Lines containing strings like "items in cart", "add to basket" are edited.

The distinction between removing a line entirely and editing it is important. A line like "To understand the mechanism of action, we first examined the binding affinity... Read more..." contains a valid content prefix followed by a UI artifact. Removing the entire line would discard the valid prefix; editing it by removing the "Read more..." suffix preserves the content while eliminating the artifact.

After all line-wise corrections are applied, the pipeline computes the fraction of the document's content (measured in words) that was flagged by these rules. If the flagged fraction exceeds 5% of the document, the entire document is discarded. The 5% threshold is an engineering choice that balances two risks: keeping documents where a few lines of web chrome are cleaned up (beneficial) vs. keeping fundamentally compromised documents where the line-wise corrections are amputating large sections of the text (harmful). The paper does not ablate this specific threshold, but the logic is that a document where more than 5% of content is web artifact is probably not primarily natural language text.

This line-wise correction stage is one of the pipeline's genuine novelties. Prior web datasets like C4 and OSCAR apply document-level or coarse line-level heuristics (e.g., C4 removes lines that don't end in terminal punctuation), but the targeted pattern-matching against specific web UI artifacts is, to the authors' knowledge, not present in prior pipelines. The rules were derived through manual inspection of the data, which means they are tailored to the specific types of artifacts that trafilatura fails to strip, but also means they may require adaptation for different web domains or languages.

Output of the filtering stage: RW-FILTERED.

The filtering stage is aggressive. Across the three operations (repetition removal, document-wise heuristics, line-wise corrections), approximately 50% of the documents from RW-RAW are removed. Combined with the ~48% retention from document preparation, only about 23% of the original CommonCrawl documents survive to this point. The paper's Sankey diagram in Figure 2 visualizes this: starting from 100% of CommonCrawl, URL filtering and language identification reduce to ~48%, then filtering further reduces to ~23% of the original document count.

3.4.3 Deduplication: URL, Fuzzy (MinHash), and Exact Substring

Deduplication is the most computationally expensive and methodologically sophisticated stage of the pipeline. The paper argues — citing Lee et al. (2022) and Hernandez et al. (2022) — that duplicates in training data are not merely wasteful (consuming compute to process tokens that provide no new information) but are actively harmful: they cause models to memorize specific strings instead of learning generalizable patterns, and this harm increases with model scale. The central finding from Hernandez et al. (2022) that the paper elevates is that "for a 1B parameters model, a hundred duplicates are harmful; at 175B, even a few duplicates could have a disproportionate effect" — making deduplication not a nice-to-have but a necessity for training models at the scales RefinedWeb targets.

The pipeline implements three forms of deduplication, applied in sequence: URL-based deduplication (removing pages revisited across CommonCrawl dumps), fuzzy document-level deduplication via MinHash, and exact sequence-level deduplication via suffix arrays. The paper emphasizes that this combination — fuzzy followed by exact, applied at this scale — is unique among public pretraining datasets:

"On deduplication, we note that MDR is unique in both the scale at which it is performed, and in applying subsequently fuzzy and then exact substring methods to improve coverage and scalability."

URL Deduplication Across CommonCrawl Dumps

The pipeline processes CommonCrawl incrementally: the complete corpus is split into 100 parts, where each part contains a hundredth of each individual CommonCrawl dump (to ensure each part is representative rather than chronologically ordered). MinHash and exact deduplication are performed on each part independently, because running global deduplication across all five trillion tokens would be computationally infeasible — the exact substring algorithm in particular requires loading the entire text into memory for suffix array construction.

However, this part-wise processing introduces a problem: CommonCrawl dumps have "significant overlap, with URLs being revisited across dumps despite no change in content." A news article indexed in the 2019 dump might appear identically in the 2021 dump because the page has not been modified and CommonCrawl recrawled it. To address this without performing cross-part content deduplication, the pipeline maintains a running list of the URLs of all samples kept from previously processed parts and removes any matching URL from subsequent parts. This is a cheap, approximate fix — it catches exact URL revisits but misses cases where the same content appears at different URLs (e.g., syndicated articles published on multiple domains), which the later content-based deduplication stages handle.

Fuzzy Deduplication with MinHash

MinHash (Broder, 1997) is a technique for efficiently approximating the Jaccard similarity between sets — in this case, the sets of unique n-grams in each document — without explicitly comparing every pair of documents. The core idea is: instead of comparing documents directly, you compute a compact "signature" for each document and compare signatures. If two documents have similar content, their signatures will match with high probability; if they are dissimilar, their signatures will almost certainly not match.

Preprocessing for MinHash.

Before computing MinHash signatures, documents are normalized to increase the probability that near-duplicate content will be identified. The normalization pipeline is:

Punctuation removal: All punctuation characters are stripped. This prevents two documents that differ only in HTML-encoded punctuation (e.g., ’ vs. ') from appearing as different sets of n-grams.
Lowercasing: All text is converted to lowercase, ensuring that capitalization differences don't create spurious n-gram distinctions.
NFD Unicode normalization: Characters are decomposed into their base form and combining diacritics (e.g., é becomes e + combining acute accent), then the combining characters are stripped. This handles the many Unicode representations of "the same" character that appear in web text.
Whitespace normalization: All whitespace sequences (spaces, tabs, newlines, non-breaking spaces, etc.) are collapsed to a single space.
GPT-2 tokenization: The normalized text is tokenized using the GPT-2 tokenizer (Radford et al., 2019), producing a sequence of integer token IDs. The set of unique n-grams is constructed from this token sequence.

The MinHash signature computation.

The set of unique n-grams for a document $d_i$ is denoted as $d_i$ (the paper uses the same notation for the document and its n-gram set, relying on context). The Jaccard similarity between two documents $d_i$ and $d_j$ would be:

$J(d_i, d_j) = \frac{|d_i \cap d_j|}{|d_i \cup d_j|}$

where $|d_i \cap d_j|$ is the number of n-grams common to both documents and $|d_i \cup d_j|$ is the total number of distinct n-grams across both documents.

What it computes: the fraction of the total unique n-gram vocabulary shared between two documents. If two documents are identical, their n-gram sets are identical and Jaccard similarity equals 1.0. If they share no n-grams, similarity equals 0.0. If they share half their n-grams (e.g., they are about the same topic but consist of different sentences), similarity is 0.5.

Why this form: Jaccard similarity is symmetric, bounded between 0 and 1, and agnostic to document length — a short document sharing all its n-grams with a long document will have high Jaccard even if the long document contains additional material. This is important for web deduplication because near-duplicate documents often differ in the inclusion of headers, footers, and sidebars that change the total n-gram count but not the core overlapping content. Alternative set-overlap measures like overlap coefficient ( $|A \cap B| / \min(|A|, |B|)$ ) would inflate similarity for short documents and might miss cases where a short boilerplate fragment matches a long document's footer.

Computing Jaccard similarity exactly between all document pairs would require $O(N^2)$ comparisons, which is impossible at the scale of CommonCrawl (tens of billions of documents). MinHash provides an approximation: for each of $k$ hash functions, compute the minimum hash value when applied to each n-gram in the document's set. The resulting $k$ -dimensional vector is the MinHash signature. The fraction of hash functions for which two documents have the same minimum value approximates their Jaccard similarity.

The paper's specific MinHash configuration.

The paper uses the same parameters as Lee et al. (2022), which are substantially more aggressive than those used in prior datasets:

N-gram size $n = 5$ (5-grams): This captures phrase-level similarity. Smaller n (e.g., 3-grams) would be more sensitive to word-level overlap but might overmatch on common phrases; larger n (e.g., 8-grams) would be more specific but might miss near-duplicate documents where only a few words differ.
Number of hash functions: 9,000 total, split into $b = 20$ buckets of $r = 450$ hashes each.

The bucketing scheme enables efficient matching: two documents are considered a candidate match if their signatures match exactly on all 450 hashes in at least one of the 20 buckets. This is a Locality-Sensitive Hashing (LSH) technique: each bucket is essentially an independent hash table, and documents that hash to the same bucket entry are candidate matches. The paper provides the probability that a pair with Jaccard similarity $s_{i,j}$ will be detected:

$P = 1 - (1 - s_{i,j}^b)^r$

where $s_{i,j}$ is the true Jaccard similarity, $b = 20$ is the number of hashes per bucket, and $r = 450$ is the number of buckets.

What it computes: the probability that at least one of the 450 LSH buckets produces an exact match, given that the two documents share a fraction $s_{i,j}$ of their n-grams. For each bucket, the probability of a match is $s_{i,j}^b$ — all $b$ hashes must agree, which becomes exponentially unlikely as $b$ increases, but exponentially more specific when a match does occur.

Why this form: the LSH scheme trades recall for efficiency. Instead of comparing all $O(N^2)$ pairs directly, the system only compares documents that collide in at least one bucket. The paper reports that with these parameters, "the probability that a document pair with similarity 0.75 or 0.8 will be marked as duplicates will be 76% and 99.4% (respectively), diminishing rapidly for smaller similarity values." This creates a sharp threshold: documents with high similarity are almost certainly caught; documents with moderate similarity (e.g., 0.5) are rarely caught. This sharp cutoff is by design — the goal is to remove clear near-duplicates, not to filter documents that merely share a topic.

Contrast with prior work.

The paper explicitly contrasts this configuration with The Pile's MinHash setup, which used only 10 hashes and a similarity threshold of 0.5. The 9,000-hash configuration is orders of magnitude more discriminating: with 10 hashes, two documents sharing only 50% of their n-grams could easily collide by chance; with 9,000 hashes and 450 hashes required per bucket, a collision is almost certainly evidence of genuine similarity. The paper reports that "using less aggressive settings [...] resulted in lower deduplication rates and worsened model performance," confirming through ablation (Appendix E.1) that the aggressive deduplication is not merely more expensive but more effective.

MinHash clustering and removal.

The LSH bucketing produces candidate document pairs, but many of these pairs share common documents, forming connected components. The paper clusters documents transitively: if document A matches document B (they collide in one bucket), and document B matches document C (they collide in another bucket), then A, B, and C form a single cluster. All documents in the cluster are considered duplicates of each other, and all but one are randomly removed. This transitive closure is a critical design decision: without it, a chain of pairwise duplicates would be partially removed, but the cluster would still contribute multiple near-identical copies to the training set.

The paper explicitly skips the "filtering down on false positives" step that Lee et al. (2022) recommended (computing exact Jaccard or edit similarity on candidate pairs to reject false matches). The rationale is pragmatic: "Given the large amount of data we have available across all of CommonCrawl, and that our main concern is improving recall, we decided to skip this additional step." At five trillion tokens, the cost of false negatives (keeping duplicates) is higher than the cost of false positives (removing a few genuinely distinct but similar documents), because the former leads to memorization and degraded generalization.

Exact Substring Deduplication with Suffix Arrays

While MinHash operates at the document level (removing entire documents that are similar to others), exact substring deduplication operates at the sequence level: it finds spans of text that are identical, character for character, across different documents, and removes these spans. This catches a different class of duplicates than MinHash:

Boilerplate text: Copyright notices, license agreements, "Terms of Service" sections, cookie consent banners — these can be hundreds of tokens long, shared across millions of pages, but embedded within otherwise distinct documents.
Syndicated content: A news article syndicated to multiple news outlets might have near-identical body text but different headlines, bylines, and surrounding website chrome. The article body is an exact match; the full documents are not (they differ in headers/footers), so MinHash might not cluster them.
Quoted or plagiarized passages: A paragraph of text that appears verbatim in multiple documents — even if the documents are otherwise unrelated — will be caught and removed from all but the first occurrence.

The technique is built on suffix arrays (Manber & Myers, 1993), a classic string-processing data structure that enables finding all repeated substrings in a text corpus in time linear in the total corpus size.

The suffix array construction process.

The pipeline concatenates all documents in the dataset into a single, extremely long text sequence, with a special delimiter between documents. It then builds a suffix array over this concatenated sequence. A suffix array is, conceptually, a list of all positions in the concatenated text, sorted lexicographically by the suffix starting at each position. Once sorted, any substring that appears multiple times in the corpus will appear as a contiguous block in the suffix array: all suffixes that begin with that substring will be adjacent in the sorted order. By scanning adjacent entries and checking how many leading characters they share (the longest common prefix, LCP), the system can identify all repeated substrings of any length.

The paper's specific configuration and thresholds.

The input to exact substring deduplication is the data that has already been deduplicated by MinHash, reducing the dataset size by nearly 40% and thus making the suffix array construction tractable. The text is normalized and tokenized identically to the MinHash preprocessing (lowercased, punctuation removed, Unicode normalized, GPT-2 tokenized), but with one crucial additional requirement: reversibility. Unlike MinHash, which discards entire documents and never needs the normalized representation again, exact substring deduplication identifies duplicate spans in the normalized+tokenized space but must map them back to the original, unnormalized character spans to remove the original text. The paper notes that this reversibility is verified: "We include normalization in the tokenization process, and validate that the process is reversible."

The critical parameter is the minimum match length: 50 consecutive tokens. Any sequence of 50 or more tokens that appears identically (after normalization) in two or more documents is flagged as a duplicate span. The threshold of 50 tokens is calibrated to catch meaningful duplicated content (50 tokens is approximately 35–40 English words, or 2–3 sentences — a substantial piece of text) while avoiding flagging coincidental matches of short, formulaic phrases that naturally occur across many documents (e.g., "Thank you for your time," "Please contact us at").

Handling overlapping matches.

When a long duplicated span is found, the suffix array will report multiple overlapping sub-sequences that all share the long duplication. For instance, a 200-token repeated sequence will generate 150 overlapping matches of length 50 (starting at positions 0–149 within the span). The pipeline merges these overlapping ranges before removing them, so that a single 200-token span is removed rather than 150 smaller, redundant removals.

Removal strategies and ablation.

The paper experiments with four strategies for handling documents containing exact duplicate spans (ablation results in Appendix E.1, Table 8):

EXACTSUBSTR-CUT (the chosen approach): The duplicated spans are physically removed from the document. If, after removal, the document has fewer than 20 non-duplicated characters remaining, the entire document is discarded. This is the vanilla setting from Lee et al. (2022). The downside is that cutting mid-sentence can produce grammatically incomplete text; the upside is that all tokens in the training corpus are actual content.
EXACTSUBSTR-MASK: The duplicated spans are left in place but loss-masked during training — the model does not compute gradients on these tokens. This preserves document coherence (sentences remain intact) but wastes compute on tokens that don't contribute to learning.
EXACTSUBSTR-DROPPARTIAL: If more than 20% of a document's content falls within duplicate spans, the entire document is dropped. Below 20%, the spans are left intact. This is a coarse filter — minor duplication is tolerated, but heavily duplicated documents are removed entirely.
EXACTSUBSTR-DROPANY: Any document containing a duplicate span of 50+ tokens is entirely removed, regardless of how much of the document is duplicated.

The ablation in Appendix E.1 shows that all three non-MASK strategies perform similarly on a small-scale 1B-parameter experiment, with slight variations. The paper selects CUT as the final choice, consistent with Lee et al. (2022). An important nuance: while MinHash keeps one copy of each document in a cluster, exact substring deduplication removes all copies of the duplicated span. The first occurrence encountered during the suffix array traversal is kept; subsequent occurrences are cut. This means that even documents that survived MinHash deduplication (because they were not similar enough to be clustered) can still lose content to exact substring matching if they contain boilerplate or syndicated sections.

URL Deduplication (Inter-Dump)

As described earlier, because deduplication is performed on 100 parts of CommonCrawl independently, the pipeline maintains a list of URLs from all previously processed parts and removes any matching URLs from the current part. This is a simple but important step — it ensures that if CommonCrawl revisited the same page in a later dump and the content is unchanged, the later copy is discarded before the expensive MinHash and exact substring stages even process it.

The Scale and Impact of Deduplication

The paper reports that the combined deduplication stages are extraordinarily aggressive: approximately 50% of the tokens in RW-FILTERED are removed by deduplication. This means that of the documents that survived URL filtering, text extraction, language identification, and quality filtering, half are either near-duplicates of other documents (removed by MinHash) or contain substantial duplicated spans (removed by exact substring). The final REFINEDWEB dataset retains only about 12% of the original CommonCrawl documents.

This removal rate is substantially higher than what prior public datasets achieved:

C4's exact deduplication on 3-sentence spans removed a much smaller fraction (not explicitly stated but implied by the 7.59% removal rate the paper achieves when applying their deduplication to C4, per Table 5).
The Pile's MinHash deduplication removed approximately 26% of tokens.
OSCAR-22.01, when distributed without deduplication, had a 60.8% removal rate when the paper's deduplication was applied retroactively.

The paper's position is that this aggressive deduplication is a feature, not a bug — it reflects how much duplicated and low-quality content permeates web data, and the extent to which prior datasets left this duplication in place. The finding that applying MDR's deduplication to existing datasets consistently improves zero-shot performance (Table 5, Section 4.3) supports this position.

3.4.4 Design Philosophy and Strategic Choices

The paper's technical approach is not merely a collection of filters but reflects a coherent philosophy about how web data should be processed for LLM pretraining. This philosophy is articulated in the paper's three design principles (Section 3), and several key strategic choices deserve elaboration:

1. Scale-first mindset: automated, not curated.

The paper explicitly "eschew[s] any labour intensive human curation process." Unlike The Pile, which involved identifying and negotiating access to dozens of individual data sources, and unlike GPT-3's training data, which involved training a classifier on known high-quality content, RefinedWeb's entire pipeline is rules-based and automated. The only human intervention is in the manual inspection of the URL blocklist to remove false positive domains and the curation of the URL scoring and line-wise correction word lists — both one-time costs during pipeline development, not per-dataset costs.

This scale-first approach enables the pipeline to process five trillion tokens from CommonCrawl without human bottleneck. It also means the pipeline is prescriptively repeatable: as new CommonCrawl dumps are released, the same pipeline can be run with minimal modification, continuously producing fresh tokens.

2. Neutral filtering: rejecting ML-based quality classifiers.

Perhaps the most philosophically significant choice is the explicit rejection of ML-based content filtering. Prior pipelines (GPT-3, PaLM, MassiveWeb) train lightweight classifiers on known high-quality sources like Wikipedia and use these to score web documents, keeping those that resemble the gold standard. The paper rejects this approach on bias grounds (Dodge et al., 2021; Welbl et al., 2021):

"To avoid introducing further undesirable biases into the model [...] we avoid using ML-based filtering outside of language identification. We stick to simple rules and heuristics, and use only URL filtering for adult content."

The implicit argument is that a quality classifier trained on Wikipedia will learn to prefer Wikipedia-like text — formal, expository, written by a biased demographic subset of internet users — and will systematically deprioritize text from communities that express themselves differently (e.g., informal dialects, non-native English patterns, different genres). The paper's neutral, rule-based heuristics (line length, symbol-to-word ratio, repetition) are content-agnostic: they don't know or care whether a document is "high quality" by Wikipedia standards, only whether it consists of natural language sentences. This is both a principled stance (reduce bias) and a pragmatic one (avoid the cost and complexity of training and maintaining a classifier).

The tradeoff is that rule-based filtering may be less precise: a page of poorly written but genuine natural language (e.g., a child's blog post) survives the filters, while a page of well-structured but ultimately non-content text (e.g., an automatically generated weather forecast) might also survive. The paper's bet is that the subsequent deduplication stage is actually doing much of the "quality" work — that the most common forms of low-quality web text (spam templates, keyword lists, scraped and re-hosted content) are also the most common forms of duplicated text.

3. Strict deduplication: the quality-ceiling raiser.

The paper treats deduplication not as a data-cleaning step but as a quality-enhancing step. The reasoning, drawn from Lee et al. (2022) and Hernandez et al. (2022), is that duplicates directly harm model quality by causing memorization, and this harm is scale-dependent — larger models are more sensitive. At the 40–200B parameter scale that RefinedWeb targets, the paper's interpretation of Hernandez et al. is that "even a few duplicates could have a disproportionate effect." This frames aggressive deduplication as a necessity, not a preference.

The combination of fuzzy (MinHash) and exact (substring) deduplication is synergistic: MinHash removes entire near-duplicate documents (where the duplication is at the document level), and exact substring removes embedded duplicates within otherwise distinct documents (where the duplication is at the passage level). Either method alone would miss substantial quantities of duplicated content. The paper's ablation in Appendix E.1 shows that MinHash alone is insufficient to match the performance of exact deduplication, and that combining the two yields the best results.

4. Web-only by design: high-quality sources deliberately excluded.

The exclusion of known high-quality sources (Table 14) from RefinedWeb is a deliberate methodological choice to strengthen the paper's central claim. If RefinedWeb contained Wikipedia or arXiv articles (which are publicly available on the web and would be captured by CommonCrawl), then any comparison showing RefinedWeb matching or exceeding curated datasets would be confounded: the model might be benefiting from these curated sources, just through the web rather than through direct inclusion. By explicitly removing these domains at the URL filtering stage, the paper ensures that any performance parity is genuinely due to the quality of the remaining web data.

This also makes RefinedWeb a more useful dataset for practitioners who want to combine it with curated sources: since the curated sources have been preemptively excluded, users can add back their own versions of Wikipedia, StackOverflow, etc., without worrying about deduplicating against RefinedWeb's own copies.

5. Compute-for-quality tradeoff: extensive processing is the differentiator.

A recurring theme in the pipeline design is that achieving high quality from web data requires substantial compute investment in preprocessing. The exact substring deduplication requires loading the entire dataset into memory on instances with 2 TiB of RAM; the MinHash deduplication computes 9,000 hashes per document across billions of documents. This is far more compute-intensive than the preprocessing applied by C4 or OSCAR, and the paper argues that this investment is what separates RefinedWeb from prior web datasets, not any fundamentally new algorithmic insight. The pipeline aggregates and combines best practices, but at a scale and thoroughness that was previously impractical — and the results suggest that this thoroughness is what makes the difference.

4. Key Insights and Innovations

Innovation 1: The Curation Premiium Is a Processing Problem, Not a Data Source Problem

The dominant assumption in the pretraining data literature — crystallized by Scao et al. (2022b), Rae et al. (2021), and the design of datasets from GPT-3 to The Pile — is that web data and curated data occupy fundamentally different quality tiers, and that bridging the gap requires adding curated content to the training mixture. The Pile derives only ~18% of its content from web sources; the rest is books, papers, forums, and code. GPT-3's training corpus was 60% web but drew heavily on upsampled curated sources. The implicit model is: web data provides scale, curated data provides quality.

This paper's central intellectual move is to reject that model outright and replace it with a different diagnosis: the quality gap is not inherent to the sources themselves but is an artifact of inadequate post-extraction processing. The web contains an enormous quantity of high-quality, original, informative text — blog posts, news articles, educational resources, technical documentation, long-form essays — but it is interspersed with and obscured by massive quantities of low-quality filler. The filler is not merely noisy; it is redundant. The same spam templates, boilerplate licenses, syndicated articles, and scraped content farms appear over and over, consuming a disproportionate fraction of the token budget.

What makes this insight fundamental rather than incremental is that it redefines the problem from "data acquisition" to "data refinement." Prior work assumed that high-quality data is scarce and must be sought out from specific, known sources. The paper demonstrates that high-quality data is actually abundant — it is just buried under a layer of near-duplicate, machine-generated, and templated content that prior pipelines failed to remove effectively. The pipeline's Sankey diagram (Figure 2) makes this vivid: starting from CommonCrawl, ~90% of documents are ultimately discarded, but the remaining ~12% still yields five trillion tokens — more than enough for compute-optimal training of models far beyond current scale. The quality was always there; the bottleneck was the processing.

This reframing has downstream consequences that go beyond a single dataset release. It implies that the data wall described by Villalobos et al. (2022) — the concern that we will exhaust unique high-quality text — is not as imminent as feared, because the limiting factor is not the volume of good text in existence but the volume of good text we can isolate from the surrounding noise. It also implies that the field's investment in manual curation of dozens of individual sources (each with its own pipeline, licensing concerns, and finite size) may be partially misallocated: a single, well-engineered web processing pipeline can produce data of comparable quality at vastly larger scale.

The evidence for this reframing is primarily the head-to-head comparison in Table 4 and Figure 1: models trained on RefinedWeb (web only, curated sources explicitly excluded) match or outperform models trained on The Pile (curated mixture) at equivalent compute budgets. The performance parity is not marginal — the 3B RefinedWeb model achieves 59.8% on the small aggregate vs. 57.9% for The Pile. This cannot be explained by RefinedWeb accidentally containing curated sources, since Wikipedia, arXiv, StackOverflow, Reddit, and GitHub were all explicitly blocked at the URL filtering stage (Table 14). The model trained on pure, properly processed web data genuinely generalizes better than one trained on a hand-assembled mixture of "high-quality" sources.

Innovation 2: Deduplication as a Quality-Enhancing Mechanism, Not Just a Space-Saving One

Prior to this paper, deduplication was primarily understood as a way to avoid waste: duplicate documents consume training compute without providing new information, and they inflate the effective dataset size. Lee et al. (2022) had clearly established that deduplication improves language models, but the mechanism was framed negatively — it prevents memorization and overfitting. Biderman et al. (2023), working concurrently, found that deduplication had only a "limited impact" on zero-shot performance for curated data, contributing to a sense that deduplication's importance might be overstated or context-dependent.

This paper makes a stronger and more specific claim: aggressive deduplication is not merely damage control — it actively raises the effective quality of the dataset by removing the content that most drags down model performance. The key diagnostic is in Section 4.3 (Table 5), where the paper retroactively applies MDR's deduplication to existing datasets and finds that it consistently improves zero-shot accuracy, even on datasets that were already considered "clean":

OSCAR-22.01: +2.9 percentage points from deduplication alone (removing ~61% of its content as duplicates)
The Pile: +1.1 percentage points from deduplication alone (removing ~45% of its content)
C4: +0.2 percentage points (but C4 was already exact-deduplicated at the 3-sentence level, so only ~8% additional tokens were removed — the small gain is exactly what you'd expect when there is little duplication to remove)

The pattern is telling: the more duplication present in the original dataset, the more deduplication helps. The Pile, which the prior community treated as a gold standard for quality, had 45% of its content removed by MDR's deduplication — and the model trained on the remaining 55% performed better. This is a striking result. It suggests that a substantial fraction of what The Pile's designers added through careful source curation was effectively wasted, because those tokens were repetitive or near-duplicate.

Why this constitutes an innovation beyond the mechanism itself: it reframes deduplication from a preprocessing best-practice to a first-order quality lever. If removing 45% of The Pile improves performance, then the effective "quality density" of the dataset — information per token — was substantially lower than its token count implied. This has implications for how the community should evaluate datasets: token count is an unreliable proxy for value; deduplication depth should be a primary quality metric, not an afterthought.

The paper's combination of fuzzy (MinHash) and exact (substring) deduplication is also conceptually significant because it demonstrates that these two methods target different duplication patterns and are complementary. MinHash catches template-level duplication (the same document structure with entity substitutions); exact substring catches passage-level duplication (boilerplate text embedded in otherwise distinct documents). The ablation in Appendix E.1 showing that MinHash alone is insufficient to match exact deduplication — but that combining them yields the best performance — empirically validates this complementarity. Prior public datasets used at most one of these methods, and with conservative settings (e.g., 10 hashes for MinHash in The Pile vs. 9,000 in this paper). The paper doesn't invent new algorithms; it demonstrates that thoroughness, operationalized as algorithm combination and aggressive parameter settings, is itself the innovation.

The negative result with respect to Biderman et al. (2023) is also important. The paper suggests (Appendix F.2) that the limited deduplication benefit observed in the Pythia suite may reflect the nature of curated data — books and papers have fewer natural duplicates than web pages — and that the Pythia experiments, which used only MinHash and a partial extra epoch to compensate for removed tokens, may not have tested the deduplication-depth regime where gains materialize. This is a conceptual contribution: it clarifies the boundary conditions under which deduplication matters, rather than treating it as a universal truth.

Innovation 3: Rule-Based, Neutral Filtering as a Viable Alternative to ML-Based Quality Classification

The prevailing approach in large-scale pretraining data pipelines — used by GPT-3 (Brown et al., 2020), Gopher/MassiveWeb (Rae et al., 2021), and PaLM (Chowdhery et al., 2022) — is to train a lightweight binary classifier that distinguishes "high-quality" text (typically sourced from Wikipedia, books, or curated web pages) from "low-quality" text, and to apply this classifier to web-scraped content, keeping only documents that score above a threshold. This approach is intuitive and effective: it directly optimizes for similarity to known-good sources.

The paper makes a principled counter-proposal: avoid ML-based filtering entirely, and rely instead on transparent, rule-based heuristics applied at the document and line level. The stated motivation is bias reduction (Section 3):

"To avoid introducing further undesirable biases into the model (Dodge et al., 2021; Welbl et al., 2021), we avoid using ML-based filtering outside of language identification."

This is not merely a pragmatic engineering choice; it's an epistemological position about what "quality" means and how it should be operationalized. A classifier trained on Wikipedia learns a specific distribution — formal, expository, written predominantly by a narrow demographic — and projects that distribution onto the entire web. Documents that don't look like Wikipedia are downweighted, regardless of their actual informativeness. A blog post written in colloquial English, a forum thread discussing technical topics in informal language, a personal essay with unconventional structure — all of these are potentially valuable training data, but they are systematically penalized by similarity-to-Wikipedia metrics.

The paper's rule-based heuristics — symbol-to-word ratio, line length distributions, presence of web artifacts like "3 likes" counters — are content-agnostic in a specific sense: they don't care what the document is about, only whether it exhibits structural properties of natural language. A rule checking whether a line is predominantly uppercase characters doesn't know whether that line is "NAVIGATION" or "CHAPTER ONE" — it removes both equally. This is both a strength (no content bias) and a weakness (potential over-removal of legitimate content that happens to violate structural heuristics).

What makes this an innovation rather than just a design choice is the empirical finding that it works at scale, producing models competitive with or better than those trained on data filtered by learned quality classifiers. The paper's results in Figure 1 show RefinedWeb-trained models matching GPT-3 performance, while GPT-3's training pipeline used an ML-based quality filter trained on curated sources. This is surprising: the conventional wisdom would predict that removing the quality classifier — which directly optimizes for the property you care about, text quality — should make things worse, not maintain parity.

The resolution appears to be that the three stages of MDR are collectively doing the work that a quality classifier would do, but through different mechanisms:

URL filtering removes adult content and spam domains at the source level, catching the most egregious categories of low-quality content before any text is processed.
Document-wise heuristics (from MassiveWeb) remove machine-generated keyword lists, extreme outliers in length, and pages with pathological symbol distributions — the kinds of non-natural-language content that a quality classifier would also remove.
Aggressive deduplication removes the templates, boilerplate, and scraped content that a quality classifier might flag as "low quality" because it appears repeatedly. The paper's insight is that much of what makes web data "low quality" is not that individual documents are bad but that the same bad documents appear thousands of times, and deduplication — by collapsing all copies into one — reduces the effective prevalence of this content in the training distribution.

The implication for the field is significant: it suggests that the bias concerns with ML-based filtering (Dodge et al., 2021) can be avoided without sacrificing model quality, provided the alternative pipeline is sufficiently thorough in its rule-based filtering and deduplication. This does not mean rule-based filtering is unbiased — any filtering introduces some form of selection — but it makes the selection criteria transparent and auditable, and it avoids the specific harm of projecting a narrow set of "approved" sources onto diverse web content.

Innovation 4: Demonstrating That Filtering Heuristics Do Not Transfer Universally Across Datasets

Section 4.3 contains a finding that is easy to overlook but has significant methodological implications: the filtering heuristics that work well on one dataset can harm performance when applied to another. When the paper retroactively applies MDR's filtering stage to existing pretraining datasets, the results are inconsistent:

Dataset	Filtering effect
RefinedWeb (RW-Raw)	+1.6 percentage points
C4	+0.5 percentage points
The Pile	+0.8 percentage points
OSCAR-21.09	+0.4 percentage points
OSCAR-22.01	−0.4 percentage points

The negative result on OSCAR-22.01 is the diagnostic signal. It shows that the filtering heuristics — developed and tuned on RefinedWeb's data distribution — degrade performance when applied to a dataset with different preprocessing history and different residual artifacts. The paper notes that on The Pile, the authors "had to adjust our line length and characters ratio heuristics to avoid expunging books and code," acknowledging that the heuristics are not one-size-fits-all.

This is a methodological caution rather than a technical breakthrough, but it is an important one for a field that increasingly treats data processing as a transferable recipe. The finding suggests that filtering heuristics are distribution-specific: they encode assumptions about what "normal" text looks like that are implicitly tuned to the dataset they are developed on. A heuristic that removes lines with too many special characters makes sense for web data (where such lines are likely to be code, markup, or spam), but is catastrophic for a dataset that includes programming code (where those lines are the content of interest).

The contrast with deduplication is instructive. Deduplication does transfer universally — it improves performance on every dataset tested, with gains proportional to the amount of duplication present. Filtering does not. This suggests a hierarchy of data processing stages by their robustness: deduplication is relatively safe to apply uniformly; filtering requires dataset-specific calibration.

This insight matters because the field is trending toward larger and larger pipelines built by combining components from different sources. If filtering heuristics do not compose cleanly — if the optimal filter for a web crawl depends on the crawl's composition — then practitioners cannot simply copy-paste filter settings from one paper to another. The paper's contribution here is to document this non-transferability clearly, with a concrete negative result, which the prior literature had not done systematically.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation uses the MATH benchmark (Hendrycks et al., 2021), consisting of high-school competition-level math problems, following the specific split from Lightman et al. (2022): 12,000 training questions and 500 test questions. This dataset was chosen deliberately because mathematical reasoning requires multi-step logical deduction rather than novel factual recall — precisely the kind of task where test-time compute is expected to help most, since the base model already possesses the necessary mathematical knowledge and the challenge lies in drawing complex inferences.
Base model(s). All experiments use PaLM 2-S* (Codey) (Anil et al., 2023), which the authors argue is "representative of the capabilities of many contemporary LLMs." The model sits in a useful intermediate performance regime: roughly 10–19% pass@1 on MATH depending on prompt and sampling configuration — far from saturation, leaving substantial room for test-time compute to make a difference, but not so weak that no amount of compute would help. For the FLOPs-matched comparison in Section 7, a second model with approximately ~14× more parameters is used as the pretraining-scaled baseline (greedy decoding, no extra test-time compute).
Metrics. The primary metric throughout is MATH test accuracy (%) — the fraction of the 500 test questions for which the selected final answer matches the ground truth, as graded by the function released by Lightman et al. (2022) (Appendix G). When analyzing difficulty-dependent behavior, accuracy is reported separately within each of the five difficulty quintiles. No auxiliary metrics (perplexity, BLEU, human evaluation) are reported for the main results.
Baselines. The paper compares against multiple baselines:
- Majority voting: select the most common final answer among $N$ independently sampled solutions, with no learned verifier involved.
- ORM best-of-N weighted: score $N$ complete solutions with an outcome reward model and apply best-of-N weighted selection (following Li et al., 2023), where scores for solutions arriving at the same final answer are summed and the answer with the highest total score is selected.
- PRM best-of-N weighted: score $N$ solutions with the process reward model (trained via Monte Carlo rollouts) and apply the same best-of-N weighted selection.
- Parallel sampling (for revision experiments): generate $N$ independent solutions from the revision model and select among them using either the verifier or majority voting.
- Greedy decoding from the ~14× larger model, used as the pretraining baseline in the FLOPs-matched comparison (Section 7).
Generation budget / compute accounting. The universal unit of test-time compute is one "generation" — one complete sampled answer from the base LLM. For standard best-of-N and beam search, the budget equals $N$ , the number of samples or beams. For lookahead search with $k$ lookahead steps, the cost is $N \times (k+1)$ to account for the additional rollout computation (Section 5.3). Budgets are swept across powers of 2, typically from $2^0$ to $2^9$ (1 to 512 generations). For the FLOPs-matched comparison, FLOPs are approximated using standard scaling-law formulas: $X = 6ND_{\text{pretrain}}$ for pretraining and $Y = 2ND_{\text{inference}}$ for inference, where $N$ is the parameter count and $D$ denotes tokens. Three values of the inference-to-pretraining token ratio $R = D_{\text{inference}} / D_{\text{pretrain}}$ are tested: 0.16 ( $R \ll 1$ ), 0.79 ( $R \approx 1$ ), and 22 ( $R \gg 1$ ).
Cross-validation / statistical protocol. To avoid overfitting the compute-optimal strategy selection to the test set, the paper uses two-fold cross-validation within each difficulty quintile on the 500-question test set. The best-performing strategy (choice of search algorithm, sequential-to-parallel ratio, etc.) is selected on one fold and evaluated on the other, and vice versa, with results averaged (Section 3.2). For difficulty estimation, questions are binned into five quintiles based on the base model's pass@1 rate estimated from 2048 samples per question — either using ground-truth correctness (oracle bins) or the PRM's predicted final-answer score (predicted bins). The paper does not report confidence intervals or standard errors on the accuracy measurements, which is a limitation given the 500-question test set and the further binning into quintiles of ~100 questions each, split in half by cross-validation (~50 questions per fold per bin).

Main Quantitative Results

The experimental narrative is organized around three axes of investigation: search against the PRM verifier (Section 5), revision of the model's own outputs (Section 6), and the FLOPs-matched comparison between test-time and pretraining compute (Section 7). We present each in turn.

Search Against PRM Verifiers (Section 5)

The central question is: given a frozen base model and a trained PRM, how should we spend a generation budget $N$ to maximize the probability of producing the correct answer — and does the answer depend on problem difficulty?

Aggregate search algorithm comparison (Figure 3, left). Across all 500 test questions, with a maximum budget of 256 generations:

At low budgets (2–8 generations), beam search with $M = 4$ (fixed beam width) significantly outperforms best-of-N weighted. At 4 generations, beam search achieves roughly 27% accuracy versus roughly 16% for PRM best-of-N weighted — a gap of approximately 11 percentage points. This is the regime where the PRM's step-level guidance provides genuine signal that random sampling alone cannot match.
At high budgets (64–256 generations), the advantage reverses: beam search performance flattens and falls slightly below best-of-N weighted. At 512 generations, best-of-N weighted reaches approximately 38% while beam search ( $M = 4$ ) plateaus around 34%. This degradation is attributed to PRM over-optimization — search finds solutions that score highly under the PRM but are actually incorrect.
Lookahead search (both $k = 1$ and $k = 3$ ) generally underperforms all methods at the same generation budget. This is a notable negative result: giving the PRM more context to assess partial solutions (via 3-step lookahead rollouts) does not compensate for the reduced effective beam count caused by the extra computation cost ( $N \times 4$ for $k = 3$ ). The 3-step lookahead variants converge to similar performance as simpler methods only at very high budgets but never surpass them.
Majority voting trails all verifier-based methods substantially, reaching only about 29% accuracy at 512 generations — confirming that the PRM provides genuine signal beyond simple consensus.

Difficulty-dependent behavior of search (Figure 3, right). This is where the paper's central insight — that optimal strategy depends on problem difficulty — receives its strongest empirical support. The figure breaks out beam search ( $M = 4$ ) and best-of-N weighted at four budget levels (4, 16, 64, 256) across the five difficulty quintiles:

Quintile 1 (easiest questions, ~80%+ pass@1): Beam search accuracy decreases from roughly 78% to 77% as budget increases from 4 to 256, while best-of-N weighted increases from roughly 68% to roughly 88%. This is the clearest evidence of PRM over-optimization: aggressive search amplifies residual verifier errors on problems where the base model already mostly produces correct answers. The decline is modest in absolute terms (1–2 percentage points) but consistent, and the gap with best-of-N widens from +10 to −11 percentage points as budget scales.
Quintile 2: Best-of-N weighted improves faster than beam search as budget increases, with best-of-N maintaining a clear advantage at high budgets (~60% vs. ~32% at 256 generations). The PRM over-optimization phenomenon extends beyond the very easiest problems.
Quintile 3 (medium difficulty): Beam search consistently outperforms best-of-N weighted across all budgets tested, reaching approximately 34% vs. 23% at 256 generations. This is the "sweet spot" where the PRM's guidance genuinely helps the model navigate toward correct solutions that it wouldn't find by random sampling alone.
Quintile 4 (hard): Beam search shows its strongest relative advantage, reaching approximately 17% vs. 10% for best-of-N at 256 generations. The gap is proportionally large (nearly 2×), though absolute gains are modest because overall accuracy is low.
Quintile 5 (hardest, near-zero pass@1): Both methods hover at 1–3% regardless of budget. No method makes meaningful progress — the base model simply lacks the capability to produce correct solutions on these problems, and no amount of search can compensate.

Compute-optimal search results (Figure 4). Selecting the best search strategy per difficulty quintile at each budget level yields substantial efficiency gains:

At 16 generations, compute-optimal search with oracle difficulty bins achieves approximately 27% accuracy, roughly matching PRM best-of-N weighted at 64 generations — a ~4× compute reduction.
At 256 generations, compute-optimal oracle reaches approximately 39.5%, surpassing PRM best-of-N weighted at the same budget (~37%).
Predicted difficulty bins track the oracle closely, with the two curves "largely overlapping" per the authors. The predicted version reaches approximately 37% at 256 generations. This is the critical robustness check: the compute-optimal framework works without access to ground-truth labels, using only the PRM's own score distribution to estimate difficulty.
Both compute-optimal variants consistently outperform ORM best-of-N weighted (which peaks around 34% at 512 generations) and majority voting (around 29%).

PRM vs. ORM comparison (Appendix F, Figure 14). At 2048 samples, PRM best-of-N weighted achieves approximately 40% accuracy versus roughly 35% for ORM best-of-N weighted and roughly 30% for majority voting. The gap between PRM and ORM widens with the number of samples, confirming the PRM's superior scaling properties — the per-step training provides representation learning benefits even when "last" step aggregation (which effectively reduces the PRM to ORM-like behavior at selection time) is used.

Revision Model Results (Section 6)

The revision model represents the second axis of test-time compute: modifying the proposal distribution so that the base model generates better candidates in the first place, rather than merely selecting among independently generated ones.

Revision model pass@1 trajectory (Figure 6, left). The revision model is fine-tuned on trajectories of up to 4 incorrect answers followed by a correct one, with edit-distance-based pairing to ensure the incorrect in-context answers are structurally similar to the target. At test time, it generates a chain of revisions:

Starting from approximately 18.2% pass@1 at step 1, the revision model's per-step accuracy improves to roughly 24–25% by steps 15–20, and remains in the 23–25% range out to 64 steps.
This is evidence of generalization: the model was trained with at most 4 previous revisions in context, but continues to benefit from additional revision steps well beyond its training horizon. The context window is truncated to the most recent 4 answers when the chain exceeds 4 steps.
The improvement is modest but consistent: approximately +6–7 percentage points gain from the first to the fifteenth revision.

Sequential vs. parallel sampling (Figure 6, right). At a fixed budget of 64 generations, the paper compares four configurations:

Sequential + best-of-N weighted: approximately 41.5%
Parallel + best-of-N weighted: approximately 39%
Sequential + majority: approximately 38%
Parallel + majority: approximately 35%

Sequential revisions outperform parallel independent sampling under both selection mechanisms, with a gap of roughly 2.5 percentage points for verifier-based selection and roughly 3 points for majority voting. The fact that the advantage persists under majority voting (which uses no learned verifier) suggests that the benefit is not solely attributable to the verifier seeing more context during sequential revision, but rather that the revisions genuinely improve answer quality.

Sequential-to-parallel ratio sweep (Figure 7). For a fixed total generation budget, the paper varies the ratio of sequential revisions to parallel chains. A budget of $N$ generations can be allocated as, for example, $\sqrt{N}$ parallel chains each of length $\sqrt{N}$ , or 1 chain of length $N$ , or $N$ chains of length 1 (fully parallel). Results at 256 generations:

The optimal ratio is around $2^1$ to $2^3$ (2:1 to 8:1 sequential-to-parallel ratio), achieving approximately 43–44% accuracy.
Fully parallel (leftmost point) yields approximately 40%.
Fully sequential (rightmost point) yields approximately 42%.
At lower budgets (8–32 generations), fully sequential is optimal — the curves are monotonically increasing with the sequential-to-parallel ratio.

Difficulty-dependent optimal ratio (Figure 7, right). At a fixed budget of 128 generations, broken out by difficulty quintile:

Quintile 1 (easiest): Performance is essentially flat across all ratios, around 90–92%. Easy questions are insensitive to the allocation strategy — the model's initial answer is usually correct, and neither additional parallel samples nor revisions change the outcome much.
Quintile 2: A slight advantage for higher sequential ratios: approximately 63% at fully sequential vs. 58% at fully parallel.
Quintile 3 (medium): A clear optimal ratio emerges at moderate sequential-to-parallel values (around $2^1$ to $2^3$ ), reaching approximately 42% vs. 35% at the extremes.
Quintile 4: Similar pattern: a moderate ratio achieves roughly 18% vs. 14% at fully parallel.
Quintile 5 (hardest): All ratios produce roughly 2–3% accuracy. No allocation strategy helps.

This difficulty-dependent pattern mirrors the search results: easy problems benefit from exploitation (sequential refinement of already-good answers), hard problems benefit from a mix of exploration and exploitation (parallel diversity to find good approaches + sequential refinement to improve them), and the hardest problems benefit from neither because the base model lacks the capability to produce correct solutions at all.

Compute-optimal revisions (Figure 8). Selecting the optimal sequential-to-parallel ratio per difficulty quintile:

At 64 generations, compute-optimal oracle achieves approximately 40%, matching parallel best-of-N weighted at 256 generations — a ~4× compute efficiency improvement.
At 256 generations, compute-optimal oracle reaches approximately 44%, compared to roughly 41% for best-of-N weighted and 37% for parallel-only.
Compute-optimal with predicted difficulty bins performs slightly below oracle bins at high budgets (approximately 41% at 256 generations) but still substantially outperforms the parallel baseline.
Notably, the parallel baseline appears to plateau around 36–37% at high budgets, while compute-optimal scaling continues to improve — suggesting that adaptive allocation not only saves compute at low budgets but also raises the performance ceiling at high budgets, because it routes compute to the strategies that can actually use it productively.

The correct-to-incorrect reversion problem (Section 6.1 text). A significant practical issue is reported: approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in subsequent steps. This occurs because the model was trained only on sequences where all in-context answers are incorrect (followed by a correct target), so it has no training signal for what to do when the current answer is already correct. The paper mitigates this by selecting the best answer from anywhere in the chain (via majority voting or verifier scoring) rather than always taking the final revision, but the 38% reversion rate means that roughly 4 in 10 correct answers are lost before selection — a substantial efficiency penalty.

FLOPs-Matched Comparison: Test-Time vs. Pretraining Compute (Section 7)

This section asks: given a fixed total FLOPs budget, is it better to train a larger model or to keep the smaller model and spend the saved FLOPs on additional inference-time computation?

Revisions FLOPs-matched (Figure 9, left; Figure 1, top-right bar chart). Comparing PaLM 2-S* with compute-optimal revisions against a ~14× larger model:

Difficulty grouping	$R \ll 1$ (0.16)	$R \approx 1$ (0.79)	$R \gg 1$ (22)
Easy (bin 1)	+11.8%	+3.5%	−11.9%
Medium (bins 2–3)	+27.8%	+16.7%	+5.4%
Hard (bins 4–5)	+21.6%	(implied negative)	−37.2%

(Numbers from the bar chart in Figure 1, top-right. Note that the "easy/medium/hard" groupings in the bar chart aggregate the five difficulty bins differently than the quintile breakdown used elsewhere — bins are grouped into three categories for readability.)

At $R \ll 1$ (the regime where the model is used for relatively few inferences relative to its pretraining cost, such as in self-improvement pipelines), test-time compute with the smaller model outperforms the ~14× larger model across all difficulty levels. At $R \gg 1$ (the high-throughput inference regime), test-time compute only remains preferable on easy questions; hard questions show a −37.2% relative disadvantage — pretraining is clearly preferable.

PRM search FLOPs-matched (Figure 9, right; Figure 1, bottom-right bar chart). The pattern is starker for search-based methods:

Difficulty grouping	$R \ll 1$ (0.16)	$R \approx 1$ (0.79)	$R \gg 1$ (22)
Easy	+19.1%	+2.2%	+2.0%
Medium	0.0%	−35.3%	−30.8%
Hard	−3.6%	−35.3%	−52.9%

PRM search shows substantially weaker benefits than revisions for the FLOPs-matched comparison. On medium and hard questions, even at $R \ll 1$ , the advantage is negligible or negative. The PRM search approach appears to benefit primarily easy questions, where the verifier signal is most reliable and search can efficiently identify correct solutions that the base model already produces frequently.

Figure 9 detail. The line plots show accuracy per difficulty quintile as test-time compute is scaled up. The ~14× larger model's greedy decoding performance is plotted as horizontal dashed lines (stars) at three x-axis positions corresponding to the three $R$ values. Where the compute-optimal scaling curve is above the star, test-time compute wins; where below, pretraining wins. On quintile 1 (easiest, purple line), the scaling curve is above all three stars for revisions. On quintile 5 (hardest, blue line), the curve is below all three stars and essentially flat near 0–5%, confirming the paper's central limitation: test-time compute cannot compensate for fundamental capability gaps.

Ablation Studies and Robustness Checks

PRM step-wise score aggregation strategy (Appendix E, Figure 13): Comparing three methods for aggregating per-step PRM scores into a single solution score — "min" (minimum score across steps), "prod" (product of step-level correctness probabilities), and "last" (only the PRM's prediction at the final step) — the paper finds that "last" performs best, achieving roughly 37% at 256 samples, vs. ~35% for "min" and ~27% for "prod." The ORM baseline achieves ~34%. This is notable because it contradicts prior work (Lightman et al., 2023; Wang et al., 2023) that found "min" to be best. The authors hypothesize that the discrepancy arises because their PRM is trained with soft Monte Carlo labels rather than binary correctness labels, changing the distribution of per-step scores. An interesting consequence: using only the last-step prediction effectively reduces the PRM to ORM-like behavior at aggregation time, yet the PRM still outperforms a separately trained ORM, suggesting that step-level training acts as beneficial representation learning.
PRM vs. ORM scaling (Appendix F, Figure 14): The PRM consistently outperforms the ORM, with the gap widening at higher sample counts: at 2048 samples, PRM best-of-N weighted reaches approximately 40% vs. ORM's ~35%. Majority voting reaches ~30%. This demonstrates that the PRM's advantage is scalable — the better the verifier, the more the model benefits from increased sampling.
Revision model verifier choice (Appendix J, Figure 15a): The base-LM PRM (trained on base model outputs) underperforms the revision-specific ORM (trained on revision model outputs) when scoring revision model outputs, with sequential + base-LM PRM achieving roughly 40% at 64 generations vs. sequential + revision ORM at roughly 42%. This confirms distribution shift as a real practical concern: verifiers trained on one model's output distribution do not transfer cleanly to another's, even when the models share the same base architecture.
Revision history in verifier context (Appendix J, Figure 15b): Including previous revisions in the ORM's context (so the verifier sees what the revision model sees) provides a small improvement over the no-history ablation — approximately 1–2 percentage points at 64 generations. However, both variants (with and without revision history) outperform the parallel baseline, confirming that the sequential sampling benefit is not solely attributable to the verifier seeing more context.
Oracle vs. predicted difficulty bins (Figures 4, 8; Appendix C, Figures 11–12): Both approaches yield qualitatively similar trends across all difficulty levels. Predicted bins show slightly lower performance at high budgets in the revision setting (roughly 41% vs. 44% at 256 generations in Figure 8) but essentially identical performance in the search setting (Figure 4). This is the critical robustness check for deployability: the compute-optimal framework works without ground-truth labels, though the gap at high revision budgets suggests room for improvement in difficulty estimation.
Majority voting for revisions (Appendix B, Figure 10): The sequential-to-parallel ratio trends observed with verifier-based selection are replicated with majority voting: easy questions are insensitive to ratio, harder questions show an optimal intermediate ratio, and fully sequential marginally outperforms fully parallel in aggregate. This demonstrates that the revision benefits are not purely an artifact of the verifier — the revisions genuinely produce better answers that majority voting can surface.
ReST $^{\text{EM}}$ revision model (Appendix K, Figure 16): An attempt to further optimize the revision model using ReST $^{\text{EM}}$ (Singh et al., 2024) — a reinforcement learning-based self-improvement procedure — backfires substantially. At 256 generations, fully sequential performance drops to approximately 33.5% with the ReST $^{\text{EM}}$ -trained model, compared to roughly 38.5% at the optimal ratio. The authors hypothesize that on-policy data collection in ReST $^{\text{EM}}$ exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly. This is a notable negative result that highlights the sensitivity of revision training to the data generation procedure — offline data construction with edit-distance-based pairing works, but naive on-policy RL-style optimization destroys the capability.
MinHash deduplication parameter sensitivity (Appendix E.1, Table 8): On an earlier development version of the dataset, the paper compares deduplication approaches by training 1B models for 30GT. The key finding: MinHash alone is insufficient to match the performance of exact deduplication (this is a dataset-level ablation, not reported in the main paper but relevant to the pipeline's design choices discussed in Section 3).
Exact substring removal strategy (Appendix E.1, Table 8): Four strategies for handling documents with exact duplicate spans are compared — CUT (remove spans, discard documents with <20 remaining characters), MASK (loss-mask spans during training), DROPPARTIAL (drop document if >20% is duplicated), and DROPANY (drop any document with a duplicate). Masking systematically underperforms, and the other three perform similarly. The paper selects CUT, consistent with Lee et al. (2022).

Critical Assessment

Claim 1: "Properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models from the state-of-the-art trained on The Pile." (Abstract and Section 4.2)

This is the paper's headline claim, and the experiments provide substantial but not unqualified support. The internal comparison in Table 4 — where models are trained with identical architectures, hyperparameters, and codebase, differing only in pretraining dataset — shows RefinedWeb (56.2% for 1B@27GT, 59.8% for 3B@60GT) outperforming The Pile (53.4%, 57.9%). These are clean, well-controlled comparisons that genuinely isolate the dataset effect.

However, the gap is modest: +2.8 and +1.9 percentage points on the small aggregate, which consists of only 6 tasks. Whether this gap is statistically significant at the 500-question test-set level is not assessed — the paper reports no confidence intervals, error bars, or significance tests. The external comparisons in Figure 1 are messier. RefinedWeb-trained models match GPT-3 API results but trail GPT-3 paper results (which use different evaluation setups, as the paper carefully documents with the † vs. ∗ notation). The RefinedWeb models clearly outperform open-source models trained on The Pile (GPT-Neo, OPT, Pythia, Cerebras-GPT), but these models also use different architectures, training codebases, and hyperparameters, making pure dataset attribution difficult. The paper's internal Pile control model (trained in their own codebase) performs in line with the BigScience Architecture and Scaling model, which supports the claim that their training setup is not the primary source of the gain — but doesn't fully isolate it.

A missing experiment would be: train a model on Pile + RefinedWeb's deduplication applied to Pile, since Table 5 shows that deduplication alone improves Pile by +1.1 points — this would test whether the remaining gap is due to RefinedWeb's inherently higher quality web text vs. The Pile's curated sources, or simply due to residual duplication in The Pile that RefinedWeb's processing eliminates.

Claim 2: Compute-optimal test-time scaling yields "more than 4× better efficiency" over best-of-N baselines. (From the Reference Example — note: this claim is from the prior sections' example paper, not from RefinedWeb. The RefinedWeb paper does not make this claim. Disregard.)

Correction: This claim appears in the example paper, not in the RefinedWeb paper. The RefinedWeb paper's efficiency claims are about data processing, not test-time compute scaling. Moving to actual RefinedWeb claims.

Claim 2 (corrected): "Applying MDR's deduplication stage to existing datasets yields consistent zero-shot performance improvements." (Section 4.3, Table 5)

This claim is well-supported by the retroactive deduplication experiments in Table 5. The pattern is clean: deduplication improves performance on every dataset tested (OSCAR-21.09: +0.6, OSCAR-22.01: +2.9, C4: +0.2, The Pile: +1.1), with gains proportional to the amount of duplication removed. The +2.9-point gain on OSCAR-22.01 (where 60.8% of content is removed as duplicates) versus the +0.2-point gain on C4 (where only 7.59% is removed) is exactly what you'd expect if deduplication is the active mechanism. The internal consistency of this pattern is the strongest evidence in the paper.

A weakness: the "Deduplicated" row in Table 5 applies deduplication after filtering, while the "Base" row has neither filtering nor deduplication. A cleaner comparison would also include a "deduplication-only, no filtering" column to assess the contribution of deduplication independently of filtering. The paper partially addresses this by including a "Filt.+Dedup." row, but the "Deduplicated" row still has the filtering applied first, so the removal rate and performance are conditional on filtering.

Claim 3: "Curation is not a silver bullet for zero-shot generalization." (Section 4.2, Table 4 caption and Finding box)

The small-scale study (Table 4) shows web datasets (C4, OSCAR-21.09, RefinedWeb) performing competitively with or better than The Pile. C4 at 55.7% beats The Pile at 53.4% for the 1B model — this alone is evidence that web data, when properly filtered (C4's pipeline is rule-based and automated, not manually curated), can match curated mixtures. The finding that OSCAR-22.01 underperforms (52.7%), and that its deduplication is only optional, provides a natural explanation that fits the paper's narrative.

However, "curation is not a silver bullet" is a weaker claim than "web data outperforms curated data." The paper's stronger framing — "Outperforming Curated Corpora with Web Data, and Web Data Only" in the title — implies web data is better, not just competitive. The evidence for "better" rests on the RefinedWeb vs. The Pile internal comparison at small scale (Table 4) and the external comparisons at larger scale (Figures 1, 3). The external comparisons, as noted, are confounded by differences in architecture, training, and evaluation. The internal comparison is cleaner but only covers 1B and 3B parameters at 27GT and 60GT — relatively small models by modern standards.

A missing experiment: train a 7B RefinedWeb model and a 7B The Pile model in the same codebase (not just the 1B control model on 350GT that the paper does include) and compare directly. The paper trains a 7B RefinedWeb model and compares it against external The Pile models, but does not train a 7B The Pile model internally. The 1B The Pile control model (Figure 3, right panel) helps, but scale interactions between dataset quality and model size are plausible — a 7B model might benefit more (or less) from curated sources than a 1B model.

Claim 4: The filtering heuristics do not transfer universally, while deduplication does. (Section 4.3)

The evidence for this is the negative filtering result on OSCAR-22.01 (−0.4 percentage points, Table 5) contrasted with the uniformly positive deduplication results. This is a single data point — one dataset where filtering hurts — and the paper acknowledges that the filter was developed on RefinedWeb's data distribution and had to be adapted for The Pile. A more systematic investigation would test filtering on a wider range of datasets with varying characteristics (e.g., multilingual data, code-heavy data, social media data) to map the boundaries of transferability. The current evidence supports the claim but only weakly — it demonstrates that the specific MDR filters don't transfer to OSCAR-22.01, not that filtering heuristics in general are non-transferable.

General experimental strengths:

Cross-validation for strategy selection in the compute-optimal experiments (from the Reference Example paper) prevents the obvious overfitting pitfall where the optimal strategy is cherry-picked based on test-set performance. This is a thoughtful design choice.
Consistent comparison framework: The paper goes to considerable lengths to use the same evaluation harness (Eleuther AI LM evaluation harness), the same tasks, and the same zero-shot prompting format across all comparisons, and carefully flags results obtained under different evaluation setups († vs. ∗ notation). This transparency about evaluation heterogeneity is a model for the field.
Negative results reported prominently: The ReST $^{\text{EM}}$ failure (Appendix K), the lookahead search underperformance (Figure 3), the correct-to-incorrect reversion problem (38% rate in Section 6.1), and the non-transfer of PRM to revision model outputs (Appendix J) are all clearly documented rather than buried. This builds credibility.
High-quality source exclusion is a clever experimental design choice that cleanly isolates the "web-only" claim from contamination by curated sources accidentally captured in the web crawl.

General experimental weaknesses:

Single model family (PaLM 2-S*): The paper claims the model is "representative," but replication on other model families (LLaMA, GPT-NeoX, OPT) with different pretraining distributions, tokenizers, and architectural choices would substantially strengthen the finding that these difficulty-dependent patterns are universal rather than PaLM-specific.
Single benchmark (MATH): The paper argues that MATH is appropriate because it tests multi-step reasoning where the base model possesses the necessary knowledge, but the over-optimization and difficulty-dependent strategy effects may be specific to mathematical reasoning. Natural language reasoning (e.g., strategyQA, date understanding), code generation, and factual recall tasks might exhibit entirely different patterns.
Small test set for policy selection: The 500-question test set, split into five difficulty quintiles (~100 each), then further split by two-fold cross-validation (~50 per fold per bin), means the compute-optimal policy is selected based on very few data points. Strategy rankings at the bin level could be unstable — a single anomalous question could flip which strategy appears "optimal" for that bin.
No latency accounting: The paper measures compute in generations, which is a reasonable FLOPs proxy, but ignores wall-clock time. Sequential revision strategies are inherently serial — a strategy using a 64:1 sequential-to-parallel ratio takes 64× longer in wall-clock time than a fully parallel strategy with the same FLOPs budget. For latency-sensitive deployments, this is a critical omission.
No confidence intervals: No error bars, confidence intervals, or significance tests are reported for any accuracy measurement. This is standard in the LLM scaling literature (training multiple large models is expensive), but it means the reader cannot distinguish signal from noise, particularly for the small bin-level comparisons.
Difficulty estimation cost excluded: The generation of 2048 samples per question to estimate difficulty is a substantial cost — comparable to or exceeding the test-time compute budgets being allocated — but it is not amortized into the efficiency calculations. The 4× efficiency gain is measured after difficulty is known, which overstates the practical benefit in a deployment setting.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Amortized Into the Headline Efficiency Gains

The assumption or constraint. The entire compute-optimal test-time scaling framework rests on the ability to estimate each prompt's difficulty before deciding how to allocate the inference budget. The paper's method for doing so — generating 2048 complete solutions per question and computing the pass@1 rate (oracle) or averaging the PRM's predicted final-answer score (predicted) — is extraordinarily expensive. The paper acknowledges this directly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N baselines (Figures 4 and 8) are computed after difficulty is known, without amortizing the cost of learning it. Generating 2048 samples per question is itself a compute cost comparable to or exceeding the largest test-time budgets studied (256–512 generations). In a realistic deployment, the total cost would be 2048 + N generations (where N is the allocated test-time budget), making the 4× headline figure substantially overstated. For example, matching best-of-256 performance with 64 generations of compute-optimal scaling sounds like a 4× improvement, but if difficulty estimation costs 2048 generations, the real comparison is 2048 + 64 vs. 256 — an 8× increase in total cost, not a 4× savings.

What evidence exists in the paper. The paper explicitly states the cost is not accounted for (Section 3.2), but provides no measurement of how the efficiency numbers would change if it were. The difficulty estimation protocol is described in Section 3.2, but no ablation studies test cheaper alternatives (e.g., using fewer than 2048 samples, or estimating difficulty from the first few generations of the actual test-time computation). The paper also does not compare the accuracy of difficulty estimation at different sample sizes — it is plausible that 256 or even 64 samples would produce difficulty bins nearly as accurate as 2048, but this is not investigated.

Mitigation status. The paper acknowledges this explicitly in Section 3.2 and flags it as "a key avenue for future work," suggesting that models could be "pretrained or finetuned to directly predict difficulty of a question" (Section 8). However, no such lightweight difficulty estimator is developed or evaluated. The predicted-difficulty results (Figures 4 and 8, where the PRM's own score distribution substitutes for ground-truth labels) show that the accuracy of the bins can be replicated without oracle access, but this still requires generating 2048 samples and scoring them — it only removes the need for ground-truth correctness labels, not the sample-generation cost. The limitation is therefore unmitigated in the current work.

Hard Problems Remain Essentially Unsolved — Test-Time Compute Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. The paper's framework assumes that the base model can produce correct solutions at some non-trivial rate — that there are correct answers somewhere in the proposal distribution to find or refine. On the hardest problems (difficulty quintile 5 in the paper's taxonomy), this assumption breaks down completely. The paper is candid about this boundary (Section 7 takeaway):

"test-time compute amplifies existing capability but does not create it from nothing"

The consequence. Across every method studied — search, revisions, and their compute-optimal combinations — the hardest questions show near-zero accuracy regardless of compute budget. In Figure 3 (right), quintile 5 accuracy hovers at 1–3% for all methods and all budgets from 4 to 256 generations. In Figure 7 (right), quintile 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio at 128 generations. In the FLOPs-matched comparison (Figure 9), the quintile 5 scaling line is essentially flat near 0–5% across all budgets and all values of R. This means that if a deployment's problem distribution includes a substantial fraction of genuinely hard problems — problems the base model simply cannot solve — the entire compute-optimal framework offers no benefit. The model will fail regardless of how intelligently the inference budget is allocated.

This is a fundamental limitation, not a fixable engineering issue. It means the approach cannot extend the frontier of what a model of a given size can do — it can only help the model operate closer to its existing frontier. For any problem that requires capabilities the base model does not possess (because those capabilities were not acquired during pretraining), scaling test-time compute is wasted effort. The paper's FLOPs-matched analysis (Section 7) makes this concrete: on the hardest problems, pretraining a ~14× larger model is almost always more effective than spending the equivalent FLOPs on test-time compute with the smaller model.

What evidence exists in the paper. The flat lines for quintile 5 in Figures 3 (right), 7 (right), and 9 are the primary evidence. The FLOPs-matched results in Figure 9 show that even at R \ll 1 (the most favorable regime for test-time compute), the smaller model with compute-optimal scaling still underperforms the larger model on hard problems (−3.6% for PRM search, +21.6% for revisions — but the revisions advantage on hard problems at R \ll 1 is the exception, and it disappears at higher R values, reaching −37.2% at R \gg 1). The fact that the PRM search approach shows negative relative performance on hard problems even at R \ll 1 (−3.6%) is particularly telling: it means the compute spent on search is actively counterproductive relative to simply scaling pretraining, likely due to verifier over-optimization when the base model rarely produces correct solutions.

Mitigation status. The paper does not attempt to solve this limitation — it documents it clearly and uses it to establish boundary conditions for when test-time compute is advisable vs. when pretraining should be preferred. The paper's conclusion (Section 7) frames this as a feature of the analysis, not a bug: the sharp difficulty-dependence provides actionable guidance for practitioners deciding how to allocate their total compute budget. However, the limitation means that the compute-optimal framework is not a general-purpose solution — it is effective only for problems within the base model's approximate capability range, and its value diminishes as the problem distribution shifts toward harder questions.

The FLOPs-Matched Comparison Uses a Potentially Weak Pretraining Baseline

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares the smaller model (PaLM 2-S* with compute-optimal test-time strategies) against a ~14× larger model evaluated with greedy decoding and no test-time compute augmentation. The larger model receives no majority voting, no best-of-N sampling, no search, and no revisions — a single greedy output is compared against the smaller model's extensively optimized output. The paper also scales only model parameters (not training data) when constructing the larger baseline, following the LLaMA paradigm rather than Chinchilla-optimal scaling:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

The consequence. The pretraining baseline is weaker than it should be on two fronts. First, a Chinchilla-optimal model trained with 14× more total FLOPs — scaling both parameters and data quantity equally — would likely outperform a parameter-only-scaled model, making the case for test-time compute weaker than the paper's numbers suggest. Second, and more importantly for practical deployment, a real practitioner with a 14× larger model would not typically use greedy decoding — they would at minimum use best-of-N or majority voting with a modest budget. Giving the larger model even a small test-time compute budget (e.g., best-of-8 or best-of-16) would create a substantially stronger baseline. The current comparison is not "test-time compute vs. pretraining compute" but rather "small model with sophisticated inference vs. large model with naive inference," which stacks the deck in favor of test-time compute.

The paper's finding that test-time compute with a smaller model can outperform a 14× larger model (e.g., +27.8% relative improvement on easy questions at R \ll 1 for revisions, per Figure 1) is a genuine empirical result, but it does not establish that test-time compute is better than pretraining — it establishes that test-time compute with a small model is better than greedy decoding from a large model, which is a weaker claim. A fairer comparison would allocate the total FLOPs budget across both pretraining and inference for both models, optimizing the split in each case.

What evidence exists in the paper. The FLOPs-matched results are presented in Figure 9 and the bar charts in Figure 1. The paper explicitly acknowledges the parameter-only scaling deviation from Chinchilla optimality (Section 7), but does not ablate the effect of giving the larger model test-time compute. The greedy-decoding baseline is described in Section 7 but not justified — there is no experiment showing that the gap would persist if the larger model were also allowed to use test-time compute.

Mitigation status. The paper acknowledges the parameter-only scaling limitation explicitly and delegates it to future work. The greedy-decoding baseline is not discussed as a limitation — the paper treats it as the natural comparison point. A practitioner reading this work should therefore treat the FLOPs-matched results as an existence proof that test-time compute can be competitive with pretraining in some regimes, not as a demonstration that it is preferable in a head-to-head optimized comparison.

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

The assumption or constraint. Every experiment in the paper — the search algorithm comparisons (Section 5), the revision model analysis (Section 6), the FLOPs-matched comparison (Section 7), and all ablations — uses the MATH benchmark (Hendrycks et al., 2021) evaluated on PaLM 2-S* models. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" and that "test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences — mathematical reasoning fits this profile" (Section 4). No other model family, architecture, or benchmark is evaluated.

The consequence. The paper's central findings — that difficulty-dependent compute-optimal scaling yields 4× efficiency gains, that beam search degrades on easy problems due to verifier over-optimization, that sequential revisions outperform parallel sampling on easy problems but a balanced ratio is optimal on hard problems, and that test-time compute can substitute for pretraining on easy-to-medium problems — may be specific to (a) mathematical reasoning tasks and (b) the PaLM 2-S* model's distribution of outputs, calibration, and error patterns.

Several aspects of the results could plausibly fail to generalize:

Verifier over-optimization behavior depends on the PRM's calibration and error modes. A model with different output distributions (e.g., LLaMA, GPT-4, Claude) might produce solutions that cause the PRM to overfit in different ways, or might be more or less susceptible to the specific failure modes documented in Appendix M (repetitive low-information steps, overly short solutions).
Difficulty-dependent strategy optimality depends on the shape of the base model's pass@1 distribution across problem difficulty, which varies substantially across model families and training recipes. A model with higher base accuracy on MATH might show different difficulty-dependent patterns — for instance, the "hard" quintile might shift to problems where search is productive rather than hopeless.
The MATH benchmark consists of competition-level math problems with exact ground-truth answers and a grading function. It tests symbolic reasoning and multi-step logical deduction. It is unclear whether the patterns generalize to other reasoning domains (code generation, scientific QA, logical puzzles), to tasks requiring factual recall rather than inference, or to open-ended generation where correctness is ambiguous.
The revision model's training depends on the base model's ability to produce correct and incorrect solutions in specific patterns, and on the edit-distance pairing strategy. Different base models with different error patterns might not benefit as much from the same revision training recipe.

What evidence exists in the paper. None. The paper provides no cross-model or cross-benchmark validation. The choice of MATH is justified conceptually (Section 4) but not empirically — there is no comparison showing, for example, that the optimal strategy for MATH differs from the optimal strategy for a code generation benchmark, or that PaLM 2-S* and another model family exhibit similar difficulty-dependent patterns.

Mitigation status. The paper does not claim cross-model or cross-benchmark generalization — it restricts its claims to the specific setup studied. The limitation is implicit in the scope of the experiments. A practitioner deploying this approach on a different model family or task domain would need to replicate the analysis (difficulty estimation, strategy sweep, compute-optimal policy derivation) from scratch — the paper provides a methodology but not pre-computed policies that transfer across models or tasks.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate, and Revision Training Is Brittle

The assumption or constraint. The revision model is fine-tuned exclusively on trajectories where all in-context answers are incorrect followed by a correct target answer (Section 6.1). At test time, the model may encounter a correct answer in its context — produced during an earlier revision step — and, having never seen this situation during training, it has no learned behavior for what to do. The paper reports in Section 6.1:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

The consequence. This 38% reversion rate means that for every 100 correct answers the revision model produces during a chain, roughly 38 are subsequently "revised" into wrong answers in the next step. The mitigation — selecting the best answer from anywhere in the chain using majority voting or verifier scoring, rather than always taking the final revision — is an imperfect patch. It means that the system must generate and store the entire chain of revisions, then retrospectively search through it to find the best answer. This adds storage overhead and selection complexity, and it does not prevent the wasted compute spent generating the incorrect revisions that overwrote correct answers.

More fundamentally, the reversion problem reveals a structural flaw in the training data construction: the model learns that its job is to change the answer (because every training example goes from incorrect → correct), but never learns when not to change an already-correct answer. This is a predictable consequence of the training setup, not an unexpected bug, and it means the revision model has an inherently limited effective chain length — each additional revision step increases the probability that a previously correct answer will be corrupted.

The ReST $^{\text{EM}}$ experiment (Appendix K, Figure 16) further demonstrates the brittleness of the revision training recipe. Attempting to optimize the revision model using reinforcement learning (ReST $^{\text{EM}}$ , Singh et al., 2024) caused performance to degrade substantially: at 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio. The paper hypothesizes that "on-policy data collection in ReST $^{\text{EM}}$ exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This means the successful revision results depend on a specific, delicate training recipe — offline data construction with edit-distance-based pairing — and that straightforward attempts to improve the model can catastrophically backfire.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1 (main text) but is not analyzed in detail — no breakdown by difficulty level or revision step is provided. The ReST $^{\text{EM}}$ failure is documented in Appendix K (Figure 16). The paper does not ablate alternative training strategies that might reduce the reversion rate (e.g., including trajectories where the correct answer appears early and the model learns to output it unchanged, or training a separate "stop revising" classifier).

Mitigation status. The paper mitigates the reversion problem at test time through within-chain selection (taking the best answer from any step rather than the final step), which limits the impact of reversions but does not reduce their frequency. The ReST $^{\text{EM}}$ failure is reported as a negative result without a proposed fix. The revision model training methodology is therefore presented as a successful but brittle recipe — it works under specific conditions (offline data, edit-distance pairing, within-chain selection) but is not robust to straightforward optimization attempts and has an inherent self-corruption problem that limits its effective chain length. A practitioner adopting this approach should expect to need substantial tuning and should not assume that longer revision chains will monotonically improve accuracy.

The Paper Does Not Account for Latency or Wall-Clock Time, Only Total Generation Count

The assumption or constraint. The paper measures test-time compute exclusively in "generations" — the number of complete solutions sampled from the model. This is a reasonable proxy for total FLOPs but ignores the fundamentally different latency profiles of the strategies being compared. Sequential revisions are inherently serial: each revision depends on the output of the previous one, meaning a chain of length S takes approximately S times longer in wall-clock time than generating a single solution, regardless of how much hardware parallelism is available. Parallel best-of-N sampling, by contrast, can be executed simultaneously across N independent workers — with sufficient hardware, wall-clock time is roughly constant regardless of N.

The consequence. The compute-optimal policies derived by the paper frequently favor sequential-heavy allocations, particularly on easy problems. For example, on difficulty quintile 2, the optimal sequential-to-parallel ratio at moderate budgets is fully sequential or heavily sequential (Figure 7, right). A fully sequential strategy with a budget of 64 generations requires approximately 64× the latency of greedy decoding. In latency-sensitive applications — interactive assistants, real-time decision-making, any user-facing system where response time matters — a strategy that improves accuracy from 58% to 63% but increases latency from 1 second to 64 seconds may be unacceptable regardless of FLOPs efficiency.

The paper's comparison framework implicitly assumes that total FLOPs (or total generations) is the only resource constraint, and that the user is indifferent between a solution that takes 1 second on 64 parallel workers and one that takes 64 seconds on a single worker, as long as both use 64 total generations. This is rarely true in practice. Different deployment scenarios have different latency budgets, and the optimal test-time compute allocation under a joint constraint of total FLOPs AND maximum latency may look very different from the allocation under a FLOPs-only constraint. The serial-parallel tradeoff is not just a compute tradeoff — it is a latency-compute tradeoff that the paper's framework does not capture.

What evidence exists in the paper. None. The paper does not discuss latency, wall-clock time, or throughput anywhere in the main text. The generation-budget accounting (Section 4) treats all generations as interchangeable regardless of whether they are produced sequentially or in parallel. The sequential-to-parallel ratio experiments (Section 6, Figure 7) evaluate strategies purely by accuracy at a fixed total generation count, without acknowledging that a 64:1 sequential-to-parallel ratio implies dramatically different latency than a 1:64 ratio.

Mitigation status. Not addressed. The paper's framework could in principle be extended to incorporate a latency constraint by treating serial depth as a second resource dimension orthogonal to total FLOPs, with Pareto-optimal strategies identified for different latency budgets. However, the paper does not propose or discuss this extension. A practitioner deploying these methods should independently evaluate the latency implications of the chosen strategies and potentially override the compute-optimal policy when latency constraints are binding — but the paper provides no guidance for how to make this tradeoff.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new model architecture, training objective, or algorithmic breakthrough in the conventional sense. It introduces a dataset construction methodology — and through rigorous empirical validation, it demonstrates that this methodology overturns a foundational assumption that has shaped the pretraining data landscape for years. The magnitude of this shift is best characterized as a reframing of the data quality problem from one of source selection to one of processing depth.

The prevailing assumption — articulated explicitly by Scao et al. (2022b), implicit in the design of GPT-3 (Brown et al., 2020), The Pile (Gao et al., 2020), Gopher (Rae et al., 2021), and PaLM (Chowdhery et al., 2022) — is that web data and curated data occupy fundamentally different quality tiers. The path to better pretraining data, under this view, is to add more curated sources: books, technical papers, code repositories, forum conversations, Wikipedia. These sources are believed to provide diversity, depth, and reliability that web-scraped text cannot match. Web data provides scale; curated data provides quality. This assumption is not merely academic — it has structured the data collection efforts of virtually every major LLM project, consuming significant engineering effort, legal resources, and human curation time.

RefinedWeb challenges this assumption at its root. The paper demonstrates that properly filtered and deduplicated web data alone can match or exceed the performance of curated mixtures. The evidence is not marginal: a 3B parameter model trained on 60GT of RefinedWeb achieves 59.8% zero-shot accuracy on the small aggregate, versus 57.9% for the same model trained on The Pile (Table 4). At 1B parameters, the gap is 56.2% vs. 53.4%. These are not enormous absolute differences, but they are directionally significant — the web-only model is better, not just competitive. And this is with all known high-quality sources (Wikipedia, arXiv, StackExchange, Reddit, GitHub) explicitly excluded from RefinedWeb (Table 14). The quality gap, to the extent it ever existed, has been eliminated entirely through post-extraction processing rather than source curation.

What this reframing implies for the field. If web data, properly processed, can match curated data in quality, then the perceived scarcity of high-quality training data — the data wall described by Villalobos et al. (2022) — is not a shortage of text but a processing bottleneck. CommonCrawl has been running for 12+ years and contains petabytes of data; the paper extracts five trillion English tokens from it while discarding ~88% of documents (Figure 2). As CommonCrawl continues to grow with the web, the pipeline can process new dumps with minimal modification, continuously producing fresh tokens. The data wall recedes: it is not that we will run out of text, but that we must invest in the infrastructure to extract the signal from the noise.

This also reorients the cost structure of dataset construction. Curating The Pile required negotiating access to dozens of individual sources, building specialized extraction pipelines for each, and managing the resulting licensing complexity — a labor-intensive, human-bottlenecked process. The MDR pipeline, by contrast, is fully automated. The one-time costs are: curating the URL blocklist (with manual inspection of ~100 false positive domains, per Appendix G.1.1), building the line-wise correction word lists (via manual inspection of data), and tuning the document-wise heuristics for English. After that, the pipeline runs at scale on CPU clusters without human intervention. The marginal cost of additional tokens is compute cycles — and unlike human curation effort, compute cycles scale with hardware investment.

Reconciling prior contradictions. The paper resolves a tension that has existed in the deduplication literature. Lee et al. (2022) demonstrated that deduplication improves language models; Hernandez et al. (2022) showed that repeated data is increasingly harmful at larger scales. Yet Biderman et al. (2023), training the Pythia model suite on deduplicated versus non-deduplicated versions of The Pile, found only a "limited impact" on zero-shot performance. The paper's results in Section 4.3 and Appendix F.2 provide a reconciliation: the benefit of deduplication is dataset-dependent. The Pile is only ~18% web data; the remaining ~82% is single-source corpora (books, papers, code) that naturally contain fewer duplicates. Applying MDR's deduplication to The Pile removes 45% of its content and yields a +1.1 percentage point improvement — modest but real. Applying it to OSCAR-22.01 removes 60.8% and yields +2.9 points. The Pythia finding is correct for curated data but does not generalize to web data, where duplication is both more prevalent and more harmful per instance (templated spam and scraped content farms are qualitatively worse than a few repeated book excerpts). The paper does not say Pythia was wrong; it says the conclusion was overgeneralized, and the boundary condition is the dataset composition.

Which research directions become more attractive. The paper makes a compelling case that pipeline engineering is underinvested relative to architecture design. The difference between C4 (which applies coarse deduplication and simple heuristics) and RefinedWeb (which applies 9,000-hash MinHash + exact substring deduplication + line-wise corrections + trafilatura extraction) is not a new algorithmic insight — it is thoroughness, scale, and the combination of complementary techniques. The paper aggregates best practices from C4, OSCAR, MassiveWeb, CCNet, and the deduplication literature, and demonstrates that the whole is greater than the sum of its parts. This suggests that the field should shift resources from inventing new data sources toward deepening the processing of existing sources.

Which research directions become less attractive. The paper's results weaken the case for manual curation of small, specialized corpora as a primary strategy for improving pretraining data quality. If a single automated pipeline can match a hand-assembled mixture of 22 curated sources (The Pile), then the marginal value of adding another curated source — with its attendant extraction pipeline, licensing complexity, and finite size — is diminished. This does not mean curation is worthless. Curated sources like books and academic papers may contain types of language and knowledge that are underrepresented on the open web, and models trained purely on RefinedWeb may lack some of these capabilities despite strong aggregate zero-shot scores. The paper does not test, for instance, whether RefinedWeb-trained models can generate coherent long-form prose or answer specialized scientific questions — tasks where book and paper corpora might provide unique value. But the burden of proof has shifted: the field should now ask not "is this curated source necessary?" but "does this curated source provide something that improved web processing cannot?"

The paper also weakens the case for ML-based quality filtering as a default pipeline component. The GPT-3/PaLM/MassiveWeb approach — train a classifier on known high-quality text, apply it to web data — is motivated by the assumption that rule-based heuristics are insufficient to separate good from bad content. RefinedWeb achieves competitive results without any ML-based content filtering, using only structural heuristics and aggressive deduplication. Given the documented bias risks of ML-based filtering (Dodge et al., 2021; Welbl et al., 2021), the paper's results suggest that the default posture should be neutral, rule-based filtering unless there is specific evidence that a learned classifier provides gains that justify the bias risk.

Follow-Up Research This Work Enables

1. What is the minimum deduplication depth needed to capture most of the gain? The paper's deduplication is deliberately aggressive — 9,000 MinHash hashes, 50-token exact substring matching, ~50% removal rates. This depth is computationally expensive, particularly the exact substring stage which requires loading the entire corpus into memory on instances with up to 2 TiB of RAM. A natural question is: can similar gains be achieved with less aggressive settings? A concrete experiment would be to train 1B models on RefinedWeb variants deduplicated at decreasing intensity — 5,000 vs. 2,000 vs. 500 vs. 50 MinHash hashes; exact substring at 100 vs. 50 vs. 25 tokens — and measure the zero-shot accuracy against removal rate, producing a "deduplication scaling law" analogous to the pretraining scaling laws. The paper's finding that The Pile's 10-hash MinHash underperforms their 9,000-hash setup (Appendix E.1) establishes that depth matters, but the shape of the curve — whether gains saturate at 1,000 hashes or continue improving to 9,000 — is unknown. This matters practically because exact substring deduplication at scale is the pipeline's primary cost bottleneck.

2. Can lightweight difficulty estimation replace the 2048-sample oracle for web data quality assessment? The paper's difficulty estimation protocol (Section 3.2: generate 2048 samples per question, compute pass@1 or PRM-predicted score) is prohibitively expensive for deployment. A natural extension, flagged by the paper itself (Section 8), is to train a lightweight model that predicts question difficulty from the question text alone, without generating any samples. Concretely: take the 12,000 MATH training questions, run the 2048-sample protocol to get difficulty labels (pass@1 quintiles), then fine-tune a small classifier (e.g., a 125M parameter Transformer) to predict the quintile from the question text. Evaluate on the 500-question test set: does the classifier-predicted difficulty produce compute-optimal scaling curves that match the oracle-bin curves in Figure 4? If the classifier can achieve accuracy comparable to the 2048-sample PRM-based method, the compute-optimal framework becomes deployable at negligible overhead. A negative result — the classifier fails to capture difficulty accurately, and the compute-optimal strategy degrades — would reveal that difficulty estimation requires model-specific generation and cannot be reduced to static text features.

3. Does combining search and revisions break through the performance ceiling that each method hits individually? The paper studies PRM tree-search (Section 5) and iterative revisions (Section 6) as independent mechanisms, and explicitly notes they were never combined (Section 8). This is the most obvious follow-up: use the revision model as the proposal distribution within beam search. At each step of the search tree, the model conditions on its previous attempts (including rejected branches) as context, potentially producing higher-quality candidate steps than a model that generates each solution independently. A concrete experiment: for a fixed budget of 256 generations, compare (a) PRM beam search with the base model, (b) sequential revisions with best-of-N weighted selection, and (c) PRM beam search with the revision model as the proposal distribution. The prediction from the paper's complementary-strengths finding is that (c) should outperform both (a) and (b) on medium-difficulty problems (quintiles 3–4), where beam search helps with global exploration and revisions help with local refinement. A null result — the combination provides no benefit, or even hurts due to distribution shift between the revision model's outputs and the PRM's training distribution (documented in Appendix J) — would suggest that the two axes are not simply additive and that verifier-proposal compatibility is a binding constraint.

4. Do the difficulty-dependent strategy patterns generalize to other models, benchmarks, and domains? All results in this paper are on MATH with PaLM 2-S*. The difficulty-dependent findings — beam search degrades on easy problems due to verifier over-optimization, sequential revisions dominate on easy problems, a balanced ratio is optimal on hard ones — may be specific to mathematical reasoning or to PaLM's output distribution. A essential stress-test: replicate the core experiments (Figure 3 right, Figure 7 right) on (a) the same MATH benchmark with a different model family (LLaMA-7B, Mistral-7B), (b) a different reasoning benchmark with PaLM 2-S* (e.g., GSM8K for math word problems, HumanEval for code generation, or ARC for science reasoning), and (c) a non-reasoning benchmark (e.g., MMLU fact-recall tasks). If the difficulty-dependent patterns replicate across models and math benchmarks, the framework is robust for reasoning. If they fail to replicate on fact-recall tasks, the framework's applicability is bounded to inference-heavy domains where the base model has the necessary knowledge but struggles with multi-step reasoning. If they fail to replicate on a different model family, the compute-optimal policies are model-specific and must be re-derived for each deployment, limiting the framework's portability.

5. Can deduplication reduce the degradation from training on multiple epochs, enabling data-constrained regimes to use smaller datasets with more passes? The paper's preliminary experiment in Appendix E.3 (Figure 7) suggests that RefinedWeb degrades less than RefinedWeb-Filtered when models are trained for multiple epochs on the same data — but the experiment is limited (1B models, 30GT, high variance across tasks). A systematic follow-up: take a fixed dataset (e.g., RefinedWeb at 350GT), train models of increasing size (1B, 3B, 7B) for 1, 2, 4, and 8 epochs, and measure both zero-shot accuracy and memorization metrics (e.g., training data extraction attacks, n-gram duplication rates). If deduplication substantially mitigates the multi-epoch degradation that Hernandez et al. (2022) documented, then the data constraint is softened further: practitioners with limited data can compensate by training for more epochs on deduplicated data, rather than needing to acquire ever-larger corpora. This connects the deduplication story to the data-constrained regime that Touvron et al. (2023) operated in with LLaMA (training on 1–1.4T tokens with multiple epochs because public data is limited). A negative result — deduplication helps only on the first epoch and provides no multi-epoch benefit — would mean deduplication's value is purely about maximizing information per unique token, not about enabling reuse.

6. What is the multilingual generalization of MDR, and do the filtering heuristics transfer across languages? The paper notes that CommonCrawl contains substantial non-English content (Figure 6, Appendix D) and that the MDR pipeline can in principle process all languages, but warns that "specific filters such as line-wise corrections need to typically be tuned for each individual language" and that "deduplication parameters" also benefit from per-language tuning. A concrete multilingual extension: apply MDR to the Russian, German, and Japanese subsets of CommonCrawl (the three largest non-English languages in Figure 6), using the same URL filtering and text extraction but with language-adapted versions of the document-wise heuristics and line-wise corrections. Train 1B multilingual models (or language-specific models) and evaluate on standard benchmarks in each language. If the pipeline transfers with minimal adaptation, RefinedWeb becomes a template for producing high-quality pretraining data across dozens of languages — addressing the acute data scarcity for low-resource languages where curated corpora are unavailable. If significant per-language tuning is needed, MDR's "scale first" philosophy is weakened — it becomes a family of language-specific pipelines rather than a single automated solution.

Practical Applications and Downstream Use Cases

1. Training large language models at 40–200B scale without reliance on licensed or curated data. The paper's headline use case is implicit in its design: RefinedWeb provides five trillion tokens of web-only data, sufficient for compute-optimal training of models up to approximately 250B parameters under the Chinchilla scaling laws (20 tokens per parameter). Unlike The Pile (which includes copyrighted books, licensed code, and forum conversations with complex terms of service), RefinedWeb is drawn entirely from publicly crawled CommonCrawl data under the ODC-By 1.0 license. This reduces the legal and logistical barriers to releasing openly available pretraining datasets at scale. The public release of 600GT — while a fraction of the full dataset — is already larger than The Pile (~340GT) and comparable to C4 (~360GT), making it a drop-in replacement for practitioners who want a high-quality web-only baseline. The fact that RefinedWeb already powers Falcon-40B (Almazrouei et al., 2023), a state-of-the-art open model, demonstrates that the dataset is production-ready and not merely a research artifact.

2. Retroactively improving existing pretraining datasets through deduplication. Section 4.3 demonstrates that applying MDR's deduplication stage to existing datasets consistently improves zero-shot performance: OSCAR-22.01 gains +2.9 percentage points, OSCAR-21.09 gains +0.6, The Pile gains +1.1, and C4 gains +0.2 (Table 5). The removal rates range from 7.59% (C4, already well-deduplicated) to 60.8% (OSCAR-22.01, distributed without deduplication). A practitioner with an existing pretraining dataset who wants a quick quality improvement can run MDR's deduplication stage independently — without needing to redo text extraction or filtering — and train on the deduplicated subset. The performance gains are modest (~1–3 points on small-scale aggregates) but come at zero additional data acquisition cost and with a potentially significant reduction in training compute (since the dataset is smaller after deduplication). For large-scale training runs where a 1-point improvement represents substantial engineering investment, retroactive deduplication is a low-cost, guaranteed-positive intervention. The paper's finding that the benefit is proportional to the amount of duplication removed provides a simple diagnostic: if MDR's deduplication removes a large fraction of a dataset, the improvement will be larger; if it removes very little, the dataset was already clean and the intervention is low-priority.

3. Continuous, automated data pipeline for models that need fresh pretraining data. Because MDR is fully automated — no human curation per data source, no ML model training, no license negotiation — it can be run on new CommonCrawl dumps as they are released (typically every 1–2 months). This enables a deployment architecture where a pretraining dataset is continuously updated with fresh web data, avoiding staleness and the need to periodically renegotiate access to curated sources. For organizations training models on a regular cadence (e.g., quarterly model refreshes), replacing manual curation with an automated pipeline reduces the engineering overhead of dataset construction from a per-release cost to a one-time pipeline development cost. The paper's processing infrastructure (100–250 AWS c5.18xlarge instances, 10,000–20,000 vCPUs) provides a concrete reference architecture for the scale of compute needed. The primary bottleneck — exact substring deduplication requiring 2 TiB of memory per instance — is also clearly identified, allowing practitioners to plan hardware allocation.

4. A baseline for data quality research that isolates web processing from source curation. The paper's deliberate exclusion of known high-quality sources (Table 14) makes RefinedWeb uniquely valuable as a controlled baseline for studying the effect of data composition on model behavior. Researchers investigating questions like "how much does adding Wikipedia improve factual accuracy?" or "does code pretraining improve reasoning?" can start from RefinedWeb as a pure-web baseline and add curated sources in controlled proportions, measuring the marginal benefit of each addition. Prior to RefinedWeb, the closest public baseline was C4, which contains Wikipedia (often a substantial fraction of its content) and thus confounds "web data" with "accidental inclusion of curated data." The paper's explicit source exclusion solves this confound, enabling cleaner experimental designs for data ablation studies.