Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

🎯 Pitch

The best AI models are built on secret data recipes, but Dolma smashes this barrier by releasing a fully documented 3-trillion-token training corpus. Surprisingly, their 1.2B model nearly matches the perplexity of one trained on The Pile despite using mostly web data, proving open, massive-scale data curation is viable—though it struggles with math reasoning.

1. Executive Summary

This paper introduces and releases Dolma, an open English-language pretraining corpus of three trillion tokens sourced from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials, designed to facilitate transparent scientific research on language model pretraining where commercial and even many open models obscure their training data. The authors document their entire curation pipeline—spanning language filtering (FastText-based English identification), quality filtering (combining Gopher heuristics with a C4 punctuation rule), content filtering (custom FastText toxicity classifiers and PII redaction via regular expressions), and multi-stage deduplication (URL, document, and paragraph-level Bloom filters)—and validate each design decision through controlled data ablations using 1.2B parameter decoder-only models trained to 150B tokens on an 8-task evaluation suite. The resulting corpus enables training OLMo-1B, a 1.2B parameter model that outperforms the comparably-sized TinyLlama on 4 of 8 downstream tasks (averaging 60.3% accuracy vs. 59.4%) while nearly matching the perplexity curve of the diverse-source Pile despite containing a much larger fraction of web data, establishing that multi-source corpora can scale to trillions of tokens without sacrificing domain coverage—though the hardest abstraction (whether these patterns generalize beyond the MATH-analogous benchmarking regime used here) remains untested across non-English languages and larger model scales.

2. Context and Motivation

The Core Problem: Pretraining Data Opacity Paralyzes Scientific Research

The fundamental problem this paper addresses is that the pretraining data used to train state-of-the-art language models is increasingly secret, even for so-called "open" models. This creates a bottleneck for scientific progress: researchers cannot study how training data composition influences model behavior because they don't know what's in that data, and they cannot reproduce or audit published results because the data isn't available.

The authors frame this not as a minor inconvenience but as a systemic barrier to scientific inquiry (Section 1). When a model generates biased outputs, hallucinates facts, or memorizes copyrighted text, the natural scientific question is: "What in the training data caused this?" Without the data, that question becomes unanswerable—or answerable only through indirect forensic analysis (the "data archaeology" approach the authors cite for GPT-4 in Appendix C.2).

The paper identifies several specific downstream impacts:

Task performance attribution. Kandpal et al. (2023) showed that LLMs perform worse on long-tail knowledge, and Razeghi et al. (2022) demonstrated that pretraining data frequency directly correlates with downstream task performance. Without knowing what was in the training data, it's impossible to understand why a model succeeds or fails on specific tasks—you can only measure the outcomes, not trace their causes.
Bias and fairness auditing. Feng et al. (2023) and Navigli et al. (2023) established that social biases in models trace back to their pretraining data. If you can't inspect the data, you can't systematically study bias provenance, and models that appear unbiased on benchmark tests may harbor undetected biases that only surface in deployment.
Memorization and copyright. Carlini et al. (2022) showed that LLMs can memorize and regurgitate training data, including copyrighted material and personal information. Without access to the training corpus, measuring memorization rates or implementing meaningful data removal requests becomes impossible.
Benchmark contamination. Magar and Schwartz (2022) demonstrated that test set examples leaking into pretraining data inflate performance metrics. When training data is hidden, contamination can only be detected through post-hoc analysis—after the model is already trained and published.

The Gap Between Model Release and Data Release

The paper carefully documents (Section 2, Appendix C) a growing asymmetry in the language modeling community: models are increasingly released publicly, but their training data is not.

The authors catalog the spectrum of opacity:

Fully closed models (GPT-4, PaLM 2, Claude) disclose essentially nothing—not corpus size, not data provenance, not filtering methods. The GPT-4 technical report (OpenAI, 2023) doesn't even report the number of pretraining tokens. The authors note that for PaLM 2 (Anil et al., 2023), information is "unreported other than it's larger than what was used to train PaLM."
"Open-weight" models without data (Llama 2, Mistral, Yi, Qwen) release model weights but not their training data, and often provide only high-level descriptions of data composition. For Llama 2 (Touvron et al., 2023b), the authors document critical missing information: data provenance is listed as "N/A aside from they avoided using Meta user data," quality filtering details are unreported, and deduplication methodology is not described. This means even researchers with the model weights cannot determine what the model was trained on.
Models with partial data release (Falcon) release weights and some training data, but the data is often incompletely documented or only partially representative of what was actually used for training.
Fully open models with data (T5 on C4, GPT-J on Pile, GPT-NeoX on Pile, Pythia on Pile, INCITE on RedPajama v1) are rare, and the paper argues they have limitations in scale, diversity, or contemporary relevance.

The historical trend concerns the authors. They note that prior to the commercial wave of LLMs, major models like BLOOM (Leong et al., 2022) on ROOTS (Piktus et al., 2023) and the Pythia suite (Biderman et al., 2023) on Pile (Gao et al., 2020) maintained data transparency. But the most powerful open models of 2023–2024—Llama 2 (2T tokens, unreleased data), Mistral (unknown tokens, unreleased data)—represent a retreat from that norm. The paper positions Dolma as a counterweight to this trend.

Why Existing Open Corpora Are Insufficient

The authors do not dismiss prior work—they explicitly acknowledge C4, Pile, ROOTS, RedPajama, and others as important contributions. Instead, they identify specific limitations in each existing corpus that prevent it from serving the research community's current needs (Section 2):

1. Scale limitations. C4 (Raffel et al., 2020) provides 175B tokens, and Pile (Gao et al., 2020) provides 387B tokens. While these datasets have "demonstrated use in training language models" and remain high-quality benchmarks, the authors argue they are "unfortunately limited in scale" for contemporary research needs. The Chinchilla scaling laws (Hoffmann et al., 2022) established that compute-optimal training requires maintaining a fixed ratio between model parameters and training tokens, and the Llama 2 results showed that performance continues to improve when training on 2T tokens. To study scaling behavior at modern model sizes, researchers need corpora at the trillion-token scale.

2. Language scope mismatches. ROOTS (Piktus et al., 2023) is large (approximately 400B tokens) but is designed as a multilingual corpus. Its English-only portion constitutes only about 30% of the total—roughly 120B tokens. For researchers studying English-only language modeling (which the authors note is a common focus, citing the need to "better leverage known curation practices and maximize generalizability of scientific work"), this is insufficient. The authors acknowledge the limitation of their English-only choice and frame it as a deliberate scope decision rather than a value judgment about multilingual research.

3. Source diversity limitations. Falcon (Almazrouei et al., 2023; 580B tokens) and RedPajama v2 (Together Computer, 2023b; 30T tokens) meet the scale criterion but are derived exclusively from Common Crawl web pages. The authors argue this lacks the source diversity "commonly targeted when curating data for the largest language models," which typically include scientific papers, code, books, and encyclopedic content alongside web text. Training on web-only data may bias models toward certain content distributions and away from the structured knowledge found in curated reference materials.

4. Curation burden on users. RedPajama v2 distributes "content output by CCNet mostly as-is," meaning the user must decide on and implement their own filtering pipeline before training. This shifts the curation burden to individual researchers, who may lack the resources to properly validate their filtering decisions through ablation experiments. The paper positions Dolma's extensively filtered and validated output as removing this barrier.

5. Data quality issues. The authors cite recent work by Soboleva et al. (2023) and Elazar et al. (2023) identifying "data quality issues suggesting significant additional cleanup of RedPajama v1 is recommended before costly language model training." This is a pointed criticism: the most directly comparable previous corpus (RedPajama v1, a reproduction of LLaMA's training data at roughly 1.2T tokens) contains quality problems that make it unsuitable for direct use without further processing.

6. Missing data sources. RedPajama v1 targeted a specific reproduction of LLaMA's training recipe. Dolma has "a broader reproduction target which required diving into data sources that RedPajama v1 did not pursue, including larger collections of scientific papers and social media forums like Reddit." The inclusion of Reddit is particularly notable—it represents social conversational data that few prior open corpora have included at scale, yet both WebText (Radford et al., 2019) and the OPT training corpus (Zhang, 2022) used Reddit-derived data, suggesting it's a standard (but undocumented) component of many closed recipes.

The Documentation Gap in Data Curation Practices

Beyond data access, the paper identifies a related problem: even when some information about data curation is disclosed, it's rarely sufficient for reproduction (Appendix C). The authors meticulously document what is and isn't known about major models:

For PaLM 2, for example, they note that while the paper mentions "web documents, books, code, mathematics, and conversational data" as data sources, it provides no information on "PII filtering (beyond stating it was performed), toxicity filtering (beyond mentioning Perspective API without thresholds), quality filtering (mentioned but undescribed), or deduplication (mentioned but undescribed)." These omissions make reproduction impossible even if one had access to the same raw sources.

For Llama 2, the situation is similar: "data provenance is not disclosed beyond stating they avoided Meta user data," and while they report post-hoc language distribution analysis using FastText at a 0.5 threshold, actual curation decisions around language ID, quality filtering, and deduplication are left unspecified.

The paper's Appendix C.2 on GPT-4 is particularly striking: nearly every category of curation detail is listed as "N/A" or "unreported." The authors note that what little is known comes from external "data archaeology"—for instance, Chang et al. (2023) demonstrating through memorization probes that GPT-4 can generate sequences from copyrighted books.

This documentation gap means that researchers cannot build on prior data curation work incrementally. Each new corpus effort starts largely from scratch, making design decisions based on incomplete public information rather than accumulated scientific knowledge.

How Dolma Positions Itself

The paper explicitly defines four design principles that position Dolma in the landscape (Section 3):

1. Consistency with prior recipes. Rather than proposing a novel data curation approach, Dolma aims to reproduce and combine known practices: "By matching data sources and methods used to create other language modeling corpora, to the extent they are known, we enable the broader research community to use our artifacts to study (and scrutinize) language models being developed today, even those developed behind closed doors." This is a deliberate choice that trades novelty for generalizability—if Dolma replicates common curation patterns, findings about it are more likely to transfer to commercial systems.

2. Evidence-backed decisions. Where prior recipes conflict or are silent, the authors run controlled ablation experiments. This is the paper's key methodological contribution: validating curation choices through systematic experiments rather than relying on heuristics or precedent. The paper describes this approach: "When in doubt, make evidence-backed decisions. In such cases, we prioritize decisions that maximize performance of language models trained on Dolma over a diverse suite of tasks and datasets."

3. Large scale. The 2–3T token target is explicitly linked to the Chinchilla scaling framework: "We aim for a sufficiently large corpus—2–3T tokens—to allow further study of the relationship between model and dataset size." This scale is roughly comparable to the training sets used by Llama 2 (2T tokens) and LLaMA (1.4T tokens), making Dolma suitable for training models that can be meaningfully compared to these systems.

4. Openness as a constraint. The paper acknowledges that openness sometimes forces deviations from known recipes: "This requirement means we occasionally must deviate from known recipes due to additional practical, legal or ethical considerations that arise when pursuing dataset research in the open." The most significant deviation is avoiding sources with active copyright litigation—specifically Books3 (Gao et al., 2020), which was used in LLaMA and RedPajama v1 but is the subject of ongoing lawsuits (Knibbs, 2023). The authors substitute Project Gutenberg (public domain books) despite it providing far fewer tokens (6B vs. the potential for much more from Books3), making a deliberate tradeoff between representativeness and legal safety.

This framing is crucial: Dolma is not claiming to be the best possible corpus but rather the most openly documented one at this scale. The paper implicitly argues that transparency has scientific value independent of any specific performance metric. By releasing both the data and the tools to process it, the authors create infrastructure that others can use, modify, and improve—a platform for research rather than a fixed artifact.

The Specific Gap This Fills

Synthesizing the above: prior to Dolma, the research community had access to small diverse corpora (Pile, C4) that couldn't study scaling effects, or large undiverse corpora (RedPajama v2, RefinedWeb) that didn't represent real LLM training mixtures, or a somewhat diverse large corpus (RedPajama v1) with documented quality issues. There was no open corpus at the trillion-token scale that combined diverse, curated sources with documented, validated curation decisions. Dolma fills that gap, explicitly targeting the regime where scaling behavior becomes interesting while maintaining the source diversity that makes training data ecologically valid for studying modern LLMs.

3. Technical Approach

3.1 Reader Orientation

Dolma is not a single algorithm or model but a curated dataset—a three-trillion-token collection of English text assembled from six distinct sources and processed through a common pipeline of filtering, cleaning, and deduplication operations. The system being built is the curation pipeline itself: a reproducible sequence of transformations that takes raw heterogeneous data (HTML pages, code repositories, social media posts, academic PDFs, books, and wiki dumps) and produces a unified corpus suitable for training large language models. The core idea is that by making every curation decision explicit and validating each one through controlled experiments—training 1.2B parameter models on dataset variants and measuring downstream task performance—the authors can produce a corpus whose properties are understood and documented, enabling other researchers to use it as a substrate for studying how pretraining data composition affects model behavior.

3.2 Big-Picture Architecture (Diagram in Words)

The Dolma system has five major components connected in a sequential pipeline, with a meta-level experimental apparatus wrapped around it:

Data Acquisition (per-source). Raw data is downloaded from each of six sources using source-specific protocols. This component handles the heterogeneity of formats: HTML for Common Crawl and C4, code files for GitHub, JSON for Reddit submissions/comments, structured academic paper objects for Semantic Scholar, plain text for Project Gutenberg, and XML dumps for Wikipedia/Wikibooks. Its output is raw text with associated metadata (URLs, thread IDs, repository names) but no cross-source normalization.
Language Filtering (shared across most sources). A FastText language identification model scores every document for English-ness. This component operates at the document or paragraph level depending on source, and produces a binary keep/discard decision based on a threshold (typically ≥ 0.5 English score). It eliminates non-English content early to reduce downstream processing costs.
Quality and Content Filtering (source-specific heuristics + shared PII/toxicity). This is a collection of rule-based and model-based filters applied in stages: Gopher heuristics (Rae et al., 2021) remove boilerplate, short documents, and degenerate text; a C4 punctuation rule removes paragraphs lacking sentence-final punctuation; custom FastText toxicity classifiers flag and remove hate speech and NSFW content at the sentence level; and regular expression-based PII detectors identify and either mask or remove email addresses, phone numbers, and IP addresses. The filtering is mostly subtractive—text is removed or replaced with placeholder tokens.
Deduplication (multi-stage across all sources). Three types of duplicate removal operate in sequence: URL-based deduplication (exact match on source URL) removes re-crawls of the same page; document-level exact-match deduplication (SHA hash of full text) removes identical content from different URLs; and paragraph-level deduplication (Bloom filter over paragraph hashes) removes common boilerplate that appears across many documents. Each stage uses the Bloom filter implementation from the Dolma Toolkit.
Mixing (post-hoc, not prescribed by Dolma). The final corpus does not mandate a specific mixture of sources. Instead, the authors provide the cleaned subsets and demonstrate several mixing strategies (Section M of Appendix), leaving the choice to downstream model trainers. The mixing component takes per-source shards and combines them according to a user-specified sampling ratio, applying upsampling or downsampling as needed.

Wrapping these components is the data ablation apparatus: a separate experimental pipeline that trains 1.2B parameter models on controlled variants of the dataset (e.g., with versus without paragraph deduplication) and evaluates them on an 8-task downstream suite plus a stratified perplexity benchmark (Paloma). This apparatus is not part of Dolma itself but is essential to its construction, since every filtering decision is motivated by ablation results.

Information flows linearly: raw source data → language filtering → quality/content/PII filtering → deduplication → cleaned subset. The six source subsets are processed independently through the same stages (with source-specific filter configurations) and then optionally mixed by the downstream user.

3.3 Roadmap for the Deep Dive

First, the Dolma Toolkit infrastructure (Section 4.1 in the paper), because it provides the shared computational substrate—the filtering and mixing primitives—that all source-specific pipelines build upon. Understanding the toolkit's design (how filters are configured, how Bloom filter deduplication works, what "tagging and removal" means operationally) is prerequisite to understanding any individual pipeline.
Second, the data ablation methodology (Section 4.2), because it is the paper's key scientific mechanism. Every curation decision in Sections 5–8 is justified by ablation results, so understanding the experimental setup (model architecture, training regime, evaluation suite) is necessary to interpret those decisions.
Third, the web pipeline (Section 5), the largest and most complex curation operation, which processes Common Crawl from 175 TB of raw HTML down to 2.28T cleaned tokens. This section introduces the core filtering concepts (CCNet pre-processing, Gopher quality rules, toxicity classification, PII masking) that are reused in simplified form for other sources.
Fourth, the code pipeline (Section 6), which applies adapted versions of the web filters to GitHub repositories, plus code-specific heuristics from RedPajama and StarCoder.
Fifth, the Reddit social media pipeline (Section 7), which adds a novel investigation of thread linearization formats and uses voting scores as a quality signal.
Sixth, the remaining sources (Section 8)—C4 reprocessed, Semantic Scholar academic papers, Project Gutenberg books, and Wikipedia/Wikibooks—which are handled with lighter-touch processing since they come pre-curated or from relatively clean sources.
Finally, the mixing and tokenization analyses (Sections 9 and Appendix F, M), which examine how source composition affects model behavior and reveal domain-specific tokenization inefficiencies (particularly the code subset's high fertility).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a dataset curation paper whose core idea is that transparent, evidence-backed data pipelines—where every filtering decision is validated through controlled model training experiments—can produce a three-trillion-token corpus that matches or exceeds the quality of closed-source alternatives while enabling scientific research on data composition effects.

The Dolma Toolkit: Unifying Filtering and Mixing

The Dolma Toolkit is a high-performance, open-source Rust-based framework designed to process hundreds of terabytes of text through configurable curation pipelines. It provides two fundamental operations that serve as building blocks for all source-specific pipelines.

Filtering operation. The toolkit unifies all data transformations into a single configurable abstraction. A filter is specified by three components: a text unit (what chunk of text to score—document, paragraph, sentence, or line), a scoring method (how to produce a numerical score for that chunk—a linear classifier, a language model perplexity, a regular expression match, or a heuristic rule), and a removal policy (what to do with chunks that meet the removal criterion—delete the chunk, replace it with a placeholder string, or, if the chunk is a sub-document unit, delete the parent document if enough chunks are flagged).

The key design insight is that this abstraction covers all the filtering types used in the pipeline:

Language identification uses a FastText classifier as the scoring method applied to the document text unit, with a threshold-based removal policy (score < 0.5 → remove document).
Quality heuristics use rule-based scoring methods (does a line end in punctuation? does a document have too few words?) applied to document or paragraph text units.
Toxicity filtering uses a custom FastText classifier as the scoring method applied to the sentence text unit.
PII detection uses regular expression matching as the scoring method applied to the sentence text unit, with a different removal policy depending on PII density (mask if ≤ 5 spans, remove document if ≥ 6 spans).

The toolkit parallelizes filtering by distributing documents across CPU cores and applying multiple filters in a single pass. The paper reports that in internal tests replicating the C4 filtering recipe, the toolkit processed data at 122 CPU hours per terabyte. For context, processing the full 200 TB of raw Dolma input files on a c6a.48xlarge EC2 instance with 192 vCPUs would take approximately 5 days. This throughput is essential because the filtering must operate at a scale where naive single-threaded processing would be completely infeasible (processing 200 TB at even 10 MB/s single-threaded would take over 230 days).

Mixing operation. The toolkit's mixing module handles cross-file operations: combining multiple shards into a single output, up-sampling or down-sampling specific sources by reading file paths multiple times, and performing deduplication across file boundaries. The core data structure is a Bloom filter—a probabilistic set-membership data structure that uses multiple hash functions to test whether an item has been seen before, with a configurable false positive rate but zero false negatives (if the filter says "not seen," the item is definitely novel; if it says "seen," there is a small probability it's a false positive).

For deduplication, the Bloom filter is seeded with hashes of previously seen documents (or paragraphs, or URLs). When a new candidate arrives, its hash is checked against the filter; if the filter reports "seen," the candidate is treated as a duplicate and removed. This enables linear-time deduplication—each candidate requires only a constant number of hash lookups, regardless of how many items have been seen before—which is critical for operating at the trillion-token scale.

The Bloom filter is repurposed for test-set decontamination: first, test examples are seeded into the filter (the filter "remembers" the test hashes without storing them); then, when processing pretraining data, any document whose hash matches the filter is flagged as contaminated and removed. This approach avoids the quadratic cost of comparing every training document against every test example explicitly.

Design choices. The authors chose to implement the toolkit in Rust rather than Python for performance reasons—Rust's zero-cost abstractions and lack of garbage collection enable the sustained high throughput needed for terabyte-scale processing. The unified filtering abstraction was chosen over ad-hoc per-filter implementations to make the pipeline configurable and reproducible: a YAML configuration file specifies the entire filtering pipeline, so users can understand exactly what was done and modify it for their own purposes. This contrasts with many prior data curation efforts where filtering code is embedded in research scripts that are difficult to reuse or audit.

Data Ablation Methodology: Making Evidence-Backed Decisions

The paper's most distinctive methodological contribution is its approach to making curation decisions: rather than relying on precedent or intuition, the authors run controlled data ablation experiments—training identical models on dataset variants that differ in only one curation choice and measuring the impact on downstream performance.

The experimental setup is standardized across all ablations (Appendix D.1). Every ablation uses a 1.2 billion parameter decoder-only transformer model from the OLMo family (Groeneveld et al., 2024). The architecture is fixed: 16 layers, 16 attention heads, 2048-dimensional hidden states, ALiBi positional embeddings (Ofir Press et al., 2021), SwiGLU activation functions (Shazeer, 2020), mixed precision training, and a context size of 2048 tokens. The tokenizer is EleutherAI's GPT NeoX tokenizer (Black et al., 2022), which has a vocabulary of 50,280 tokens.

Training uses the LionW optimizer (Chen et al., 2023a) with a peak learning rate of $1 \times 10^{-4}$ , 2000 warmup steps, cosine learning rate decay, and weight decay of $1 \times 10^{-2}$ . The batch size is 1024 sequences of 2048 tokens each (approximately 2.1 million tokens per batch). Ablation runs are terminated at 150 billion tokens—roughly 71,000 steps—which is far short of full training but sufficient to observe relative differences between dataset variants. The hardware configuration is 64 AMD Instinct MI250X accelerators, each containing two logical compute units for a total of 128 compute units across 16 nodes, with a micro-batch size of 8 per logical unit.

The choice of 1.2B parameters is deliberate and aligned with prior work: Le Scao et al. (2022) used similar model sizes for ablation experiments, and the authors argue this size is "in line with similar model sizes that have been used for ablations in prior work." Training larger models for each ablation would be prohibitively expensive given the number of decisions being evaluated—the paper presents ablation figures for at least 8 distinct experimental comparisons, each requiring a full 150B-token training run.

The evaluation suite (Appendix D.3) is carefully designed to capture diverse capabilities. The authors select 8 tasks that have been "used in prior language model pretraining evaluation" and "capture a range of language model knowledge and capabilities":

AI2 Reasoning Challenge (ARC-Easy) (Clark et al., 2018): Grade-school science multiple-choice questions. Tests factual knowledge and simple reasoning.
BoolQ (Clark et al., 2019): Yes/no reading comprehension questions with supporting passages. Tests natural language understanding and factual reasoning.
HellaSwag (Zellers et al., 2019): Multiple-choice sentence completion requiring commonsense reasoning about physical and social situations. Tests grounded understanding of everyday scenarios. The paper uses HellaSwag as its primary illustrative example throughout the ablation figures.
OpenBookQA (Mihaylov et al., 2018): Science questions requiring both factual recall and multi-step reasoning, modeled on open-book exams. Tests scientific knowledge.
PIQA (Bisk et al., 2019): Physical commonsense reasoning about everyday tasks. Tests understanding of physical causality.
SciQ (Welbl et al., 2017): Crowdsourced science questions spanning physics, chemistry, and biology. Tests domain-specific factual knowledge.
WinoGrande (Sakaguchi et al., 2019): Pronoun resolution problems requiring commonsense reasoning, adversarially constructed to be harder than the original Winograd Schema Challenge.

Additionally, the authors track perplexity on the Paloma benchmark (Magnusson et al., 2023), a stratified collection of hundreds of fine-grained textual sources designed to measure how well a language model fits diverse domains. The Paloma evaluation is central to the domain-fit analysis in Section 9.2.

Evaluation uses zero-shot in-context prompting with every task cast as ranked text classification, following the truncation approach from Min et al. (2022) and prompts from PromptSource (Bach et al., 2022). The authors use an in-house evaluation harness "similar to the Eleuther harness" (Gao et al., 2023).

An important constraint: avoiding test set contamination. The paper explicitly notes that its evaluation suite was selected to avoid tasks whose test sets appear in Dolma. Appendix L documents a contamination analysis using WIMBD (Elazar et al., 2023), finding that several popular benchmarks are heavily contaminated—the Winograd Schema Challenge, SICK, GLUE AX, SemEval 2014 Task 1, COPA, and SuperGLUE AXb all show 100% contamination, primarily from the code subset (public GitHub repositories often contain copies of these datasets). The contaminated datasets are excluded from evaluation, which is why the suite does not include some commonly used benchmarks.

Ablation comparison protocol. For each curation decision, the authors train a model on the "intervention" dataset (e.g., with paragraph deduplication) and compare it to a "baseline" dataset (e.g., without paragraph deduplication), measuring performance on all 8 tasks plus perplexity across the Paloma suite at regular intervals during training (the x-axes of the ablation figures are labeled in billions of tokens seen). The comparison is visualized as learning curves, with the intervention and baseline plotted as separate lines; a consistent gap favoring the intervention across training steps is interpreted as evidence that the curation decision improves data quality.

Perplexity evaluation suite (Appendix D.2). Beyond the downstream tasks, the authors track perplexity on a diverse set of domains from Paloma, including C4, mC4 (English), Pile, WikiText-103, Penn Tree Bank, M2D2 (S2ORC academic papers and Wikipedia articles), C4 100 domains, Gab, ICE (International Corpus of English, with subsets for multiple English varieties), Twitter AAE (African American English), Manosphere forums, and 4chan. They also curate held-out sets from Dolma itself and other open corpora (RedPajama v1, Falcon RefinedWeb, Dolma 100 subreddits, Dolma 100 programming languages). This broad coverage enables measuring not just overall model quality but domain-specific fit—how well a model trained on a particular dataset variant handles academic prose, social media slang, or code tokens.

Web Pipeline (Section 5): From 175 TB of Common Crawl to 2.28T Tokens

The web pipeline is the largest and most complex curation operation in Dolma, processing 25 Common Crawl snapshots from May 2020 to June 2023. The pipeline transforms approximately 175 TB of raw HTML (before any processing) into 2.28 trillion cleaned tokens, a roughly 60× reduction in data volume. The processing occurs in five sequential stages.

Stage 1: Acquisition and CCNet pre-processing. The authors acquire Common Crawl snapshots from the official S3 bucket (s3://commoncrawl/), downloading only enough shards to meet their 2–3T token target after the expected ~10× reduction from all downstream processing. The CCNet pipeline (Wenzek et al., 2020) performs three operations:

HTML linearization: Each web page is converted from HTML to plain text (WET file generation), stripping all markup, scripts, and styling. This is lossy—page layout information and structural semantics are discarded.
Language identification: A FastText language ID model (Joulin et al., 2016a,b) scores each document for English, and documents with a score below 0.5 are removed. This eliminates 61.7% of the data by byte size. The threshold of 0.5 is relatively permissive—a document could be 49% non-English and still be retained. The authors validate this choice through an external audit (Appendix G) on the International Corpus of English, finding that even at a 0.90 threshold, the majority of English documents from nine countries are correctly identified, suggesting the 0.5 threshold is conservative and unlikely to filter out legitimate English content.
Paragraph-level near-duplicate removal: CCNet groups shards within each snapshot into small sets and removes exact-duplicate paragraphs within each set. This primarily eliminates navigation headers, footers, and other boilerplate that appears identically across many pages from the same site. The authors modify the original CCNet approach slightly: rather than grouping by 2% of each snapshot's size, they use fixed-size sets of 20 GB each for more predictable resource usage. This step removes approximately 70% of paragraphs.

Overall, CCNet filters out 84.2% of the content in Common Crawl, reducing the data from 175.1 TB to 27.7 TB.

Stage 2: URL-level deduplication. Before any content-based filtering, documents are deduplicated by their source URL using exact string matching. This removes 53.2% of documents—re-crawls of the same page across different snapshots, or the same content accessed via different URL variants that CCNet normalizes. URL deduplication is computationally cheap (no content comparison needed) and eliminates a large fraction of redundant data before expensive quality and toxicity classifiers are applied.

Stage 3: Quality filtering—Gopher heuristics and C4 punctuation rule. The paper conducts an ablation study comparing four approaches to quality filtering on web data (Figure 1, Appendix O.4):

Gopher All: All quality heuristics from Rae et al. (2021). These are rule-based filters applied at the document level, each removing documents that match specific degenerate-text patterns:
- Fraction of characters in most common n-gram above a threshold: For bigrams, 0.20; for trigrams, 0.18; for 4-grams, 0.16. Documents dominated by a single repeated phrase are removed.
- Fraction of characters in duplicate n-grams above a threshold: For 5-grams, 0.15; for 6-grams, 0.14; for 7-grams, 0.13; for 8-grams, 0.12; for 9-grams, 0.11; for 10-grams, 0.10. Documents with excessive repetition are removed.
- Contains fewer than 50 or more than 100,000 words: Documents that are too short (likely navigation pages) or too long (likely data dumps) are removed.
- Median word length is less than 3 or greater than 10: Documents with words that are consistently too short (likely encoding artifacts) or too long (likely non-text) are removed.
- Symbol-to-word ratio greater than 0.10: Documents with excessive non-alphabetic content are removed.
- Fraction of words with alpha character less than 0.80: Documents where fewer than 80% of words contain at least one alphabetical character are removed.
- Contains fewer than 2 of a set of required common English words ("the", "be", "to", "of", "and", "that", "have", "with"): Documents lacking basic English function words are removed.
- Fraction of lines starting with bullet point greater than 0.90: Documents consisting almost entirely of lists are removed.
- Fraction of lines ending with ellipsis greater than 0.30: Documents with excessive trailing ellipses (suggesting truncated content) are removed.
- Fraction of duplicated lines greater than 0.30, or fraction of characters in duplicated lines greater than 0.30: Documents with excessive line-level repetition are removed.
C4 All: The full set of C4 quality heuristics from Raffel et al. (2020). These include additional rules beyond what Gopher covers, designed for the specific characteristics of the April 2019 Common Crawl snapshot used to create C4.
C4 NoPunc: A single C4 heuristic: removing paragraphs that do not end in a sentence-final punctuation character (".", "?", "!", or '"'). This is a lightweight filter that targets boilerplate text, navigation elements, and malformed HTML conversions—content that typically lacks proper sentence structure.
C4 NoPunc + Gopher All: The combination of both approaches.

The ablation results (Figure 1, shown for HellaSwag; full results across all 8 tasks and Paloma domains in Appendix O.4) reveal a non-trivial pattern: C4 NoPunc alone outperforms both C4 All and Gopher All individually, and the combination of Gopher All + C4 NoPunc achieves the best performance. This is significant because it suggests the filters are complementary—C4 NoPunc removes structural boilerplate that Gopher's lexical rules might miss, while Gopher catches degenerate text patterns that might still end in punctuation.

The paper reports that Gopher All tagged 15.23% of UTF-8 characters for removal, while C4 NoPunc tagged 22.73% of characters for removal. These fractions are measured on the CCNet-preprocessed data, so they represent additional removal beyond the 84.2% already removed by CCNet.

A surprising finding: model-based and heuristic filters are orthogonal. CCNet also provides quality scores using KenLM (Heafield, 2011) perplexity, which measures how Wikipedia-like a document is. Documents are grouped into three buckets: high quality (21.9% of remaining data), medium quality (28.5%), and low quality (49.6%). The authors report that their heuristic filtering rules "did not affect these proportions"—in other words, the heuristics remove roughly equal fractions from each perplexity bucket. This suggests that model-based perplexity scoring captures signals (such as coherence, grammaticality, or topical content) that are orthogonal to the surface-level patterns detected by Gopher and C4 rules. A document can have high-KenLM-score Wikipedia-like prose but still be tagged by C4 NoPunc (if it happens to lack sentence-final punctuation due to formatting), or vice versa—a document with low KenLM score might still pass all heuristic checks if it's grammatically well-formed but on a topic far from Wikipedia.

Stage 3 (continued): Repetitive token removal. An additional quality filter removes any document containing "a token or sequence of tokens repeating over 100 times." This targets degenerate cases like a single word repeated hundreds of times or a short phrase looped. The authors tokenize using allenai/gpt-neox-olmo-dolma-v1_5 to identify tokens (using the GPT NeoX tokenizer). This lightweight filter affects only 0.003% of remaining characters but catches edge cases that might otherwise degrade language model training.

Stage 4: Content filtering—toxicity and PII. The authors train custom FastText classifiers rather than relying on existing tools, motivated by the need for high-throughput classification at web scale.

Toxicity classifier training data. The Jigsaw Toxic Comments dataset (cjadams et al., 2017) provides forum comments tagged with multi-label categories: "toxic," "severe toxic," "threat," "insult," "obscene," and "identity hate," alongside unlabeled comments. The authors construct two binary classifiers:

Hate detector: Positive examples are all comments tagged with any of the categories except "obscene-only" comments, which are grouped with unlabeled comments as negatives. This creates a broad hate-speech detector.
NSFW detector: Positive examples are all comments tagged "obscene"; negatives are everything else. The paper notes this "only filters toxic content that mentions sexual or obscene topics, not sexual content in general."

Both classifiers are bigram FastText models, chosen because FastText supports subword n-gram features that help handle misspellings and morphological variations common in web text, and because FastText classification is significantly faster than neural alternatives at the throughput needed for processing hundreds of terabytes.

Toxicity filtering procedure. The classifiers are applied at the sentence level—each sentence in a document is independently scored, and sentences above the threshold are removed. The paper evaluates two thresholds:

Low threshold: $\tau = 0.0004$ . This removes even slightly toxic content, eliminating 29.1–34.9% of sentences (the range depends on which classifier—hate tends to remove more).
High threshold: $\tau = 0.4$ . This removes only clearly toxic content, eliminating 5.5–7.3% of sentences.

The ablation results (Figure 2, shown for HellaSwag; full results across all tasks in Appendix O.5) are counterintuitive: the Low threshold (removing MORE content) yields higher downstream task performance than the High threshold, despite reducing the total available tokens. This suggests that even mildly toxic content has a negative effect on language model training, perhaps because toxic text patterns conflict with the kind of neutral, factual language that downstream evaluation tasks reward, or because toxic content is correlated with other undesirable text properties (incoherence, extreme viewpoints, low factual accuracy) that the toxicity classifier picks up as a side effect.

Despite the performance advantage, the authors choose the High threshold for the released Dolma corpus. The reason is pragmatic: "the cause of this was surprising: Our quality, content, and deduplication filters overlap very little in which texts they remove (Figure 9), resulting in a compounded filtering effect when combining them." In other words, each filter removes a different subset of documents, and the cumulative removal across all filters was large enough that the Low toxicity threshold would reduce the corpus below the 2–3T token target. The authors explicitly state this as a tradeoff: "Weighing the tradeoff between dataset scale ('High') and performance maximization ('Low'), we adopt the more permissive 'High' threshold to ensure we meet our minimum token count requirement. In future versions of Dolma, we will start with more shards of Common Crawl and adopt stricter filter thresholds."

Filter correlation analysis (Appendix J, Figure 9). The authors compute Pearson correlation between different filters at the document level—two filters are positively correlated when any span in a document is tagged by both. The results for the High, Medium, and Low KenLM perplexity buckets show:

Generally low correlations across filters (most correlations are near zero), confirming that the Gopher heuristics, deduplication, toxicity filtering, PII filtering, and decontamination are capturing different signals.
Positive correlation between PII and hate speech filters (0.02–0.05), which the authors attribute to hate speech "often being directed at people"—personal attacks naturally involve personal information.
Negative correlation between Gopher rules and deduplication, especially in the high-perplexity tail (correlation of -0.36 for the Low bucket). This is because "Gopher rules remove many high-perplexity documents such as random strings, which are not caught by deduplication due to their randomness." Random strings are by definition unique, so they survive deduplication but are caught by Gopher's n-gram repetition checks.

This analysis is important because it validates the pipeline design: if filters were highly correlated, some could be removed without affecting corpus quality. The low correlations mean each filter provides independent value, and all are needed.

PII filtering. Following Subramani et al. (2023), the authors target three PII types detectable with high precision via regular expressions:

Email addresses: The regex [.\s@,?!;:)(]*([^\s@]+@[^\s@,?!;:)(]+?)[.\s@,?!;:)(]?[\s\n\r] matches patterns containing an "@" symbol with allowable characters on either side, excluding whitespace and common punctuation.
IP addresses: The regex \s+$?(\d{3})$?[-\.]*(\d{3})[-. ]?(\d{4}) matches sequences of three digit groups that could represent IP address octets or phone numbers (the same regex is used for both, since the formats overlap in unstructured text).
Phone numbers: The regex (?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9]{1,2})\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9]{1,2}) matches valid IPv4 address patterns (0.0.0.0 through 255.255.255.255).

The processing depends on PII density within a document:

Low density (≤ 5 PII spans): Spans are replaced with special placeholder tokens: |||EMAIL_ADDRESS|||, |||PHONE_NUMBER|||, or |||IP_ADDRESS|||. This preserves the document structure and surrounding context while removing the actual personal information. The special tokens are added to the tokenizer vocabulary when training models, so they receive learned embeddings. This affects 0.02% of documents.
High density (≥ 6 PII spans): The entire document is removed. This targets pages that are essentially collections of contact information (directories, leaked databases) where masking individual spans would be insufficient to prevent PII leakage. This affects 0.001% of documents.

The ablation results (Appendix O.3) show that "execution details around PII (e.g., removal versus special token replacement) had no effect on model performance, which is expected given the tiny percentage of affected data." This is a validation that the PII filtering is effectively costless in terms of model quality—it removes a negligible fraction of training data while providing privacy benefits.

Stage 5: Deduplication. After URL-based deduplication (53.2% removal, done in Stage 2) and all quality/content filtering (done in Stages 3–4), the pipeline performs two additional deduplication passes:

Exact document deduplication: The full text of each document is hashed, and duplicate hashes are removed. This catches identical content from different URLs—for example, the same news article syndicated across multiple newspapers, or the same forum post mirrored on different sites. Empty documents (those reduced to zero length by previous filtering) are also removed at this stage. This removes 14.9% of URL-deduped documents.
Paragraph-level deduplication: Using a Bloom filter seeded with paragraph hashes, any paragraph that has already been seen in the corpus is removed. This targets boilerplate text: copyright notices, "cookie consent" banners, site navigation footers, author bylines, and other content that appears identically across many pages. Unlike document-level deduplication, this operates at the sub-document level—a document may be partially preserved if only some of its paragraphs are duplicates. This removes 18.7% of paragraphs (equivalent to 19.1% of UTF-8 characters).

The multi-stage approach is designed for efficiency: URL deduplication is extremely cheap and eliminates a large fraction of data early, reducing the load on subsequent computationally expensive steps. Document-level deduplication catches exact copies that URL deduplication misses (different URLs, same content). Paragraph-level deduplication cleans boilerplate that survives both URL and document deduplication.

Compounding effects of all filter stages. Figure 3 (shown for HellaSwag; full results in Appendix O.5) demonstrates the cumulative benefit of stacking filters. The four conditions are:

Baseline: No additional filtering beyond CCNet pre-processing.
Quality Filters: CCNet + Gopher All + C4 NoPunc (the quality filtering stage alone).
Quality Filters + Dedup: Quality filtering plus paragraph-level deduplication.
Quality Filters + Dedup + Content Filters: The full pipeline including toxicity and PII filtering.

Each additional stage produces a visible improvement in the HellaSwag learning curve, with the full pipeline achieving the highest accuracy at all training steps. This compounding effect validates the decision to include all filtering stages despite each removing additional data—the quality improvement per remaining token outweighs the quantity reduction.

The full web pipeline details provide a snapshot of the massive scale: from 175.1 TB of raw HTML in Common Crawl, CCNet filters out 84.2% (leaving 27.7 TB), Gopher All tags 15.23% of remaining characters, C4 NoPunc tags 22.73%, URL dedup removes 53.2% of documents, document dedup removes 14.9% of remaining, paragraph dedup removes 18.7% of remaining paragraphs, and toxicity/PII filters remove small additional fractions. The net result is 2.28 trillion tokens from an initial pool several orders of magnitude larger.

Code Pipeline (Section 6): From GitHub to 411B Tokens

The code pipeline processes data from The Stack (Kocetkov et al., 2022), a pre-deduplicated collection of permissively-licensed GitHub repositories. Unlike the web pipeline, which starts from raw HTML, the code pipeline benefits from The Stack having already performed initial deduplication using MinHash and Locality-Sensitive Hashing (the approach from Allal et al., 2023). The raw data was collected in March 2023.

Language filtering. The pipeline removes files associated with data-heavy formats: CSV, JSON, JSON5, JSONLD, JSONIQ, and SVG files are excluded because they contain structured data rather than code. Assembly language files are also excluded, though the paper doesn't specify the rationale (likely because assembly is rarely the target of code language models and has different tokenization characteristics).

Quality filtering: combining RedPajama v1 and StarCoder heuristics. The authors run an ablation comparing two filtering configurations:

RedPajama v1 rules only: These target boilerplate and low-quality code:
- Remove copyright statement preambles from file headers (code license and provenance is preserved in metadata).
- Remove files with maximum line length exceeding 1000 characters (likely minified code or data dumps).
- Remove files with average line length exceeding 100 characters.
- Remove files where fewer than 25% of characters are alphanumeric.
- Remove files where the ratio of alphabetical characters to tokens (whitespace-separated) is below 1.5.
RedPajama v1 rules + StarCoder rules: Add additional filters from the StarCoder pipeline (Li et al., 2023b):
- Remove files containing XML template code.
- Remove HTML files where the code-to-text ratio is ≤ 0.2 (pages that are mostly markup rather than executable code).
- Remove Java, JavaScript, and Python files where the code-to-comment ratio is ≤ 0.01 (almost no comments—possibly autogenerated code) or > 0.8 (almost all comments—possibly documentation rather than code).

The ablation finds that combining both rule sets "leads to lower perplexity on code datasets (e.g., HumanEval) and improved performance on datasets in our evaluation suite" (Figure 56–62 in Appendix O.7 show these results across all tasks and Paloma domains). The RedPajama v1 rules alone remove 41.49% of data, and the StarCoder rules remove additional content. The combination is adopted for the final Dolma code subset.

Content filtering. The code pipeline applies the same PII regex patterns as the web pipeline (email, IP address, phone number detection), plus an additional filter: the detect-secrets library (Yelp, 2013) scans for code secrets (API keys, authentication tokens, cryptographic keys) and removes any document containing a match. This is code-specific—secrets embedded in source code are a known problem in public repositories, and their inclusion in training data could lead models to generate valid-looking secrets during code completion.

Deduplication. The Stack is already deduplicated at the file level, so no additional deduplication is applied. The MinHash-based approach used by The Stack catches near-duplicate files (files with high Jaccard similarity), which is important for code because exact deduplication misses files that differ only in variable names or minor formatting.

The Reddit pipeline introduces several novel challenges not present in the web or code domains: thread structure linearization, voting-based quality signals, and the specific characteristics of short-form conversational text.

Data acquisition. The dataset is derived from Pushshift (Baumgartner et al., 2020), which provides monthly dumps of Reddit submissions and comments. The authors use 378M posts from December 2005 through March 2023, including both submissions (top-level posts initiating discussions) and comments (replies to submissions or other comments). The data was downloaded from Pushshift's file servers between March and May 2023, before Reddit's API changes restricted access.

Thread linearization: an ablation study. The tree-like structure of Reddit threads (submissions with nested comment trees) creates a representation problem: how should this structured conversational data be flattened into linear text for language model training? The authors evaluate three approaches (detailed in Appendix E):

Atomic Content: Treat every submission and comment as an independent document with no connection to its thread context. A comment that says "I agree" would appear in isolation, without the post it's responding to. This loses conversational structure but keeps formatting simple.
Partial Threads: Assemble comments from the same thread into multi-round dialogues between users, with submissions as separate documents. The assembled dialogues are limited to a maximum parent depth, producing snippets of threads rather than complete conversations. User identities are preserved in the formatting.
Full Threads: Combine an entire submission with all its child comments into a single document, using code-like indentation to represent comment depth in the thread hierarchy. This preserves maximal conversational context but creates very long documents with complex formatting.

The ablation results (Figure 4, shown for HellaSwag; full results across all tasks in Appendix O.9) are clear: Atomic Content consistently outperforms both Partial and Full Threads across all evaluation metrics. The authors hypothesize that "artificial formatting introduced when combining thread elements negatively impacts language model training"—the indentation markers, user tags, and multi-turn dialogue structure may be unnatural input patterns that the model struggles to learn from. Short repeated comments (like "lol" or "this") that make sense in thread context become noise when concatenated into dialogue documents.

The final pipeline uses the Atomic Content format, treating each submission and comment as a standalone document.

Quality filtering. The pipeline applies length-based and voting-based filters:

Minimum length: Comments shorter than 500 characters and submissions shorter than 400 characters are removed. The asymmetry (submissions can be slightly shorter) reflects the observation that "qualitative inspection of the data suggested that submissions are of higher quality than comments." Submissions are more likely to be well-formed even when short (e.g., a question title), while short comments are often reactions ("nice," "agreed") that add little value for language modeling.
Maximum length: Documents exceeding 40,000 characters are removed to prevent very long documents from dominating training batches (which would require truncation or cause memory issues with the 2048-token context window).
Voting threshold: Comments with fewer than 3 net votes are removed. The vote count is computed as upvotes minus downvotes. The rationale draws on prior work: lower-scoring comments are more likely to be deeply nested in conversational threads (Weninger et al., 2013) or to result in "emotionally-charged discourse" (Davis and Graham, 2021). Voting was used as a quality signal in WebText (Radford et al., 2019) and OpenWebText (Peterson, 2020), both of which used Reddit votes to select "high-quality" web content. By applying voting directly to Reddit content, the authors are following an established but rarely-documented practice.
Deletion and moderation signals: Documents deleted by their authors, removed by moderators, or labeled as "over 18" are discarded. These signals indicate content that violates community norms or was withdrawn by its creator.
Subreddit blocklist: Documents from 26,123 banned or NSFW subreddits are excluded. The blocklist was compiled by merging multiple community-maintained lists of problematic subreddits, plus any subreddit where over 10% of posts are tagged NSFW. This is a more aggressive filtering approach than the toxicity classifier used for web data—rather than scoring individual sentences, entire communities are excluded based on their overall content characteristics.

Content filtering. The same toxicity classifiers and PII regex patterns from the web pipeline are applied. However, due to the short length of many Reddit documents, PII masking is replaced with full document removal—if a short comment contains an email address, masking it and keeping the remaining few words would produce a document too short to be useful.

Deduplication. Only document-level deduplication is applied (no paragraph-level), since Reddit documents are typically too short for paragraph-level dedup to be meaningful. This removes "copypasta" (identical text repeated across comments and subreddits for comedic effect) and other repetitive content.

Toxicity filtering ablation for Reddit (Appendix O.10). The paper includes an additional ablation specific to the Reddit pipeline, comparing three conditions: no filtering, NSFW + hate filtering, and NSFW + hate + PII filtering. The full combination (all three filters) produces the best downstream performance, consistent with the web pipeline's compounding pattern.

Remaining Sources (Section 8): Pre-Curated Data with Light Processing

Four additional sources are included in Dolma with substantially simpler processing, since they start from relatively clean inputs:

C4 (reprocessed). The original C4 corpus (Raffel et al., 2020) is included as a supplement to the web subset, following the practice of LLaMA and Llama 2, which used C4 alongside custom Common Crawl processing. The authors reprocess C4 through their full web pipeline (excluding URL deduplication, since C4 is already deduped), applying all quality, content, and paragraph-dedup filters. This removes additional content beyond C4's original filtering, including low-quality text and PII that survived the initial C4 curation.

Semantic Scholar academic papers (peS2o). The peS2o dataset (Soldaini and Lo, 2023) is a preprocessed collection of approximately 40 million open-access academic papers derived from S2ORC (Lo et al., 2020). It has already been "cleaned, filtered, deduplicated, and formatted for pretraining language models," so the authors use it as-is. The peS2o dataset was specifically designed for language model pretraining, making it a natural fit. The only additional processing is removing documents containing tokens or sequences repeating over 100 times (the same repetitive-token filter used in the web pipeline).

Project Gutenberg books. The authors collect the Project Gutenberg archive (April 2023), which contains over 70,000 public domain books. Processing includes:

Language identification at the paragraph level: each paragraph is scored for English using FastText, and the average score across all paragraphs must exceed 0.5 for the book to be retained.
Removal of pages with fewer than 25 UTF-8 words.
Removal of documents with repetitive token sequences (same filter as above).
Deduplication by exact match on book title.

The small size of this subset (6B tokens) reflects the limitation of public-domain books—most books that would be in a commercial training corpus are under copyright and cannot be included in an open dataset. The authors explicitly contrast this with Books3 (Gao et al., 2020), which was used in LLaMA and RedPajama v1 but is excluded from Dolma due to active copyright litigation (Knibbs, 2023).

Wikipedia and Wikibooks. English and Simple English editions from March 2023 Wikimedia dumps are processed using WikiExtractor (Attardi, 2023), which converts MediaWiki markup to plain text. Documents with 25 or fewer UTF-8 words are removed (targeting stub pages and XML parsing errors). The dataset is naturally deduplication-free by design, since Wikipedia articles are unique.

Mixing Strategies (Appendix M): How Source Composition Affects Model Behavior

Dolma does not prescribe a single mixture of sources for training. Instead, the authors provide cleaned subsets and investigate how different mixtures affect model behavior, offering guidance for downstream users.

How much code is needed? The paper investigates three C4 + Stack mixtures with 0%, 5%, and 15% code (Table 3 in Appendix M). The models are evaluated on reasoning tasks:

bAbI (Weston et al., 2015): synthetic reasoning tasks requiring state tracking. The C4-only model scores 0.0% (fails entirely), while 5% code achieves 8.8% and 15% code achieves 10.1%. This replicates findings from Muennighoff et al. (2023b) that "adding code to pre-training data improves ICL performance on bAbI and WebNLG," attributed to code improving "long-range state-tracking capabilities."
WebNLG (Gardent et al., 2017): structured data-to-text generation measuring Rouge-2. Performance improves monotonically with code fraction: 16.8% → 19.3% → 22.0%.
GSM8K (Cobbe et al., 2021): grade-school math word problems. All models score 0.0% in standard in-context learning and even when fine-tuned on the training set. However, when fine-tuned using program-aided language models (PAL; Gao et al., 2022)—where math problems are solved by writing Python code—the 15% code model achieves 14.7% accuracy (vs. 11.8% for 0% code). This demonstrates that "models pre-trained on code can leverage code generation to answer challenging reasoning tasks even when the original task does not directly involve code."

Comparing four mixing strategies. The paper evaluates four mixtures (Table 4 in Appendix M) on Paloma perplexity (Figure 5 in main text, Figures 63–68 in Appendix O.8):

Naïve Mix: Equal sampling from all sources (100% of each source's size). Composition: 83.5% web, 13.8% code, 2.5% reference material (papers, Wikipedia, books), 0.2% books.
Web Only: 100% web data, no other sources. This tests the contribution of non-web sources.
Reference+ Mix: Upsamples knowledge-intensive content: 2× papers, 2× books, 2× Wikipedia/Wikibooks. Composition: 81.2% web, 13.5% code, 4.9% reference material, 0.4% books.
Gopher-like Mix: Heavily biased toward curated reference material, following the approach in Rae et al. (2021). Uses 17% of web, 8% of code, but upsampled 2× for papers and books. Composition: 68.4% web, 5.4% code, 24.2% reference material, 2.0% books.

The key finding from Figure 5 (and Figures 12, 63–68): on web-domain data (C4 100 domains), all mixtures perform similarly—even the Web Only mix achieves comparable perplexity to the diverse mixtures. However, on specialized domains, excluding those domains from training causes large perplexity increases: the Web Only mix performs poorly on HumanEval (code) and M2D2 S2ORC (academic papers). Adding even small amounts of domain-specific data helps substantially: the Reference+ mix (only 4.9% reference material) achieves nearly identical perplexity to the Gopher-like mix (24.2% reference material) on academic papers, "suggesting that even a relatively small percentage of in-domain data is sufficient to achieve good domain fit."

The mixing experiments are not the primary contribution. The paper emphasizes that Dolma doesn't mandate a specific mixture and that users should "sample subsets according to their needs." The mixing results are provided as guidance, not prescription.

Tokenization Analysis (Appendix F): Code Has 2× the Fertility of Text

The paper includes a detailed tokenization analysis using the GPT NeoX tokenizer, revealing important domain-specific differences in how the tokenizer handles different Dolma sources.

Fertility is defined as the average number of tokens produced per Unicode word (Acs, 2019; Scao et al., 2022). A fertility of 1.0 means one token per word on average; higher fertility means the tokenizer fragments words into multiple subword tokens, which increases sequence length for a given amount of semantic content and thus increases computational cost.

The results (Figure 6b) show:

Most text sources have fertility between 1.15 (Reddit) and 1.28 (books). This is close to one token per word, indicating efficient tokenization.
The code subset has fertility of 2.45—more than twice the token cost per word compared to other sources.

The cause (Figure 6c) is whitespace tokenization asymmetry: "while a blank space before a word is tokenized as part of that word (e.g., 'I love you' → 'I', ' love', ' you'), other whitespace characters yield separate tokens (e.g., 'I\n\tlove\n\tyou' → 'I', '\n', '\t', 'love', '\n', '\t', 'you')." In code, newlines, tabs, and other whitespace characters are frequent and each produces a separate token, whereas in prose, spaces are absorbed into the following word token. This means "the costs of processing the code subset—be they computational or financial in nature (Petrov et al., 2023)—are more than twice as high compared to the other data sources."

The authors suggest a practical mitigation: "it might be advisable to add special tokens to the tokenizer (e.g., '\nif'; Hong et al., 2021)" to handle common code patterns more efficiently. They note that "this observation applies to most tokenizers in use today (e.g., the tokenizer used by GPT-4), which tend to lack tokens such as '\nif'."

Tokenizer vocabulary coverage. Of the 50,280 tokens in the GPT NeoX vocabulary, 50,057 are present in Dolma—223 tokens (0.4%) are never used. These unused tokens "mostly consist of combinations of whitespace characters (e.g., '\n\n ', two newline characters followed by two blank space characters)." The input embeddings for these unused tokens would not receive gradient updates during training.

There is a strong correlation between token ID and frequency in Dolma (Spearman's $r = 0.638$, $p < 0.001$): smaller token IDs (which correspond to byte pairs merged earlier during tokenizer training, and thus more frequent in the tokenizer's training data) appear more frequently in Dolma. This suggests "a good fit of the GPTNeoX tokenizer to Dolma."

Decontamination (Appendix L): How Test Set Leakage Is Prevented

The paper takes test set contamination seriously, performing both pre-training filtering and post-hoc analysis.

Perplexity evaluation decontamination. Using the paragraph deduplication Bloom filter, any paragraph in Dolma is flagged if (a) it is longer than 13 Unicode tokens and (b) it appears in any document in the Paloma benchmark suite. Documents containing any flagged paragraph are removed. The impact is minimal: "≤ 0.001% characters in Dolma contaminated, ≤ 0.02% of documents removed." This approach is conservative (prone to false positives—a document might be removed because one paragraph happens to match a Paloma example, even if the document as a whole doesn't reveal test answers), but the tiny removal rate means the conservatism has negligible impact on corpus size.

Downstream task decontamination. Using WIMBD (Elazar et al., 2023), the authors analyze which popular benchmarks appear in Dolma. Figure 11 shows contamination percentages: several datasets are 100% contaminated (Winograd Schema Challenge, SICK, GLUE AX, SemEval 2014 Task 1, COPA, SuperGLUE AXb) and others are > 90% contaminated (HumanEval at 92.7%, WIC at 96.1%, ESNLI at 97.2%, SNLI at 97.2%). The contamination source is identified: "many of these sets are contaminated in our code subset, as public repositories in GitHub often contain copies of these datasets." This is a systematic issue—code repositories frequently include academic benchmark datasets as test cases or examples, making any GitHub-derived corpus likely to contain benchmark contamination. The contaminated datasets are excluded from Dolma's evaluation suite, explaining the specific task selection in Section 4.2.

4. Key Insights and Innovations

Innovation 1: Data Curation as an Empirical Science — Ablation-Driven Design Replaces Precedent-Driven Recipes

The most fundamental conceptual contribution of this paper is not the Dolma corpus itself but the methodology used to build it: treating data curation decisions as testable hypotheses to be validated through controlled experiments rather than as inherited wisdom from prior work. Before Dolma, the dominant approach to pretraining data curation—even among relatively transparent efforts—was to follow established recipes (the LLaMA pipeline, C4 filtering, Gopher heuristics) and assume they would generalize. When decisions deviated from precedent, they were made on intuition or engineering convenience, not empirical evidence.

The paper systematically overturns this assumption by demonstrating that curation choices interact in non-obvious ways, that filtering stages compound rather than saturate, and that simple heuristics can outperform more complex or widely-used alternatives—but only when tested directly. The ablation apparatus (Section 4.2, Appendix D) is the methodological innovation: training 1.2B parameter models on controlled dataset variants and measuring downstream performance on a diverse 8-task suite plus the Paloma perplexity benchmark.

The significance is that this transforms data curation from a craft into a science. Consider what the ablations revealed that precedent would not have predicted:

C4 NoPunc outperforms the full C4 filter set and the full Gopher filter set individually (Figure 1). The dominant assumption in the field was that comprehensive filtering (many rules covering many failure modes) was always better. The ablation showed that a single, well-targeted heuristic—removing paragraphs lacking sentence-final punctuation—was more effective than either of the more complex systems used in major prior work. This is not a small refinement; it's a counterintuitive result that challenges the default of stacking every available filter.
Low-threshold toxicity filtering improves model quality despite removing more data (Figure 2). The field's implicit assumption was that filtering is a tradeoff—more aggressive filtering reduces dataset size, which harms performance. The ablation showed the opposite: removing 29–35% of sentences for low-level toxicity actually improves downstream task accuracy compared to removing only 5–7% at a higher threshold. This reveals that toxic content is not just ethically undesirable but quantitatively harmful to model quality, and that the quantity-versus-quality tradeoff can invert in unexpected ways.
Quality filters, deduplication, and content filters are nearly orthogonal (Figure 9, Appendix J), with Pearson correlations near zero between most filter pairs. This validates the multi-stage pipeline design empirically rather than assuming it—each filter removes different content, so removing any stage would degrade corpus quality. The negative correlation between Gopher rules and deduplication (the more random a document, the less likely deduplication catches it, but the more likely Gopher catches it) reveals a complementary relationship that would not be obvious from filter descriptions alone.
Code improves reasoning capabilities (Table 3). While prior work (Madaan et al., 2022; Muennighoff et al., 2023b) had shown code mixtures improve reasoning, the Dolma ablations replicate this finding in controlled conditions, showing that even 5% code in the training mixture enables bAbI state-tracking tasks that a pure-text model completely fails (0.0% vs. 8.8% accuracy), and that code pretraining enables program-aided mathematical reasoning that would otherwise be impossible.

This methodology is fundamental rather than incremental because it changes how future data curation should be approached. Rather than asking "what did prior models do?" the right question becomes "what does the evidence say about this specific decision in our specific pipeline?" The paper provides a template for answering that question—a standardized protocol (1.2B model, 150B tokens, 8-task suite, Paloma) that could be adopted by other data curation efforts. The fact that the authors found multiple cases where their initial assumptions were wrong validates the approach: without the ablations, they would have made suboptimal choices confidently.

The limitation is that this methodology has only been validated at one scale (1.2B parameters, 150B tokens) and on English-only data. The paper explicitly acknowledges this: "This choice was dictated by the need to efficiently iterate over many possible configurations, but it might result in design decisions that are not relevant at larger model sizes." Whether the ablation results transfer to 7B or 70B models remains an open question, but the approach of testing curation decisions empirically rather than following precedent is transferable regardless of scale.

Innovation 2: The "Orthogonal Filters" Finding Resolves a Longstanding Design Tension

A persistent debate in data curation has been whether to invest in multiple specialized filters or to develop a single, sophisticated quality model. Proponents of model-based filtering (using perplexity scores from language models like KenLM, or neural classifiers) argue that learned quality estimates can capture complex signals that rule-based heuristics miss. Proponents of heuristic filtering argue that rules are interpretable, fast, and avoid the biases inherent in models trained on specific distributions (e.g., KenLM's Wikipedia-centric notion of "quality").

The Dolma paper provides empirical evidence that largely resolves this debate in favor of diversification: the filters are orthogonal, so you need both types. The key finding is in Section 5.2:

"CCNet also provides quality scores using KenLM perplexity that groups documents based on Wikipedia-likeness; these buckets are often interpreted as high (21.9%), medium (28.5%), or low (49.6%) quality content... To our surprise, we found our heuristic filtering rules did not affect these proportions, suggesting that such model-based quality filters may capture other signals orthogonal to heuristic filters."

In other words, applying the Gopher and C4 heuristic filters removes roughly equal fractions of documents from the high, medium, and low KenLM perplexity buckets. The heuristics are not merely a crude approximation of what a perplexity model would detect—they are detecting different properties entirely. A document can score well on KenLM (Wikipedia-like prose) but be flagged by the punctuation filter (no sentence-final punctuation due to formatting), or score poorly on KenLM (atypical vocabulary) but pass all heuristic checks (well-formed sentences with normal word lengths and no excessive repetition).

This finding is significant for several reasons:

It provides intellectual justification for hybrid filtering pipelines. Before this result, one could argue that using both KenLM and heuristic filters was redundant—if heuristics were just approximating perplexity, you could drop them. The orthogonality evidence shows this argument is wrong. You need both because they capture different axes of "quality."
It explains why prior work reached different conclusions about filtering. Studies that used only model-based quality scores (and discarded heuristics) would miss surface-level formatting issues. Studies that used only heuristics (and discarded perplexity) would miss semantic coherence issues. The success of Dolma's hybrid approach suggests that both camps were capturing real but different quality dimensions, and the apparent disagreement was an artifact of incomplete filtering.
It connects to the broader literature on data quality in a non-obvious way. The finding that even the "low quality" bucket (49.6% of CCNet output) still contains useful data after heuristic filtering challenges the common practice of simply discarding low-perplexity documents. If heuristic filters remove orthogonal undesirable content regardless of perplexity bucket, then a low-perplexity document that survives heuristics may be perfectly serviceable training data—it's just about a topic far from Wikipedia.

The filter correlation matrix in Figure 9 (Appendix J) reinforces this point: most pairwise correlations are near zero, and the few non-zero correlations have interpretable explanations (PII and hate speech are correlated because hate speech often targets individuals; Gopher rules anti-correlate with deduplication because random strings are unique but catchable by n-gram repetition checks). This matrix is essentially a decomposition of data quality into independent axes, each addressed by a different filter.

This innovation is incremental in its mechanism but fundamental in its implications. The specific finding (heuristic and perplexity filters are orthogonal) could have been observed before—the raw data to compute it existed in any pipeline that used both. But nobody had thought to check, because the implicit assumption was that all quality filters measure the same underlying construct ("goodness" of text). By demonstrating that they don't, the paper changes how practitioners should think about filter design: instead of asking "which filter is best?" the question becomes "which orthogonal quality axes are we missing coverage for?"

The paper's investigation of Reddit thread linearization (Section 7.1, Appendix E) yields a counterintuitive result with implications beyond this specific dataset: flattening structured conversational data into independent documents consistently outperforms preserving thread structure, across both perplexity and downstream task metrics (Figure 4).

This is surprising because the natural intuition—and the approach taken by several prior efforts—is that preserving conversational structure provides useful context. A comment like "I agree" or "That's wrong" is semantically impoverished in isolation but informative in thread context. Partial Threads and Full Threads attempt to preserve this context by concatenating related comments into multi-turn dialogues with user identifiers and nesting markers.

The ablation results (Figure 4, with full results across 12 Paloma domains and 8 downstream tasks in Appendix O.9) show that Atomic Content—treating every post as an independent document—outperforms both Partial Threads and Full Threads on HellaSwag and across nearly all evaluation tasks. The authors hypothesize that "artificial formatting introduced when combining thread elements negatively impacts language model training," but the deeper insight may be that language models struggle with heterogeneous conversational formats that differ from the continuous prose they're optimized for.

Consider what happens in Partial Threads format: a document might contain 10 comments from different users, each potentially short, using different writing styles, separated by user tags and newlines, with conversational dependencies that aren't explicitly marked (sarcasm, inside jokes, references to deleted parent comments). This is a challenging text distribution—highly variable sentence lengths, abrupt topic shifts, fragmented grammar—that differs substantially from the books, articles, and encyclopedic text that dominate the training mixture. The model likely learns to ignore or poorly model these patterns, and the computational cost of processing them (via tokenization, attention) isn't recovered in improved representations of individual utterances.

There's a subtle connection to the tokenization analysis in Appendix F: Reddit content has the lowest fertility of any source (1.15 tokens per word, compared to 1.28 for books and 2.45 for code). This means Reddit text is already efficiently tokenized—short, common words that map to single tokens. Thread formatting introduces additional tokens (user tags, nesting markers, newlines) that increase fertility without adding semantic content, potentially diluting the signal.

This innovation is incremental in scope but conceptually transferable. It's a specific finding about one dataset, but the principle—that preserving structure from structured sources can backfire for language model pretraining—may apply broadly. Other structured data sources (forum discussions, chat logs, email threads, Q&A pairs) could exhibit similar patterns where flattening produces better training data than structure preservation. The paper doesn't investigate this generalization, but the ablation methodology it establishes provides a template for testing it in other domains.

The practical implication is clear: for forum or conversational data, the default should be atomic processing, and any structural enrichment should be treated as a hypothesis requiring ablation validation, not an obvious improvement.

Innovation 4: The Contamination Bottleneck — Code Datasets Systematically Leak Benchmarks

The paper's decontamination analysis (Appendix L) reveals a finding that, while not positioned as a major contribution, has significant implications for the field: GitHub-derived code datasets systematically contaminate popular NLP benchmarks, and this contamination is hard to avoid because the benchmarks are distributed as test cases and examples in public code repositories.

Figure 11 shows the scale of the problem:

6 datasets are 100% contaminated: Winograd Schema Challenge, SICK, GLUE AX, SemEval 2014 Task 1, COPA, and SuperGLUE AXb.
4 additional datasets are > 90% contaminated: HumanEval (92.7%), WIC (96.1%), ESNLI (97.2%), and SNLI (97.2%).
Several widely-used benchmarks show 50–70% contamination: GLUE QNLI (62.3%), GLUE RTE and SuperGLUE RTE (68.0% each).

The contamination source is explicitly identified: "many of these sets are contaminated in our code subset, as public repositories in GitHub often contain copies of these datasets." This isn't a Dolma-specific problem—it's a property of GitHub-derived code corpora in general, and by extension, of any large language model trained on GitHub data. The authors' response—excluding all contaminated datasets from their evaluation suite—is correct but raises a deeper issue: models trained on code-augmented data (which is essentially all modern LLMs) may have memorized benchmark answers through code exposure, not just through web text exposure as previously studied.

This finding connects to broader concerns about benchmark validity that the field is only beginning to grapple with. Magar and Schwartz (2022) demonstrated that data contamination inflates performance metrics, but their analysis focused on web text contamination. The code contamination vector is less studied and potentially more insidious because:

It's harder to detect. Web contamination often occurs through discussion of benchmark questions in articles or forums. Code contamination occurs through literal copies of benchmark datasets in repository test suites—the text is identical to the evaluation examples, making exact-match detection possible but still requiring the computational cost of checking.
It's harder to prevent. Repositories that include benchmark datasets as test fixtures are often legitimate (e.g., an NLP library's test suite uses SNLI examples to verify its data loaders). Removing these repositories would eliminate useful code. The alternative—detecting and redacting only the benchmark-containing files—requires benchmark-specific knowledge that general-purpose filtering pipelines lack.
It complicates the interpretation of code ablation studies. The finding that adding code improves reasoning (Table 3) could partially reflect contamination rather than genuine reasoning improvement—if the model has memorized benchmark answers from code exposure, it may perform better on evaluation without actually being better at the underlying reasoning task.

The paper handles this responsibly by excluding contaminated datasets, but the finding itself is a contribution because it quantifies a previously underappreciated contamination pathway and establishes a methodology (using WIMBD for systematic contamination analysis against PromptSource datasets) that other data curation efforts should adopt. The implication for future work is clear: any corpus that includes GitHub data must be analyzed for code-mediated benchmark contamination, and any model trained on code-augmented data must be evaluated on contamination-controlled benchmarks. The paper's exclusion of popular benchmarks like SNLI and HumanEval from their evaluation suite is not a limitation but a necessary methodological discipline that sets a standard for the field.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation benchmark is the Paloma perplexity suite (Magnusson et al., 2023), a stratified collection of hundreds of fine-grained textual sources designed to measure how well a language model fits diverse domains. For downstream task evaluation, the authors use an 8-task suite: ARC-Easy (Clark et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), OpenBookQA (Mihaylov et al., 2018), PIQA (Bisk et al., 2019), SciQ (Welbl et al., 2017), and WinoGrande (Sakaguchi et al., 2019). These tasks were selected because they (i) have been used in prior language model pretraining evaluation, (ii) capture diverse capabilities (commonsense reasoning, factual knowledge, reading comprehension), and (iii) are not contaminated in Dolma—Appendix L documents that several popular benchmarks (Winograd Schema Challenge, SNLI, HumanEval, etc.) show 90–100% contamination, primarily from code repositories in the GitHub subset, and were therefore excluded. All downstream evaluation uses zero-shot in-context prompting, casting every task as ranked text classification with prompts from PromptSource (Bach et al., 2022) and truncation following Min et al. (2022). For perplexity tracking, the Paloma benchmark includes subsets spanning web text (C4, mC4-en, C4 100 domains), reference text (WikiText-103, Penn Tree Bank), academic papers (M2D2 S2ORC), social media (Gab, 4chan, Twitter AAE, Manosphere), and code (HumanEval), plus held-out samples from Dolma itself and other open corpora (RedPajama v1, Falcon RefinedWeb).

Base model(s). All data ablation experiments use a 1.2 billion parameter decoder-only transformer from the OLMo family (Groeneveld et al., 2024): 16 layers, 16 attention heads, 2048-dimensional hidden states, ALiBi positional embeddings (Ofir Press et al., 2021), SwiGLU activations (Shazeer, 2020), mixed precision, 2048-token context window, and EleutherAI's GPT NeoX tokenizer (Black et al., 2022) with 50,280-token vocabulary. Training uses the LionW optimizer (Chen et al., 2023a) with peak learning rate $1 \times 10^{-4}$ , 2000 warmup steps, cosine decay, weight decay $1 \times 10^{-2}$ , and batch size 1024 sequences (~2.1M tokens/batch) on 64 AMD Instinct MI250X accelerators (128 logical compute units across 16 nodes). This model size was chosen following Le Scao et al. (2022), who used comparable scales for ablation experiments, and the authors argue it represents a computationally feasible size for the many ablation runs required (each decision requires a full 150B-token training run). For the final validation model (OLMo-1B, Section 9), they scale up: batch size doubles to 2048, compute doubles to 256 units, optimizer switches to AdamW (due to LionW instabilities at scale), and training continues to 739,328 steps (~3.1T tokens), i.e., a full pass over Dolma. For comparison with other open corpora (Section 9.2), identical 1.2B models are trained on 150B-token samples from C4, mC4 (English-only), RedPajama v1, RefinedWeb, Pile, and Dolma.

Metrics. The primary metrics are downstream task accuracy (percentage of correctly answered multiple-choice questions in zero-shot prompting) for the 8-task suite and perplexity on the Paloma benchmark suite. Perplexity is computed token-by-token and averaged across documents within each Paloma subset; lower perplexity indicates better domain fit. Training cross-entropy loss is also tracked but serves as a secondary diagnostic. For downstream tasks, accuracy is reported individually per task, with HellaSwag serving as the primary illustrative example throughout the ablation figures (the paper notes this choice in the text but does not provide a specific justification—likely because HellaSwag's commonsense reasoning is broadly representative of the capabilities these ablations are designed to probe). For the final OLMo-1B comparison (Table 2), average accuracy across all 8 tasks is reported alongside per-task scores.

Baselines. The ablation experiments are self-contained within the Dolma pipeline—each curation decision is evaluated by comparing an "intervention" dataset variant against a "baseline" variant that differs only in the specific filtering step being tested. The baselines vary by experiment:

Quality filter ablations (Figure 1): Baseline is CCNet-preprocessed web data with no additional quality filtering. Intervention variants include C4 NoPunc alone, C4 All (full C4 heuristics), Gopher All (full Gopher heuristics from Rae et al., 2021), and C4 NoPunc + Gopher All combined.
Toxicity filter ablations (Figure 2): Baseline is CCNet + quality-filtered data before toxicity filtering. Interventions include Hate Filter at Low Threshold ( $\tau = 0.0004$ ), NSFW Filter at Low Threshold, Hate Filter at High Threshold ( $\tau = 0.4$ ), NSFW Filter at High Threshold.
Full pipeline stacking ablations (Figure 3): Baseline is CCNet output alone. Intervention stages are Quality Filters alone, Quality Filters + Dedup, and Quality Filters + Dedup + Content Filters stacked sequentially.
Reddit thread format ablations (Figure 4): Baselines are Partial Threads and Full Threads. Intervention is Atomic Content format (with and without dedup/PII/toxic filtering).
Code pipeline ablations (Appendix O.7): Baseline is RedPajama v1 code rules alone. Intervention is RedPajama v1 + StarCoder rules combined.
Domain fit comparison (Figure 5, Section 9.2): No explicit baseline; 1.2B models trained on 150B-token samples from Pile, RefinedWeb, Dolma, C4, RedPajama v1, and mC4 (English) are compared simultaneously on Paloma perplexity.

For the final OLMo-1B evaluation (Table 2), the baselines are three similarly-sized open models: StableLM2 1.6B (trained on 2T tokens, 2 epochs, data composition undisclosed), Pythia 1.1B (Biderman et al., 2023; trained on ~300B tokens of Pile), and TinyLlama 1.1B (trained on ~3T tokens, dataset unspecified but described as "most comparable" by the authors).

Generation budget / compute accounting. In the context of this dataset curation paper, "compute" is not measured in model generations or FLOPs-matched comparisons. Instead, two distinct forms of compute accounting are used:

Data processing budget: The Dolma Toolkit's filtering throughput is reported as 122 CPU hours per terabyte (Section 4.1) when replicating the C4 recipe. For the full 200 TB of raw Dolma input, this projects to approximately 5 days on a c6a.48xlarge instance with 192 vCPUs. The paper does not report actual total CPU-hours consumed for the full corpus curation.
Model training budget for ablations: Each ablation run trains a 1.2B model for 150 billion tokens (approximately 71,000 steps with batch size 1024 and 2048-token sequences). The paper does not report wall-clock time or total FLOPs for these runs, though the hardware configuration is specified (64 AMD Instinct MI250X accelerators). The final OLMo-1B training extends to 739,328 steps (~3.1T tokens) with doubled compute (256 logical units).

The CCNet quality score buckets—"high" (21.9%), "medium" (28.5%), "low" (49.6%)—are provided as descriptive statistics for the pre-processed web data but are not used as a filtering criterion themselves; their orthogonality to heuristic filters is an empirical finding rather than a design choice.

Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing in the traditional sense. The ablation methodology uses a different form of rigor: controlled variation with matched training regimes. For each curation decision, the intervention and baseline models are trained with identical hyperparameters, identical random seeds (implied by the fixed training recipe), and evaluated at regular token-count intervals (every few billion tokens) to produce learning curves. The "statistical protocol" is visual: if the intervention curve consistently lies above the baseline curve across multiple training steps and multiple evaluation tasks, the intervention is judged beneficial. The paper presents learning curves for HellaSwag in the main text (Figures 1–4) and all 8 downstream tasks plus 12+ Paloma domains in Appendix O (Figures 13–92), providing a comprehensive view of consistency. However, the paper does not report standard deviations across multiple training runs, bootstrapped confidence intervals, or formal hypothesis tests. The final OLMo-1B comparison (Table 2) reports point estimates without error bars. For the domain-fit analysis (Figure 5), multiple corpora are compared on the same perplexity axes with learning curves, but no statistical test is applied to claim that one corpus produces significantly better fit than another.

Main Quantitative Results

Quality Filtering on Web Data: C4 NoPunc + Gopher All Is the Winner

The headline finding from the quality filter ablation (Figure 1, HellaSwag; full results in Appendix O.4, Figures 35–41) is that the combination of a single C4 punctuation rule (NoPunc) with the full Gopher heuristic set achieves the best downstream performance, outperforming either filter set alone and substantially outperforming a no-filtering baseline.

At 150B tokens of training, the HellaSwag accuracy ranking is:

C4 NoPunc + Gopher All: ~0.49 (highest)
C4 NoPunc alone: ~0.48
Gopher All alone: ~0.47
C4 All alone: ~0.46
No-filtering baseline: ~0.44 (lowest)

The ordering is consistent throughout training (the curves do not cross), indicating that the filtering benefit is sustained rather than being a transient advantage that disappears with more data. The gap between C4 NoPunc + Gopher All and the no-filtering baseline is approximately 5 percentage points in absolute accuracy—an 11% relative improvement over a baseline that already represents CCNet-preprocessed data (which itself removed 84.2% of raw Common Crawl content). This demonstrates that "even after CCNet's aggressive initial filtering, additional quality filtering provides meaningful improvements."

The magnitude of data removal is substantial: Gopher All tags 15.23% of UTF-8 characters for removal, and C4 NoPunc tags 22.73%. Critically, these are not redundant—the combination outperforms either alone, confirming that the filters are complementary. The paper does not report the exact overlap fraction, but the performance evidence implies it is low.

The full ablation suite (Appendix O.4, Figures 35–41) shows that the C4 NoPunc + Gopher All advantage generalizes across tasks: it outperforms on OpenBookQA (Figure 39, left), ARC-Easy (Figure 39, center), PIQA (Figure 40, right), and SciQ (Figure 40, left), with varying gap sizes. On WinoGrande (Figure 39, right), the advantage is smaller but consistent. The perplexity results (Figures 35–38) show the same pattern across Paloma domains: C4 NoPunc + Gopher All achieves the lowest perplexity on C4 100 domains (Figure 35, right), Pile (Figure 35, center), and most other subsets.

A surprising result: C4 NoPunc alone outperforms the full C4 filter set. The C4 All configuration—which includes C4 NoPunc plus additional heuristics—consistently underperforms C4 NoPunc alone. This suggests that some C4 heuristics may be counterproductive, perhaps removing text that, while flagged by C4's rules, remains useful for language model training. The paper does not investigate which specific C4 rules cause the degradation.

Toxicity Filtering: Lower Thresholds Improve Quality Despite Reducing Data Volume

The headline finding from the toxicity filter ablation (Figure 2, HellaSwag; full results in Appendix O.6, Figures 49–55) is that the Low Threshold ( $\tau = 0.0004$ ) for both hate and NSFW classifiers yields better downstream performance than the High Threshold ( $\tau = 0.4$ ), despite removing substantially more data. This is counterintuitive because it inverts the typical quantity-versus-quality tradeoff in data curation—removing more content does not just maintain quality but actually improves it.

At 150B tokens, the HellaSwag accuracy ranking is:

NSFW Filter (Low Threshold): ~0.48 (highest among toxicity variants)
Hate Filter (Low Threshold): ~0.47
Hate Filter (High Threshold): ~0.46
NSFW Filter (High Threshold): ~0.46
Baseline (no toxicity filtering): ~0.44

The data removal fractions are starkly different:

Low Threshold ( $\tau = 0.0004$ ): removes 29.1–34.9% of sentences (the range reflects that the hate classifier is slightly more aggressive than the NSFW classifier).
High Threshold ( $\tau = 0.4$ ): removes 5.5–7.3% of sentences.

The Low Threshold removes 4–6× more content but produces a model with 2–4 percentage points higher HellaSwag accuracy. The authors note that the score distribution is bimodal: "Manual inspection of the distribution of sentence scores revealed a bi-modal distribution with peaks near 0.0 and 1.0." This means that most sentences are either clearly toxic (score near 1.0) or clearly benign (score near 0.0), with relatively few ambiguous cases. The Low Threshold of 0.0004 essentially sets the bar just above "definitely not toxic," catching even mildly suspicious content.

The full ablation suite (Appendix O.6) shows the pattern holds across tasks: the Low Threshold variants outperform High Threshold variants on ARC-Easy, OpenBookQA, PIQA, and SciQ, though the margin varies. On WinoGrande, the differences are smaller. The perplexity results are more mixed: the Low Threshold reduces perplexity on some Paloma domains (C4 100 domains, Pile, mC4-en) but shows negligible difference on others (Gab, Manosphere, Twitter AAE).

The authors' actual choice diverges from the ablation recommendation. Despite the Low Threshold's performance advantage, the released Dolma corpus uses the High Threshold. The reason is explicitly stated as a tradeoff: "Weighing the tradeoff between dataset scale ('High') and performance maximization ('Low'), we adopt the more permissive 'High' threshold to ensure we meet our minimum token count requirement. The cause of this was surprising: Our quality, content, and deduplication filters overlap very little in which texts they remove (Figure 9), resulting in a compounded filtering effect when combining them." In other words, the cumulative removal from all pipeline stages (CCNet → Gopher → C4 NoPunc → URL dedup → document dedup → paragraph dedup → toxicity → PII) was large enough that adding the Low Threshold toxicity filter would have pushed the corpus below the 2–3T token target. The paper frames this as a limitation to be addressed in future versions: "In future versions of Dolma, we will start with more shards of Common Crawl and adopt stricter filter thresholds."

Full Web Pipeline Stacking: Filters Compound Rather Than Saturate

The headline finding from the pipeline stacking ablation (Figure 3, HellaSwag; full results in Appendix O.5, Figures 42–48) is that each additional filtering stage provides incremental performance improvements, with no evidence of saturation. The learning curves for the four conditions are ordered top-to-bottom at nearly all training steps:

Quality Filters + Dedup + Content Filters: ~0.50 (highest)
Quality Filters + Dedup: ~0.48
Quality Filters: ~0.46
Baseline (CCNet only, no additional filtering): ~0.44

The gap between the full pipeline and baseline is approximately 6 percentage points on HellaSwag—a 14% relative improvement. The gaps between successive stages are roughly equal, suggesting that each filtering component contributes comparable value. This validates the multi-stage design: if filters were redundant, adding more would produce diminishing returns, but the roughly linear improvement with each additional stage suggests they are capturing independent quality dimensions.

The full results (Appendix O.5, Figures 42–48) confirm the pattern across tasks:

OpenBookQA (Figure 46, left): Full pipeline achieves ~0.34 vs. baseline ~0.31.
ARC-Easy (Figure 46, center): Full pipeline achieves ~0.55 vs. baseline ~0.50.
PIQA (Figure 47, right): Full pipeline achieves ~0.69 vs. baseline ~0.66.
SciQ (Figure 47, left): Full pipeline achieves ~0.82 vs. baseline ~0.76.

On WinoGrande (Figure 46, right), the gains are smaller (full pipeline ~0.54 vs. baseline ~0.53), suggesting that web text quality has less impact on pronoun resolution tasks.

Perplexity results (Figures 42–45) show a broadly similar pattern: the full pipeline achieves the lowest perplexity on C4, C4 100 domains, Pile, and mC4-en. On social-media-heavy domains (4chan, Gab, Twitter AAE, Manosphere), the differences are smaller, which aligns with the expectation that quality filters targeting "good web prose" are less effective at distinguishing quality in these already-atypical textual domains.

Reddit Thread Linearization: Atomic Content Consistently Outperforms Structured Formats

The headline finding from the Reddit format ablation (Figure 4, HellaSwag; full results in Appendix O.9, Figures 69–75) is that treating Reddit submissions and comments as independent documents (Atomic Content) outperforms both Partial Threads (multi-turn dialogues) and Full Threads (entire conversation trees) across all evaluation metrics. The learning curves at 60B tokens show:

Atomic Content + Dedup + PII + Toxic: ~0.48 (highest overall)
Atomic Content (no filtering): ~0.47
Partial Threads + Dedup: ~0.46
Partial Threads (no filtering): ~0.45
Full Threads: ~0.44 (lowest)

The Atomic Content advantage over Partial Threads is approximately 2 percentage points, and over Full Threads is approximately 4 percentage points—a non-trivial gap given the relatively small fraction of total training data that Reddit represents. The ordering is consistent across training steps, suggesting the structural format is an inherent property of the data representation rather than an interaction with training duration.

A secondary finding is that content filtering (dedup, PII, toxicity) provides an additional small benefit on top of the format choice: Atomic Content + filtering outperforms Atomic Content without filtering by roughly 1 percentage point on HellaSwag. This is consistent with the compounding filter benefits observed in the web pipeline.

The full results (Appendix O.9, Figures 69–75) show the same pattern across tasks: Atomic Content + filtering achieves the highest accuracy on OpenBookQA, ARC-Easy, PIQA, and SciQ, with Full Threads consistently at the bottom. The perplexity results (Figures 69–72) are more nuanced: on some domains (C4, C4 100 domains), Atomic Content shows clear perplexity advantages; on others (Gab, Manosphere), the differences between formats are negligible; and on 4chan, Atomic Content surprisingly underperforms Partial Threads in perplexity (Figure 69, left)—the paper does not comment on this exception, but it may reflect that 4chan's own conversational structure is more similar to Reddit threads than to independent documents.

The authors hypothesize that "artificial formatting introduced when combining thread elements negatively impacts language model training," suggesting that the user tags, indentation markers, and multi-turn dialogue structure are unnatural text patterns that confuse the model. An alternative (and compatible) explanation is that thread-format documents are much longer than individual posts, requiring the model to handle longer-range dependencies in a domain (conversational text) where it receives relatively little training exposure compared to continuous prose.

Code Pipeline: Combining RedPajama v1 and StarCoder Rules Outperforms Either Alone

The headline finding from the code pipeline ablation (Table 3 and Appendix O.7, Figures 56–62) is that adding StarCoder filtering rules to RedPajama v1 rules improves both code-specific perplexity and downstream task performance. The paper reports that "RedPajama v1 and StarCoder rules combined lead to lower perplexity on code datasets (e.g., HumanEval) and improved performance on datasets in our evaluation suite."

The ablation comparison (Appendix O.7, Figures 56–62) shows consistent but modest advantages for the combined rule set. At 50B tokens (the code ablation training limit):

HumanEval perplexity: Combined rules achieve approximately 4–5 vs. RedPajama v1 alone at approximately 5–6 (Figure 12, center panel, though this figure appears in Appendix M rather than O.7).
Downstream tasks: Combined rules show small improvements on OpenBookQA (~0.26 vs. ~0.25), ARC-Easy (~0.38 vs. ~0.37), and PIQA (~0.57 vs. ~0.56) with negligible differences on other tasks.

The RedPajama v1 rules alone remove 41.49% of data from the raw Stack corpus. The additional removal from StarCoder rules is not reported explicitly but can be inferred to be smaller, since the combined performance gains are modest.

Mixing Strategy and Code Inclusion: Code Improves Reasoning but All Mixtures Handle Web Data Similarly

The headline findings from the mixing experiments (Appendix M, Table 3, Table 4, Figure 12, Figures 63–68) are twofold:

Finding 1: Code inclusion enables reasoning capabilities that pure-text models completely lack. Table 3 shows:

bAbI (ICL): 0% Code model scores 0.0% ± 0.0 on all bAbI tasks (complete failure). 5% Code achieves 8.8% ± 0.9. 15% Code achieves 10.1% ± 2.8. The jump from 0% to non-zero is the critical result—a small amount of code enables a capability that doesn't exist without it.
WebNLG (ICL, Rouge-2): Monotonically improves from 16.8% ± 1.1 (0% code) → 19.3% ± 1.1 (5% code) → 22.0% ± 1.3 (15% code).
GSM8K (standard ICL and standard fine-tuning): All models score 0.0% regardless of code fraction. This isolates the difficulty—grade-school math is beyond all these 1B models without program-aided methods.
GSM8K + PAL (fine-tuned with program-aided language models): 0% Code achieves 11.8% ± 0.8, 5% Code achieves 14.2% ± 1.3, 15% Code achieves 14.7% ± 0.9. The key finding: code-pretrained models can leverage program generation to solve math problems that are otherwise unsolvable, and the benefit increases with code fraction, though with diminishing returns.

Finding 2: Source mixture affects domain-specific perplexity but not general web text perplexity. Figure 12 (Appendix M) and the Paloma learning curves in Appendix O.8 (Figures 63–68) show:

On C4 100 domains (Figure 12, left): All four mixtures (Naïve, Web Only, Reference+, Gopher-like) achieve nearly identical perplexity curves—roughly 20–40 across training, with curves largely overlapping. This means that for modeling standard web text, the specific mixture matters little; web data alone is sufficient.
On HumanEval (Figure 12, center): The Web Only mix shows dramatically higher perplexity (~30–50) compared to all code-containing mixes (~4–10). The Naïve, Reference+, and Gopher-like mixes all perform similarly despite different code fractions (13.8%, 13.5%, and 5.4% respectively). This suggests that even a small amount of code (~5%) is sufficient to achieve good domain fit on code evaluation—diminishing returns kick in quickly.
On M2D2 S2ORC academic papers (Figure 12, right): The Web Only mix shows the highest perplexity (~60–80). Adding reference material reduces perplexity: the Reference+ mix (4.9% reference) and Gopher-like mix (24.2% reference) achieve nearly identical perplexity (~30–40), while the Naïve mix (2.5% reference) is slightly higher (~40–50). As with code, even a small fraction of in-domain data provides most of the benefit—the 5× difference in reference material between Reference+ and Gopher-like produces negligible perplexity difference.

The downstream task results (Appendix O.8, Figures 67–68) are less differentiated: all mixtures perform similarly on most tasks, with the Gopher-like mix showing slight advantages on PIQA and SciQ. The practical implication is that mixture design primarily affects domain-specific perplexity (how well the model models text from a given domain) rather than general reasoning capabilities (how well it answers questions), at least at the 1.2B scale and 150B-token training budget.

Final OLMo-1B Evaluation: Competitive with Comparable Models on Downstream Tasks

The headline finding from the final OLMo-1B evaluation (Table 2) is that OLMo-1B, trained on the full Dolma corpus for one epoch (~3.1T tokens), performs competitively with similarly-sized open models, outperforming the most comparable model (TinyLlama 1.1B) on 4 of 8 tasks and achieving a higher average accuracy (60.3 vs. 59.4).

Per-task comparison (Table 2):

ARC-Easy: OLMo-1B 58.1 vs. TinyLlama 53.2 (+4.9). OLMo-1B trails StableLM2 63.7 and outperforms Pythia 50.2.
ARC-Challenge: OLMo-1B 34.5 vs. TinyLlama 34.8 (-0.3). Essentially tied. StableLM2 leads at 43.8.
BoolQ: OLMo-1B 60.7 vs. TinyLlama 64.6 (-3.9). OLMo-1B underperforms here.
HellaSwag: OLMo-1B 62.5 vs. TinyLlama 58.7 (+3.8).
OpenBookQA: OLMo-1B 46.4 vs. TinyLlama 43.6 (+2.8).
PIQA: OLMo-1B 73.7 vs. TinyLlama 71.1 (+2.6).
SciQ: OLMo-1B 88.1 vs. TinyLlama 90.5 (-2.4).
WinoGrande: OLMo-1B 58.9 vs. TinyLlama 58.9 (tied).

Averaged across all 8 tasks: OLMo-1B 60.3, TinyLlama 59.4, Pythia 54.5, StableLM2 66.5. StableLM2's higher average is confounded by its larger size (1.6B vs. 1.1–1.2B) and its training on 2T tokens for 2 epochs (4T tokens total, though data composition is undisclosed), making it an imperfect comparison.

The paper emphasizes that "zero-shot evaluations of such tasks are often challenging for smaller 1B models," noting that all models perform substantially above random chance (which varies by task but is typically 25% for 4-way multiple choice). The key interpretive claim is that Dolma enables training a model that is "competitive" with the best similarly-sized open models, validating the corpus quality without claiming it to be superior.

Domain Fit: Dolma Nearly Matches Pile's Diversity Despite Much Larger Web Fraction

The headline finding from the domain-fit analysis (Figure 5, Section 9.2; full results in Appendix O.1, Figures 13–20) is that a model trained on Dolma achieves nearly identical Paloma perplexity to a model trained on Pile despite Dolma containing a much larger fraction of web data, and substantially outperforms single-source corpora (C4, mC4-en, RefinedWeb) on domain diversity metrics.

Figure 5 shows Paloma perplexity learning curves for 1.2B models trained on 150B-token samples from six corpora:

Pile: Achieves the lowest overall perplexity (~14 at 150B tokens), consistent with its composition of many diverse, hand-curated sources.
Dolma: Nearly matches Pile's perplexity (~15 at 150B tokens), with the curves largely parallel and separated by only ~1 perplexity point.
RedPajama v1: Intermediate perplexity (~17 at 150B tokens), below Pile and Dolma but above the single-source corpora.
C4, mC4 (English), RefinedWeb: All three single-source web corpora cluster together at higher perplexity (~19–22 at 150B tokens), with RefinedWeb performing slightly better than C4.

The gap between Dolma and the single-source corpora is approximately 4–7 perplexity points at 150B tokens, which is substantial in perplexity space—it indicates that models trained on web-only data are significantly worse at modeling the diverse textual domains in Paloma than models trained on multi-source data.

Beyond the aggregate Paloma score, the full results (Appendix O.1, Figures 13–20) reveal domain-specific patterns:

On web-text domains (C4, C4 100 domains, mC4-en, Figures 17 right, 14 left, 17 left): All corpora perform similarly, with Dolma and C4 nearly tied. This echoes the mixing finding that web text is easy to model regardless of source mixture.
On specialized reference text (Penn Tree Bank, WikiText-103, M2D2 S2ORC, M2D2 Wiki, Figures 14 center, 13 center, 17 center, 15 center): Dolma and Pile outperform single-source corpora by substantial margins, confirming that including non-web reference material improves fit on non-web domains.
On social media and fringe communities (4chan, Gab, Manosphere, Twitter AAE, Figures 13 left, 14 right, 16, 15 right): The patterns are complex. Pile and Dolma generally outperform single-source corpora, but the gaps are smaller than for reference text. On 4chan (Figure 13, left), Dolma actually underperforms RefinedWeb and mC4-en after ~50B tokens—the paper does not discuss this anomaly, but it may reflect that 4chan's text distribution is closer to certain web text patterns than to the curated sources that distinguish Dolma from web-only corpora.
On downstream tasks (Figures 18–19): Dolma-trained models roughly match or slightly outperform RedPajama v1-trained models across ARC-Easy, HellaSwag, PIQA, SciQ, and WinoGrande, with Pile-trained models often leading.

The key interpretive claim is that "larger multi-source datasets like Dolma and, to a lesser extent, RedPajama v1 yield models with similar coverage of diverse domains to Pile." The Paloma metric is designed to "surface how models fit more heterogeneous data, because it samples marked domains from each source equally rather than by their unequal proportions in the source." This means the low Paloma perplexity of the Dolma model indicates it has learned to model diverse text types (academic papers, social media, code) despite those types being minority fractions of the training mixture. The contrast with single-source corpora shows that this diversity is a property of the multi-source curation, not an inevitable consequence of training on web text at scale.

Ablation Studies and Robustness Checks

PII filtering implementation (masking vs. removal): The ablation comparing PII removal (≥5 spans = remove document) + masking (<5 spans = replace with placeholder tokens) against PII removal for all matches showed "no effect on model performance, which is expected given the tiny percentage of affected data" (Appendix O.3, Figures 28–34). The affected fractions are 0.02% of documents for masking and 0.001% for removal. This validates that PII filtering can be applied with negligible impact on model quality—the privacy benefits come at essentially zero performance cost.

Paragraph deduplication in web pipeline: The dedup-only ablation (comparing baseline vs. paragraph deduplication alone, Appendix O.2, Figures 21–27) confirmed that paragraph-level deduplication improves performance. At 150B tokens, the dedup condition achieves lower perplexity on most Paloma domains (C4, Pile, C4 100 domains, mC4-en, M2D2 Wiki) and performs similarly or slightly better on downstream tasks (HellaSwag ~0.47 vs. ~0.46 baseline). The improvement is consistent but smaller than the quality filter improvements.

CCNet perplexity bucket analysis: The finding that heuristic quality filters do not alter the proportion of data in CCNet's high (21.9%), medium (28.5%), and low (49.6%) KenLM perplexity buckets (Section 5.2) serves as an implicit ablation—if heuristics and perplexity-based filtering were measuring the same quality signal, applying heuristics would shift data toward higher-quality buckets. The null result confirms orthogonality.

Language identification audit: An external validation on the International Corpus of English (Appendix G, Figure 7) confirmed that the FastText language ID classifier with a 0.5 threshold correctly identifies all English documents across nine countries' English varieties. Even at a threshold of 0.90, the majority of documents remain classified as English. This validates that the 0.5 threshold is conservative and unlikely to disproportionately filter non-American English dialects—a concern given that certain English varieties (African American English, Indian English) might have linguistic features that the classifier misidentifies as non-English.

Toxicity classifier dialectal bias check: An analysis of the FastText toxicity classifier's behavior across regional English varieties (Appendix H, Figure 8) found that for any two country-specific subreddits, there is <5% difference in the fraction of comments marked as toxic across classifier thresholds. The score distribution is bimodal (peaks near 0.0 and 1.0), suggesting "any reasonable threshold (lying between 0.1 to 0.9) to predict toxicity will lead to similar outcomes." This provides some evidence that the toxicity classifier is not systematically biased against specific dialects, though the proxy (country-based subreddits) is crude and may not capture within-country dialect variation.

Filter correlation analysis: The Pearson correlation matrix across filters (Appendix J, Figure 9) computed on 24M (High), 20M (Medium), and 43M (Low) CCNet perplexity bucket documents confirmed generally low correlations (most near zero), with two interpretable exceptions: positive PII-Hate correlation (0.02–0.05, attributed to hate speech often directing at individuals) and negative Gopher-Dedup correlation in the low-perplexity bucket (-0.36, attributed to Gopher catching random strings that dedup misses). This provides quantitative evidence for the qualitative claim that filters are orthogonal.

Reddit filtering compounding: The Reddit-specific ablation (Appendix O.10, Figures 76–82) confirmed that adding toxicity filtering (NSFW + Hate) on top of dedup and PII processing improves performance, and that the full combination (PII + NSFW + Hate) performs best. The pattern mirrors the web pipeline's compounding results.

Mixing strategy comparison (ablation within Appendix M): The four-mixture comparison (Naïve, Web Only, Reference+, Gopher-like) on downstream tasks (Figures 67–68) showed mostly overlapping performance, with small advantages for the Gopher-like mix on PIQA and SciQ. The lack of strong differentiation suggests that for downstream task accuracy (as opposed to domain-specific perplexity), the exact mixture is relatively unimportant at the 1.2B/150B scale.

ReSTᴱᴹ revision model negative result (from the reference example, NOT present in this paper): [Not applicable—this paper does not discuss revision models.]

OLMo-1B training stability (Appendix D.4): The switch from LionW to AdamW for the final OLMo-1B training was motivated by "instabilities we found in the LionW optimizer." This is a negative result about optimizer choice that affected the experimental protocol—the 150B-token ablations used LionW, while the full 3.1T-token training used AdamW, introducing a confound between training duration and optimizer choice. The paper does not investigate whether the ablation results would differ under AdamW, leaving open the possibility that some curation decisions validated under LionW would not transfer to the optimizer used for the final model.

Code ablation on reasoning tasks (Table 3): The finding that 0% Code models completely fail on bAbI (0.0 ± 0.0 across 5 seeds) while even 5% Code achieves 8.8 ± 0.9 is a robustness check that validates prior work (Muennighoff et al., 2023b) in the specific context of Dolma-sourced training data. The GSM8K + PAL results (0% Code: 11.8 ± 0.8, 15% Code: 14.7 ± 0.9) provide moderate evidence for code improving mathematical reasoning, though the confidence intervals overlap.

Test set contamination analysis (Appendix L, Figure 11): The WIMBD-based contamination scan serves as a robustness check for the evaluation methodology—it validates that the chosen 8-task suite is contamination-free. This is not an ablation of Dolma but an ablation of potential confounds in the experimental design. The finding that several popular benchmarks are 100% contaminated (and thus excluded) is itself a robustness result that strengthens the credibility of the reported numbers.

Tokenization fertility analysis (Appendix F, Figure 6): This is a diagnostic rather than an intervention ablation—it measures how efficiently the tokenizer handles different Dolma sources without testing alternatives. The finding that code has 2.45 fertility vs. 1.15–1.28 for text sources reveals a potential confound: models trained on code-heavy mixtures process fewer semantic units per token, which could affect perplexity comparisons. The paper does not control for this in the mixing experiments.

Critical Assessment

Claim 1: "Dolma offers a larger pool of tokens at comparable quality while maintaining diverse data composition compared to other publicly-available pretraining corpora."

What the experiments demonstrate: The domain-fit analysis (Figure 5) supports this claim within the specific experimental frame. A 1.2B model trained on 150B tokens of Dolma nearly matches the Paloma perplexity of a Pile-trained model (~15 vs. ~14 at 150B tokens) while being roughly 8× larger than Pile in total token count, and substantially outperforms same-size samples from C4, mC4-en, and RefinedWeb (~19–22 perplexity). On downstream tasks (Figures 18–19), Dolma-trained models are roughly competitive with RedPajama v1 and Pile, though they don't consistently lead.

What the experiments do NOT demonstrate: The claim of "comparable quality" is tested only at the 1.2B parameter scale and only on 150B-token training runs. Whether the quality comparison holds at larger model scales (7B–70B parameters) or full training budgets is untested. The paper's own methodology is clear that 150B tokens is an early-stopping point chosen for computational feasibility, not because the relative quality ordering stabilizes. If Dolma's advantages accrue from source diversity, they might become more pronounced at larger scales (where diverse data prevents overfitting) or less pronounced (if web-only corpora eventually catch up). The paper provides no evidence in either direction.

Additionally, "comparable quality" is measured only on downstream multiple-choice question answering and perplexity. The paper does not evaluate on generative tasks (summarization, translation, dialogue), code generation tasks (beyond perplexity on HumanEval), or safety/toxicity metrics of the trained models. The quality metric is relatively narrow, and a corpus that does well on these specific benchmarks might underperform on other important dimensions.

The comparison with RedPajama v1 is missing a direct model comparison. The paper trains models on Dolma and RedPajama v1 samples and compares them (Figure 5 shows Dolma outperforming RedPajama v1 on Paloma perplexity), but this is a 150B-token sample comparison, not a full-training comparison. The OLMo-1B model is trained on Dolma and compared against models trained on other corpora (Pythia on Pile, TinyLlama on unknown data), but there is no full-training comparison between Dolma and RedPajama v1. The claim that Dolma addresses "data quality issues" in RedPajama v1 (citing Soboleva et al., 2023; Elazar et al., 2023) is referenced but not experimentally validated—no ablation directly shows that Dolma's quality filtering produces better models than RedPajama v1's filtering at comparable scale.

Claim 2: "Data curation decisions are validated through controlled data ablations."

What the experiments demonstrate: Each major curation decision (quality filtering, toxicity filtering, deduplication, thread formatting, code filtering, mixing strategy) is tested through a controlled ablation with a matching 1.2B model trained on the intervention versus the baseline. The experiments are well-designed in terms of isolating single variables while holding architecture, hyperparameters, and evaluation protocol constant. The results are presented transparently, with learning curves showing performance across all 8 tasks (in Appendix O) rather than cherry-picking favorable outcomes.

What the experiments do NOT demonstrate: The ablation results are purely observational—they show that intervention > baseline on a fixed set of tasks, but they do not explain why. The paper offers hypotheses (e.g., "artificial formatting introduced when combining thread elements negatively impacts language model training") but never tests them through diagnostic experiments. For example, the Reddit thread format ablation could have been complemented by probing whether the model learns to track speaker identity, understand conversational dependencies, or parse thread structure in the Partial/Full Thread conditions—experiments that would distinguish between competing explanations for the Atomic Content advantage.

The ablation protocol has several methodological limitations:

Single training run per condition. The paper does not report training multiple models with different random seeds and computing confidence intervals. The learning curves are smooth and the gaps are typically consistent across training steps, reducing the likelihood that the results are noise, but without error bars or statistical tests, the reliability of small differences (e.g., the ~1 percentage point advantage of C4 NoPunc + Gopher All over C4 NoPunc alone in Figure 1) is uncertain.
Single model scale. All ablations use 1.2B parameter models. The paper explicitly acknowledges this limitation: "This choice was dictated by the need to efficiently iterate over many possible configurations, but it might result in design decisions that are not relevant at larger model sizes." This is a genuine weakness. If, for example, the benefit of toxicity filtering diminishes at larger scales (because larger models are better at learning to ignore noisy patterns in the data), then the Low Threshold's advantage in the ablation would not generalize to production-scale training.
Single training duration (150B tokens). The paper terminates ablations at 150B tokens, which is less than 5% of the full OLMo-1B training budget (3.1T tokens). The learning curves for downstream tasks (Figures 1–4) are still rising at 150B tokens rather than plateauing, meaning the relative ordering of conditions could change with more training. This is especially concerning for the toxicity filter ablation: if the Low Threshold removes 30% of data, the model sees fewer tokens over the same number of training steps, and the apparent advantage at 150B "tokens seen" could partially reflect the data removal (the intervention model has actually processed fewer unique tokens than the baseline at each x-axis position).
Single optimizer for ablations, different optimizer for final model. The ablations used LionW; OLMo-1B used AdamW due to instabilities. The paper does not verify that ablation conclusions transfer across optimizers. Optimizer choice can interact with data quality—for instance, AdamW's adaptive learning rates might be more or less sensitive to noisy/token-corrupted training data than LionW's sign-based updates.

Claim 3: "Models trained on Dolma are competitive with similarly-sized open models."

What the experiments demonstrate: Table 2 shows that OLMo-1B (60.3 average) slightly outperforms TinyLlama 1.1B (59.4) and substantially outperforms Pythia 1.1B (54.5), while underperforming StableLM2 1.6B (66.5). The comparison with TinyLlama is the most valid, since both were trained on roughly 3T tokens (OLMo-1B on ~3.1T in one epoch, TinyLlama on ~3T). The per-task breakdown shows OLMo-1B winning on 4 tasks, losing on 3, and tying on 1.

What the experiments do NOT demonstrate: The comparison has significant confounds that prevent attributing performance differences to corpus quality:

TinyLlama's training data is undisclosed. TinyLlama's paper (or the version available at the time of this manuscript) does not fully specify its training corpus composition, so we cannot know whether the performance difference reflects Dolma's data quality versus TinyLlama's data choices, or simply reflects different architectural decisions, tokenizer choices, or training hyperparameters.
Pythia was trained on significantly fewer tokens (~300B vs. ~3T for OLMo-1B). The paper acknowledges this (Table 2 caption: "Pythia was trained on nearly 10 times fewer tokens"), which makes the comparison largely uninformative—we expect more training data to improve performance regardless of corpus quality. The Pythia comparison demonstrates that Dolma's scale enables longer training, not that Dolma's data is better per token.
StableLM2's comparison is confounded by model size, training tokens, and data composition. At 1.6B parameters and 2T tokens × 2 epochs = 4T tokens total, StableLM2 has advantages in all three dimensions, making its higher average accuracy unsurprising and uninformative about Dolma's relative quality.
The 8-task suite is narrow. All tasks are multiple-choice English question answering. A model that performs well on these specific tasks might not generalize to code generation, multilingual tasks, long-form generation, or dialogue. The paper's domain-fit analysis (Figure 5) partially addresses this by showing Dolma models fit diverse textual domains, but that analysis is limited to perplexity—it doesn't show that the model can effectively generate or manipulate text in those domains.

Missing comparison: What if Dolma were trained with the same data but different curation decisions? The most informative test of "the Dolma corpus is high-quality" would be to train OLMo-1B on a variant of Dolma with a specific curation decision reversed (e.g., High Threshold toxicity instead of Low Threshold, No paragraph dedup, Atomic Content instead of Partial Threads) and compare. This would isolate the effect of the curation decisions that the ablations validated at 1.2B/150B scales but at the final 1.2B/3.1T scale. The paper does not perform such a validation, meaning the bridge between "this filtering decision helps at 150B tokens" and "this filtering decision was correct for the released corpus" is an extrapolation, not a demonstration.

Claim 4: "The ablation methodology provides evidence-backed decisions that resolve open questions in data curation."

What the experiments demonstrate: The ablations do provide concrete answers to specific curation questions:

C4 NoPunc + Gopher All is better than either alone (Figure 1).
Low-threshold toxicity filtering improves quality despite removing more data (Figure 2), though the released corpus uses High Threshold for scale reasons.
Filters compound rather than saturate (Figure 3).
Atomic Content outperforms thread-structured formats for Reddit (Figure 4).
Even small amounts of code (5%) substantially improve reasoning capabilities (Table 3).
Model-based and heuristic quality filters are orthogonal (implicitly, via the CCNet bucket distribution finding).

What the experiments do NOT demonstrate: The paper does not investigate several curation questions that a truly comprehensive evidence-backed approach would address:

Interaction effects between curation decisions. All ablations vary one factor at a time. There are no 2×2 factorial experiments (e.g., Atomic Content × Toxicity Threshold) that would reveal whether the optimal toxicity threshold depends on thread format. The filter correlation analysis (Figure 9) measures output overlap, not interaction effects on downstream model performance. It's possible, for instance, that the benefit of toxicity filtering is larger for thread-structured formats (where toxic comments are juxtaposed with benign ones in the same document) than for atomic format (where toxic comments are isolated documents that can be removed without affecting neighbor context).
Scaling behavior of filtering benefits. The paper observes that filters compound at 150B tokens, but does not test whether the compounding continues, saturates, or reverses at larger token counts. This is especially important for the toxicity filter: if toxic content primarily causes problems early in training (when the model is learning basic language patterns) but becomes irrelevant later, the optimal threshold might depend on training duration, and the Low Threshold's advantage at 150B tokens might not hold at 3T tokens.
Optimal filter ordering. The pipeline order (URL dedup → quality → content → paragraph dedup) is specified but not ablated. There's a computational argument for ordering (cheap URL dedup first to reduce downstream cost), but there's no evidence that this ordering is optimal for model quality. If, for example, quality filtering removes some documents that then reappear via paragraph dedup, the dedup stage would be less effective than if it preceded quality filtering.
Alternative toxicity filtering approaches. The paper uses custom FastText classifiers trained on Jigsaw, but does not compare against alternatives like Perspective API, HateBERT, or lexicon-based approaches. The choice of FastText is justified by throughput, but the paper doesn't quantify the throughput-quality tradeoff (i.e., how much quality would a slower but more accurate classifier provide?).
Threshold sensitivity curves. The toxicity ablation tests only two thresholds (0.0004 and 0.4). A more thorough analysis would sweep multiple thresholds and identify a performance-maximizing point (or a Pareto frontier trading off data retention against model quality). The choice of exactly 0.4 as "High" is not justified beyond the bimodal score distribution observation.
Reproducibility of the exact dataset. The paper provides the Dolma Toolkit and configuration files, which should enable deterministic reproduction given the same raw inputs. However, the paper does not demonstrate that reproduction is actually achievable by an independent group. Reproducing the full 3T-token corpus would require the same Common Crawl snapshots, GitHub repository collection, Reddit dumps, and other sources at the same timestamps—some of which (Reddit Pushshift dumps) are no longer officially distributed. This means Dolma as a specific artifact is not fully reproducible, only Dolma as a pipeline applied to future data snapshots.

Overall Assessment

The experimental analysis provides solid evidence for the paper's central methodological claim—that data curation decisions can and should be validated through controlled ablation experiments. The ablations are well-designed for their purpose, the results are presented transparently (with full learning curves for all tasks in the appendix), and the conclusions drawn are appropriately cautious. The paper does not overclaim: it acknowledges the scale limitation of the ablations, the tradeoff that forced adoption of High Threshold toxicity filtering, and the narrowing of evaluation to English-only multiple-choice tasks.

The strongest experimental contributions are the findings that survive robustness checks and align with prior work: the benefit of quality filtering, deduplication, and code inclusion are consistent with existing literature, and the ablation methodology provides clean confirmation within a unified experimental framework. The domain-fit analysis (Figure 5) is the most informative result for the paper's central value proposition—it demonstrates that a multi-source corpus at trillion-token scale can maintain the domain diversity previously associated with smaller, hand-curated corpora like Pile.

The weakest link is the extrapolation from 1.2B/150B-token ablations to 1.2B/3.1T-token OLMo-1B training. The paper provides no direct evidence that the filtering decisions validated at the smaller scale produce the expected benefits at full scale. The OLMo-1B evaluation (Table 2) shows competitive performance, but this could reflect the corpus's scale and source diversity rather than the specific filtering choices—a less-aggressively-filtered version of Dolma might perform equally well or better at 3.1T tokens, and the paper provides no evidence to distinguish these possibilities. The most important missing experiment is a full-scale validation: training OLMo-1B on a minimally-filtered version of Dolma and comparing against the fully-filtered version. This would close the loop between the 150B-token ablation findings and the claim that the released corpus's specific curation choices are optimal.

6. Limitations and Trade-offs

The Ablation Scale Gap: Curation Decisions Are Validated Only at 1.2B Parameters and 150B Tokens

The assumption or constraint. The paper's entire evidence-backed curation methodology rests on data ablations conducted at a single, fixed scale: 1.2 billion parameter models trained for 150 billion tokens (Section 4.2, Appendix D.1). Every filtering decision—quality heuristics, toxicity thresholds, deduplication strategy, thread formatting, code inclusion—was validated by comparing learning curves from these small, early-stopped training runs. The assumption is that the relative ordering of curation choices at 1.2B/150B tokens transfers to larger models and longer training. The paper explicitly acknowledges this limitation in its Limitations section:

"while many language models are in the 7 billion to 70 billion parameters range, we train 1 billion parameter models... This choice was dictated by the need to efficiently iterate over many possible configurations, but it might result in design decisions that are not relevant at larger model sizes."

The consequence. If curation decisions do not scale monotonically with model size, some of Dolma's filtering choices could be suboptimal for the 7B, 13B, or 70B models that practitioners would actually train on this corpus. The direction of potential failure is not obvious a priori. For example, aggressive toxicity filtering that helps at 1.2B (by removing noisy patterns that small models struggle to ignore) might hurt at 7B (by removing useful signal that larger models could learn to contextualize). Conversely, quality filtering that provides modest gains at 1.2B might provide compounding benefits at larger scales as the model better leverages clean data. The paper provides no evidence about which, if any, of its validated decisions would flip at larger scales.

This is not merely a theoretical concern. The OLMo family (Groeneveld et al., 2024) includes 7B models trained on Dolma, so a direct empirical test of scaling behavior exists in principle—but the paper does not report whether the 7B model's training would have benefited from different curation decisions than those chosen by the 1.2B ablations. The ablation methodology, while internally rigorous at its chosen scale, provides a lower-bound guarantee only: it tells us which decisions are better at 1.2B/150B, but cannot tell us whether those decisions remain optimal or even beneficial at production scales.

What evidence exists in the paper. The paper's own final model evaluation (Section 9, Table 2) demonstrates a related scaling inconsistency: OLMo-1B was trained with AdamW because "instabilities we found in the LionW optimizer" (Appendix D.4) forced an optimizer switch that did not occur during the ablations (which used LionW). This means the 150B-token ablations and the 3.1T-token final training used different optimizers, introducing a confound that the paper does not control for. If LionW and AdamW interact differently with data quality (e.g., AdamW's adaptive per-parameter learning rates might be more or less sensitive to noisy training examples than LionW's sign-based updates), then ablation decisions validated under LionW might not hold for the AdamW-trained OLMo-1B. The paper does not run any ablation under AdamW to test this.

Mitigation status. The paper acknowledges this limitation explicitly but does not attempt to mitigate it. The authors describe the 1.2B/150B protocol as a necessary compromise driven by computational constraints: training full-scale models for every ablation would be "prohibitively expensive." They frame the ablation results as the best available evidence given resource constraints, and leave validation at larger scales to future work or to downstream model developers: "We expect downstream model developers to scrutinize Dolma before using it to train their language models." This is a reasonable position but means that the central claim—that Dolma's curation decisions are "evidence-backed"—is true only within the specific experimental regime where the evidence was collected. Whether the evidence transfers to the regime where Dolma will actually be used remains an open question.

The Difficulty Estimation Cost Problem: Reproducing the Ablation Methodology Requires Massive Compute

The assumption or constraint. The paper's core methodological contribution—running controlled data ablations to validate curation decisions—is presented as a template for "further research in large-scale data curation" (Section 1). However, the paper does not account for the computational cost of this methodology in its headline framing. Each ablation requires training a 1.2B parameter model for 150B tokens on 64 AMD Instinct MI250X accelerators, and the paper presents at least 8 distinct ablation experiments (quality filtering, toxicity filtering, pipeline stacking, thread formatting, code rules, PII handling, deduplication, toxicity for Reddit). The total compute for these ablations is equivalent to training multiple 1B-scale models to partial completion—a cost that few academic research groups can afford.

The consequence. The paper inadvertently creates a practical barrier: the very methodology it advocates—and that distinguishes Dolma from prior corpora—is too expensive for most potential adopters to replicate. A research group wanting to curate a new dataset (e.g., for a non-English language, a specialized domain, or a different scale regime) cannot simply adopt Dolma's curation pipeline; they would need to run their own ablation experiments to validate that the filtering decisions transfer to their setting, and those experiments would require the same order of magnitude of compute that the paper consumed. The paper provides no guidance on how to reduce ablation costs—no investigation of whether smaller models (e.g., 300M parameters) produce similar rankings, whether fewer training tokens suffice, or whether certain curation decisions are more sensitive to compute scale than others.

This is analogous to the "difficulty estimation cost" problem in the compute-optimal inference paper (Section 3.2 of that work), where generating 2048 samples per prompt to estimate difficulty cost more compute than the test-time budget being optimized. In both cases, the experimental apparatus that produces the headline gains consumes resources that are not included in the headline numbers. For Dolma, the "headline number" is the corpus quality, but producing that quality required ablation experiments that are externalized from the corpus's total cost.

What evidence exists in the paper. The paper reports throughput for the data processing pipeline (122 CPU hours per TB, Section 4.1) and specifies the hardware configuration for ablations (64 MI250X accelerators, Appendix D.1), but does not report the total GPU-hours or FLOPs consumed by the ablation experiments. The paper also does not report the transferability of ablation results across computational budgets. The finding that even small amounts of in-domain data achieve good perplexity (Section 9.2, Figure 5: Reference+ mix with 4.9% reference material nearly matches Gopher-like mix with 24.2%) hints that some curation decisions might reach diminishing returns quickly and thus could be validated with less compute, but this is not systematically investigated.

Mitigation status. The paper does not directly address this limitation, but three aspects partially mitigate it. First, the release of the Dolma Toolkit (Section 4.1) means that downstream users can apply Dolma's filtering pipeline to new data without re-running the ablations—the tools and configurations are provided, so users can adopt the same filtering decisions validated in the paper for their own similar corpora without replicating the experimental work. Second, the paper's detailed documentation of which decisions were empirically tested and which followed precedent (Section 3) provides transparency about where the ablation methodology was applied, allowing users to assess whether untested decisions apply to their use case. Third, the paper implicitly suggests that certain decisions (quality filtering, deduplication) are robust across sources and scales based on consistency with prior work, which could justify adopting them without re-ablation. However, none of these mitigations addresses the core issue: a research group wanting to curate a truly novel corpus (different languages, different domains, different scale) with the same evidence-backed rigor would face the same computational burden.

English-Only Scope Is a Fundamental Capability Boundary

The assumption or constraint. Dolma is explicitly and deliberately English-only. The paper states this clearly in its design principles (Section 3):

"this also means scoping Dolma to English-only text to better leverage known curation practices and maximize generalizability of scientific work on Dolma to existing language models."

The paper acknowledges the limitation in its Limitations section: "Dolma reinforces the expectation of English being the 'default' language for NLP." All language filtering uses a FastText classifier with an English score threshold of 0.5 (Section 5.1), meaning non-English content is actively removed.

The consequence. Any model trained primarily on Dolma will have essentially no capability in non-English languages. The paper's language audit (Appendix G) shows that the 0.5 threshold is conservative—it correctly identifies all English documents in the International Corpus of English—but the WIMBD language distribution analysis (Figure 10c) reveals that after filtering, 0.86% of remaining documents have "un" (unidentified) language and only 0.06% are identified as Chinese (the largest remaining language). A model trained on Dolma would see Chinese text in roughly 0.06% of its training data—one document per ~1,667 documents—which is far below any meaningful exposure threshold. For tasks requiring multilingual understanding, code-switching, or cross-lingual transfer, a Dolma-trained model would perform near chance.

This limitation is especially consequential given that Dolma is positioned as a research corpus for studying "how training data impacts model capabilities and limitations" (Section 1). A researcher using Dolma to investigate, say, "how does pretraining data composition affect factual knowledge recall?" would only be able to study this question for English factual knowledge. The finding would not necessarily transfer to multilingual settings, where different data sources, filtering challenges, and tokenization issues arise (the tokenization analysis in Appendix F shows that even within Dolma, fertility varies 2× between code and text—multilingual data would introduce substantially larger tokenization disparities that Dolma's English-only design does not address).

What evidence exists in the paper. The language distribution analysis (Figure 10c) directly quantifies the near-total absence of non-English content. The tokenization analysis (Appendix F) provides indirect evidence of what adding non-English data would entail: if code (with its non-standard whitespace patterns) raises fertility from ~1.2 to 2.45, languages with non-Latin scripts or different morphological structures would likely show even larger fertility disparities, changing the effective token budget for non-English content. The paper's contamination analysis (Appendix L) shows that 100% of Winograd Schema Challenge, SICK, and other English benchmarks are contaminated in Dolma—a multilingual corpus would face an even larger decontamination challenge, as benchmark datasets exist in dozens of languages.

Mitigation status. The paper partially acknowledges this limitation and gestures toward future work: "Recognizing that this focus reinforces the assumption of English as the 'default' language, we hope to expand Dolma to more languages in the future. We release our data curation tools to support such efforts." The release of the Dolma Toolkit is the primary mitigation—it provides the technical infrastructure for multilingual curation—but this shifts the burden to the community. The paper provides no evidence about whether its specific filtering decisions (FastText language ID thresholds, Gopher quality heuristics, toxicity classifiers trained on English Jigsaw data) would transfer to non-English languages. The PII regex patterns (Section 5.3), for instance, are designed for English-format email addresses, US-format phone numbers, and IPv4 addresses; they would miss region-specific PII formats (e.g., national ID numbers, non-US phone formats) that are important for non-English corpora. The toxicity classifiers, trained on English forum comments, would fail entirely on non-English toxic content and might exhibit cross-lingual biases that the dialectal bias check (Appendix H, Figure 8) could not detect.

Hard Content Thresholds Create a Fixed Curation Regime That Cannot Adapt to Downstream Needs

The assumption or constraint. Dolma applies fixed, hard thresholds across all its filtering stages—and in one notable case, explicitly chooses a suboptimal threshold for scale reasons—producing a single canonical version of the corpus. The toxicity filter uses a High Threshold of $\tau = 0.4$ despite Low Threshold ( $\tau = 0.0004$ ) performing better in ablations, because the Low Threshold would have reduced the corpus below the 2–3T token target (Section 5.3). The quality filters are applied uniformly across all CCNet perplexity buckets (high, medium, low) even though the optimal filtering strategy might differ by bucket. The Reddit pipeline uses fixed minimum length thresholds (500 characters for comments, 400 for submissions) and a fixed vote threshold (3 votes), without investigating whether these thresholds interact with subreddit topic, comment sentiment, or other content characteristics.

The paper explicitly frames this as a tradeoff:

"Weighing the tradeoff between dataset scale ('High') and performance maximization ('Low'), we adopt the more permissive 'High' threshold to ensure we meet our minimum token count requirement."

The authors acknowledge this as temporary: "In future versions of Dolma, we will start with more shards of Common Crawl and adopt stricter filter thresholds."

The consequence. The released Dolma corpus represents a single point on a multi-dimensional filtering Pareto frontier—and not necessarily the optimal point even by the paper's own ablation evidence. A downstream model trainer who values quality over quantity (e.g., training a small model for a specific downstream task) might prefer the Low Threshold toxicity filtering, which the paper's own Figure 2 shows produces better HellaSwag accuracy at 150B tokens. A trainer who values quantity over quality (e.g., training a very large model where data volume is the primary constraint) might prefer even more permissive filtering than the High Threshold. A trainer building a code-focused model might want more aggressive code-specific filtering (beyond RedPajama v1 + StarCoder rules). Dolma's single release provides none of these options.

This is a practical limitation because the optimal filtering strategy is likely task-dependent and scale-dependent, but Dolma provides no mechanism for users to configure filtering post-hoc. The paper's own mixing experiments (Appendix M) demonstrate that source composition affects domain-specific perplexity—a model for code generation needs different source proportions than a model for scientific QA. By analogy, a model for different purposes likely needs different quality/toxicity/PII filtering thresholds, but Dolma's hard-filtered output cannot be "unfiltered" to recover removed content. A user who wants Low Threshold toxicity filtering must obtain the raw CCNet output and re-run the entire pipeline—effectively starting from scratch.

What evidence exists in the paper. Figure 2 directly shows the performance gap between Low and High thresholds: the Low Threshold achieves ~0.48 HellaSwag accuracy at 150B tokens vs. ~0.46 for High Threshold, a consistent gap of ~2 percentage points. The filter correlation analysis (Figure 9) shows that filters are largely orthogonal, meaning the choice of toxicity threshold does not affect which documents are removed by Gopher rules, deduplication, or PII filtering. Thus, the performance gap between Low and High thresholds is attributable specifically to the toxicity filter, not confounded with other filtering stages. The paper's mixing experiments (Appendix M) demonstrate that even small variations in source composition (e.g., 5.4% vs. 13.8% code) produce measurable perplexity differences on domain-specific evaluations—suggesting that similar sensitivity would apply to filter thresholds if they were configurable.

Mitigation status. The paper provides partial mitigation through the Dolma Toolkit release: users can reconstruct a differently-filtered version by re-running the pipeline with modified thresholds, provided they have access to the same raw source data. However, this mitigation has two critical gaps. First, some raw data sources (Reddit Pushshift dumps) are no longer officially distributed, making exact reproduction impossible. Second, even with data access, re-running the full pipeline for three trillion tokens requires the same 122 CPU hours per terabyte throughput reported in Section 4.1—a substantial computational investment that undermines the convenience of using a pre-built corpus. The paper's suggestion that future versions will adopt stricter thresholds indicates the authors view this as a fixable limitation rather than a fundamental design choice, but the current release remains locked to a single filter configuration that is explicitly acknowledged as suboptimal on the performance-maximization axis.

Reproducibility Is Compromised by Unavailable Source Data

The assumption or constraint. Dolma is released as a processed corpus, but the paper also claims that the Dolma Toolkit and documentation enable "reproduction of our work" (Section 1). This assumes that the raw source data used to create Dolma remains accessible and stable. The Datasheet (Appendix N.3) lists specific acquisition timestamps for each source: Common Crawl snapshots from May 2020 to June 2023, Reddit Pushshift dumps from June 2005 to March 2023 (acquired March–May 2023), GitHub Stack from March 2023, Project Gutenberg from April 2023, Wikipedia/Wikibooks from March 2023, and Semantic Scholar peS2o V2 from June 2023.

The consequence. At least one major source is no longer available in its original form: Reddit Pushshift dumps are no longer distributed following changes to the Reddit API terms (the Datasheet notes this explicitly: "Pushshift no longer distributes this dataset due to changes to the Reddit API's terms. Unofficial copies of the data might be available through torrents and some public web archives."). This means that Dolma cannot be bit-for-bit reproduced from the documented pipeline applied to accessible raw data. The best an independent researcher can do is obtain approximate copies from unofficial archives, which may differ in content, completeness, or format from the original dumps the authors used.

This reproducibility gap undermines several of the paper's stated goals. Scientific replication—confirming that the same pipeline run on the same inputs produces the same outputs—becomes impossible for the Reddit component. Auditing—checking whether Dolma's filtering decisions are correctly implemented or biased in undocumented ways—cannot fully verify the Reddit subset because the unfiltered input is inaccessible. Extension—building a new corpus that adds or modifies Reddit processing—cannot guarantee comparability with the original Dolma Reddit subset because the starting data differs.

Even for sources that remain accessible, bit-for-bit reproduction is not guaranteed. Common Crawl snapshots should be stable on S3, but the paper acquired "only enough shards of Common Crawl to meet our target 2–3T token corpus size" (footnote 7, Section 5.1) without specifying exactly which shards. The GitHub Stack collection may evolve as repositories are added, removed, or relicensed. Project Gutenberg continuously adds new public-domain books. Wikipedia dumps are versioned but the specific March 2023 dump may not be indefinitely retained.

What evidence exists in the paper. The Datasheet (Appendix N.2) documents the acquisition parameters for each source, including timestamps and versions. The Datasheet also notes the Reddit distribution issue explicitly. The WIMBD-based data distribution analysis (Figure 10) provides aggregate statistics (domain distribution, date distribution, language distribution) that could be used to verify approximate reproduction—if a reproduced corpus has similar distributions, it is likely close to the original. However, approximate reproduction is not the same as exact reproduction, and the paper's contamination analysis (Appendix L) shows that even tiny contamination rates (≤0.001% of characters) can include entire benchmark test sets—so small differences between a reproduction and the original could have large evaluation consequences.

Mitigation status. The paper partially mitigates this through transparency: the Datasheet specifies which versions of each source were used, and the Dolma Toolkit provides the exact filtering code and configuration. For sources that remain accessible (Common Crawl, GitHub, Project Gutenberg, Wikipedia), this should enable functional reproduction—the pipeline will produce a dataset with very similar statistical properties to the original Dolma, even if some individual documents differ. The paper also distributes the processed corpus directly, so researchers who only need the final dataset (not the ability to re-derive it) face no reproduction barrier. For the Reddit subset specifically, the paper acknowledges the distribution issue but provides no solution—users who need the Reddit data must either trust the released processed version or track down unofficial archives. The paper's broader framing of Dolma as a "platform for research rather than a fixed artifact" (Section 3) partially reframes the goal: the value is in the tools and methodology, not in the specific bytes. But for researchers who want to verify, audit, or extend the exact corpus used to train OLMo models, the unavailable Reddit source data is an irrecoverable gap.

Quality and Toxicity Filters Encode Ideological Judgments That Are Not Validated for Bias

The assumption or constraint. The paper applies multiple content-based filters—Gopher quality heuristics, C4 punctuation rules, FastText toxicity classifiers—that make implicit judgments about what text is "good" or "bad" for language model training. The paper is self-aware about the normative nature of these judgments:

"The term 'quality filter,' while widely used in literature, does not appropriately describe the outcome of filtering a dataset. Quality might be perceived as a comment on the informativeness, comprehensiveness, or other characteristics valued by humans. However, the filters used in Dolma and other language models efforts select text according to criteria that are inherently ideological (Gururangan et al., 2022)."

"Similar to 'quality', there is no single definition for 'toxicity'. Rather, specific definitions vary depending on task (Vidgen and Derczynski, 2020) and dataset curators' social identities (Santy et al., 2023); annotators' beliefs also influence toxic language detection (Sap et al., 2021). Predicting toxicity remains challenging (Welbl et al., 2021; Markov et al., 2023), especially as existing methods have been shown to discriminate against minoritized groups (Xu et al., 2021)."

The consequence. Despite acknowledging these concerns, the paper validates its filtering decisions only on model performance metrics—downstream task accuracy and perplexity—and not on bias, fairness, representation, or content diversity outcomes. The Gopher quality rules, for instance, filter out documents where fewer than 80% of words contain an alphabetical character, where the median word length is less than 3 or greater than 10, or where certain common English function words ("the", "be", "to", "of", "and", "that", "have", "with") are absent. These rules implicitly define "quality" as conformity to formal written English prose, and would disproportionately filter texts in non-standard dialects (African American English, Chicano English, regional British dialects), poetry and experimental writing, technical documentation with non-prose formatting, and code-switched or translanguaged content.

The toxicity classifiers present an even sharper bias risk. The paper trains on the Jigsaw Toxic Comments dataset (cjadams et al., 2017), which is known to contain biases in its labeling—research has shown that comments in African American English are more likely to be labeled as toxic by classifiers trained on standard English datasets (Sap et al., 2021; Xu et al., 2021). The paper's bias audit (Appendix H, Figure 8) attempts to check for dialectal bias by comparing toxicity rates across country-specific subreddits and finding <5% difference. However, this audit uses country as a proxy for dialect—a crude measure that would miss within-country dialect variation (e.g., African American English vs. Standard American English within the US) and does not control for the possibility that some country-specific subreddits actually have different rates of toxic content. The audit also uses "any reasonable threshold (lying between 0.1 to 0.9)" to claim robustness, but this is exactly the range where the bimodal score distribution makes the threshold effectively meaningless—if all scores are near 0 or 1, the threshold only controls false positives vs. false negatives, and the 0.0004 and 0.4 thresholds tested in the ablation represent qualitatively different filtering regimes, not a continuous sensitivity curve.

What evidence exists in the paper. The bias audit (Appendix H, Figure 8) provides some evidence against gross dialectal bias, but it is methodologically limited: it assumes equal underlying toxicity rates across country subreddits, which is untestable without ground-truth labels, and it does not examine within-country biases. The filter correlation analysis (Figure 9) shows low correlations between quality filters and toxicity filters, suggesting they are capturing different signals, but does not examine whether the combination of filters systematically excludes content from particular communities, dialects, or viewpoints. The paper does not analyze the demographic or linguistic composition of filtered-versus-retained content—how much African American English, how much LGBTQ+ discourse, how much political speech from marginalized viewpoints, how much non-Western cultural reference text was removed by the combined effect of Gopher + C4 NoPunc + toxicity + PII + deduplication. The WIMBD-based data distribution analysis (Figure 10) provides aggregate domain and language statistics but does not break down by dialect, community, or content type in ways that would reveal bias.

The paper's downstream evaluation suite (Section 4.2) includes no bias or fairness benchmarks—no measures of stereotypical associations, no tests of dialect robustness, no analysis of representation in model outputs. The Paloma perplexity benchmark (Magnusson et al., 2023) includes some dialectal sources (Twitter AAE, ICE country-specific English varieties), and the domain-fit analysis (Figure 5, Appendix O.1) shows that Dolma-trained models achieve lower perplexity on these sources than single-source corpora—but lower perplexity on African American English could mean the model has learned to model it well, or that the model has been trained on enough AAE-like text to reproduce its patterns, including any stereotypical or harmful associations present in that text. The paper does not distinguish these possibilities.

Mitigation status. The paper acknowledges the ideological nature of filtering in its footnotes (Section 4.1, footnotes 4 and 5) and its Datasheet (Appendix N.4, footnote 34), but does not address the consequences beyond acknowledgment. The ethical considerations section mentions that "we engaged with legal and ethics experts early in the project and evaluated data design decisions based on their feedback on a case-by-case basis" and that "we will provide tools to request data removal," but these are governance processes, not empirical validations. The paper does not perform any bias or fairness evaluation of models trained on Dolma, does not audit the filtering pipeline for disparate impact on specific communities or dialects, and does not provide users with the tools to perform such audits (e.g., by releasing the unfiltered data alongside the filtered version so that removal patterns can be analyzed). The paper's framing of future work—"we will start with more shards of Common Crawl and adopt stricter filter thresholds"—implies that stricter filtering is unambiguously better without addressing the possibility that stricter filtering might amplify representation biases. A practitioner deploying Dolma for applications where dialectal fairness, cultural representation, or viewpoint diversity matters (e.g., educational technology, public-facing chatbots, content moderation tools) would need to perform their own bias audits that the paper does not provide.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around open language model research from a focus on model weights as the primary artifact of openness toward a view where training data transparency is equally essential for scientific progress. Before Dolma, the prevailing norm—even among nominally "open" model releases—was to distribute model weights while withholding detailed information about what those models were trained on. Llama 2, Mistral, Yi, and Qwen all release weights but not data; GPT-4 and PaLM 2 release neither. Dolma establishes a counterexample: a three-trillion-token corpus with documented provenance, validated curation decisions, and open-source processing tools, demonstrating that the technical and legal barriers to transparent data release are surmountable at scale.

The practical implication is a re-centering of the reproducibility standard. Prior to Dolma, a model release was considered "open" if weights were available—the data was treated as a separate, optional concern. After Dolma, a model release without accompanying training data can be compared against a concrete alternative that provides both. This does not guarantee that all future model developers will release their data, but it changes the baseline against which openness claims are evaluated. When Meta releases Llama 3 without training data, the response is no longer "well, no one releases training data" but rather "Dolma managed to release three trillion tokens with full documentation—why can't you?" The paper's detailed catalog of what is not disclosed by major models (Appendix C) serves as an implicit benchmark that future releases will be measured against.

The paper also introduces a methodological template for evidence-backed data curation that transforms how the research community should approach pretraining corpus design. Before Dolma, data curation decisions—which quality heuristics to apply, what toxicity threshold to use, whether to deduplicate at the paragraph or document level—were made primarily on precedent or intuition. The paper's ablation apparatus (Section 4.2) demonstrates that these decisions can be treated as testable hypotheses: train controlled model variants, measure downstream performance, and let the evidence determine the optimal configuration. This is most striking in the finding that C4 NoPunc, a single heuristic, outperforms both the full C4 filter set and the full Gopher filter set individually (Figure 1)—a result that no amount of precedent-based reasoning would have predicted, and that reveals how inherited curation recipes encode assumptions that do not survive empirical scrutiny.

This methodology is likely to be the paper's most lasting contribution because it is transferable to any data curation effort, regardless of language, domain, or scale. The specific findings—that C4 NoPunc + Gopher All is the best quality filter combination, that low-threshold toxicity filtering improves model quality despite reducing data volume—may or may not generalize beyond English web text at the 1.2B-parameter scale. But the approach of running controlled ablations to validate curation decisions generalizes trivially, and the paper provides both a protocol (1.2B model, 150B tokens, 8-task suite, Paloma perplexity) and open-source tools (the Dolma Toolkit) that lower the barrier to adoption. A research group curating a corpus for Arabic scientific text, or Japanese legal documents, or multilingual code, can adopt the same methodology even if none of Dolma's specific filter thresholds apply to their domain.

The paper also resolves a longstanding tension in data curation between model-based quality filtering and heuristic-based quality filtering. The dominant assumption—visible in the design of CCNet, RefinedWeb, and other web-scale curation pipelines—was that KenLM perplexity scores (or similar language model-based quality estimates) capture a comprehensive notion of "text quality" that subsumes simpler heuristics. The finding that heuristic filters (Gopher, C4 NoPunc) do not alter the proportion of data in CCNet's high/medium/low perplexity buckets (Section 5.2) empirically refutes this assumption. Heuristic and perplexity-based filters are measuring orthogonal signals, and hybrid pipelines that combine both types outperform either alone. This finding explains why different prior efforts reached different conclusions about optimal filtering (they were measuring different quality axes) and provides a principled justification for the multi-stage filtering approach that Dolma uses.

Finally, the paper identifies—though does not fully address—a contamination bottleneck that threatens the validity of code-augmented language model evaluation. The finding that GitHub-derived code datasets systematically contain copies of popular NLP benchmarks (Figure 11, Appendix L), with six datasets showing 100% contamination and four more exceeding 90%, reveals that the standard practice of training on code alongside text may be leaking evaluation answers through an underappreciated pathway. This matters because the field increasingly trains models on mixtures that include code (Llama 2, GPT-4, PaLM 2 all use code data), and performance on reasoning benchmarks is often attributed to genuine capability improvements from code exposure. If some fraction of that improvement reflects memorized benchmark answers rather than transferable reasoning skill, the conclusions of an entire line of research become suspect. The paper does not resolve this—it simply excludes contaminated benchmarks—but by quantifying the problem and identifying the mechanism (public repositories distributing benchmark datasets as test fixtures), it provides the evidence needed for the field to develop mitigation strategies.

Follow-Up Research This Work Enables

Scaling behavior of curation decisions: Do the ablation results at 1.2B/150B tokens transfer to larger models? The paper's central methodological limitation is that all curation decisions were validated on 1.2B-parameter models trained for only 150B tokens—less than 5% of the full OLMo-1B training budget. A high-priority follow-up would train model pairs at 7B parameters (the scale of OLMo-7B, which was also trained on Dolma) on dataset variants that differ in a single curation choice validated by the 1.2B ablations, and check whether the relative ordering holds. For example: train a 7B model on Dolma with the Low Threshold ( $\tau = 0.0004$ ) toxicity filter versus the High Threshold ( $\tau = 0.4$ ) for at least 500B tokens, and measure whether the Low Threshold advantage observed in Figure 2 persists, shrinks, or reverses. If it reverses (i.e., the High Threshold becomes better at scale because larger models learn to ignore noisy toxic patterns), then Dolma's decision to use the High Threshold—made for scale reasons rather than performance reasons—would be retrospectively validated. If the Low Threshold advantage persists or grows, it would mean the released Dolma corpus is materially suboptimal, and future releases should prioritize acquiring enough raw data to apply strict filtering while maintaining the target token count. This experiment requires substantial compute but is directly actionable given the existing OLMo training infrastructure.

Interaction effects between filtering stages: Are the "orthogonal filters" finding and the "compounding benefits" finding robust to the ordering and combination of filters? The paper's filter correlation analysis (Figure 9) establishes that filters tag mostly non-overlapping documents, and the pipeline stacking ablation (Figure 3) shows that adding more stages improves performance. However, the paper never tests whether the order of filtering matters, or whether there are interaction effects where the benefit of one filter depends on another filter's threshold. A factorial ablation varying two curation decisions simultaneously—for example, Atomic Content vs. Partial Threads for Reddit × Low Threshold vs. High Threshold toxicity filtering—would reveal whether the thread format decision interacts with content filtering choices. If Atomic Content benefits more from aggressive toxicity filtering (because toxic comments appear as isolated documents that can be cleanly removed) than Partial Threads (where toxicity removal might fragment multi-turn dialogues), this would justify per-source filter configurations rather than the uniform thresholds currently applied. A strong experiment would also test filter ordering: run the web pipeline with paragraph deduplication before quality filtering and compare against the current ordering, testing whether the current design (URL dedup → quality → content → paragraph dedup) is optimal or merely computationally convenient.

Cost-efficient difficulty estimation for curation: Can the ablation protocol be compressed to make evidence-backed curation accessible to lower-resource groups? The paper's ablation methodology consumes substantial compute—each 1.2B/150B-token training run requires 64 MI250X accelerators—placing it out of reach for many academic labs. A systematic study could investigate how much the ablation protocol can be compressed without losing decision-relevant signal. Specific dimensions to vary: (a) model size (do 300M-parameter models produce the same ordering of curation decisions as 1.2B models?), (b) training duration (do the relative rankings of curation decisions stabilize at 50B tokens, or are full 150B tokens needed?), (c) evaluation suite size (can HellaSwag alone serve as a reliable proxy for the full 8-task suite, or does the optimal curation decision differ across tasks?). If, for example, 300M-parameter models trained for 50B tokens produce the same curation rankings as 1.2B models trained for 150B tokens for 90% of decisions, the cost of evidence-backed curation drops by roughly an order of magnitude, making the methodology accessible to groups without large-scale GPU access. The paper's data provides some indirect evidence: the learning curves for different filtering conditions (Figures 1–4) often show stable relative ordering by 50B tokens, suggesting early stabilization, but this was not systematically analyzed. A targeted study would directly measure the correlation between ablation rankings at different compute budgets.

Cross-lingual and cross-domain transfer of curation heuristics: Do English-validated filters work for other languages and domains? The paper's English-only scope means that every curation decision—FastText language ID thresholds, Gopher quality rules (which check for English function words), C4 punctuation rules (which assume English punctuation conventions), toxicity classifiers trained on English Jigsaw data, and PII regex patterns designed for US/English-format identifiers—was optimized for a single language. A multi-language or multi-domain Dolma extension would need to answer: which of these heuristics are language-universal (e.g., removing documents with excessive n-gram repetition, or filtering by median word length) and which are English-specific (e.g., the required function words list, the punctuation-ending rule for languages with different sentence-boundary conventions)? A controlled study would replicate the quality filter ablation for a non-English language (e.g., Arabic, Japanese, or Hindi), comparing English-validated heuristics against language-adapted heuristics (e.g., replacing "the, be, to, of, and" with high-frequency function words in the target language) and measuring whether the adapted version improves over the direct transfer. If language-adapted heuristics show consistent gains across multiple languages, this would motivate building language-specific filter configurations into the Dolma Toolkit, making it genuinely multilingual. If the gains are negligible, it would suggest that the surface-level patterns captured by Gopher rules are largely language-independent, simplifying multilingual curation.

Verifier over-optimization analog for data filtering: Does aggressive filtering hurt model robustness even when it improves aggregate metrics? The paper's toxicity filter ablation (Figure 2) shows that Low Threshold filtering improves HellaSwag accuracy while removing 29–35% of sentences. But what else is being removed? A follow-up study would train models on the Low Threshold and High Threshold filtered corpora and evaluate them on a broader set of capabilities that Dolma's 8-task suite does not cover: dialectal robustness (does the Low Threshold model perform worse on African American English or other non-standard varieties because its training data was more aggressively filtered for "toxic" language that correlates with those dialects?), viewpoint diversity (does the Low Threshold model show narrower or more stereotyped outputs on tasks requiring reasoning about social or political topics?), and content generation safety (does the more aggressively filtered corpus produce a model that is actually less likely to generate toxic text when prompted, or does the filtering merely shift the distribution of generated toxicity?). The paper's bias audit (Appendix H, Figure 8) provides preliminary evidence that the toxicity classifier does not exhibit gross dialectal bias across country subreddits, but this is far from sufficient to rule out within-country or within-community biases. A rigorous study would curate evaluation datasets stratified by dialect, register, and community, and test whether models trained with different filtering thresholds show systematic performance disparities. If aggressive filtering improves aggregate metrics but widens performance gaps between standard and non-standard language varieties, the community would face a direct tradeoff between aggregate model quality and linguistic fairness—a tradeoff that should be explicitly characterized rather than implicitly encoded in filtering thresholds.

Long-range effects of code contamination: Does GitHub-derived benchmark leakage actually inflate downstream task performance in realistic training regimes? The paper's contamination analysis (Appendix L) identifies six datasets with 100% contamination in Dolma (primarily from the code subset) and excludes them from evaluation. But this is a detection study, not an intervention study—it does not measure how much the contaminated data would have affected model performance if it had been included. A controlled experiment would take a Dolma variant with all contaminated documents removed (the clean baseline) and a variant with contaminated documents deliberately left in, train matching models on both, and measure the performance difference on the contaminated benchmarks. This would quantify the "contamination inflation factor"—how much of the apparent performance on a contaminated benchmark is due to genuine capability versus memorized exposure. If the inflation factor is small (e.g., 1–2 percentage points on a 60% accuracy benchmark), then the field's concern about contamination may be overstated, and excluding contaminated benchmarks represents an unnecessary constraint. If the inflation factor is large (e.g., 10+ percentage points), then essentially all code-augmented models trained on GitHub data are reporting inflated numbers on popular benchmarks, and the contamination problem is substantially worse than currently appreciated. This experiment is feasible with the existing Dolma infrastructure: the decontamination metadata already identifies which documents are contaminated, so constructing the contaminated-vs-clean dataset variants is a simple filtering operation. The main cost is training the comparison models.

Practical Applications and Downstream Use Cases

Training data attribution and memorization research. The paper explicitly positions Dolma as enabling the research community to "study (and scrutinize) language models being developed today, even those developed behind closed doors" (Section 3). The most immediate application is training data attribution (TDA) research: given a model trained on Dolma, researchers can trace model outputs back to specific training documents with known provenance, because the entire corpus is open and documented. Prior TDA work (Grosse et al., 2023; Hammoudeh and Lowd, 2022) has been limited by the opacity of training data—you can estimate influence functions, but you can't verify whether they're identifying the correct training examples because the actual training corpus is unknown. With Dolma-trained models, TDA methods can be ground-truthed: if an influence function claims that document X was highly influential for a particular model output, researchers can ablate document X from the training set, retrain, and measure the actual causal effect. This enables the first systematic validation of TDA methods at realistic model scales. Similarly, memorization studies (Carlini et al., 2022) can use Dolma to precisely measure what fraction of training documents are extractable from the trained model, under what conditions, and with what dependence on deduplication strategy. The paper's deduplication ablation (Figure 3) shows that deduplication improves downstream task performance, but the memorization implications—does paragraph-level dedup reduce extractable memorization of boilerplate, or of rare factual content?—remain unexplored and are directly testable with the released corpus.

Pre-training data audits by third parties. The paper's detailed documentation of filtering decisions (Section 5–8, Appendix N) and the availability of the full corpus enable external audits that were previously impossible for any corpus at this scale. An auditor can test specific hypotheses about corpus composition: does the toxicity filtering disproportionately remove content from LGBTQ+ communities, as prior work on toxicity classifiers would predict (Xu et al., 2021)? Does the Gopher quality filter systematically exclude certain dialects or registers, as its function-word requirements suggest? Does the subreddit blocklist (26,123 banned/NSFW subreddits) contain false positives—communities that were banned for reasons other than content quality, or that were added to blocklists based on outdated or incorrect information? These questions cannot be answered by the paper's aggregate statistics, but they can be investigated by anyone with access to Dolma and the motivation to perform a targeted audit. The paper's WIMBD-based data distribution analysis (Figure 10) provides a template for such audits, showing how to query domain distributions, date distributions, and language distributions from the corpus metadata. A concrete audit project would extend this to demographic or dialectal representation: extract all documents from country-specific or community-specific subreddits, run the full filtering pipeline on them, and measure the subreddit-level removal rate. If certain communities show systematically higher removal rates that are not explained by observable content differences, the filtering pipeline has a bias that should be documented and, ideally, corrected.

Self-improvement data generation pipelines. The paper's finding that even small amounts of high-quality non-web data substantially improve domain fit (Section 9.2, Figure 5: Reference+ mix with 4.9% reference material nearly matches Gopher-like mix with 24.2% on academic paper perplexity) has direct implications for self-improvement pipelines. If a team is using a model to generate synthetic training data for fine-tuning (as in STaR, ReSTᵉᵐ, or instruction-tuning data generation), the Dolma ablation methodology provides guidance on how to mix generated data with curated sources to maximize domain-specific performance without drowning the signal in generic web text. Concretely: a group generating synthetic scientific reasoning data could combine it with the peS2o academic paper subset (which Dolma already provides in cleaned, deduplicated form), using the mixing ratios from Appendix M as a starting point, and validate the mixture through their own small-scale ablations following the paper's protocol. The open-source Dolma Toolkit means the generation pipeline can use the same filtering and deduplication infrastructure as the pretraining corpus, ensuring consistency between curated and synthetic data.

Benchmark contamination detection as a standard step in model evaluation. The paper's contamination analysis (Appendix L) demonstrates that code-augmented corpora systematically leak benchmark answers, and provides both a methodology (WIMBD-based exact-match scanning across PromptSource datasets) and a set of contamination statistics that can serve as a baseline for future corpora. Model evaluation frameworks (EleutherAI's lm-eval, HELM) could incorporate a contamination-checking step: before reporting benchmark scores for a model, check whether the model's training corpus (if available) contains exact or near-duplicate matches of test examples, and flag any contaminated benchmarks with a warning. The Dolma Toolkit's Bloom filter decontamination module (Section 4.1, Appendix L) provides a production-ready implementation that seeds a Bloom filter with test examples and flags matching training documents, operating at the throughput needed for trillion-token corpora. If this became a standard part of model evaluation, it would address the contamination crisis that the paper documents—not by preventing contamination (which is impractical given how benchmark datasets propagate through public repositories) but by ensuring that contamination is always disclosed alongside reported performance numbers, allowing the community to calibrate their trust in benchmark results.