Datasheet for the Pile

🎯 Pitch

The Pile is an 825 GiB corpus purpose-built for large language model training at a time when no comparable public resource existed, yet its explicit inclusion of copyrighted books and papers—and reliance on a fair use argument—makes every downstream model a potential litigation target. This datasheet reveals the precise legal and ethical fault lines running through one of open-source AI’s foundational datasets.

1. Executive Summary

This paper documents the composition, collection, and ethical considerations of the Pile, an 825 GiB English text dataset assembled by EleutherAI for large-scale language model pretraining. Spanning 22 diverse sources—from academic repositories like arXiv and PubMed Central, to web scrapes like Pile-CC and OpenWebText2, to domain-specific corpora like GitHub code and FreeLaw legal opinions—the Pile was created to provide the research community with a single, publicly accessible training corpus when no such resource existed, and has since been used to train models including GPT-Neo, GPT-J, Jurassic-1, Wu Dao, and GPT-NeoX. The datasheet inventories each component's instance types, collection procedures, preprocessing decisions, funding sources, and known subpopulation identifications, while transparently disclosing that copyrighted material appears in several subsets (notably Books3, arXiv, and PhilPapers), establishing that the dataset is provided under an MIT License with the authors' assertion that its use for language modeling falls under U.S. fair use doctrine.

2. Context and Motivation

The Core Problem: No Public, High-Quality Training Corpus Existed for Large Language Models

When EleutherAI began work on the Pile in mid-2020, the landscape of large-scale language model research suffered from a fundamental asymmetry: the techniques for training large models were increasingly well-documented in the literature (Radford et al., 2019; Brown et al., 2020), but the data that made these models possible remained locked away inside private organizations. OpenAI's GPT-2 and GPT-3 were trained on WebText, an internal dataset derived from highly-upvoted Reddit links, and later on undisclosed mixtures of web text, books, and Wikipedia. Microsoft's Turing-NLG used a proprietary corpus. Google's models used internal data pipelines. For researchers outside these organizations—academic labs, independent scientists, open-source collectives—there was simply no publicly available dataset of sufficient scale and quality to train a competitive large language model from scratch.

This was not merely an inconvenience. It represented a structural barrier to entry that threatened to concentrate language model capability exclusively within a handful of well-resourced industrial labs, creating what the broader community increasingly recognized as a democratization crisis in AI research. Without access to training data, independent researchers could not reproduce reported results, could not systematically study how dataset composition affects model behavior (biases, factual knowledge, reasoning capabilities), and could not build the kinds of open, auditable models that would be necessary for safety research and public accountability.

The gap was particularly acute because training a competitive language model requires not just any large text corpus, but one that is diverse across domains (to produce broadly capable models), reasonably filtered for quality (to avoid training on SEO spam, boilerplate, and machine-generated text), and large enough to saturate the compute budgets that modern hardware enables. Public web crawl data existed—Common Crawl had been releasing snapshots since 2011—but in its raw form it was dominated by low-quality, repetitive, and auto-generated content. Projects like C4 (Raffel et al., 2020) had demonstrated that aggressive filtering could extract a usable subset from Common Crawl, but C4 was designed primarily for masked language models like T5, and its filtering heuristics (dropping any page containing curly braces, for example) removed substantial amounts of high-quality technical content including code and mathematics. More fundamentally, a single-domain corpus—even a well-filtered one—could not produce the kind of broad knowledge and reasoning capabilities that made models like GPT-3 so compelling.

Why This Gap Mattered: The Stakes of Training Data Access

The importance of solving this problem extended well beyond the practical desire to train bigger models. Several interconnected concerns made it urgent:

Scientific reproducibility. When training data is proprietary, reported benchmark results are inherently irreproducible. If OpenAI claimed GPT-3 achieved a certain accuracy on a reasoning task, there was no way for independent researchers to determine whether that performance was attributable to the model architecture, the training procedure, or some artifact of the particular data mixture. This made it impossible to conduct rigorous ablation studies or to distinguish genuine advances in modeling from advances in data curation.

Bias and fairness research. Understanding how training data composition influences model outputs—stereotypes, factual errors, toxic generations—requires being able to examine, audit, and modify the training corpus itself. Researchers studying these phenomena were largely limited to analyzing model outputs after the fact, probing what biases had been learned without being able to trace them back to specific sources or distributions in the training data.

Safety and alignment. A growing community of researchers argued that making powerful language models safe would require broad, transparent, and participatory governance of the data they were trained on. If only a few companies controlled the data pipelines, they also controlled the values, knowledge, and perspectives that models would learn. Open training data was increasingly seen as a prerequisite for open safety research.

Domain-specific applications. Industrial models trained on proprietary web corpora showed impressive general capabilities but often lacked depth in specialized domains—legal reasoning, biomedical literature, mathematical proofs, structured code. Researchers in these fields had access to domain-specific corpora but lacked the infrastructure to combine them with general web text at the scales needed for competitive pretraining. A modular, composable dataset could enable domain experts to contribute their corpora and benefit from the broader language model ecosystem.

The economic barrier. Training data collection at web scale is expensive. Common Crawl snapshots are petabytes in size, and processing them requires significant compute resources. Individual academic labs could not afford to replicate the data pipelines that industrial labs had built over years with dedicated engineering teams. A shared, community-maintained dataset could amortize these costs across the entire research community.

Prior Approaches and Their Shortcomings

Several prior efforts had attempted to fill this gap, but each fell short in ways that the Pile was specifically designed to address.

Common Crawl alone (raw or lightly filtered). Common Crawl is the most obvious source of public web text, and several research projects had attempted to use it directly or with minimal filtering. The problem, as the authors note indirectly through their extensive preprocessing (documented in Gao et al., 2020), is that raw Common Crawl is overwhelmingly low-quality. A study cited in related work found that only a small fraction of Common Crawl pages contain "coherent, well-formed English text" suitable for language model training. The rest is boilerplate, machine-generated spam, duplicate content, and non-text artifacts from HTML parsing. Training directly on this data produces models that are measurably worse than those trained on curated corpora. The Pile-CC component represents an attempt to solve this through heavy filtering (using jusText for boilerplate removal, deduplication, and quality heuristics), but the authors recognized that even a well-filtered web crawl was insufficient on its own—it would lack the depth of domain-specific knowledge that comes from curated sources like academic papers, legal documents, and books.

C4 (Colossal Clean Crawled Corpus). Released by Google as part of the T5 project (Raffel et al., 2020), C4 was the most prominent attempt to create a public, filtered version of Common Crawl for language model pretraining. It applied a series of heuristic filters—removing pages with curly brackets ("{"), with the word "lorem ipsum," with the phrase "terms of service," and so on—to extract roughly 750 GiB of "clean" English text from the April 2019 Common Crawl snapshot. While C4 was widely adopted and served as an important proof of concept, it had several limitations that the Pile's creators sought to address. First, its filters were overly aggressive in ways that systematically removed valuable content: code (which uses curly brackets for control flow and data structures), mathematics (LaTeX uses curly brackets for grouping and commands), and any page that happened to include templated legal language. Second, it was a single-source corpus—all text came from the web, with no representation of books, academic papers, legal documents, code repositories, or other structured knowledge sources. Third, its filtering heuristics were designed without community input, optimized for general-purpose text-to-text transfer rather than the diverse set of downstream applications the broader NLP community cared about.

OpenWebTextCorpus (OWTC). Developed by Gokaslan and Cohen (2019), OWTC was an attempt to reproduce OpenAI's WebText dataset—the corpus used to train GPT-2—using publicly available data. WebText was constructed by scraping all outbound links from Reddit submissions that received at least 3 karma, a heuristic designed to select for "high-quality" human-written content that people had found valuable enough to engage with. OWTC replicated this methodology, producing a corpus of roughly 38 GiB. While this was a landmark effort in dataset democratization, OWTC suffered from several limitations that the Pile's OpenWebText2 component sought to address. At 38 GiB, it was too small to train models at the scale of GPT-3 (which was trained on hundreds of billions of tokens). The original OWTC used a single Reddit dump and did not incorporate newer submissions, meaning it was frozen in time and could not capture evolving language or knowledge. And like C4, it was a single-source web corpus that could not provide the domain coverage needed for truly general-purpose language models.

BookCorpus. The original BookCorpus (Zhu et al., 2015) was one of the most influential datasets in NLP, used to train or fine-tune models including Skip-Thought Vectors, unsupervised Sentiment Neuron discovery, and BERT. It consisted of approximately 11,000 books from the self-publishing platform Smashwords, selected to represent "unpublished authors" (which, in practice, meant books that had been made freely available). However, by 2020, the original BookCorpus had become effectively unavailable—the official download link had gone offline, and the dataset's legal status was ambiguous since the books were redistributed without explicit authorization from their authors. This created a reproducibility crisis: papers reporting results using "BookCorpus" were actually using different, often undocumented subsets or reconstructions. EleutherAI's BookCorpus2 component (17,868 books) was created using similar methodology but with expanded coverage and clearer documentation, partially addressing this reproducibility gap.

Existing domain-specific corpora in isolation. Several of the datasets that the Pile incorporates—PubMed Central, arXiv, GitHub, USPTO backgrounds, FreeLaw—existed as standalone resources. Researchers in biomedical NLP had access to PubMed; researchers in code generation had access to GitHub. However, training a large language model on these sources individually produced models that were deeply knowledgeable in one domain and nearly useless in others. The innovation was not in accessing these sources (most were publicly available) but in combining them into a single, carefully-weighted mixture that could produce models with genuinely broad capabilities while preserving depth in specialized domains. This required solving the engineering challenge of processing 22 different data formats into a consistent representation, and the curation challenge of deciding how much weight to assign each source so that rarer but valuable domains (like mathematical reasoning) were not drowned out by abundant but less informative sources (like web text).

Models trained on undisclosed data mixtures. By far the most successful language models in 2019–2020 were trained on proprietary, undisclosed data. GPT-3's training corpus was described only in broad strokes (60% filtered Common Crawl, 22% WebText2, 16% books, 3% Wikipedia) without release of the actual data. This was the status quo the Pile sought to disrupt: a world where the most important decisions about what knowledge, perspectives, and linguistic patterns a model would learn were made behind closed doors, with no mechanism for community input, audit, or replication.

How the Pile Positions Itself

The Pile positions itself not as a theoretical contribution but as an infrastructure contribution—a piece of shared research infrastructure that enables a broad class of future work that would otherwise be impossible or impractical. This is a fundamentally different kind of contribution from the algorithmic innovations that dominate NLP conference proceedings, and the datasheet's structure reflects this. Rather than presenting experimental results comparing the Pile against alternatives (though such comparisons exist in Gao et al., 2020, the accompanying technical paper), the datasheet focuses on exhaustive documentation of what is in the dataset, where it came from, how it was processed, and what caveats apply to its use.

Several positioning decisions are notable:

Transparency about limitations and unknowns. The datasheet repeatedly uses phrases like "we do not know," "we have no idea," "as far as we are aware," and "it is always a possibility." This is a deliberate rhetorical strategy that distinguishes the Pile from industrial datasets whose contents are treated as trade secrets. By being explicit about the boundaries of their knowledge—acknowledging that copyrighted material appears in the dataset, that subpopulations are likely identified in various components, that offensive content "probably" exists in most subsets—the authors aim to provide downstream users with the information they need to make informed decisions about whether and how to use the data.

Fair use assertion for copyrighted content. The datasheet directly addresses the most legally contentious aspect of the dataset: several components (Books3, portions of arXiv and PhilPapers) contain copyrighted material. The authors assert that "under U.S. copyright law use of copyrighted texts in the Pile falls under the 'fair use' doctrine," grounding this claim in the fact that the data has been "heavily processed to aid in language modeling research" and that "no copyrighted text is contained in the Pile in its original form." This is a substantive legal position, not merely a disclaimer—the authors are arguing that the transformative use of copyrighted works for training statistical models constitutes fair use, a position that remains legally contested but which has been central to the open-source AI movement's strategy. Notably, they acknowledge that "copyright law varies by country" and advise users to "speak to an intellectual property attorney," implicitly recognizing that the fair use defense is strongest in U.S. jurisdictions and that international users may face different legal risks.

Modularity as a design principle. Rather than releasing the Pile as a monolithic blob, the authors provide the code and instructions necessary to "compile a custom remix of the datasets," explicitly encouraging users to exclude components that pose legal, ethical, or quality concerns for their particular use case. This modularity is a direct response to the one-size-fits-all nature of prior public corpora: a researcher who cannot legally use Books3 can still train on the remaining 21 components; a researcher studying biomedical NLP can upweight PubMed Central and NIH ExPORTER; a researcher concerned about toxic content can exclude web scrapes while retaining curated academic sources.

Community governance model. EleutherAI positions itself as a custodian rather than an owner, noting that "we do not accept submissions of new contributions to the dataset" but that "we have made all of the code necessary for constructing the Pile from scratch public to enable easier augmentation and improvement." The Pile is presented as a snapshot—frozen at September 1, 2020—with any future expansions released as separate datasets ("Pile Version 2"). This distinguishes it from dynamic datasets that evolve over time (complicating reproducibility) while still inviting community forks and remixes.

Documentation as a first-class artifact. The datasheet itself represents a positioning choice. Following the framework proposed by Gebru et al. (2021) for dataset documentation, the authors treat the datasheet not as supplementary material but as a core publication alongside the technical paper (Gao et al., 2020). This reflects a growing recognition in the NLP community that datasets are research contributions in their own right, deserving of the same level of rigorous documentation as algorithmic contributions—and that the absence of such documentation has led to widespread misuse, misinterpretation, and harm.

The Unstated Motivation: Building an Open Ecosystem

Reading between the lines, the Pile's creation was driven by a conviction that the future of language model research should not be determined by a handful of corporate labs. Every design decision—from the choice of 22 diverse sources (to enable broad capabilities without proprietary data) to the MIT License (maximally permissive) to the detailed documentation (enabling informed use by researchers without legal teams)—reflects a commitment to building research infrastructure that is genuinely open, auditable, and community-governed. This motivation is never stated in so many words (the datasheet's tone is factual and dry), but it is the unifying thread that connects the dataset's creation to its documentation.

The Pile was released at a pivotal moment. In early 2021, GPT-3 had demonstrated that scaling language models produced qualitatively new capabilities, but access to these capabilities was controlled entirely through OpenAI's commercial API. Researchers could query the model but could not inspect its weights, study its training data, or understand why it produced particular outputs. The Pile, alongside EleutherAI's model training efforts (GPT-Neo, GPT-J), represented an attempt to create an alternative path—one where the research community could study, critique, and improve large language models without being dependent on corporate gatekeepers. The datasheet is a key piece of that infrastructure: it is the document that tells researchers what they are actually working with when they download and train on the Pile, and what ethical and legal considerations they should weigh before doing so.

3. Technical Approach

3.1 Reader Orientation

This paper is not a description of a single algorithm or model, but a dataset documentation artifact—specifically, a "datasheet" following the framework proposed by Gebru et al. (2021)—that exhaustively inventories the composition, collection procedures, preprocessing decisions, known ethical issues, and legal considerations of the Pile, an 825 GiB text corpus assembled by EleutherAI from 22 distinct sources for large-scale language model pretraining. The "system" being described is the dataset itself, viewed as a piece of research infrastructure, and the datasheet's job is to answer for every component the questions a responsible practitioner should ask before using it: What exactly is in here? Where did it come from? Who created it, under what funding, with what consent? What biases, sensitivities, and legal risks does it carry? The architecture of the solution is the datasheet's structure: a systematic walk through eight categories of metadata (motivation, composition, collection, preprocessing, distribution, maintenance, legal/ethical considerations, and per-component details), applied in parallel to all 22 constituent datasets, so that downstream users can make informed, domain-specific decisions about whether and how to use the Pile in their work.

3.2 Big-Picture Architecture (Diagram in Words)

The Pile's architecture can be understood at two levels: the data pipeline that produced the dataset (process) and the documentation framework that describes it (meta-structure).

Data Pipeline (creation): First, source identification: EleutherAI researchers selected 22 text sources spanning web text, academic literature, books, code, legal documents, dialogue, and domain-specific corpora, chosen for diversity across knowledge domains, linguistic registers, and data modalities. Second, per-source acquisition: each source was obtained through its own mechanism—direct download from official repositories (PubMed Central, Wikipedia, USPTO, NIH ExPORTER), scraping from public websites (arXiv, GitHub, YouTube Subtitles, HackerNews), use of pre-existing datasets (Books3, DM Mathematics, PG-19, EuroParl, OpenSubtitles, Enron Emails), or custom EleutherAI scrapes and compilations (Pile-CC, OpenWebText2, BookCorpus2). Third, per-source preprocessing: each source underwent format-specific cleaning—boilerplate removal (jusText for Pile-CC, BeautifulSoup for FreeLaw HTML), format conversion (pandoc for arXiv LaTeX to Markdown), metadata stripping (all components), deduplication (MinHashLSH for Pile-CC and OpenWebText2, application-ID-based dedup for NIH ExPORTER, abstract-text dedup for PubMed), and filtering (quality thresholds, minimum text length, malformed entry removal). Fourth, interleaved mixing: documents from all 22 sources were concatenated into a single corpus and randomly shuffled while preserving each document's internal order, with source-specific weighting applied to control how frequently each source appears in training (upsampling smaller, high-value sources like DM Mathematics relative to larger ones like Pile-CC). Fifth, train/validation/test splitting: 0.1% of the data was randomly sampled for each of the validation and test sets (approximately 1.4 GiB each), with the remainder forming the training set. The final output is 211,043,181 documents totaling 825.18 GiB, distributed via direct download and community mirrors under the MIT License.

Documentation Framework (the datasheet itself): The datasheet is organized into eight sections that systematically answer the questions from Gebru et al.'s dataset documentation framework. Each section poses a set of questions and answers them for the Pile as a whole and (where answers differ) for each of the 22 components individually. The sections flow from motivation (why each source was included, what tasks it enables), to composition (what instances are, how many there are, what data they contain, what labels exist, whether the dataset is a sample or complete), to collection process (acquisition mechanisms, timeframe, personnel, compensation), to preprocessing (cleaning, filtering, deduplication), to distribution (access mechanism, license, copyright status, fees), to maintenance (hosting, update policy, obsolescence communication), to legal and ethical considerations (ethical review, confidentiality, offensive content, subpopulation identification, identifiability of individuals, sensitive data, consent, impact analysis). The repeated structure—answering the same question for each of 22 components in sequence—creates a reference format where a researcher interested in a specific component (e.g., "what subpopulations are identified in PubMed Central?") can locate the relevant subsection and find a specific, component-level answer.

3.3 Roadmap for the Deep Dive

First, the unit of analysis: what a "document" means in the Pile, since this varies radically across components (web pages vs. books vs. code files vs. IRC chat logs vs. patent abstracts) and understanding this heterogeneity is essential before any further technical discussion.
Second, the acquisition mechanisms: how each of the 22 sources was obtained, because the diversity of methods—official bulk downloads, custom scrapes, pre-existing dataset reuse, algorithmic generation—determines what metadata and provenance information is available and what preprocessing was necessary.
Third, the preprocessing pipeline per source: the concrete transformations applied to convert raw acquired data into the cleaned, deduplicated, format-normalized text documents that constitute the Pile's instances, including specific tools (jusText, pandoc, BeautifulSoup, pdfbox, MinHashLSH) and their configurations, since these choices determine what information was preserved and what was discarded.
Fourth, the mixing, weighting, and splitting strategy: how the 22 processed components were combined into a single corpus with deliberate upsampling of smaller sources, and how train/validation/test splits were constructed, since these design decisions directly affect what models trained on the Pile will learn.
Fifth, the documentation decisions: what the authors chose to disclose, what they acknowledge they do not know, and how the structure of the datasheet itself reflects deliberate choices about transparency, legal positioning (particularly around copyrighted content and fair use), and community governance.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a dataset documentation paper whose core technical contribution is not an algorithm or model architecture, but a systematic, component-by-component inventory of the Pile's composition, provenance, processing, and known issues, structured to enable downstream researchers to make informed decisions about whether and how to use each subset. The datasheet functions as a reference document: for each of the 8 top-level questions in the Gebru et al. framework, it provides a Pile-level answer followed by per-component answers that capture the heterogeneity across the 22 sources.

The Pile Document and Its Instance Types

The fundamental unit of data in the Pile is the document—a text file that has been processed to remove metadata, formatting, and non-textual artifacts, preserving only the linguistic content. However, what constitutes a "document" varies dramatically across components, and understanding this variation is essential for interpreting what models actually learn from each source.

Pile-CC documents are individual webpages after processing through Common Crawl's WARC format conversion and jusText boilerplate removal. Each document corresponds to what jusText identifies as the "main content" of a single URL, with navigation elements, sidebars, footers, and other boilerplate stripped away. The underlying data is drawn from Common Crawl snapshots, which crawl billions of URLs monthly and store the raw HTTP responses in WARC (Web ARChive) format. The conversion to text discards HTML structure, CSS, JavaScript, images, and other non-text content, producing a stream of natural-language paragraphs. After deduplication (using in-memory MinHashLSH via the DataSketch library), the Pile-CC component contains 54,953,117 documents, making it the single largest component by document count (though not by total size—it represents 227.12 GiB out of 825.18 GiB, or roughly 27.5% of the total corpus).

PubMed Central documents are full-text academic medical papers. Unlike Pile-CC webpages, these are structured scholarly documents with predictable sections (abstract, introduction, methods, results, discussion) and contain domain-specific terminology, citations, and quantitative data. Each document corresponds to one paper in the PubMed Central repository, which was complete as of June 2020. The 3,098,931 documents constitute 90.27 GiB—a smaller count than Pile-CC but a comparable total size, reflecting the fact that academic papers are substantially longer than typical webpages.

Books3 documents are published books, each document corresponding to one complete book. The Books3 dataset (created by Shawn Presser and hosted by The Eye) was included in its entirety: 196,640 books comprising 100.96 GiB. These are long-form narrative or expository texts with internal structure (chapters, paragraphs, dialogue), making them fundamentally different from the atomic webpages in Pile-CC or the structured academic format of PubMed Central. Books provide the long-range coherence that is essential for training models to track entities, maintain narrative consistency, and perform multi-page reasoning.

arXiv documents are preprints of academic papers, primarily in mathematics, computer science, physics, and statistics. Each document was originally a LaTeX source file (.tex with associated figure files and bibliography), which was converted to Markdown using pandoc 1.19.2.4. Papers that produced errors during this conversion were discarded (the datasheet does not report the discard rate, but notes it explicitly). The 1,264,405 documents total 56.21 GiB, averaging shorter than PubMed Central papers because arXiv papers span a wider range of lengths (from short notes to full-length dissertations) and the LaTeX-to-Markdown conversion may have imperfectly preserved all content.

GitHub documents are individual code files. Each document corresponds to one file from a GitHub repository, with repositories and files filtered by size: only repositories below a certain size threshold (not quantified in the datasheet but detailed in Gao et al., 2020) were included, and individual files were capped at 100 KB. From approximately 600 GiB of code files meeting these criteria, the authors subsampled to 95.16 GiB (19,021,454 documents). This is the only component where documents are primarily non-natural-language: code files contain programming language syntax, comments (which are natural language but embedded in code context), documentation strings, and structural patterns (indentation, function definitions, control flow) that differ fundamentally from prose.

OpenWebText2 documents are webpages, similar in kind to Pile-CC documents but drawn from a different underlying source. Rather than a general web crawl, OpenWebText2 was constructed by first extracting all URLs from Reddit submissions (up to April 2020) along with their associated metadata, deduplicating URLs, computing an aggregate score (the sum of scores/karma across all submissions linking to that URL), and discarding URLs with an aggregate score below 3. The surviving URLs were then scraped and processed with Newspaper scraper (a Python library for article extraction, distinct from the jusText pipeline used for Pile-CC). Deduplication was performed at the document level using in-memory MinHashLSH through the DataSketch library (the same approach as Pile-CC, ensuring cross-component deduplication). The 17,103,059 documents total 62.77 GiB.

FreeLaw documents are legal documents from the Free Law Project's bulk data. The datasheet specifies that only the subset consisting of "court opinions" was retained, excluding "dockets, people, retention votes, and citation data" because they were "broadly administrative." Court opinions came in either plaintext or HTML format; for HTML opinions, all formatting was discarded and raw text was extracted using BeautifulSoup. The 3,562,015 documents total 51.15 GiB, averaging longer than typical webpages because judicial opinions are detailed legal analyses that can run to dozens or hundreds of pages.

Wikipedia (en) documents are pages from the English Wikipedia, taken from the wikipedia/20200301.en dataset published via TensorFlow Datasets. Each document is a single Wikipedia page, consisting of the article text with wiki markup, infoboxes, categories, and other metadata stripped away (these processing steps were performed by DeepMind, who created the dataset; the Pile authors used the pre-processed version). The 6,033,151 documents total only 6.38 GiB—a small fraction by size because Wikipedia articles are relatively short and highly dense in information, but valuable out of proportion to their size because Wikipedia text is heavily curated, fact-checked, and written in a consistent encyclopedic register that models benefit from learning.

StackExchange documents are question-answer pairs from the Stack Exchange network. Each "document" consists of a question and up to three answers that meet the inclusion criteria (at least three upvotes each). The data was obtained by downloading and parsing every Stack Exchange database dump as of July 1, 2020. The authors "extract the top three answers with at least three upvotes, discarding all other responses," and "only include the plain text question and response and do not incorporate any metadata." This creates a structured dialogue-like format where each document contains a technical question followed by expert answers, providing models with examples of explanatory technical writing, problem-solving discourse, and community-moderated knowledge curation. The 15,622,475 documents total 32.20 GiB.

USPTO Backgrounds documents are the "Background" sections extracted from patents granted by the United States Patent and Trademark Office. The USPTO publishes bulk archives of full patent text from 1976 to September 2020; from these, the Pile authors "extract the Background sections, along with key grant-specific metadata, such as the inventor, assignee, and classification information." The 5,883,037 documents total 22.90 GiB. The Background section of a patent is written by the inventor (or their attorney) to describe the problem that the patent addresses and the prior state of the art, making it a corpus of technical problem descriptions written in a distinctive legal-technical register.

PubMed Abstracts documents are abstracts of 30 million publications from the PubMed index. Unlike the PubMed Central full-text component, this contains only the abstract (and associated metadata like title and authors), not the full paper. About one-third of the articles were excluded because they "were missing or contained a malformed title or abstract," and articles that already appear in the full-text PubMed Central component were removed to avoid overlap. The 15,518,009 documents total 19.26 GiB, averaging only about 1.2 KB per abstract.

Project Gutenberg (PG-19) documents are books published before 1919, consisting of the complete Project Gutenberg 1919 dataset as created by DeepMind (Rae et al., 2019). The 28,602 documents total 10.88 GiB. These are primarily classic literature in the public domain, representing 19th-century and early 20th-century English prose with distinctive vocabulary, syntax, and cultural references that differ from contemporary text.

OpenSubtitles documents are movie and television subtitles, with each document corresponding to the complete subtitle track for one movie or episode. Only files tagged as English by Tiedemann (2016) were included; all metadata was discarded. The 446,612 documents total 12.98 GiB. Subtitles consist primarily of dialogue, making this the largest source of conversational language in the Pile, with distinctive features like sentence fragments, interruptions, colloquialisms, and turn-taking patterns that are absent from formal written text.

DM Mathematics documents are groups of related mathematics problems, algorithmically generated by DeepMind (Saxton et al., 2019) to cover algebra, arithmetic, calculus, number theory, probability, and other mathematical domains. Each document consists of a natural-language problem statement (e.g., "Calculate the derivative of f(x) = 3x^2 + 2x - 1") possibly followed by its answer, formatted as a natural-language prompt. The 1,014,997 documents total 7.75 GiB. This is the only component whose data was not authored by humans—it is entirely synthetic, generated from templates and mathematical rule systems.

BookCorpus2 documents are books from the self-publishing platform Smashwords, similar in kind to the original BookCorpus (Zhu et al., 2015). The creation methodology followed Kobayashi (2018)'s approach to reconstructing BookCorpus from Smashwords data, yielding 17,868 books totaling 6.30 GiB—a larger collection than the original BookCorpus's 11,038 books "due to the expanding nature of the underlying data."

Ubuntu IRC documents are chat logs from Ubuntu-related channels on the Freenode IRC server, covering July 5, 2004 through September 1, 2020. Each document corresponds to one week of chat logs, with system messages (joins, disconnects, nick changes) removed but user actions (messages sent with the /me command) preserved. The 10,605 documents total 5.52 GiB. This is the richest source of informal, multi-party dialogue in the Pile, featuring technical support conversations, developer coordination, and community discussion.

EuroParl documents are minutes from meetings of the European Parliament, covering 1996 to 2012. Each document corresponds to one speaker's intervention in a parliamentary session. The raw data (downloaded from statmt.org/europarl/) contains XML tags identifying speakers and languages; the Pile authors "remove all basic tag information and only retain the name of each document as a title" and "extract the body of each document, discarding those that are shorter than 200 characters." A distinctive feature is that "a large proportion of the data contains the same text repeated in different languages" because parliamentary proceedings are transcribed in all official EU languages, and these multilingual parallel versions are preserved as separate documents. The 69,814 documents total 4.59 GiB.

YouTube Subtitles documents are closed-caption subtitle tracks for YouTube videos, obtained by scraping YouTube directly. Each document corresponds to the subtitles for one video, and the dataset includes videos from YouTube's founding (February 14, 2005) through July 1, 2020. The 173,651 documents total 3.73 GiB. These are primarily human-generated captions (YouTube also offers auto-generated captions, but the datasheet specifies "human generated closed-captions"), providing a source of spoken-language transcription that differs from scripted subtitles (OpenSubtitles) in that it captures spontaneous speech, hesitations, false starts, and informal conversational patterns.

PhilPapers documents are open-access philosophy publications from the PhilPapers database maintained by the Center for Digital Philosophy at the University of Western Ontario. The data spans 2009–2020. The Pile authors "kept machine-readable entries and non-English entries" but "entries which could not be parsed by pdfbox were ignored." The 33,990 documents total 2.38 GiB. These are academic humanities papers, providing a source of abstract philosophical reasoning, argumentation structure, and humanities-specific vocabulary that complements the STEM-heavy arXiv and PubMed components.

NIH ExPORTER documents are abstracts of awarded grant applications from the NIH and other U.S. Department of Health and Human Services agencies (ACF, AHRQ, CDC, HRSA, FDA) as well as the VA, covering fiscal years 1985 to the present (via the ExPORTER system) and fiscal years 1970–2009 (via the legacy CRISP system). The two data sources were merged, deduplicated on application ID, excluded if abstract text was missing or too short, further deduplicated on abstract text (to remove small boilerplate-only administrative grants), and minimally parsed to remove administrative boilerplate like "DESCRIPTION (provided by applicant)." The 939,668 documents total 1.89 GiB, averaging only about 2 KB per abstract—the smallest component by total size but providing a concentrated source of scientific project descriptions written in a distinctive grant-application register.

HackerNews documents are conversation threads from the Hacker News link aggregator operated by Y Combinator. The data was scraped from the Y Combinator API, covering approximately October 9, 2006 to September 20, 2020 (the first 24,531,712 posts). Each document is a conversation thread consisting of a submission (link plus title) and all comments and replies in nested discussion. The 831,198 documents total 3.90 GiB. This is a source of technically-oriented community discussion, debate, and knowledge sharing, with a distinctive register combining technical precision with informal conversational norms.

Enron Emails documents are emails between Enron employees from January 6, 1998 until February 4, 2004, originally collected by U.S. federal agents during the investigation into Enron's accounting fraud and later prepared for research use by the CALO Project. The 517,401 documents total 0.88 GiB, averaging roughly 1.7 KB per email—the smallest component by total size. This is the Pile's only source of private, professional email communication, providing a window into organizational communication patterns, internal corporate language, and the linguistic conventions of email as a genre.

Document Counts, Sizes, and the Weighting Scheme

The Pile contains what the datasheet reports as "211,043,181 documents (unweighted), totaling 825.18 GiB." However, this raw count and size are misleading because the Pile is not constructed by simply concatenating these documents at their natural frequencies. The authors deliberately upsample certain components, making the effective size of each component in the training data different from its raw size.

The datasheet defers the details of the weighting scheme to Gao et al. (2020), citing Figure 1 of that paper for the effective sizes, but the principle is clear from context: smaller, high-quality, or domain-specific sources (like DM Mathematics, which is only 7.75 GiB raw, or PubMed Abstracts at 19.26 GiB) are repeated during training to ensure they are not drowned out by the much larger web scrape components (Pile-CC at 227.12 GiB and OpenWebText2 at 62.77 GiB). This is a deliberate design choice: without upsampling, a language model trained on the Pile would see web text roughly 30 times more frequently than domain-specific academic text, and might fail to learn the specialized knowledge and linguistic patterns those domains contain. The upsampling effectively treats rarer but higher-quality sources as having higher "value per token" for model training, and compensates for their smaller size through repetition.

The train/validation/test split is constructed by randomly sampling 0.1% of the data for each of validation and test (about 1.4 GiB each), with the remaining approximately 99.8% forming the training set. The sampling is performed at the document level from the combined corpus (not per-component), meaning that the validation and test sets preserve the same component distribution as the training set. This is a standard practice: it ensures that held-out evaluation measures how well the model generalizes to new documents from the same distribution, rather than testing on a fundamentally different composition that would conflate distribution shift with model capability.

Acquisition Mechanisms and Provenance

The datasheet's Section 4 (Collection Process) and Section 2 (Motivation, which includes funding sources) together document how each component was acquired, who funded its creation, and—critically—what the Pile authors do and do not know about its origins. This documentation serves multiple purposes: it enables downstream users to assess data quality (a legally-mandated government database has different reliability properties than a volunteer-submitted web forum), to evaluate legal risks (data scraped from copyrighted sources carries different restrictions than data released under open licenses), and to understand consent and privacy implications (data generated by professional transcribers for public parliamentary proceedings has fundamentally different privacy expectations than private email communications).

The acquisition mechanisms fall into five categories:

Category 1: Direct download from official repositories. Several components were obtained by downloading complete, officially-published datasets from their institutional hosts. PubMed Central was downloaded from the NCBI's official database, complete as of June 2020. Wikipedia (en) was obtained from the wikipedia/20200301.en dataset on TensorFlow Datasets, which itself was created by DeepMind from official Wikimedia dumps uploaded to Kaggle. USPTO Backgrounds were "obtained from the U.S. Patent and Trademark Office directly" through their published bulk archives. PubMed Abstracts were "downloaded directly from PubMed by EleutherAI." NIH ExPORTER was "downloaded directly from the ExPORTER database by EleutherAI." These official sources carry the strongest provenance guarantees: the data is exactly what the hosting institution intended to make available, with documented collection procedures, update schedules, and (in many cases) legal mandates governing their publication. A researcher using these components can trace the data back to a specific institutional release and verify its contents against the official record.

Category 2: Custom scrapes of public websites. Several components were obtained by EleutherAI members writing custom scrapers to collect data from publicly accessible websites. arXiv was obtained by "scraping [arXiv] ourselves" using arXiv's S3 Bulk Source File Access to download all paper sources up to the July 2020 dump (the last file included was arXiv_src_2007_068.tar). GitHub was obtained by "scraping [GitHub] ourselves," filtering repositories by size and files by 100 KB, then subsampling from approximately 600 GiB of qualifying files to the final 95.16 GiB. StackExchange was obtained by "scraping the StackExchange Network ourselves" from the July 1, 2020 database dumps. YouTube Subtitles were obtained by "scraping [YouTube] ourselves" for videos from February 14, 2005 through July 1, 2020. PhilPapers was obtained by "scraping [PhilPapers] ourselves." HackerNews was obtained by scraping "the Y Combinator API directly by EleutherAI." These custom scrapes give the Pile authors direct control over the collection parameters (date ranges, filtering criteria, rate limiting), but require them to document those parameters precisely so that other researchers can reproduce or audit the collection. The datasheet provides these parameters for most components, though at varying levels of detail—the GitHub sampling procedure (from 600 GiB to 95.16 GiB) is described qualitatively but the exact sampling fraction is not specified.

Category 3: Reuse of pre-existing datasets. Several components were incorporated from existing, third-party datasets without modification or with minimal reformatting. Books3 was taken from the dataset created by Shawn Presser and hosted on The Eye; its collection methodology is "not all publicly disclosed." PG-19 was taken from DeepMind's pre-existing dataset (Rae et al., 2019). DM Mathematics was taken from DeepMind's dataset (Saxton et al., 2019). OpenSubtitles was taken from the dataset created by Tiedemann (2016), using only the English-tagged entries. EuroParl was downloaded from the pre-existing corpus at statmt.org/europarl/. Enron Emails was taken from the dataset prepared by the CALO Project. For these components, the Pile authors are not the original data collectors; they are redistributing data that was collected by others, often for different purposes. The provenance chain is longer and less transparent, and the datasheet's answers to questions about collection methodology, consent, and ethical review often take the form of directing the reader to the original data creators (e.g., "Contact Shawn Presser or the Eye for further information," "Contact the Free Law Project for further details," "We do not know how the data for OpenSubtitles was collected").

Category 4: Custom EleutherAI compilations from public data. Two major components were created by EleutherAI specifically for the Pile by processing publicly available data through custom pipelines. Pile-CC was created by taking data from Common Crawl WARC files, converting them to text using jusText, and applying additional filtering (detailed in Gao et al., 2020). OpenWebText2 was created by extracting URLs from Reddit submissions up to April 2020, filtering by aggregate score (≥3), scraping the resulting webpages with Newspaper scraper, and deduplicating with MinHashLSH. BookCorpus2 was created by downloading books from SmashWords following the methodology of Kobayashi (2018). These custom compilations represent the Pile's most significant data collection contribution: they are large, high-quality corpora that did not exist in any prior public dataset and required substantial engineering effort to produce.

Category 5: Data from legal or investigative processes. Two components have unusual provenance because they were created through legal mandates or investigations. FreeLaw's underlying data was collected by the Free Law Project as part of their mission to provide free public access to primary legal materials, and was "downloaded from the Free Law Project's bulk downloader by EleutherAI." Enron Emails were originally collected by the U.S. Federal Energy Regulatory Commission (FERC) during their investigation of Enron's accounting fraud, and were later "collected and prepared by the CALO Project" for research use. Neither component was created with language model training in mind; both were repurposed from their original legal or investigative contexts.

Funding and compensation disclosures. The datasheet systematically answers "Who funded the creation of the dataset?" for each component, revealing a diverse funding landscape. The Pile itself was "created by individuals working on their own time without funding." Several components come from government agencies: PubMed Central from NIH/NCBI, USPTO from the U.S. Office of the Chief Economist, NIH ExPORTER from NIH and other HHS agencies, Enron Emails from FERC. Several come from non-profits: Common Crawl (used for Pile-CC), the Free Law Project (501(c)(3)), the Wikimedia Foundation (501(c)(3)), Project Gutenberg (funded by universities including UNC Chapel Hill). Several come from corporate entities: DeepMind (Alphabet subsidiary, created Wikipedia dataset, PG-19, and DM Mathematics); Canonical Ltd. (funds Ubuntu); Y Combinator (funds HackerNews). Books3 was created by an independent developer (Shawn Presser). OpenSubtitles is funded by VIP memberships. EuroParl's funding is uncertain ("We believe that this dataset was funded by the School of Informatics of the University of Edinburgh, Scotland, but were unable to confirm this fact").

The datasheet also answers "Who was involved in the data collection process and how were they compensated?" with a notable disclosure: "People involved with collecting new datasets for the Pile were compensated with an invitation to be an author of Gao et al. [2020]." This is a form of academic credit rather than monetary payment—the contributors who built the scrape pipelines and processing code received co-authorship on the accompanying paper, which is a meaningful incentive in research communities but does not constitute financial compensation.

Preprocessing Per Source

The datasheet's Section 5 (Data Preprocessing) is deliberately brief, deferring the full details to Gao et al. (2020) and noting that "the data was extensively preprocessed" with public code available on GitHub. However, the per-component descriptions in Sections 3 and 4 collectively specify the preprocessing steps applied to each source, allowing a reconstruction of the pipeline architecture.

Common preprocessing across all components. Every component underwent metadata removal: any non-textual data associated with the instances (timestamps, author information, URLs, format-specific markup, database identifiers) was stripped, leaving only the textual content. The goal was to produce clean, plain-text documents suitable for tokenization and language model training without structured metadata that models might learn to exploit as shortcuts. Additionally, all components were format-normalized: HTML was converted to plain text (BeautifulSoup, Newspaper scraper), LaTeX was converted to Markdown (pandoc 1.19.2.4), XML tags were removed (EuroParl), and proprietary formats were discarded, producing a uniform representation across all 22 sources.

Source-specific preprocessing details. The datasheet provides the following per-component processing descriptions:

Pile-CC: The raw Common Crawl WARC files were converted from HTML to plain text using jusText (Endrédy and Novák, 2013), a heuristic algorithm that identifies the main content block of a webpage by analyzing the distribution of text density in the DOM tree and discarding boilerplate regions (navigation, sidebars, footers, advertisements). The algorithm works by computing a "text density" score for each paragraph-like element, and classifying elements with low text density as boilerplate. The resulting text was then filtered and deduplicated (details in Gao et al., 2020). The datasheet acknowledges that "some of this junk undoubtedly made it through our processing and is embedded in the text files."
PubMed Central: The papers were downloaded as full-text articles from the NCBI database. The datasheet does not describe specific format conversion for this component (PubMed Central provides articles in XML with structured markup; the conversion likely stripped XML tags and extracted paragraph text, but the exact procedure is deferred to Gao et al., 2020).
Books3: No preprocessing details are provided beyond the fact that the dataset was used as-is from Presser's release. The original books were presumably in formats like EPUB or PDF, and Presser's processing converted them to plain text, but the datasheet notes that "not all details about the creation of Books3 have been publicly disclosed."
arXiv: The TEX sources of all papers up to the July 2020 dump were downloaded via arXiv's S3 Bulk Source File Access, and pandoc 1.19.2.4 was used to convert these LaTeX source files to Markdown. The conversion process handles LaTeX commands, math environments, citations, and cross-references, translating them into a more uniform text representation. Papers that produced errors during conversion were discarded (the datasheet does not report the discard rate, but it is likely a small fraction, as LaTeX compilation errors are relatively rare in submitted preprints).
GitHub: The scraping filtered repositories by size (small only) and files by maximum size (100 KB). The subsampling from 600 GiB to 95.16 GiB was deterministic (the datasheet specifies "all sampled datasets were sampled deterministically"), though the exact sampling algorithm is deferred to Gao et al., 2020. Code files are stored as-is in their original programming language syntax.
OpenWebText2: URLs from Reddit submissions up to April 2020 were extracted, deduplicated (keeping unique URLs with aggregate scores), and filtered for aggregate score ≥ 3. The surviving URLs were scraped and processed with Newspaper scraper, a Python library that extracts article text from news websites by analyzing HTML structure and identifying the main content. Newspaper differs from jusText in that it is designed specifically for news articles (with title, author, publication date extraction) whereas jusText is a general-purpose boilerplate removal tool. Finally, document-level deduplication was performed using in-memory MinHashLSH through the DataSketch library. MinHashLSH (MinHash Locality-Sensitive Hashing) works by computing a set of hash values (the MinHash signature) for each document's n-gram set, and using locality-sensitive hashing to efficiently find near-duplicate pairs. Documents with high Jaccard similarity (as approximated by MinHash collision) were identified as duplicates and one copy was retained.
FreeLaw: The raw data was filtered to retain only court opinions, excluding administrative records. For opinions in HTML format, BeautifulSoup (a Python HTML parsing library) was used to extract raw text by discarding all formatting tags. Plaintext opinions were used as-is. The processing discards structural information (case citations, party names, court identifiers embedded in metadata) that might have been valuable for legal reasoning models but would have been inconsistent with the Pile's goal of producing plain-text linguistic data.
Wikipedia (en): No additional preprocessing was performed by EleutherAI. The wikipedia/20200301.en dataset from TensorFlow Datasets includes Wikipedia articles with wiki markup removed and article text extracted, as processed by DeepMind. The Pile authors used this pre-cleaned version directly.
StackExchange: The July 1, 2020 database dumps were parsed to extract question text and the top three answers with at least three upvotes. All other responses were discarded, and "only the plain text question and response" was included "and do not incorporate any metadata." This filter ensures that each document contains high-quality, community-vetted answers—the "at least three upvotes" threshold acts as a quality signal, selecting answers that the StackExchange community judged to be correct and well-written. The discarding of metadata removes vote counts, timestamps, user information, and question tags, which would have been useful for many NLP tasks but were judged to be out of scope for the Pile's language modeling purpose.
USPTO Backgrounds: From the USPTO's bulk patent archives covering 1976 to September 2020, the Background sections were programmatically extracted from the full patent text. The extraction logic identifies the Background section boundary (patent documents have a standardized structure with labeled sections) and retains the text within that section. "Key grant-specific metadata, such as the inventor, assignee, and classification information" was extracted alongside the text—unlike most other components where metadata was discarded—suggesting that this information was preserved because patent classification codes and assignee data are linguistically informative (they use domain-specific technical vocabulary and legal entity references).
PubMed Abstracts: From the 30 million publications in PubMed's index, about one-third were excluded because they "were missing or contained a malformed title or abstract." This is a substantial discard rate (approximately 10 million records), likely including records where the abstract field was empty (older publications that were never abstracted), where XML parsing failed, or where the abstract consisted only of non-English text or formatting artifacts. Additionally, any publications that already appeared in the full-text PubMed Central component were removed to avoid overlap between the two PubMed-derived components. The remaining abstracts were stored as standalone text documents.
PG-19: No preprocessing by EleutherAI. The PG-19 dataset as created by DeepMind (Rae et al., 2019) was used directly. DeepMind's processing likely involved extracting the plain text of Project Gutenberg e-books and splitting into documents, but the details are not reported in the datasheet.
OpenSubtitles: The English-language subset created by Tiedemann (2016) was used. "We discarded any provided metadata," meaning that file-level metadata (movie titles, release years, genre tags) was removed. However, the subtitles themselves contain timing information (timestamps indicating when each subtitle appears on screen), and the datasheet does not specify whether these were stripped—if they were retained, the documents would contain interspersed timestamp tokens that language models would need to learn to ignore.
DM Mathematics: The dataset was used as-is from Saxton et al. (2019). The problems are algorithmically generated from templates, producing clean, consistent formatting without preprocessing needs.
BookCorpus2: The books were downloaded from SmashWords following the methodology of Kobayashi (2018). SmashWords provides books in EPUB format; the processing likely involves EPUB-to-text conversion, but the datasheet defers details to the technical paper.
Ubuntu IRC: The raw IRC logs (covering July 5, 2004 through September 1, 2020) were processed to "discard all system messages, such as joins, disconnects, nick changes, etc." while retaining "actions (i.e using /me)." This leaves only the conversational content—the actual messages exchanged between users—and removes the server-level protocol messages that are artifacts of the IRC system rather than human-generated text. The /me action messages (used to narrate actions in third person, e.g., "/me waves" renders as "* username waves") are preserved because they are user-authored content, albeit in a distinctive IRC-specific format. Chats are chunked by week into documents.
EuroParl: The raw corpus from statmt.org/europarl/ was processed to "remove all basic tag information and only retain the name of each document as a title" and "extract the body of each document, discarding those that are shorter than 200 characters." This strips the XML markup that encodes speaker identity (turning <SPEAKER ID=77 LANGUAGE="NL" NAME="Pronk"> into just Pronk as a title) and removes the document's structural markup, leaving the transcribed speech. The 200-character minimum threshold filters out very brief interventions (procedural motions, votes, brief interjections) that lack substantive linguistic content.
YouTube Subtitles: The subtitles were scraped directly from YouTube. The datasheet does not describe format conversion; YouTube subtitles are available in various formats (SRT, VTT) containing timestamped text segments, and the conversion likely stripped timestamps to produce continuous text, but this is not specified.
PhilPapers: "Machine-readable entries and non-english entries were kept, but entries which could not be parsed by pdfbox were ignored." PDFBox is a Java library for extracting text from PDF documents. The use of PDFBox indicates that PhilPapers documents were stored as PDFs (typically academic paper preprints) and needed to be converted to plain text. Entries that PDFBox could not parse—likely scanned documents without embedded text, heavily formatted papers, or documents with corrupt PDF structures—were discarded. The discard rate is not reported.
NIH ExPORTER: The ExPORTER data (fiscal years 1985–present) and CRISP data (fiscal years 1970–2009) were merged. Deduplication was performed on application ID (to remove records that appeared in both systems) and on abstract text (to remove small administrative grants whose abstracts consisted solely of boilerplate text like "This is an administrative supplement"). The text was "minimally parsed to remove administrative boilerplate," with the example "description: (provided by applicant)" as a pattern that was systematically stripped. This targeted removal of known boilerplate patterns is different from the general-purpose boilerplate removal used for Pile-CC (which relies on jusText's statistical heuristics)—it reflects the fact that grant abstracts follow a highly standardized format where specific header phrases can be identified and removed with regular expressions. The filtering also excluded entries where "abstract text was missing or too short" (exact minimum length not specified).
HackerNews: The first 24,531,712 posts were scraped from the Y Combinator API, covering approximately October 9, 2006 to September 20, 2020. The API returns posts as structured JSON with author, timestamp, vote count, and parent-child relationships. The preprocessing likely flattened the nested comment tree into a linear text representation, but the datasheet does not specify the exact format—whether threads are represented as sequential comments with indentation, as reply chains, or in some other structure that preserves conversation flow.
Enron Emails: The CALO Project's preprocessed version of the Enron corpus was used. The original data consisted of email messages with headers (From, To, Subject, Date, CC, BCC) and body text. The preprocessing removed some headers and attachment content, but the extent of cleaning relative to the raw PST files is not detailed in the datasheet (it is deferred to the original corpus documentation; Klimt and Yang, 2004).

The Mixing, Weighting, and Splitting Strategy

Once each component had been individually acquired and preprocessed, the documents from all 22 sources were combined into the final Pile corpus. The datasheet describes this process at a high level but defers the quantitative details of weighting to Gao et al. (2020).

Interleaved mixing with source-specific upsampling. The documents from all 22 sources were concatenated and then randomly shuffled, with the constraint that each document's internal order is preserved (paragraphs within a book chapter, lines within a code file, turns within an IRC log are not shuffled—only the ordering of entire documents is randomized). This interleaving means that consecutive training examples are drawn from different sources, preventing the model from learning source-specific patterns (like "after reading 100 Wikipedia articles, I should expect Wikipedia syntax") and forcing it to learn representations that generalize across all 22 domains.

However, the random shuffling is not uniform across sources. The Pile's construction uses deliberate upsampling of smaller, higher-quality sources. The datasheet states: "Note that the Pile has had its datasets weighted. The sizes reported here are the raw sizes. For the effective sizes in the Pile, see Figure 1 of Gao et al. [2020]." This means that during training, documents from sources like DM Mathematics (7.75 GiB raw) appear more frequently than their raw size would dictate, while documents from Pile-CC (227.12 GiB raw) appear less frequently. The upsampling is implemented by repeating documents from smaller sources multiple times during the shuffling process, such that the effective training distribution has a more balanced representation across domains.

The motivation for upsampling is that language model capacity is finite: a model trained on a uniform mixture would spend the vast majority of its training compute on web text (which dominates by volume), and would likely underfit the specialized linguistic patterns, knowledge, and reasoning structures present in the smaller, curated sources. By upsampling DM Mathematics, the model sees mathematical reasoning problems with sufficient frequency to learn arithmetic and algebraic patterns. By upsampling PubMed Central, the model learns biomedical terminology that would be statistically rare in a web-dominated corpus. The upsampling factors are detailed in Gao et al. (2020), Figure 1; the datasheet's role is to flag that weighting exists so that users do not misinterpret the raw component sizes as the effective training distribution.

Train/validation/test split construction. The random 0.1%/0.1%/99.8% split is drawn uniformly from the combined, shuffled corpus, not independently from each component. This ensures that the validation and test sets have the same component distribution as the training set. If the splits were instead constructed by sampling 0.1% from each component independently (and concatenating), the validation and test sets would have exactly the same number of documents from each source, but would not reflect the upsampled distribution—they would over-represent rare, upsampled sources and under-represent common, downsampled sources relative to what the model actually saw during training. The uniform sampling approach avoids this mismatch.

The 0.1% fractions represent approximately 211,000 documents and 1.4 GiB each for validation and test. For language model evaluation, these held-out sets are used to compute perplexity: the model's average log-likelihood per token on the held-out text. The test set measures generalization to unseen documents from the same distribution mix, while the validation set is used for hyperparameter tuning and early stopping decisions during training.

Documentation Decisions and Transparency

The datasheet's structure is itself a technical artifact that reflects deliberate design choices about what to disclose, how to handle uncertainty, and how to position the Pile legally and ethically. Several documentation decisions merit detailed examination.

Acknowledged unknowns as documentation strategy. The datasheet repeatedly uses phrases like "we do not know," "as far as we are aware," "we have no idea," and "it is always a possibility." This is not sloppiness—it is a deliberate documentation strategy that distinguishes the Pile from industrial datasets whose contents are treated as opaque trade secrets. By explicitly cataloging the boundaries of their knowledge, the authors enable downstream users to make risk assessments that account for uncertainty. Examples include:

On OpenSubtitles collection: "We do not know how the data for OpenSubtitles was collected."
On OpenSubtitles timeframe: "We have no idea how old the contents of OpenSubtitles are."
On BookCorpus2 timeframe: "We have no idea how old the contents of BookCorpus2 are."
On Wikipedia (en) funding: "We believe that its costs are currently paid for by UNC Chapel Hill, but are not certain."
On EuroParl funding: "We believe that this dataset was funded the School of Informatics of the University of Edinburgh, Scotland, but were unable to confirm this fact."
On the presence of confidential data: "We are not aware of any confidential data in the Pile, though it is always a possibility."

This pattern of acknowledged uncertainty serves an important function: it prevents downstream users from assuming that the absence of reported issues means the absence of issues. By saying "we have no idea" rather than leaving a question unanswered, the authors signal that the relevant investigation was not performed, and that users who care about the answer should investigate themselves.

The fair use assertion and its legal framing. The datasheet directly addresses the most legally contentious aspect of the Pile: the presence of copyrighted material in several components. The key passage in Section 6 states:

"Some of the documents in the datasets that the Pile is based on are copyrighted. In particular, Books3 is almost entirely comprised of copyrighted works, and a substantial portion of arXiv and PhilPapers are as well. Other datasets, such as PubMed Central and GitHub contain documents that may be under limited licensing, but are not copyrighted as far as we are aware. All data contained in the Pile has been heavily processed to aid in language modeling research and no copyrighted text is contained in the Pile in its original form. As far as we are aware, under U.S. copyright law use of copyrighted texts in the Pile falls under the 'fair use' doctrine, which allows for the unlicensed use of copyright-protected works in some circumstances."

This is a substantive legal argument, not merely a disclaimer. The authors stake out a position on the fair use question that has several components:

Transformative use: The data has been "heavily processed" from its original form—books have been converted to plain text, metadata has been stripped, and documents have been interleaved into a training corpus. This processing is argued to be transformative because the data is used not for reading or distribution of the original works, but for training statistical models that learn patterns of language.
Nature of the use: The use is for "language modeling research," which the authors position as a scientific activity with societal benefits.
Amount and substantiality: The argument that "no copyrighted text is contained in the Pile in its original form" suggests that the transformation is sufficiently thorough that the Pile does not function as a replacement for the original works—a user cannot read a book by extracting it from the Pile, because the text has been processed, shuffled, and intermixed with other sources.
Market effect: While not stated explicitly, the fair use analysis typically considers whether the use competes with the market for the original work. Language model training data does not serve as a substitute for buying or reading the original books.

The authors then add a crucial caveat: "Copyright law varies by country, and there may be additional restrictions on some of these works in your country. If you are in doubt, it is always advisable to speak to an intellectual property attorney." This acknowledges that the fair use defense is strongest in U.S. jurisdictions and that international users face different legal landscapes—for example, the EU's text and data mining exceptions are narrower than U.S. fair use, and some countries have no equivalent doctrine at all.

The modularity escape hatch. The datasheet explicitly encourages users who cannot use certain components to exclude them: "If you wish to exclude some components of the Pile for legal (or any other) reason, you can compile a custom remix of the datasets using the code on the EleutherAI GitHub to do so." This modularity is a response to the fact that different components carry different legal and ethical risks for different users. A researcher in the EU who cannot rely on fair use for Books3 can train on the remaining 21 components. A researcher studying bias in language models might exclude web scrape components known to contain toxic content. A researcher training a model for medical applications might exclude all non-academic sources to reduce the risk of the model learning unreliable health information. The code's public availability makes this remixing practical rather than theoretical.

Community governance and the snapshot model. The Pile is frozen at September 1, 2020 and will not be updated. The datasheet states: "EleutherAI does not plan to update the Pile. We may release a 'Pile Version 2' which will contain texts from a variety of languages as well as updated scrapes of the data sources that increase over time. However, in the event that such a dataset is created, it will be a separate dataset." This snapshot model serves several purposes. It ensures reproducibility: a researcher reporting results on "the Pile" can be confident that others are using exactly the same data, not a newer version with different composition. It also limits maintenance burden: EleutherAI does not commit to ongoing updates, data removal requests, or quality improvements to an existing dataset. And it creates a versioned release model where improvements are captured in new, separately-named datasets that can be compared against the original Pile in controlled experiments.

The datasheet also states that "we do not accept submissions of new contributions to the dataset," clarifying that EleutherAI acts as a curator for the specific Pile snapshot but does not operate an open-contribution dataset. This distinguishes it from community-driven datasets like Wikipedia or Common Crawl (where anyone can contribute content) and positions it as a curated research artifact with a defined set of maintainers (the EleutherAI collective).

Subpopulation identification disclosure. The datasheet's Section 8 provides per-component answers to "Does the dataset identify any subpopulations?" with varying degrees of specificity. These disclosures are important because language models trained on data that identifies subpopulations may learn and reproduce stereotypes, biases, and associations related to those subpopulations.

For PubMed Central and PubMed Abstracts: "Many medical papers identify subpopulations in the course of their studies. We have confirmed the presence of papers that study race, gender, ability, nation of origin, religion, and sexual orientation." This is an active confirmation—the authors examined the data and found evidence of subpopulation identification—rather than a probabilistic guess.

For Books3: "Many books identify subpopulations in various ways. We have confirmed the presence of papers that study race, gender, ability, nation of origin, religion, and sexual orientation." The same active confirmation applies to BookCorpus2, PG-19, and OpenSubtitles (substituting "movies" for "books").

For Wikipedia (en): The datasheet provides the most detailed subpopulation disclosure, citing specific academic studies on bias in Wikipedia across dimensions of race (Adams et al., 2019; Xing and Vetter, 2020), gender (Reagle and Rhue, 2011; Wagner et al., 2015; Hargittai and Shaw, 2015; Graells-Garrido et al., 2015), ability (Phillips, 2016; Derby, 2012), nation of origin (Rask, 2008; Lee and Chun, 2017), religion (Callahan and Herring, 2011; Ball, 2021), and sexual orientation (Eisner, 2013). This extensive citation list serves double duty: it documents that Wikipedia contains subpopulation identifications, and it provides downstream users with a curated bibliography of research on Wikipedia bias that they can use to understand and mitigate potential harms in models trained on this data.

For DM Mathematics: "The DM Mathematics dataset does not contain any data about people," making it the only component definitively free of subpopulation identification.

For GitHub: "We have no reason to believe that any GitHub code identifies any subpopulations," a weaker claim acknowledging that code comments could in principle contain demographic references, but that this is unlikely to be systematic.

For all other components (Pile-CC, arXiv, StackExchange, USPTO, Ubuntu IRC, EuroParl, YouTube Subtitles, PhilPapers, NIH ExPORTER, HackerNews, Enron Emails), the datasheet states: "We do not know the extent to which [component] identifies any subpopulations, although we expect that it does." This consistent pattern—acknowledging expectation of subpopulation identification without confirming it—reflects a realistic assessment: in any sufficiently large corpus of human-authored text, some documents will discuss or identify demographic groups, but systematically auditing for this across hundreds of millions of documents is impractical.

Offensive content and sensitive data disclosures. The datasheet's handling of offensive content is notably brief. For the question "Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?" the answer is: "The answer is 'probably' for all components other than GitHub and DM Mathematics, for which the answer is 'probably not.'" This single-sentence answer across 22 components reflects a realistic understanding: in a corpus of 825 GiB of largely unfiltered human-authored text, offensive content is statistically certain to exist, especially in the web scrape components (Pile-CC, OpenWebText2) which include the unfiltered internet, and in the dialogue components (Ubuntu IRC, OpenSubtitles, YouTube Subtitles) which capture informal conversation where offensive language is common. The exceptions—GitHub (code, not natural language) and DM Mathematics (synthetic math problems)—are the only components where the datasheet assesses offensive content as "probably not" present.

For sensitive data (racial/ethnic origins, sexual orientations, religious beliefs, political opinions, financial/health data, government IDs, criminal history), the answer is similarly terse: "We do not know the extent to which this is the case, although we expect that it does." This again reflects the practical impossibility of auditing 825 GiB of text for sensitive content, while acknowledging that such content is likely present given the diversity of sources.

Consent disclosures. The datasheet's answers to consent-related questions reveal a fundamental tension in the Pile's construction. For "Were the individuals in question notified about the data collection?" the answer is: "No." For "Did the individuals in question consent to the collection and use of their data?" the answer is: "The extent to which consent was obtained varies by dataset. For details about the provenance of each dataset, see Gao et al. [2020]." For "If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent?" the answer is: "No."

These answers reflect the fact that for most components, data was collected from public sources (websites, repositories, databases) without direct engagement with the individuals who authored the text. The individuals who wrote Wikipedia articles, posted on StackExchange, submitted papers to arXiv, or commented on HackerNews did so with the knowledge that their contributions would be publicly visible, but not necessarily with the knowledge that their text would be incorporated into a language model training corpus. The lack of a revocation mechanism means that individuals who later become uncomfortable with their data's inclusion have no formal process for requesting its removal (beyond contacting EleutherAI informally, which the datasheet does not provide a mechanism for).

No ethical review. The datasheet states: "No formal ethical review process was done because EleutherAI does not have an associated IRB." EleutherAI is a decentralized research collective without institutional affiliation, and therefore has no Institutional Review Board to consult. The authors note that "ethical considerations were discussed throughout the data collection process and is documented in our paper," implying an informal, community-based ethical review rather than a formal institutional process. This is both honest about the constraints of decentralized research and potentially concerning for users who would prefer datasets that have undergone formal ethical review—the Pile provides transparency about its review process, but that process does not include the kinds of external oversight, risk assessment, or impact analysis that an IRB would require.

The datasheet also states that no "data protection impact analysis" has been conducted, consistent with the lack of formal review. This means that questions about the Pile's compliance with regulations like GDPR (which requires Data Protection Impact Assessments for high-risk processing) have not been systematically addressed in the documentation, though the use of publicly available data rather than directly-collected personal data may mitigate some of these obligations.

4. Key Insights and Innovations

Innovation 1: Dataset Documentation as a First-Class Research Contribution

The Pile datasheet does something that, at the time of its release, was genuinely unusual in the NLP community: it treats the documentation of a dataset as a standalone scholarly artifact deserving of the same rigor, detail, and intellectual care as the dataset itself. This is not merely an administrative appendix or a boilerplate README — it is a 44-page structured document that systematically walks through 8 categories of metadata for each of 22 constituent sources, answering the questions that a responsible practitioner would need to ask before using the data for research.

What makes this distinctive is not the documentation framework itself. Gebru et al. (2021) had proposed "Datasheets for Datasets" as a standardized practice, and the Pile datasheet explicitly follows that template. Rather, the innovation is in the depth, honesty, and strategic function of the documentation. Prior to the Pile, major NLP datasets were released with varying levels of accompanying information: some (like C4) had detailed technical papers describing their construction but minimal ethical or legal documentation; others (like BookCorpus) became widely used despite having no official documentation at all, with researchers relying on third-party reconstructions and secondhand descriptions. The Pile established a new baseline for what dataset documentation could and should include.

Several specific documentation moves exemplify this shift:

The practice of systematically acknowledging unknowns. The datasheet repeatedly states "we do not know," "we have no idea," "as far as we are aware," and "it is always a possibility." In Section 4, the authors say of OpenSubtitles: "We do not know how the data for OpenSubtitles was collected." Of BookCorpus2's timeframe: "We have no idea how old the contents of BookCorpus2 are." Of EuroParl's funding: "We believe that this dataset was funded by the School of Informatics of the University of Edinburgh, Scotland, but were unable to confirm this fact." This is not sloppiness — it is a deliberate strategy to prevent downstream users from treating the absence of reported problems as evidence of their absence. In an era when datasets were often released with minimal documentation (creating what Bandy and Vincent, 2021, later called "documentation debt"), the Pile's explicit catalog of unknowns served as a counter-norm: transparency about what you don't know is itself valuable information for risk assessment.

The fair use assertion as a substantive legal argument, not a disclaimer. The datasheet does not simply note that copyrighted material exists in the dataset and advise caution. It stakes out a multi-part legal position: the use is transformative (data has been "heavily processed" from its original form), the processing serves research purposes rather than content redistribution, and "no copyrighted text is contained in the Pile in its original form." This is a particular reading of U.S. fair use doctrine that was, in 2021, legally untested in the specific context of language model training data. By making this argument explicit in the documentation, the authors provided downstream users with a legal rationale they could cite, while simultaneously flagging that "copyright law varies by country" and recommending consultation with "an intellectual property attorney." This dual move — asserting fair use while acknowledging jurisdictional variation — is legally substantive rather than evasive, and it anticipated the legal debates that would intensify around training data in subsequent years (culminating in high-profile lawsuits against model developers in 2023-2024).

The modular "escape hatch" for users with different risk tolerances. The datasheet explicitly encourages "compil[ing] a custom remix of the datasets" to exclude components that pose legal, ethical, or quality concerns. This is a small design decision with large implications: it acknowledges that no single composition of the dataset can be appropriate for all users in all jurisdictions, and it provides the mechanism (public code) to operationalize that acknowledgment. This distinguishes the Pile from monolithic dataset releases where the only choices are "use all of it" or "use none of it."

Citing external research on bias as part of the dataset description. The Wikipedia (en) subpopulation identification entry cites nine separate academic studies documenting bias in Wikipedia across race, gender, ability, nation of origin, religion, and sexual orientation. This is not merely a content warning — it is a curated mini-review of relevant scholarship, providing downstream users with both evidence that the problem exists and a starting bibliography for understanding its dimensions. The contrast with earlier dataset releases, which typically mentioned bias only in passing (if at all, e.g., "our dataset may contain biases from the underlying data"), is stark.

The significance of this innovation extends beyond the Pile itself. The datasheet demonstrated that thorough dataset documentation was feasible even for enormous, heterogeneous corpora — that the cost of documentation was not prohibitive and that the result was useful to the community. In the years following its release, comprehensive dataset documentation became an expected norm in NLP (codified in conference requirements and funding agency guidelines), and the Pile's datasheet was frequently cited as an exemplar. This is a rare case where a dataset's most influential contribution may have been how it was documented rather than what it contained.

Innovation 2: The Snapshot Release Model as a Reproducibility Commitment

The Pile introduced a release model that was subtle but structurally important: the dataset is a frozen snapshot, not a living resource. The datasheet states explicitly: "EleutherAI does not plan to update the Pile. We may release a 'Pile Version 2'... However, in the event that such a dataset is created, it will be a separate dataset." This decision, and its implications, distinguish the Pile from several categories of prior work.

Before the Pile, major NLP datasets fell into roughly three categories with respect to versioning. Some were static by nature: once created, they were distributed as-is, but with no explicit commitment to versioning — if the host updated the files, there was no guarantee that old versions remained accessible (BookCorpus was a notorious example, becoming effectively unavailable when its download link went offline). Some were dynamic by design: Common Crawl releases new snapshots monthly, Wikipedia dumps are continuously updated, and C4 was periodically re-released with updated filtering. These dynamic datasets had the advantage of freshness but created a reproducibility problem: a paper reporting results on "C4" or "Common Crawl" might mean different things depending on which version was used, and often the exact version was not specified. Some were monolithic releases with ambiguous versioning: large industrial datasets like GPT-3's training corpus were released only as descriptions in papers, with no public access to any version at all.

The Pile's model — explicitly frozen, separately versioned, with the code to reconstruct it made public — addresses a specific tension in dataset governance. By freezing the Pile at September 1, 2020, the authors made a tradeoff: the dataset would become progressively "stale" as new text was created (new arXiv papers, new Wikipedia articles, new HackerNews posts), but every research result based on "the Pile" would refer to exactly the same set of documents, enabling direct comparison and reproduction. This is essentially the reproducibility-over-recency tradeoff that is standard in benchmark design (ImageNet is not continuously updated with new photographs) but was not, at the time, standard for language model pretraining corpora.

What makes this more than just a practical decision is the governance rationale it implies. By declining to accept "submissions of new contributions to the dataset" and positioning any future version as a "separate dataset," EleutherAI adopted a curator rather than a platform model. This contrasts with datasets like Wikipedia or Common Crawl, which grow through community contribution and where the maintainers' role is infrastructure provision rather than content curation. The Pile's frozen-snapshot model makes explicit that the dataset is a specific artifact created by specific people at a specific time, with deliberate compositional choices, rather than a neutral reflection of "the web" or "all available text." This curation model has implications for accountability: if the Pile has problems (biases, toxic content, legal issues), those problems can be traced to specific curation decisions made during its construction, rather than being attributed to the amorphous nature of "internet data."

The downstream impact of this model has been substantial. Research papers training models on the Pile can specify "the Pile" as their training corpus without ambiguity — a level of specificity that was not possible for "Common Crawl," "C4," or "WebText" at the time. The Pile has been used in hundreds of papers with consistent referential clarity, and the snapshot model has been adopted by subsequent dataset releases in the open-source language model ecosystem (e.g., RedPajama, Dolma, DCLM). This is a case where a process innovation (how the dataset is versioned and distributed) has had more lasting structural impact than any specific compositional choice about what text to include.

Innovation 3: The Multi-Source Composition as a Democratization Strategy

The Pile's composition — 22 sources spanning web text, academic literature, books, code, legal documents, dialogue, and synthetic mathematics — is often described as a contribution to model quality (diverse training data produces more capable models) or domain coverage (models trained on the Pile can perform specialized tasks that web-text-only models cannot). These claims are true and are supported by subsequent work showing that Pile-trained models exhibit capabilities in coding, mathematics, and biomedicine that purely web-trained models lack. But the more fundamental innovation is at the level of research infrastructure design: the Pile's multi-source composition is a democratization strategy, not merely a data mixture.

To see why this matters, consider the state of large language model training in 2020. The most capable models (GPT-3, Turing-NLG, PaLM precursors) were trained on proprietary data mixtures assembled by industrial labs with dedicated data engineering teams. A researcher outside these labs who wanted to train a competitive large language model had access to Common Crawl (via C4 or raw processing) and a handful of small, domain-specific corpora (Wikipedia, BookCorpus if they could find it). What they lacked was the ability to combine these sources at scale into a coherent training corpus with principled weighting. Each individual academic lab that wanted to train a model had to solve the same engineering problems — scraping, cleaning, deduplicating, mixing — independently, wasting enormous collective effort.

The Pile solves a coordination problem rather than a purely technical one. By assembling 22 sources — including several that EleutherAI created from scratch (Pile-CC, OpenWebText2, BookCorpus2) and several that required navigating arcane data formats (arXiv LaTeX conversion, USPTO patent parsing, FreeLaw HTML extraction) — the Pile amortized the fixed cost of building a diverse pretraining corpus across the entire research community. Any researcher could download the Pile and immediately train on a corpus whose diversity rivaled or exceeded what industrial labs had access to, without needing to replicate months of data engineering work.

The democratization aspect is visible in the choice of sources. The Pile includes components that serve the interests of specific research communities: biomedical NLP researchers get PubMed Central and PubMed Abstracts; legal NLP researchers get FreeLaw; code generation researchers get GitHub; mathematics reasoning researchers get DM Mathematics. None of these communities had, on their own, the resources to assemble a general-purpose pretraining corpus that included their domain alongside general web text. By including their data in a general-purpose corpus, the Pile enabled domain-specific researchers to participate in the large-scale pretraining paradigm without sacrificing domain depth. This is a coalitional strategy: the Pile's composition reflects an implicit bargain where each research community contributes its expertise in curating a specific component, and in return gains access to the combined corpus that none could have built alone.

The significance of this innovation is measurable in the dataset's adoption. As documented in Section 2 of the datasheet, the Pile has been used to train GPT-Neo, GPT-J, Jurassic-1, Wu Dao, GPT-NeoX, and numerous other models. These models span independent research collectives (EleutherAI itself), startups (AI21 Labs), academic collaborations (Wu Dao), and major industrial labs. The Pile became a shared substrate for language model research across organizational boundaries — a role that no prior public dataset had achieved at this scale. This is an infrastructure contribution whose value lies not in any single design choice but in the fact that it exists as a common resource that researchers can build on, compare against, and critique.

Innovation 4: Transparent Admission of Copyrighted Content as a Norm-Setting Move

Perhaps the most structurally significant decision in the Pile's documentation is the direct, unapologetic acknowledgment that copyrighted material is present in the dataset, combined with a specific legal rationale for its inclusion. The datasheet states: "Books3 is almost entirely comprised of copyrighted works, and a substantial portion of arXiv and PhilPapers are as well." This is not a leak or an oversight that the documentation is scrambling to address — it is the explicit, front-facing position of the dataset's creators.

To understand why this qualifies as an innovation, consider the prior norms around training data copyright. Before the Pile, the dominant practice was opacity: datasets were described in ways that obscured the copyright status of their contents. BookCorpus was widely used without systematic documentation of the copyright status of its books. WebText and GPT-3's training data were described only in aggregate statistics, with no per-component copyright analysis. C4's documentation focused on filtering heuristics and did not systematically address whether the filtered text included copyrighted material. This opacity served a strategic function: it allowed researchers to use copyrighted data without explicitly acknowledging that they were doing so, reducing legal exposure by avoiding the creation of paper trails that could be used as evidence in infringement claims.

The Pile's approach is the opposite: transparency with legal justification. By explicitly naming which components contain copyrighted material, describing the nature of the copyrighted works (published books, academic papers), and articulating a fair use rationale, the authors made a deliberate choice to bring the copyright question into the open. This serves several functions:

First, it forces the research community to confront the legal status of its training data rather than avoiding the question. A researcher who trains on the Pile cannot later claim ignorance about the presence of copyrighted material — the documentation makes it unambiguous. This creates informed consent at the level of dataset use: researchers choose to use the Pile with full knowledge of the copyright claims involved, rather than being passively exposed to legal risk they did not know about.

Second, it provides a legal defense framework that downstream users can adopt. The fair use argument articulated in the datasheet — transformative use for research, processed form that does not substitute for the original works, no market harm — is available to anyone training on the Pile who needs to justify their use of the data. This is not to say the argument will prevail in every jurisdiction or in every legal challenge, but it means that Pile users are not starting from scratch in constructing their legal position.

Third, it establishes a norm that dataset documentation should include copyright analysis. Subsequent major dataset releases in the open-source language model ecosystem (RedPajama, Dolma, FineWeb, DCLM) have followed this pattern, including explicit discussion of copyright status and fair use considerations as standard components of their documentation. The Pile did not invent fair use analysis of training data, but it demonstrated that such analysis could be included in dataset documentation without triggering immediate legal consequences — contrary to the fear that acknowledging copyright would invite lawsuits.

The legal landscape has evolved significantly since the Pile's release. Multiple lawsuits have been filed against language model developers alleging copyright infringement in training data, and the fair use defense remains legally contested (with some cases still in progress as of 2024). The Pile's documentation position — transparency without apology — can be seen in retrospect as an early and influential move in the ongoing legal and normative negotiation over what constitutes acceptable use of copyrighted text for language model training. Whether one views this as courageous or reckless depends on one's assessment of the underlying legal merits, but it is unquestionably a distinctive intellectual position that has shaped subsequent practice.

5. Experimental Analysis

This section differs fundamentally from a traditional experimental analysis because the Pile paper is not a paper that presents experimental results. It is a dataset documentation artifact—a datasheet following the Gebru et al. (2021) framework—whose purpose is to exhaustively inventory what the dataset contains, where it came from, how it was processed, and what legal and ethical considerations apply. There are no accuracy curves, no ablation studies, no baseline comparisons, and no FLOPs-matched benchmarks. The "experiment" is the dataset itself, and the "results" are the document's systematic disclosures about its composition, provenance, and limitations.

This creates an unusual situation for the Experimental Analysis section: rather than evaluating whether the paper's empirical claims are supported by rigorous measurements, we must evaluate whether the documentation itself functions as an adequate substitute for the traditional experimental validation that would normally accompany a dataset release. In other words: does this datasheet give a practitioner enough information to decide whether and how to use the Pile? Does it honestly report what is known and unknown about the data? Does it provide the kind of evidence that would allow a researcher to reproduce the dataset or audit its contents?

What follows is structured in the spirit of experimental analysis, adapted to a dataset documentation paper. The "evaluation methodology" describes what the datasheet purports to do. The "main quantitative results" reports the compositional statistics and per-component disclosures that constitute the paper's empirical content. The "ablation studies and robustness checks" examines what the datasheet includes and omits across its eight structured sections. The "critical assessment" evaluates whether the disclosed information supports the implicit claims that the Pile is fit for purpose as a large-scale language modeling corpus and that its documentation is sufficiently thorough for informed use.

Evaluation Methodology

Dataset. The Pile itself is the subject of the datasheet, not a benchmark on which it is evaluated. The datasheet describes the Pile as 825.18 GiB of text comprising 211,043,181 documents drawn from 22 distinct sources, with a recommended train/validation/test split of approximately 99.8%/0.1%/0.1%. The constituent sources are: Pile-CC (227.12 GiB, 54,953,117 documents), PubMed Central (90.27 GiB, 3,098,931), Books3 (100.96 GiB, 196,640), arXiv (56.21 GiB, 1,264,405), GitHub (95.16 GiB, 19,021,454), OpenWebText2 (62.77 GiB, 17,103,059), FreeLaw (51.15 GiB, 3,562,015), Wikipedia (en) (6.38 GiB, 6,033,151), StackExchange (32.20 GiB, 15,622,475), USPTO Backgrounds (22.90 GiB, 5,883,037), PubMed Abstracts (19.26 GiB, 15,518,009), PG-19 (10.88 GiB, 28,602), OpenSubtitles (12.98 GiB, 446,612), DM Mathematics (7.75 GiB, 1,014,997), BookCorpus2 (6.30 GiB, 17,868), Ubuntu IRC (5.52 GiB, 10,605), EuroParl (4.59 GiB, 69,814), YouTube Subtitles (3.73 GiB, 173,651), PhilPapers (2.38 GiB, 33,990), NIH ExPORTER (1.89 GiB, 939,668), HackerNews (3.90 GiB, 831,198), and Enron Emails (0.88 GiB, 517,401). All figures are reported in Section 3 of the datasheet under "How many instances are there in total."

Base model(s). Not applicable. The datasheet does not train or evaluate any models. However, it cites models that have been trained on the Pile, including GPT-Neo (Black et al., 2021), GPT-J (Wang and Komatsuzaki, 2021), Jurassic-1 (Lieber et al., 2021), Wu Dao (Tang, 2021), and GPT-NeoX (Andonian et al., 2021), establishing that the dataset has been used successfully for large-scale pretraining. The Pile was designed with the explicit goal of training models comparable to GPT-3 (Brown et al., 2020), which was trained on a proprietary mixture of filtered Common Crawl, WebText2, books, and Wikipedia.

Metrics. The datasheet does not define evaluation metrics in a traditional sense. Instead, its "measurements" are compositional: raw document counts, raw sizes in GiB, and qualitative descriptions of instance types. The datasheet explicitly notes that these raw sizes differ from the effective training sizes due to deliberate upsampling of smaller sources, deferring the effective weights to Figure 1 of Gao et al. (2020). The validation and test splits (0.1% each) are provided for computing language modeling perplexity, but no perplexity numbers are reported in the datasheet itself.

Baselines. The datasheet positions the Pile against several implicit baselines through its discussion of prior work and motivation. The baseline datasets are C4 (Raffel et al., 2020)—a filtered Common Crawl corpus of comparable size but from a single source—and OpenWebTextCorpus (Gokaslan and Cohen, 2019)—a Reddit-derived web corpus that is substantially smaller (38 GiB vs. the Pile's 825 GiB). The Pile is distinguished from these baselines not by quantitative comparison but by structural differences: multi-source composition (22 sources vs. 1), deliberate inclusion of non-web text (books, academic papers, code, legal documents, dialogue), and transparent documentation of contents and limitations.

Generation budget / compute accounting. Not applicable in the traditional sense. The datasheet does not report training FLOPs, inference costs, or generation budgets. The relevant "cost" is the storage and bandwidth for distributing 825.18 GiB of text, which the datasheet addresses by listing multiple distribution channels (EleutherAI GitHub, community mirrors, The Eye archive).

Cross-validation / statistical protocol. Not applicable. There is no statistical validation, no significance testing, and no confidence intervals reported. The datasheet's evidence is documentary rather than statistical: it reports what the authors observed during dataset construction and what they verified through manual inspection. For example, Section 8 reports that the authors "have confirmed the presence of papers that study race, gender, ability, nation of origin, religion, and sexual orientation" in PubMed Central, but does not quantify the frequency or distribution of such mentions.

Main Quantitative Results

The "results" of this paper are the compositional statistics and per-component descriptions reported across Sections 2 through 8. What follows organizes these disclosures by the logical axes along which the datasheet provides evidence about the Pile's fitness for use.

Compositional Coverage and Scale

The headline numbers establish that the Pile is sufficiently large and diverse for large-scale language model pretraining. At 825.18 GiB and 211 million documents, it is comparable in scale to C4 (within a factor of 2 for filtered text volume) and substantially larger than OpenWebTextCorpus (38 GiB). The 22 sources span at least six distinct textual registers: web text (Pile-CC, OpenWebText2), academic and scientific literature (arXiv, PubMed Central, PubMed Abstracts, PhilPapers, NIH ExPORTER), long-form narrative (Books3, BookCorpus2, PG-19), code (GitHub), legal and governmental documents (FreeLaw, USPTO Backgrounds, EuroParl), and dialogue (OpenSubtitles, Ubuntu IRC, YouTube Subtitles, HackerNews, StackExchange, Enron Emails). The presence of DM Mathematics—a fully synthetic dataset of 1,014,997 algorithmically generated math problems—adds a category not present in web-only corpora.

The datasheet reports the raw size of every component (Section 3, "How many instances are there in total"), enabling practitioners to understand the corpus's composition before any upsampling. Pile-CC is the largest single component at 227.12 GiB (27.5% of total), followed by Books3 (100.96 GiB, 12.2%), GitHub (95.16 GiB, 11.5%), and PubMed Central (90.27 GiB, 10.9%). The long tail of smaller sources is substantial: 11 components are under 10 GiB each, collectively representing less than 10% of raw volume. This composition reveals the Pile's intended balance: web text provides broad coverage, while curated sources provide domain depth, with upsampling used at training time to prevent the smaller sources from being statistically negligible.

The datasheet explicitly flags that raw sizes are not training sizes ("Note that the Pile has had its datasets weighted. The sizes reported here are the raw sizes"), deferring the effective training weights to Gao et al. (2020). This is a crucial caveat: without consulting the companion paper, a practitioner cannot know the actual sampling distribution their model will see during training. A researcher training on "the Pile" is not training on the distribution implied by the raw component sizes.

Provenance Transparency

The datasheet reports the acquisition mechanism for every component (Section 4, "What mechanisms or procedures were used to collect the data"). The mechanisms fall into five categories with varying provenance quality:

Official repository downloads (PubMed Central, Wikipedia, USPTO, PubMed Abstracts, NIH ExPORTER): data obtained directly from institutional hosts with documented release procedures.
Custom EleutherAI scrapes (arXiv, GitHub, StackExchange, YouTube Subtitles, PhilPapers, HackerNews): data collected by EleutherAI members writing scrapers against public websites or APIs, with collection parameters specified (date ranges, filtering criteria).
Custom EleutherAI compilations (Pile-CC, OpenWebText2, BookCorpus2): new datasets created by EleutherAI through processing of existing public data (Common Crawl, Reddit-linked webpages, SmashWords books).
Third-party dataset reuse (Books3, PG-19, DM Mathematics, OpenSubtitles, EuroParl, Enron Emails): pre-existing datasets incorporated without modification, with provenance documentation that varies from comprehensive (EuroParl's creation is documented in Koehn, 2005) to opaque ("not all details about the creation of Books3 have been publicly disclosed").
Legal/investigative provenance (FreeLaw, Enron Emails): data originally created through legal mandates or investigations and later repurposed for research.

This disclosure enables practitioners to assess provenance risk independently: a researcher training a model for commercial deployment might avoid Books3 (opaque provenance, entirely copyrighted) while being comfortable with PubMed Central (government-funded, publicly mandated release). The datasheet is explicit when provenance is unknown: "We do not know how the data for OpenSubtitles was collected" and "We have no idea how old the contents of BookCorpus2 are."

The funding disclosures (Section 2, "Who funded the creation of the dataset?") reveal a mix of government, non-profit, corporate, and individual sources. The Pile itself was "created by individuals working on their own time without funding," while components range from NIH-funded biomedical databases to Alphabet-subsidiary-created (DeepMind) datasets to an independently-created book corpus (Shawn Presser). For EuroParl, the datasheet states "We believe that this dataset was funded by the School of Informatics of the University of Edinburgh, Scotland, but were unable to confirm this fact"—a disclosure that is itself informative about the limits of provenance tracking.

Known Content Issues: Copyright, Offensive Material, and Subpopulation Identification

The datasheet's most consequential disclosures concern the presence of potentially problematic content.

Copyright. Section 6 explicitly names three components with substantial copyrighted material: "Books3 is almost entirely comprised of copyrighted works, and a substantial portion of arXiv and PhilPapers are as well." The datasheet further notes that PubMed Central and GitHub "contain documents that may be under limited licensing, but are not copyrighted as far as we are aware." This is a direct, unambiguous disclosure that distinguishes the Pile from datasets where copyright status was omitted from documentation entirely. The datasheet couples this disclosure with a fair use argument and a jurisdictional caveat ("Copyright law varies by country"), providing both a legal rationale and a warning.

Offensive content. Section 8 reports that the answer to whether the dataset contains offensive material "is 'probably' for all components other than GitHub and DM Mathematics, for which the answer is 'probably not.'" This is the entirety of the offensive content analysis—a one-sentence probabilistic assessment across all 22 components. The datasheet provides no examples, no frequency estimates, and no filtering procedures for removing such content.

Subpopulation identification. Section 8 provides per-component answers to whether the dataset identifies subpopulations. For PubMed Central, PubMed Abstracts, Books3, PG-19, OpenSubtitles, and BookCorpus2, the datasheet reports active confirmation: "We have confirmed the presence of papers that study race, gender, ability, nation of origin, religion, and sexual orientation." For Wikipedia, the datasheet provides an extensive citation list of external research documenting bias (Adams et al., 2019; Reagle and Rhue, 2011; Wagner et al., 2015; Hargittai and Shaw, 2015; Graells-Garrido et al., 2015; Phillips, 2016; Derby, 2012; Rask, 2008; Lee and Chun, 2017; Callahan and Herring, 2011; Ball, 2021; Eisner, 2013). For DM Mathematics, it states "does not contain any data about people." For all remaining components (Pile-CC, arXiv, StackExchange, USPTO, Ubuntu IRC, EuroParl, YouTube Subtitles, PhilPapers, NIH ExPORTER, HackerNews, Enron Emails), the datasheet reports: "We do not know the extent to which [component] identifies any subpopulations, although we expect that it does." This pattern—active confirmation where possible, honest uncertainty where not—provides a differentiated risk map rather than a blanket disclaimer.

Sensitive data. Section 8 reports: "We do not know the extent to which this is the case, although we expect that it does" for whether the dataset contains data that might be considered sensitive (racial or ethnic origins, sexual orientations, religious beliefs, political opinions, financial/health data, government IDs, criminal history). No per-component breakdown is provided.

Consent. Section 8 reports that individuals were not notified about data collection ("No"), that consent varied by dataset ("The extent to which consent was obtained varies by dataset"), and that no revocation mechanism exists ("No").

Adoption Evidence

Section 2 provides a list of models trained on the Pile (GPT-Neo, GPT-J, Jurassic-1, Wu Dao, GPT-NeoX) and papers studying properties of Pile-trained models (Mitchell et al., 2021; Peyrard et al., 2021; Matiana et al., 2021; Mukherjee et al., 2021; Magee et al., 2021; Lee et al., 2021). Per-component adoption is also reported: Pile-CC has been used by Luo et al. (2021), Kharya and Alvi (2021), and Askell et al. (2021); Books3 by Luo et al. (2021), Wang et al. (2021), and Kharya and Alvi (2021); and so on across all components. This adoption evidence serves as a form of implicit validation: the dataset has been used successfully (in the sense of producing working models) by multiple independent groups. However, the datasheet reports no quantitative comparisons of models trained on the Pile versus other corpora—those comparisons appear in the cited papers, not in the datasheet itself.

Documentation Completeness

The datasheet systematically answers the questions from Gebru et al.'s framework across 8 sections. For each question, it provides a Pile-level answer followed by per-component answers where components differ. The structure is exhaustive: every component is addressed (or explicitly noted as unknown) for every question in the framework. The questions cover motivation, composition, collection process, preprocessing, distribution, maintenance, and legal/ethical considerations.

Notable documentation gaps explicitly acknowledged include: the timeline of OpenSubtitles ("We have no idea how old the contents of OpenSubtitles are"), the timeline of BookCorpus2 ("We have no idea how old the contents of BookCorpus2 are"), the funding of EuroParl (uncertain), the timeframe of Books3 collection (not publicly disclosed), the methodology of Books3 collection (not publicly disclosed), and the extent of subpopulation identification in 11 of 22 components ("We do not know"). These gaps are not omissions in the sense of things the authors forgot to document—they are limitations explicitly cataloged in the documentation itself.

Ablation Studies and Robustness Checks

There are no traditional ablation studies in this paper. However, the datasheet's structure can be understood as performing a kind of documentation ablation: it systematically varies the level of detail provided, revealing which aspects of a dataset can be reliably documented at scale and which cannot.

Provenance documentation across acquisition categories: The datasheet reveals a sharp gradient in documentation quality based on how data was acquired. Components downloaded from official repositories (PubMed Central, Wikipedia, USPTO, PubMed Abstracts, NIH ExPORTER) have precise collection dates, known funding sources, and documented institutional mandates. Components obtained through custom scrapes (arXiv, GitHub, StackExchange, YouTube Subtitles, PhilPapers, HackerNews) have precise collection parameters (date ranges, API endpoints, filtering criteria) because EleutherAI controlled the collection. Components reused from third parties (Books3, PG-19, DM Mathematics, OpenSubtitles, EuroParl, Enron Emails) have documentation quality that degrades to the level of the original source—and in several cases (Books3, OpenSubtitles), the original source did not provide sufficient documentation to answer basic questions about collection methodology and timeframe. This gradient is not an experimental finding in the traditional sense, but it is an empirical result of the documentation process: when you rely on third-party datasets, you inherit their documentation debt.

Subpopulation identification by component type: The datasheet's per-component subpopulation disclosures reveal an implicit taxonomy. Components consisting primarily of academic or professional writing about people (PubMed Central, FreeLaw, NIH ExPORTER) systematically identify subpopulations. Components consisting of fictional or narrative content (Books3, PG-19, OpenSubtitles, BookCorpus2) identify subpopulations through character descriptions and narrative. Components consisting of reference or encyclopedic content (Wikipedia) identify subpopulations extensively and have been systematically studied for bias. Components consisting of code or synthetic data (GitHub, DM Mathematics) do not identify subpopulations. Components consisting of general web text or domain-specific technical writing (Pile-CC, arXiv, StackExchange, USPTO, Ubuntu IRC, EuroParl, YouTube Subtitles, PhilPapers, HackerNews, Enron Emails) fall into the "we do not know" category. This taxonomy is an empirical finding about which kinds of text corpora reliably contain demographic information, and it provides a heuristic for practitioners assessing bias risk in new datasets.

Offensive content disclosure as a binary flag: The datasheet's offensive content disclosure—"probably" for 20 of 22 components, "probably not" for GitHub and DM Mathematics—effectively treats offensive content as a binary yes/no property rather than a quantitative distribution. This is a documentation decision with empirical implications: it provides no information about frequency, severity, or component-specific patterns of offensive content. A practitioner trying to decide whether to include Ubuntu IRC (which might contain toxic gaming and tech support conversations) versus OpenSubtitles (which might contain offensive dialogue from movies) gets no differentiated guidance. The binary flag is a robustness check in reverse: it demonstrates that at 825 GiB scale and with volunteer labor, even coarse-grained content auditing is infeasible.

Absence of filtering documentation in the datasheet itself: The datasheet states that "the data was extensively preprocessed as documented in Gao et al. [2020]" (Section 5) but does not reproduce that documentation. This is a structural decision that limits the datasheet's standalone utility: a reader who wants to understand what quality filtering was applied, what deduplication thresholds were used, or what boilerplate removal heuristics were employed must consult a separate paper. The datasheet provides per-component preprocessing summaries (jusText for Pile-CC, pandoc for arXiv, BeautifulSoup for FreeLaw, PDFBox for PhilPapers) but does not report parameters (e.g., minimum text length thresholds, discard rates, deduplication Jaccard similarity cutoffs). This division of documentation between datasheet and companion paper means that the datasheet is not a complete standalone description of the Pile's construction; it is one component of a multi-document documentation package.

Absence of quantitative bias or toxicity measurements: The datasheet reports no measurements of bias, toxicity, or content quality beyond the author's qualitative assessments ("we have confirmed the presence"). There is no statistical characterization of the corpus: no n-gram distributions, no topic model results, no language identification statistics, no perplexity baselines. This is not an oversight—it reflects the division of labor between the datasheet (which documents provenance, composition, and legal/ethical considerations) and the companion paper Gao et al. (2020) (which presumably contains quantitative characterization). But it means that a practitioner concerned about, say, the prevalence of gender stereotypes in the training data gets no quantitative evidence from the datasheet itself.

Critical Assessment

The Pile datasheet makes no explicit empirical claims in the traditional sense—it does not assert that the Pile achieves a certain accuracy on a benchmark or that one preprocessing method outperforms another. Its implicit claims are structural: that the Pile is sufficiently well-documented for informed use, that its multi-source composition enables broad model capabilities, that its transparency about legal and ethical issues distinguishes it from prior datasets, and that it has been successfully adopted by the research community. What follows evaluates whether the documentation provided supports these implicit claims, and what a practitioner would still need to investigate independently before using the Pile.

Does the Datasheet Enable Fully Informed Use?

The documentation is remarkably thorough by the standards of 2021 dataset releases, but it is not sufficient for fully informed use on its own. Several categories of information that a risk-conscious practitioner would need are incomplete or deferred to other documents.

First, the effective training distribution is not specified in the datasheet. The datasheet reports raw component sizes but explicitly notes that the Pile uses deliberate upsampling, deferring the actual training weights to Figure 1 of Gao et al. (2020). A practitioner downloading "the Pile" and training on it with uniform sampling across documents would get a different training distribution than the one the Pile's creators intended and used in their own experiments. The datasheet flags this issue but does not resolve it; a responsible user must consult the companion paper to understand what distribution they should be sampling from.

Second, the filtering and deduplication parameters are described qualitatively but not quantitatively. The datasheet reports that Pile-CC used jusText for boilerplate removal, that OpenWebText2 and Pile-CC used MinHashLSH for deduplication, and that various components had minimum length filters, but it does not report thresholds, discard rates, or hyperparameter settings. A practitioner trying to reproduce the Pile's construction or to understand what fraction of the original source data was discarded cannot determine this from the datasheet alone. The datasheet defers to Gao et al. (2020) and the public code repository. This means the datasheet is one component of a multi-document documentation suite, not a self-contained artifact.

Third, the content risk characterization is coarse-grained for most components. For 11 of 22 components, the datasheet cannot say whether subpopulations are identified ("We do not know"). For 20 of 22 components, the offensive content assessment is a single word: "probably." There is no quantitative sampling, no topic distribution analysis, and no content category breakdown that would allow a practitioner to say "this component is 3× more likely to contain toxic content than that component." This is not necessarily a failure of the datasheet—systematically auditing 825 GiB of text for sensitive content is a massive undertaking that would require automated classifiers far beyond what was available in 2021—but it is a limitation that practitioners should understand. The datasheet honestly reports the limits of its authors' knowledge, which is valuable, but that honesty does not substitute for the missing information.

Fourth, the license of individual components is not fully specified. The datasheet states that the Pile as a whole is distributed under the MIT License, but it also acknowledges that "some of the documents in the datasets that the Pile is based on are copyrighted" and that "PubMed Central and GitHub contain documents that may be under limited licensing." The MIT License applies to the compilation and processing code, not necessarily to the underlying data. The datasheet does not provide a per-component license analysis (e.g., which components are CC-BY, which are public domain, which are copyrighted with fair use as the only legal basis for use). A practitioner using the Pile for commercial purposes would need to perform this analysis independently.

Does the Multi-Source Composition Enable Broad Model Capabilities?

This claim is not evaluated in the datasheet itself. The datasheet asserts that the Pile was designed for training large-scale language models and that it has been used for this purpose (citing GPT-Neo, GPT-J, Jurassic-1, Wu Dao, GPT-NeoX), but it reports no intrinsic evaluation of the corpus and no extrinsic evaluation of models trained on it. The evidence for the Pile's effectiveness is entirely external—it consists of citations to papers that have used the Pile and achieved results. The datasheet does not report perplexity on the Pile's test set for any model, does not compare Pile-trained models against models trained on other corpora, and does not ablate individual components to measure their contribution to downstream performance.

This is appropriate for a datasheet, which documents what the dataset is rather than what it achieves. But it means the claim that the Pile "enables broad model capabilities" is not supported by evidence within the datasheet itself. A practitioner deciding whether to train on the Pile versus C4 versus a custom corpus gets no comparative performance signal from this document. They must consult the companion paper (Gao et al., 2020) and the cited model papers for that evidence.

Transparency About Legal and Ethical Issues: How Far Does It Go?

The datasheet is unusually transparent relative to contemporaneous dataset releases, but its transparency has specific boundaries that are worth characterizing precisely.

What it discloses explicitly: the presence of copyrighted material (naming specific components), the absence of consent from data subjects, the absence of an IRB review, the absence of a data protection impact analysis, the likely presence of offensive content, the confirmed presence of subpopulation identification in several components, the absence of a mechanism for data removal requests, and the jurisdictional variability of copyright law.

What it does not disclose: the specific copyrighted works in Books3 (titles, authors, publishers), the frequency or severity of offensive content, the distribution of subpopulation mentions, the names or identifying information of individuals who might appear in the data, the steps taken (if any) to filter PII, and the results of any legal review (the datasheet mentions none was conducted).

The distinction between these two categories reflects a deliberate strategy: disclose what is known about systematic properties of the data (this component is mostly copyrighted works; that component probably contains offensive content) while not attempting to enumerate specific instances (which books, which offensive passages, which individuals). This strategy is pragmatic—enumerating specific copyrighted works in Books3 would arguably strengthen a copyright infringement claim by demonstrating that the dataset creators knew exactly which works they were redistributing—but it also limits the information available to practitioners who want to assess risk at a finer granularity. A publisher checking whether their specific books appear in Books3 gets no answer from the datasheet.

The consent disclosures are particularly stark: "No" to whether individuals were notified, "No" to whether revocation is possible, and a non-answer to whether consent was obtained ("varies by dataset"). These disclosures are honest, but they also reveal that the Pile was constructed under a data collection paradigm that would likely not satisfy the requirements of GDPR (which mandates notice and purpose specification) or emerging AI regulation (which increasingly requires consent or legitimate interest justifications for training data). The datasheet does not analyze GDPR compliance, and the statement that "no formal ethical review process was done" means that these questions were not systematically addressed during construction.

Are There Missing Evaluations That Would Strengthen the Documentation?

Several forms of analysis, standard in subsequent dataset releases, are absent from the Pile datasheet:

Quantitative content statistics: no n-gram frequency distributions, no topic model results, no language identification breakdown, no readability score distributions, no domain category taxonomy. These analyses would help practitioners understand the linguistic properties of the corpus beyond its source-level composition.

Toxicity and bias measurements: no Perspective API scores, no hate speech classifier results, no gender/race/religion association tests, no stereotype quantification. The datasheet acknowledges that these issues probably exist but provides no measurement. Subsequent datasets (e.g., C4, Dolma) have included such measurements as standard components of their documentation.

Per-component deduplication rates: the datasheet mentions that Pile-CC and OpenWebText2 were deduplicated with MinHashLSH, but reports no statistics on what fraction of documents were identified as duplicates and removed. Without this information, a practitioner cannot assess the redundancy of the corpus or compare it against other deduplicated corpora.

Cross-source overlap analysis: the datasheet does not report whether the same documents appear in multiple components (e.g., an arXiv paper that was also posted as a HackerNews submission, or a PubMed Central abstract that also appears in PubMed Abstracts—though for the latter, the datasheet notes that duplicates were explicitly removed). Cross-source overlap can cause models to memorize specific passages and reduce effective data diversity.

Model-based evaluation: no perplexity measurements, no downstream task benchmarks, no scaling law analysis. These would provide direct evidence about the Pile's suitability for language model training, but they properly belong in the companion paper rather than the datasheet.

PII audit: the datasheet states "we have no reason to believe that personal identifying information (PII) is contained in most of the subsets" but reports no systematic PII scan using named entity recognition or pattern matching. This is a significant gap for a dataset that includes emails (Enron), chat logs (Ubuntu IRC), and web text (Pile-CC, OpenWebText2), all of which are known to frequently contain names, addresses, phone numbers, and email addresses.

The Adoption Evidence: Does It Validate the Dataset?

The list of models trained on the Pile (GPT-Neo, GPT-J, Jurassic-1, Wu Dao, GPT-NeoX) demonstrates that the dataset has been used at scale and that the resulting models were functional (they produced coherent text and achieved non-trivial benchmark performance). However, "functional" is not the same as "optimal" or "safe." The adoption evidence does not tell us whether these models would have been better trained on a different corpus, whether their failures are attributable to the Pile's composition, or whether the Pile's inclusion of copyrighted material was necessary (versus incidental) to their performance.

The per-component adoption lists (which papers have used which subsets) are useful for identifying communities that might have expertise with specific components, but they do not establish that the components are individually valuable for language model training. A component might be widely cited because it is included in the Pile (which is widely used) rather than because it contributes meaningfully to model capabilities.

Summary: What the Datasheet Achieves and What It Leaves Open

The Pile datasheet achieves what it sets out to do: it provides a structured, per-component inventory of the dataset's composition, provenance, processing, distribution, and known legal and ethical issues, following the Gebru et al. (2021) framework. It is transparent about the limits of its authors' knowledge, explicit about the presence of copyrighted material, and honest about the absence of formal ethical review and consent mechanisms. These qualities made it a landmark in dataset documentation at the time of its release.

However, the datasheet does not—and does not claim to—provide quantitative evidence about the Pile's suitability for language model training, the distribution of sensitive or toxic content, or the comparative performance of models trained on the Pile versus alternatives. These questions are deferred to the companion paper (Gao et al., 2020) and to the downstream model papers that use the Pile. The datasheet is a necessary component of responsible Pile use but is not sufficient; a practitioner must consult multiple sources (the companion paper, the code repository, the original component documentation where available, and independent legal counsel where appropriate) to make a fully informed decision. The datasheet's most important contribution is not answering every question but rather making explicit which questions remain unanswered, so that practitioners cannot claim ignorance about the risks they are accepting.

6. Limitations and Trade-offs

The Documentation's Utility Depends on External Resources Not Contained in the Datasheet

The datasheet is explicitly not a self-contained description of the Pile. It repeatedly defers critical technical details to the companion paper (Gao et al., 2020) and the public code repository, making the documentation a multi-source puzzle rather than a single reference point. The datasheet itself acknowledges this dependency structure: the effective training distribution weights are deferred to "Figure 1 of Gao et al. [2020]" (Section 3); the extensive preprocessing is summarized with "the data was extensively preprocessed as documented in Gao et al. [2020]" (Section 5); and the per-component processing parameters—filtering thresholds, deduplication hyperparameters, discard rates, minimum text lengths—are not reported in the datasheet at all.

The consequence is that a practitioner reading this document in isolation cannot determine exactly what is in the Pile or how to reproduce its construction. Without consulting the companion paper, they do not know what fraction of Common Crawl was discarded during filtering, what Jaccard similarity threshold was used for MinHash deduplication, what the effective sampling weights are for each component during training, or how many arXiv papers were discarded during pandoc conversion. These are not arcane details—they directly affect what a model trained on the Pile will learn, and different implementations of "training on the Pile" that use different sampling weights or different preprocessing parameters would produce different models. A practitioner who trains with uniform sampling across documents (the natural default) would get a fundamentally different training distribution than the one intended by the Pile's creators, because the raw component sizes (reported in the datasheet) differ from the upsampled effective sizes (reported only in Gao et al., 2020).

The datasheet provides partial descriptions of preprocessing tools—jusText for Pile-CC, pandoc 1.19.2.4 for arXiv, BeautifulSoup for FreeLaw, PDFBox for PhilPapers, MinHashLSH via DataSketch for deduplication—but without parameter settings, these tool names indicate only the general approach, not the specific transformation applied. Two different researchers running jusText with different settings would produce different Pile-CC components. The code repository (available on GitHub) partially addresses this by providing the actual scripts, but the datasheet does not guarantee that the documented parameters match the code, and the code itself may have dependencies on specific library versions (pandoc 1.19.2.4 is explicitly versioned; other tools are not) that affect reproducibility.

The paper does not measure this limitation directly—there is no ablation showing how sensitive downstream model performance is to preprocessing parameter choices, sampling weights, or tool versions. The documentation gap is acknowledged structurally (by directing the reader to other resources) but is not analyzed as a risk. The mitigation is partial and external: the companion paper and code repository exist, but the datasheet does not reproduce their content, meaning the documentation burden on the practitioner is higher than the datasheet's structure suggests.

Copyright Status Is Asserted, Not Legally Adjudicated

The datasheet stakes out a clear legal position: the use of copyrighted texts in the Pile "falls under the 'fair use' doctrine" because the data has been "heavily processed to aid in language modeling research" and "no copyrighted text is contained in the Pile in its original form" (Section 6). This is a substantive legal argument with several components—transformative use, research purpose, lack of market substitution—and it is presented as the basis on which the dataset is distributed under the MIT License.

The limitation is that this fair use assertion had not been tested in court at the time of the Pile's release, and the legal landscape has subsequently become more contested. The datasheet itself seems to acknowledge this uncertainty by adding "As far as we are aware" before the fair use claim, and by warning that "copyright law varies by country" and advising users to "speak to an intellectual property attorney" (Section 6). These caveats indicate that the authors recognize the fair use argument is a legal position, not an established fact.

The consequence is that downstream users of the Pile accept legal risk whose magnitude is unknown and difficult to quantify. A researcher or company training a model on the Pile cannot rely on the datasheet's fair use argument as a legal shield—particularly in jurisdictions outside the United States, where fair use doctrine does not apply, or where text and data mining exceptions are narrower. The datasheet explicitly names the components with copyrighted material (Books3 "almost entirely comprised of copyrighted works," "a substantial portion of arXiv and PhilPapers") and the nature of the infringement (redistribution of full-text books and papers), meaning any downstream user who has read the datasheet cannot claim ignorance of copyright status. The modularity escape hatch—the ability to "compile a custom remix of the datasets" excluding certain components—is offered as a partial mitigation, but the default distribution of the Pile includes the copyrighted components, and the datasheet does not quantify how much of the Pile's value derives from those components versus the non-copyrighted remainder.

The paper provides no evidence on this limitation because it is a legal question, not an empirical one. There is no citation to supporting case law, no expert legal opinion attached, and no analysis of how fair use applies specifically to language model training (as opposed to text reproduction or distribution). The documentation is transparent in acknowledging the presence of copyrighted material, but the absence of a definitive legal resolution means that users are operating in a zone of legal uncertainty. The mitigation—consulting an attorney, excluding components—places the burden on the user rather than resolving the underlying ambiguity.

Content Risk Characterization Is Binary and Non-Quantitative

The datasheet's disclosures about offensive content, sensitive data, and subpopulation identification are consistently framed at the level of "probably yes" or "probably no" rather than quantitative measurement. The offensive content assessment for 20 of 22 components is the single word "probably" (Section 8). For sensitive data (racial/ethnic origins, sexual orientations, religious beliefs, political opinions, financial/health data, government IDs, criminal history), the assessment is "We do not know the extent to which this is the case, although we expect that it does" (Section 8), with no per-component breakdown. For subpopulation identification, 11 of 22 components receive "We do not know the extent to which [component] identifies any subpopulations, although we expect that it does" (Section 8).

The consequence is that the datasheet provides no differentiated risk signal across components. A practitioner trying to decide whether the Ubuntu IRC chat logs (which might contain toxic gaming and tech support conversations) pose a different content risk profile than the Enron Emails (which contain private corporate communications) or the OpenSubtitles corpus (which contains movie dialogue) gets no guidance. The binary characterization collapses potentially large differences in harmful content prevalence into a single category, making it impossible for downstream users to make component-level inclusion decisions based on content safety. A user who wants to train a model that minimizes exposure to hate speech or personally identifying information cannot determine from the datasheet whether excluding Pile-CC (web text) would be more effective than excluding OpenWebText2 (Reddit-linked web text), or whether the curated academic sources (arXiv, PubMed Central) are substantially cleaner than the dialogue sources.

This limitation is acknowledged implicitly by the datasheet itself, which attributes the coarse-grained characterization to the practical impossibility of systematic auditing: "Due to the sheer scale of the data, it is impractical to systematically search through every text to validate that it is what it purports to be" (Section 1). The authors are transparent about the fact that they did not perform quantitative content analysis, and they frame the documentation as their best effort given resource constraints. However, the datasheet does not report even a sampled analysis—no random subsample of documents was manually reviewed for offensive content or PII, and no automated classifiers were applied to estimate prevalence. The "probably" assessment is therefore based on general knowledge about the source domains rather than on empirical examination of the Pile's actual contents.

The paper does not attempt to mitigate this limitation beyond the honesty of acknowledging it. There are no reported plans for future quantitative content audits, no suggestion of community processes for flagging problematic content, and no mechanism for downstream users to report discovered issues back to the dataset maintainers. The documentation is transparently limited, but that transparency does not substitute for the missing risk characterization.

Dataset Is Frozen at September 2020 With No Update or Removal Mechanism

The Pile is a snapshot frozen at September 1, 2020, with EleutherAI explicitly stating that they "do not plan to update the Pile" and that any future version "will be a separate dataset" (Section 7). The datasheet further states that "we do not accept submissions of new contributions to the dataset" (Section 7). This snapshot model is presented as a reproducibility feature, and it does serve that function—every reference to "the Pile" unambiguously identifies the same set of documents.

The consequence is that the Pile becomes progressively stale as a representation of human language and knowledge. Knowledge cutoff becomes a structural property of any model trained on the Pile: scientific knowledge is frozen at mid-2020 (PubMed Central complete as of June 2020, arXiv up to the July 2020 dump), current events coverage ends with the HackerNews scrape (September 2020), and all web-derived content (Pile-CC, OpenWebText2) reflects the state of the internet before September 2020. This staleness is a deliberate tradeoff for reproducibility, but the datasheet does not discuss how quickly different components become outdated, or what categories of knowledge degrade fastest. A model trained on the Pile in 2024 would lack knowledge of COVID-19 developments post-2020, the Russian invasion of Ukraine, advances in large language models themselves (GPT-3 was released in 2020; everything after is missing), and all subsequently published scientific literature, legal opinions, and code.

More fundamentally, the frozen-snapshot model means there is no mechanism for removing data even when problems are discovered. The datasheet explicitly reports that no consent revocation mechanism exists (Section 8: "No"). If an individual discovers their private information in the Enron Emails, their medical data in PubMed Central, or their personal communications in Ubuntu IRC, there is no process for requesting removal. If a publisher objects to their copyrighted books appearing in Books3, the dataset remains available in its original form. If researchers discover that a specific document contains dangerous content (e.g., instructions for weapons manufacture, non-consensual intimate imagery descriptions), that document remains in the distributed dataset permanently. This is structurally different from dynamic datasets or content platforms, where content can be removed in response to takedown requests, policy violations, or ethical concerns. The Pile's snapshot model prioritizes research reproducibility over the ability to respond to harms discovered after release.

The datasheet acknowledges the snapshot model but frames it positively—as a commitment to reproducibility and a practical necessity given volunteer maintenance capacity. It does not discuss the tradeoff between reproducibility and redress, nor does it propose any process for documenting known problems with the frozen dataset so that future users can be warned. A researcher downloading the Pile in 2024 has no way to know whether subsequent analysis has revealed toxic content clusters, PII exposure, or copyright complaints related to specific components, because there is no errata mechanism, no versioned documentation updates, and no community-maintained issue tracker referenced in the datasheet.

No Formal Ethical Review Process Was Conducted

The datasheet states plainly: "No formal ethical review process was done because EleutherAI does not have an associated IRB. Ethical considerations were discussed throughout the data collection process and is documented in our paper" (Section 8). This disclosure is honest but reveals a structural gap in the dataset's governance: there was no external oversight, no systematic risk assessment, no data protection impact analysis, and no institutional review of whether the benefits of releasing the dataset justified the potential harms.

The consequence is that the ethical judgments embedded in the Pile's construction—which sources to include, whether to filter offensive content, how to handle copyrighted material, whether to attempt consent—were made by the dataset creators without the kind of formal review that would be required for human-subjects research at an academic institution. The datasheet reports no consultations with legal experts, no engagement with affected communities (authors whose copyrighted works were included, individuals whose personal communications appear in the Enron or IRC data, members of groups stereotyped in the training data), and no systematic attempt to identify or mitigate potential downstream harms from models trained on the Pile. The ethical considerations that were discussed are "documented in our paper" (Gao et al., 2020), but the datasheet—which is supposed to be the comprehensive ethics and provenance document—does not summarize those discussions or their resolution.

This limitation is partly structural: EleutherAI is a decentralized research collective without institutional affiliation, meaning there is no IRB to consult and no formal ethics infrastructure. The datasheet is transparent about this constraint. However, the absence of formal review does not eliminate the ethical obligations that such review would address; it only means those obligations were handled informally. The datasheet provides no evidence about the quality or rigor of that informal process—we do not know what specific harms were considered, what alternatives were weighed, who (beyond the dataset creators) provided input, or what criteria were used to decide that the benefits of release outweighed the risks.

The paper does not attempt to mitigate this limitation retrospectively. There is no post-hoc external ethics review, no community advisory board, and no commitment to commission an independent impact assessment. The datasheet itself is the most thorough accounting of the Pile's ethical dimensions, and it was written by the same people who created the dataset, without external validation of its completeness or accuracy. The absence of formal ethical review means that the ethical analysis in the datasheet should be read as the authors' self-assessment rather than as a independently validated evaluation—an important distinction for downstream users who might otherwise treat the datasheet as a certification of ethical adequacy.

7. Implications and Future Directions

How This Work Changes the Landscape

The Pile datasheet does not introduce a new algorithm, model architecture, or training technique. Its impact operates at a different level: it establishes that dataset documentation can be a first-class research contribution and that the norms governing training data can be shifted through transparency rather than through technical innovation alone. This is not a paradigm shift in the Kuhnian sense — it does not overturn an existing theoretical framework — but it is a normative intervention that changed what the NLP community expects from dataset releases, and its influence is visible in the documentation practices that have subsequently become standard.

Before the Pile datasheet, the dominant approach to training data for large language models was documentation by omission. Industrial labs described their training corpora in aggregate statistics (percentages of web text, books, Wikipedia) without disclosing specific sources, copyright status, or per-component content risks. Academic dataset releases varied widely in documentation quality: some had detailed technical papers describing construction, but almost none systematically answered the questions that Gebru et al. (2021) had proposed as standard — who created the data, under what funding, with what consent, containing what sensitive content, carrying what legal restrictions. The Pile datasheet demonstrated that answering these questions was feasible even for an 825 GiB corpus spanning 22 heterogeneous sources, and that the resulting document had practical value for downstream users making decisions about whether and how to use the data.

The specific way this shifted norms is measurable. In the years following the Pile's release, major open-source language model training datasets — RedPajama, Dolma, FineWeb, DCLM — have included structured documentation addressing composition, provenance, filtering parameters, and known content issues. Conference submission guidelines and funding agency requirements have increasingly mandated dataset documentation following the datasheet template. The Pile did not single-handedly cause these changes, but it provided an existence proof that thorough documentation was possible at scale, and it established a baseline against which subsequent releases could be compared. The fact that the Pile datasheet is frequently cited not for its content but as an exemplar of documentation practice indicates that its primary impact was methodological rather than substantive.

The datasheet also resolved a specific tension in the NLP community's relationship with training data. Before the Pile, there was a widespread implicit belief that transparency about copyrighted content, offensive material, and absent consent was legally or reputationally dangerous — that acknowledging these issues in documentation would invite lawsuits or public criticism. The Pile's strategy of direct, unapologetic disclosure — naming which components contain copyrighted works, stating plainly that individuals were not notified or consented, acknowledging that offensive content "probably" exists in most components — demonstrated that this fear was at least partly unfounded. The dataset was released, widely adopted, and became the training corpus for major models (GPT-Neo, GPT-J, GPT-NeoX) without triggering the legal consequences that opacity was meant to avoid. This does not mean the legal risks are negligible — the fair use status of training data remains contested in ongoing litigation — but it showed that transparency does not automatically invite enforcement, and that the reputational benefits of honest documentation (trust from the research community, informed use by practitioners) may outweigh the legal risks of disclosure.

The datasheet's treatment of copyright represents a reframing of the legal conversation around training data. Rather than treating copyrighted material as a problem to be hidden or minimized, the datasheet articulates a positive legal theory — transformative use for research, processed form that does not substitute for original works, no market harm — and presents it as the basis for the dataset's distribution under the MIT License. This transforms copyright from a liability to be managed into a position to be argued, and it provides downstream users with a legal rationale they can adopt (or reject, based on their own circumstances and jurisdiction). Whether this legal theory ultimately prevails is a separate question, but the datasheet's contribution was to move the conversation from "should we acknowledge copyrighted data?" to "under what legal theory can copyrighted data be used for training?" — a shift from evasion to engagement.

The datasheet also redirects research attention in a subtler way. By exhaustively documenting the Pile's composition — 22 sources, each with distinct acquisition mechanisms, preprocessing pipelines, and content characteristics — the datasheet makes visible the curation decisions that are invisible in aggregated datasets. This invites research on how specific components contribute to (or detract from) model capabilities and biases. A researcher who wants to know whether biomedical knowledge in a language model comes from PubMed Central versus general web text can design an ablation study that removes that component and measures the effect. A researcher concerned about gender bias can compare models trained with and without the dialogue components (Ubuntu IRC, OpenSubtitles, YouTube Subtitles) which might contain different patterns of gendered language than formal written sources. The datasheet does not answer these questions, but it provides the component-level metadata that makes such studies possible, and in doing so it enables a research program on the relationship between training data composition and model behavior that was previously infeasible for proprietary corpora.

Follow-Up Research This Work Enables

Quantitative content audits of the Pile's actual contents. The datasheet reports that offensive content "probably" exists in 20 of 22 components but provides no frequency estimates, severity distributions, or per-component breakdowns. A natural follow-up would apply modern content moderation classifiers (toxicity detection, hate speech identification, PII scanning) to stratified random samples from each of the Pile's 22 components and produce the quantitative risk characterization that the datasheet acknowledges is missing. This would answer questions like: Does Ubuntu IRC contain significantly more toxic language than OpenSubtitles? Are Pile-CC and OpenWebText2 comparable in their prevalence of hate speech, or does the Reddit-upvote heuristic used for OpenWebText2 effectively filter it? The Pile's frozen-snapshot nature makes this study permanently valuable — unlike dynamic web corpora, the results would remain valid for all future users of the dataset and would not need to be periodically recomputed. A strong study would report per-component toxicity scores, PII detection rates, and sensitive content prevalence using established benchmarks (e.g., Perspective API for toxicity, Presidio or similar for PII), and would identify specific high-risk components that practitioners might want to exclude or downsample for safety-sensitive applications.

Component ablation studies for model capabilities and biases. The Pile's 22-source composition, combined with the datasheet's per-component documentation, makes it possible to train models on systematic subsets of the corpus and measure the contribution of each source to downstream capabilities. The companion paper Gao et al. (2020) reports some aggregate evaluations, but there is no published work that ablates each component individually, measuring, for example, how much of a Pile-trained model's biomedical knowledge is attributable to PubMed Central (90.27 GiB) versus PubMed Abstracts (19.26 GiB) versus incidental medical text in Pile-CC, or whether Books3 and BookCorpus2 contribute differentially to long-range narrative coherence. A comprehensive ablation study would train a family of models on all 2^22 subsets (impractical) or use a more efficient design (leave-one-out, grouped by register) to produce a component importance matrix mapping specific Pile sources to specific model capabilities. This would transform the datasheet's qualitative descriptions into quantitative evidence about which data sources matter for which tasks — directly informing future dataset curation decisions and helping to resolve the question of whether the Pile's copyrighted components (Books3, portions of arXiv and PhilPapers) are necessary for its performance or could be replaced with open-licensed alternatives.

Cross-source memorization and contamination analysis. The Pile contains overlapping content across components: PubMed Central full-text papers and PubMed Abstracts both draw from the biomedical literature; arXiv papers may be discussed on HackerNews or referenced in StackExchange answers; web text in Pile-CC may contain copies of Wikipedia articles or GitHub README files. The datasheet notes that PubMed Abstracts explicitly excludes papers already present in PubMed Central, but does not analyze cross-source duplication more broadly. A follow-up study would measure cross-component duplication rates by applying fuzzy matching (MinHash, suffix arrays) across all 22 components, quantifying what fraction of each component's content appears verbatim or near-verbatim in other components. This matters for two reasons: duplicated content inflates effective dataset size (making models appear to be trained on more unique text than they actually see) and creates memorization risks (models may reproduce duplicated passages at test time because they saw them repeatedly during training). The study would also identify benchmark contamination: whether evaluation datasets like MATH, GSM8K, or MMLU contain problems whose source material appears in the Pile (e.g., DM Mathematics problems or StackExchange answers that match test-set questions).

Consent and data subject perspectives. The datasheet reports that individuals were not notified about data collection and that no consent revocation mechanism exists. A qualitative follow-up would interview or survey people whose text appears in the Pile — for example, arXiv authors, StackExchange contributors, GitHub developers, or Enron employees (and their families) — to understand their perspectives on having their work included in a language model training corpus. What do they see as the harms and benefits? Would they have consented if asked? Under what conditions would they find inclusion acceptable (opt-in vs. opt-out, attribution, compensation, ability to remove)? This study would ground the abstract legal and ethical discussion in the datasheet (which treats consent as a binary yes/no property) in the actual views of affected communities, and would provide empirical evidence for policy debates about training data consent requirements. The Pile's component structure makes this feasible: researchers can identify the specific communities associated with each data source and sample participants accordingly.

Legal analysis of fair use for training data across jurisdictions. The datasheet's fair use argument is U.S.-centric and explicitly warns that "copyright law varies by country." A comparative legal analysis would examine the Pile's composition against the copyright and text-and-data-mining exceptions in multiple jurisdictions (EU's DSM Directive, UK's fair dealing, Japan's broad TDM exception, Singapore's computational data analysis provision) and produce per-jurisdiction risk assessments for each of the Pile's 22 components. This would transform the datasheet's single-jurisdiction fair use assertion into a legal risk map that practitioners in different countries can use to decide which components to include. Such a study would need to be conducted by legal scholars rather than ML researchers, but the Pile's detailed provenance documentation makes it unusually tractable for legal analysis — the acquisition mechanism, copyright status, and processing applied to each component are all documented, providing the factual basis that copyright analysis requires.

Community-maintained errata and issue tracking for frozen datasets. The Pile's frozen-snapshot model creates a specific problem: problems discovered after release (copyright complaints, identified PII, toxic content clusters, factual errors) cannot be removed from the dataset, but future users should be warned about them. A follow-up project would establish a living errata document — a versioned, community-maintained repository of known issues with the Pile, indexed by component and document, that researchers can consult before training. This would test whether the reproducibility benefits of frozen snapshots can be preserved while adding a redress-like mechanism that the datasheet explicitly lacks. The project would need to develop processes for verifying reported issues, for distinguishing between different categories of concern (legal vs. ethical vs. quality), and for maintaining the errata as an independent artifact from the dataset itself (since EleutherAI has stated they will not update the Pile). This is partly a technical infrastructure project and partly a governance experiment: can a community process substitute for the institutional oversight that the datasheet acknowledges was absent during the Pile's creation?

Practical Applications and Downstream Use Cases

Informed dataset selection for academic and startup language model training. A research lab or startup deciding which pretraining corpus to use for training a new language model in 2024 can use the Pile datasheet as a risk assessment tool that is unavailable for most alternative corpora. The datasheet's per-component breakdown of copyright status, subpopulation identification, and known content issues allows practitioners to make component-level inclusion decisions based on their specific legal jurisdiction, ethical commitments, and application domain. A European startup bound by GDPR can identify which components lack documented consent mechanisms and exclude them. A lab focused on biomedical applications can verify that PubMed Central and PubMed Abstracts are included and estimate the effective training exposure to medical text. A group concerned about copyright liability can exclude Books3, arXiv, and PhilPapers and measure the performance impact. The alternative — training on a proprietary or poorly-documented corpus — offers none of this granularity. The datasheet transforms the Pile from a take-it-or-leave-it blob into a configurable training substrate, and the explicit encouragement to "compile a custom remix" (Section 6) makes this configurability practical rather than hypothetical.

Benchmarking and reproducibility infrastructure for language model research. The Pile's frozen-snapshot model, combined with its recommended train/validation/test splits (0.1% each for validation and test, approximately 1.4 GiB each), provides a stable evaluation substrate that enables direct comparison across models and over time. A researcher reporting language modeling perplexity on "the Pile test set" is referring to an unambiguous, permanently accessible set of documents. This is structurally valuable for the research ecosystem: it eliminates the "which version of the training data" ambiguity that plagues comparisons between models trained on dynamic corpora, and it enables longitudinal studies where new models can be evaluated against the same test set years after the Pile was released. The Pile has effectively become a standard benchmark for open-source language model pretraining, and the datasheet's documentation of the split construction (random sampling from the combined corpus, preserving component distribution) ensures that researchers understand what they are measuring.

Training data composition research and data-centric AI. The Pile's 22-source structure, documented in detail in the datasheet, makes it a natural platform for studying how training data composition affects model behavior — a research direction that has grown substantially since the Pile's release under the banner of "data-centric AI." Researchers can train models on controlled subsets of the Pile, vary the upsampling weights of specific components, or introduce synthetic perturbations and measure the downstream effects — all with the confidence that the base corpus is well-documented and that their experimental manipulations are applied to a known starting distribution. The datasheet provides the metadata layer that makes such experiments interpretable: when a researcher finds that removing Books3 reduces narrative coherence, they know what Books3 is (196,640 published books totaling 100.96 GiB), where it came from (Shawn Presser via the Bibliotik private tracker, hosted by The Eye), and what content characteristics it likely has (long-form narrative, copyrighted, subpopulation-identifying). This traceability from experimental result back to data characteristic is precisely what is impossible with proprietary training corpora and what the Pile datasheet was designed to enable.

When to Prefer This Method

The Pile datasheet does not propose a method to be preferred over alternatives — it documents a dataset. However, the datasheet's structure implicitly articulates a documentation methodology (the Gebru et al. framework applied exhaustively to a multi-source corpus) that practitioners can choose to adopt or not when releasing their own datasets. The paper does not explicitly compare this documentation approach against alternatives, so a formal "prefer A when, prefer B when" matrix would be a fabrication. What the datasheet does provide is a set of implicit criteria for when its approach to documentation is most valuable:

Multi-source corpora with heterogeneous provenance benefit disproportionately from per-component documentation, because the legal, ethical, and quality characteristics vary across sources and users need component-level granularity to make informed inclusion decisions. The Pile's documentation structure — answering every question for every component — would be unnecessary for a single-source corpus of uniform provenance.
Datasets intended as shared research infrastructure (used by many groups with different legal jurisdictions, ethical commitments, and application domains) benefit from the Pile's strategy of acknowledged uncertainty and explicit caveats, because no single composition or set of disclosures can satisfy all users. The honesty about unknowns becomes a feature, not a bug, by preventing users from assuming risks have been resolved when they have not.
Corpora containing known legally or ethically contested material benefit from the Pile's strategy of direct disclosure with legal argumentation, as opposed to the alternative strategy of opacity. The datasheet's approach is most appropriate when dataset creators are willing to stake out a legal position and accept the downstream consequences, rather than attempting to avoid legal exposure through non-disclosure.
Research datasets where reproducibility is a core value benefit from the frozen-snapshot model with separately-versioned releases, particularly when the underlying sources are dynamic (web crawls, continuously-updated repositories). The tradeoff — staleness for reproducibility — is appropriate when the dataset is intended as a benchmark or shared substrate, less so when it is intended as a continuously-improving resource.

These are not criteria the datasheet itself articulates as a decision framework — they are inferred from the design choices the documentation embodies — but they provide practical guidance for dataset creators deciding whether to adopt the Pile's documentation model for their own work.