Towards Best Practices for Open Datasets for LLM Training

🎯 Pitch

Major AI companies are rapidly retreating from disclosing their training data amid copyright lawsuits, yet a June 2024 convening reveals that legally robust, openly licensed LLM datasets are feasible at scale. The biggest barrier isn't law—it’s the global patchwork of missing copyright metadata and the cost of digitizing public domain works, challenges the paper maps to concrete technical fixes.

1. Executive Summary

This paper captures the proceedings and consensus recommendations from a June 2024 convening co-hosted by Mozilla and EleutherAI, articulating normative principles and technical best practices for creating openly licensed LLM training datasets in response to the growing opacity of training data among major AI companies. Through analysis of three leading open dataset efforts—EleutherAI's Common Pile, Pleias' Common Corpus, and YouTube-Commons—the paper identifies seven guiding principles (ranging from fostering a competitive LLM ecosystem to preserving data for the long term) and maps them to concrete technical recommendations spanning metadata encoding, data sourcing, processing, governance, and terms of use. The paper establishes that building openly licensed datasets at meaningful scale remains feasible but requires coordinated investment across legal, technical, and policy domains—establishing that current bottlenecks, including incomplete copyright metadata across jurisdictions and the cost of digitizing public domain works, demand community-wide infrastructure rather than isolated organizational efforts.

2. Context and Motivation

The Core Problem: Training Data Transparency Is Collapsing Just When We Need It Most

The paper addresses a specific and worsening problem: the near-total opacity of training datasets for large language models, which is occurring precisely at the moment when understanding what models are trained on has become most urgent. The historical trajectory is stark: Google described the training data for T5 in detail, Meta disclosed the composition of LLaMA 1, but today—as the authors note in the introduction—"little is known about the datasets behind the most popular models from companies like OpenAI, Anthropic, Google, and Meta." This is not a gradual shift but a rapid retreat from transparency norms that existed only a few years ago.

What makes this retreat particularly consequential is that it coincides with a period of unprecedented scrutiny of AI training data. Creative producers—authors, artists, musicians, photographers—have launched multiple high-profile copyright lawsuits against AI companies, alleging that their copyrighted works were used without permission to train commercial models. These lawsuits, including those against OpenAI, Stability AI, and Meta, have fundamentally changed the incentive calculus for dataset disclosure: companies now perceive describing their training data in detail as a litigation risk rather than a research norm. The paper articulates this feedback loop explicitly: "Regardless of these lawsuits' legal merits, they have become one of several factors discouraging companies from disclosing their data sources and governance practices."

The result is a self-reinforcing cycle of opacity: legal uncertainty discourages disclosure, which in turn increases suspicion from creative communities, which generates more litigation pressure, which further disincentivizes transparency. Breaking this cycle requires an alternative path—training on data whose provenance and permission status are unambiguous, eliminating the legal risk that drives secrecy.

Why This Problem Matters: Four Distinct Harms of Dataset Opacity

The paper identifies four dimensions of harm that flow from training data opacity, each with different affected constituencies:

First, it undermines accountability and auditing. When training datasets are proprietary, external researchers, auditors, and civil society organizations cannot examine them for the presence of harmful content, personal data, biased representations, or copyrighted material. This is not merely an academic concern: researchers at the Data Provenance Initiative have documented alarming patterns in publicly available training datasets, including the presence of private medical records, non-consensual intimate imagery, and systematically biased representations of marginalized communities. When datasets are closed, such auditing becomes impossible—and when problems are discovered post-deployment, the connection between training data and model behavior remains opaque, making root-cause analysis intractable. The paper positions dataset transparency as a prerequisite for meaningful accountability, noting that open datasets "enable scrutiny, supporting AI accountability by facilitating new lines of inquiry for auditors and researchers, particularly in areas requiring manipulation of model training data."

Second, it stifles competition and concentrates power. The current landscape creates an asymmetric playing field: large companies with substantial legal resources can afford to shoulder the litigation risk of using copyrighted material for training, while smaller actors—startups, nonprofit labs, academic researchers—cannot. The paper is explicit about this dynamic: large companies "can shoulder the risk of legal uncertainty, as well as strike deals worth hundreds of millions for exclusive content licensing for their proprietary datasets." Meanwhile, "smaller corporate actors, researchers, and public institutions depend on open datasets to compete." When only the largest incumbents can legally (or practically) access the data needed to train competitive models, the market tilts toward concentration. Open datasets serve as a counterweight, providing a "base layer" that smaller entities can build upon without incurring prohibitive legal risk.

The connection to the open source software movement is deliberate. Just as open source software created a shared infrastructure layer that enabled a competitive ecosystem of startups, services, and tools built on Linux, Apache, and other open foundations, open training datasets could serve an analogous role for AI—enabling competition not just on data access but on model architecture, training methodology, fine-tuning, and application development.

Third, it prevents reproducibility in AI research. This point is particularly acute for the research community. As the paper explains, "one cannot improve upon best processing setups if they are not known." When datasets are proprietary, research claims about model performance become difficult or impossible to verify independently. If two models achieve different results, is it because of architecture, training recipe, or data composition? Without transparency, scientific progress slows because researchers cannot isolate variables, replicate findings, or build incrementally on each other's work. The paper argues that standardized open datasets enable "rigorous evaluations of model performance because it limits the number of factors that lead to variations between them"—essentially serving as a controlled variable that makes other experimental variables interpretable.

Fourth, it denies affected communities meaningful recourse. When individuals, communities, or organizations discover that their data has been used for training, the opacity of proprietary datasets makes it impossible to verify, challenge, or seek redress. Authors cannot determine whether their books were ingested. Marginalized communities cannot assess whether their cultural heritage was appropriated. In contrast, the paper envisions a regime where "people without a technical background" can check if their data is included in a training set, and where opt-out mechanisms, removal processes, and preference signals are built into the data infrastructure from the start.

Prior Approaches and Where They Fall Short

The paper positions itself against several existing approaches to the training data problem, each of which has identifiable limitations:

Proprietary datasets with selective disclosure. Companies like OpenAI, Anthropic, and Meta have increasingly moved toward releasing minimal or no information about their training data, often citing competitive reasons or legal caution. This approach—commercially successful but scientifically and socially problematic—is what the paper explicitly rejects. The authors frame this not as an unavoidable reality but as a choice with identifiable consequences, and one that can be countered by demonstrating that open alternatives are technically viable. This is a key strategic claim: part of the motivation for building Common Pile, Common Corpus, and YouTube-Commons is to disprove the narrative that training competitive models requires using copyrighted material. As stated, EleutherAI "views Common Pile as a statement against claims by some leading AI companies that training performant LLMs without copyrighted material is impossible."

Open-access but not openly licensed datasets. Several widely used training datasets—including AI2's Dolma, the original Pile from EleutherAI, and Common Crawl-derived corpora like C4—are available for download but contain data whose licensing status is ambiguous, unverified, or demonstrably non-permissive. The paper draws a crucial distinction between downloadable/open-access datasets (freely available but without verified license compliance) and openly licensed datasets (where both the dataset and its constituent components have verified permissive licensing). This distinction is widely misunderstood: the paper notes that "many people believe that AI2's Dolma dataset is openly licensed due to the ODC-by dataset license even though the constituent data is not openly licensed." This confusion arises because dataset-level licenses (covering the arrangement and compilation) are often more permissive than the licenses on individual data points within the dataset—a legal nuance that renders many "open" datasets legally risky for commercial use.

Opt-out mechanisms as a primary solution. Some industry actors have proposed that website-level opt-outs (via robots.txt or similar mechanisms) suffice to address concerns about training data. The paper identifies several problems with this approach. First, current opt-out mechanisms are binary and crude—they control who is allowed to crawl a website, not how scraped data is used, which "leads many website owners to completely block non-profit archives like Common Crawl that are used for non-commercial purposes." This creates a tragedy of the commons where blanket blocking deprives researchers and nonprofits of data while large companies with existing scrapes are unaffected. Second, there is a documented trend toward massive opt-outs that threatens to "shrink the open web" in ways that harm the broader ecosystem, not just AI training. Third, opt-out mechanisms implemented at the network level cannot distinguish between different uses, different users, or different contexts—a news publisher who might welcome nonprofit research use cannot selectively permit it while blocking commercial AI training.

Legal compliance as a substitute for ethics. The paper is careful to distinguish legal compliance from ethical practice: "It is crucial to remember that an action can be legally compliant without being ethical. Ethics concerns itself with wider questions of, for example: justice, equality, distribution of resources, redress of harms." Some existing approaches focus narrowly on avoiding copyright infringement without addressing broader concerns about consent, representation, bias, or community impact. A dataset could be entirely public domain and legally unimpeachable, yet still contain deeply harmful content or systematically exclude marginalized perspectives. The paper argues that open dataset builders must address both legal and ethical dimensions, and that these can sometimes conflict—as when implementing a rolling opt-out mechanism that "contradicts the idea of working with a verifiably identical dataset over time."

Model-level fixes without data-level transparency. Some proposals suggest that addressing bias, toxicity, or factual errors at the model level (through alignment techniques like RLHF, constitutional AI, or filtering at inference time) can substitute for dataset transparency and curation. The paper implicitly rejects this framing by arguing that dataset-level work is foundational: without knowing what data was used, post-hoc fixes at the model level are working in the dark. The emphasis on reproducible data processing pipelines, documented filtering decisions, and auditable provenance reflects a conviction that data governance must happen upstream.

How This Paper Positions Itself

This paper is unusual in being neither a traditional research contribution (it presents no new models, evaluations, or algorithms) nor a purely normative policy document. It occupies a middle ground that is best understood as a community consensus-building artifact—an attempt to create what the authors call "a shared reference point for the emerging open LLM ecosystem" analogous to the shared artifacts (software licenses, documentation standards, governance models) that defined the early open source software movement.

The positioning strategy has several distinct elements:

Bridging normative principles and technical implementation. Rather than merely articulating values (though it does so through its seven guiding principles), the paper maps each principle to concrete technical recommendations. The section on "Encoding preferences in metadata" (Section 4.1) translates the principle of "minimizing harms and enabling preference signals" into specific technical practices: using SPDX license identifiers, preserving URLs and crawl metadata, developing machine-readable opt-out standards. This is not abstract advocacy—it is grounded in the actual implementation experience of Common Pile, Common Corpus, and YouTube-Commons, giving the recommendations the credibility that comes from having been attempted, not just theorized.

Defining a taxonomy of openness. One of the paper's most useful contributions is its three-tier framework for dataset openness, distinguishing "openly licensed" from "downloadable/open-access" from "replicable" datasets (Figure 1). This is not merely definitional housekeeping; it addresses a genuine confusion in the field. The Dolma example—where the dataset license is permissive but the constituent data is not—illustrates how the failure to make this distinction leads organizations and researchers to believe they are operating within legal safe harbors when they may not be. By providing precise terminology, the paper aims to enable more honest and informed conversations about what "open" actually means for training data.

Modeling the kind of transparency it advocates. The case studies of Common Pile (Appendix A) and Common Corpus/YouTube-Commons (Appendix B) are unusually detailed in their documentation of data sources, processing decisions, licensing verification procedures, and unresolved challenges. The Common Pile case study includes per-component descriptions (academic texts, code data, public domain books, news, web data, wikis, YouTube transcripts, government documents) with specific license determination methods, data sizes, and processing pipelines—including a flowchart (Figure 3) of the copyright renewal verification process for post-1929 books. This level of detail is not merely illustrative; it is performative, demonstrating the kind of transparency the paper advocates and providing a template for other dataset builders.

Acknowledging unresolved tensions rather than papering over them. The paper is notably honest about conflicts between its own principles. The tension between opt-out mechanisms and reproducibility is discussed explicitly: removal processes "could limit both reproducibility and transparency, if not well documented, as well as the competitiveness of open datasets and data availability for research when faced with massive opt-outs on the open web." Similarly, the paper notes that "not all potential harms and risks can be mitigated directly via interventions in the dataset" and that filtering methods "introduce their own biases." This willingness to surface contradictions rather than resolve them artificially gives the paper intellectual credibility and positions it as the beginning of a conversation rather than a final answer.

Situating itself within a broader ecosystem of standards and initiatives. The paper references a dense network of related efforts—the Data Ethics Canvas, The Model Openness Framework, the Foundation Model Development Cheatsheet, the Open Source AI Definition, the Columbia Convening on Openness in AI, Croissant metadata standards, RAIL licenses, SPDX identifiers, ISCC content codes, and many more. This is not namedropping; it reflects a deliberate strategy of positioning open dataset building as part of a larger movement toward AI openness, governance, and accountability, rather than as an isolated technical challenge. The repeated invocation of the open source software analogy reinforces this framing: dataset builders are asked to see themselves as part of a historical tradition with accumulated wisdom about how to build shared infrastructure, govern volunteer communities, and sustain public goods.

Addressing multiple audiences simultaneously. The paper attempts the difficult task of speaking to dataset builders (who need technical guidance), policymakers (who need to understand what interventions would be helpful), legal experts (who need to navigate jurisdiction-specific copyright complexities), and community organizers (who need principles for engaging affected stakeholders). This breadth is both a strength and a limitation: it means specific recommendations are sometimes underspecified ("work with affected communities" is good advice but not actionable without more detail on what meaningful engagement entails). However, the paper explicitly acknowledges this, stating that it "is not intended to be a comprehensive analysis of all considerations around open datasets for LLM training" but rather aims to "capture the ambitions and consensus that is starting to emerge."

A strategic intervention, not a neutral survey. The paper has a clear theory of change: by demonstrating that openly licensed training datasets at scale are technically feasible, by building community consensus around shared practices, and by providing templates that reduce the barrier to entry for new dataset builders, the open dataset ecosystem can counter the trend toward opacity and concentration. This is not a passive documentation of existing practices; it is an active attempt to shape and accelerate the emergence of a community around open LLM training data. The convening format itself—bringing together 30 practitioners from startups, nonprofits, and civil society—reflects the theory that progress requires coordinated action across organizational boundaries, not just individual effort.

The paper's ultimate positioning is encapsulated in its closing sentence: "Achieving this will demand collaborative effort and a commitment to the public good, but it is an essential pursuit to ensure that the next generation of AI remains transparent, accessible, and beneficial to all." This is neither the language of a technical paper nor of a policy brief—it is the language of a movement-building document, and understanding this framing is essential to interpreting the paper's recommendations, priorities, and acknowledged gaps.

3. Technical Approach

3.1 Reader orientation

This paper is a community consensus-building and best-practices document, not a traditional machine learning contribution — it proposes no new models, algorithms, or quantitative evaluations. The "system" being analyzed is the socio-technical pipeline for constructing, processing, governing, and releasing openly licensed training datasets for large language models, understood as an interdependent set of legal determinations, technical processing steps, metadata standards, community governance mechanisms, and funding models. The core problem it addresses is that building such datasets at meaningful scale — with verified permissive licensing across jurisdictions, high-quality text extraction from heterogeneous sources, and responsible governance — requires coordinating expertise across domains (copyright law, software engineering, community organizing, policy) that are typically siloed, and the paper aims to establish the shared vocabulary, principles, and concrete practices that enable this coordination.

3.2 Big-picture architecture (diagram in words)

The dataset construction pipeline described across the paper, particularly in the Common Pile and Common Corpus case studies, has five major stages:

Source identification and license verification — locating repositories of potentially open content (public domain books, Creative Commons-licensed web pages, government documents, academic preprints, code repositories, wiki dumps, YouTube transcripts) and determining, for each candidate source, whether its licensing status is compatible with open dataset inclusion under relevant jurisdictions. This stage involves both automated methods (regex matching for CC license identifiers in HTML, API queries for repository metadata) and intensive manual work (copyright renewal record analysis, jurisdiction-specific public domain determinations).
Content acquisition and digitization — obtaining the actual text from identified sources, which often requires format conversion (LaTeX to plain text, PDF to text via OCR, YouTube video to transcript via Whisper), API-based bulk downloading, and navigating access restrictions even for legally public domain materials (e.g., Google Books digitizations that are technically public domain but practically access-restricted).
Data processing and filtering — transforming acquired raw content into training-ready text through pipelines that include: heuristics for quality filtering (removing boilerplate, non-natural-language text, machine-generated spam), personal information removal (email addresses, phone numbers), license metadata preservation, and format standardization. Different source types require different processing — Common Crawl web pages go through Resiliparse extraction and quality heuristics, while academic papers require LaTeX-to-text conversion with math preservation.
Data governance and preference infrastructure — embedding mechanisms for rights-holders to express preferences (opt-outs, usage restrictions) and for downstream users to understand provenance. This includes preserving crawl metadata (URLs, timestamps, HTTP headers), implementing machine-readable preference signals (robots.txt parsing, emerging standards like TDM opt-outs under the EU AI Act), and building post-release removal processes.
Release, documentation, and versioning — packaging the dataset for distribution on platforms like Hugging Face, providing comprehensive documentation (data cards, datasheets, processing code), making decisions about update cadence (Common Pile's "stable default dataset" model vs. Pleias' "release often, release early" model), and specifying terms of use that align with the underlying content licenses.

Information flows linearly through these stages — source identification feeds acquisition, which feeds processing, which feeds governance instrumentation, which feeds release — but with critical feedback loops: processing quality issues (e.g., poor OCR) may require revisiting acquisition methods; governance requirements (e.g., opt-out mechanisms) shape what metadata must be preserved during processing; and community feedback on released datasets informs future source selection.

3.3 Roadmap for the deep dive

First, the three-tier openness taxonomy (openly licensed, downloadable, replicable), because this definitional framework underlies every subsequent decision about which sources to include, what metadata to preserve, and how to document datasets.
Second, source identification and license verification, the most labor-intensive stage, with detailed examination of the specific methods used for different source types (public domain determination for books, CC license extraction from Common Crawl, copyright renewal record analysis) to illustrate why this remains an "artisanal" rather than fully automated process.
Third, content acquisition and digitization challenges, covering the practical obstacles in obtaining machine-readable text from identified sources — PDF OCR quality, format conversion, access restrictions on digitized public domain works — since these challenges fundamentally constrain dataset scale and quality.
Fourth, data processing and quality filtering, examining the concrete heuristics and pipelines used to transform heterogeneous raw content into training-ready text, and the tension between aggressive filtering (which removes noise) and conservative approaches (which risk introducing filter-specific biases).
Fifth, metadata preservation and preference encoding, the infrastructural layer that enables downstream governance — explaining why preserving URLs, crawl dates, HTTP headers, and license identifiers is not merely documentation but a prerequisite for opt-outs, removal requests, and provenance auditing.
Sixth, data governance models, contrasting the "stable immutable dataset" model (Common Pile) with the "living community-maintained dataset" model (Pleias' vision), and analyzing the mechanisms for post-release removal, community engagement, and preference expression.
Seventh, release practices and terms of use, covering platform choices, documentation standards, versioning, and the principles guiding whether and how to impose additional terms beyond the underlying content licenses.

3.4 Detailed, sentence-based technical breakdown

This is fundamentally a normative guidance and best-practices paper, organized around the insight that building openly licensed LLM training datasets requires simultaneously solving legal, technical, and governance challenges that are deeply interdependent — a copyright determination made in isolation from technical processing decisions can create downstream liability, and a filtering decision made without governance infrastructure can foreclose opt-out mechanisms that communities need. The paper's technical contribution is not a single system but rather the codification of emerging practices from organizations actually attempting this work, structured as actionable guidance across the full dataset lifecycle.

The Three-Tier Openness Taxonomy

The paper establishes a precise vocabulary for discussing dataset openness because, as the authors observe, "we have observed substantial confusion about the licensing of machine learning datasets." This confusion has practical consequences: organizations may believe they are operating within legal safe harbors when they are not, or may forego using datasets that are actually legally viable.

The taxonomy defines three categories, ordered by increasing stringency of openness:

Tier 1: Replicable dataset. The data sources and processing steps are disclosed sufficiently that "an independent party can produce a substantially similar (albeit not identical) dataset." This tier does not require that the dataset itself be distributed — only that the recipe for reconstructing it is publicly documented. The defining feature is that data sources must be "widely accessible (e.g., not internal data or data accessed through private agreements)." A replicable dataset enables scientific verification (researchers can check whether reported results are reproducible) but does not guarantee that the dataset is legally safe to use, because the underlying data may be copyrighted and the processing recipe may produce outputs that inherit those copyrights.

Tier 2: Downloadable / open-access dataset. The dataset is "available to freely download, with no claim about license compliance." This describes many widely used datasets — including C4, the original Pile, and Dolma — that are accessible for download but whose constituent data may include copyrighted or restrictively licensed material. The critical distinction the paper emphasizes is that "the act of collecting, processing, and arranging existing data grants intellectual property interests in the arrangement" (under EU sui generis database right and similar doctrines in other jurisdictions), but "it does not grant the compiler the right to change the licensing of the underlying data." Therefore, a dataset with a permissive compilation license (e.g., ODC-by on Dolma) but restrictive underlying content licenses is in this tier, not Tier 3 — a nuance that the paper notes is "not widely known, and can cause confusion and incompatibility."

Tier 3: Openly licensed dataset. "A dataset and its components can be freely used, modified, and shared by anyone for any purpose," following the Open Knowledge Foundation's Open Definition. This requires that both the dataset compilation/arrangement AND every individual data component be under permissive licenses or in the public domain. This is the tier that Common Pile aims for, and it is the hardest to achieve because it requires verifying the licensing status of every constituent element — a task that, for web-scale datasets, requires automated methods that are currently imperfect and manual verification that is prohibitively expensive at scale.

The paper also clarifies a terminological point: "Throughout this document, when we discuss licensing, we are referring to both of these types of IP being suitably licensed, as the construction of the actual input to a machine learning model involves both." This is because training an LLM involves both the arrangement of data (the dataset structure, the selection and ordering of examples) and the individual data points themselves — each potentially with its own copyright status.

Why this taxonomy matters operationally. The taxonomy is not merely definitional; it determines what verification work is necessary. Building a Tier 3 dataset requires a fundamentally different approach than building a Tier 2 dataset: every source must be individually vetted, rather than the dataset as a whole being released under a permissive umbrella license. This means the source identification stage must be designed around license verification, not just content quality — a constraint that shapes every subsequent pipeline decision.

Source Identification and License Verification

This is the most labor-intensive and jurisdictionally complex stage of the pipeline. The paper provides extensive detail through the Common Pile case study (Appendix A), which describes per-component methods for identifying open content and verifying its licensing status. The core challenge is that "determining whether a particular document is in the public domain can require country-specific analyses, and it can require many lawyers to vet work published in multiple countries."

Public domain book identification: the copyright renewal verification pipeline. The most detailed technical description in the paper concerns the process for identifying post-1929 books that are in the public domain because their copyright was not renewed. This is a concrete illustration of why open dataset construction requires skills far beyond machine learning engineering.

The legal context: For works published in the United States between 1929 and 1964, copyright owners were required to renew their copyrights after 28 years. Works whose copyrights were not renewed entered the public domain. The New York Public Library estimates that approximately 480,000 books published in this period are in the public domain for this reason, but identifying which specific titles these are requires matching original copyright registration records with subsequent renewal records — a task complicated by the fact that "the U.S. Copyright Office does assign a numerical ID to copyrighted works, that ID is not necessarily unique because the numbering system has been changed multiple times in the past century."

The Common Pile team's pipeline (illustrated in Figure 3 of Appendix A) proceeds as follows:

Input: Two datasets from the NYPL — a list of copyright registrations and a list of copyright renewals. These were originally paper records that have been digitized via OCR, introducing quality issues.
Matching: The registration and renewal records are matched to determine which registrations were never renewed. This matching is challenging because of OCR errors in the digitized records, inconsistent identifier systems across different eras of the Copyright Office's history, and incorrectly filled-out forms.
LLM-assisted metadata extraction: For renewal records where copyright metadata is in an unstructured format (rather than structured fields), the team uses a language model (trained by Nous Research) to extract relevant fields. The paper reports validation results for this approach: "The large language model demonstrates high accuracy in extracting key copyright metadata from unstructured data, achieving an impressive 96.58% accuracy for titles and 97.20% for registration numbers. The accuracy for authors (90.25%) and dates (87.66%) is slightly lower, but mistakes in these values cannot wrongly cause a work to be classified as in the public domain." The paper explains the safety property: errors in author or date extraction might cause a public domain work to be incorrectly classified as requiring further investigation (a false negative), but cannot cause a copyrighted work to be incorrectly classified as public domain (a false positive), because title and registration number errors are required for that failure mode, and "assuming that errors in identifying titles and registration numbers are independent, the odds of a work being wrongly classified in the public domain due to incorrect LLM-identified metadata is 0.09%, or about 380 books out of 424,059."
Ambiguous match resolution: Cases where the match between registration and renewal is ambiguous are routed for further investigation. The team is also experimenting with LLM-assisted resolution of these ambiguous cases, with preliminary results indicating "an accuracy of 98.6% in determining if an ambiguous match is correct or not."
Output classification: Works are classified into three categories: clear exclusions (renewed, not public domain), requires more investigation (ambiguous), or believed public domain (not renewed).
Full-text acquisition: Once a list of books believed to be in the public domain is established, the next challenge is locating digitized copies. The Copyright Office metadata "does not reliably have useful ID numbers, so we are utilizing a range of resources to obtain cross-reference ID numbers and identify libraries or other sources that possess these texts." The team reports identifying "approximately 500,000 books that we believe to be in the public domain, though it is currently unclear how many have accessible digitizations."

This pipeline exemplifies the interdisciplinary nature of the work: it requires copyright law expertise (to know which failure modes matter), software engineering (to build the matching and extraction systems), data quality assessment (to understand OCR error propagation), and library science (to locate physical copies). No single discipline suffices.

Creative Commons license extraction from Common Crawl. For web data, the Common Pile team developed a pipeline to identify Creative Commons-licensed content within Common Crawl archives. The approach relies on the fact that "Creative Commons pages are typically labeled using a standardized identifier placed in the HTML code of a website." The pipeline:

Crawls across 52 Common Crawl snapshots (rather than a single crawl) to maximize coverage.
Applies regex matching to identify CC license identifiers in HTML metadata. The paper notes they "limit this subset to Creative Commons licenses as we found automatically detecting all Blue Oak Council Bronze or higher licenses to be infeasible due to the wide variety of ways people describe their licensing terms." This is a deliberate scope restriction: they accept that some permissively licensed content will be missed in exchange for higher precision on the content they do include.
Extracts plain text from license-matching pages using the Resiliparse library.
Applies quality heuristics "commonly employed when curating text for language model pre-training to maximize the quality of the obtained text."
Reports results: After filtering, "we end with 259,728,610 pages comprising 221,715,271,483 words."

The paper is transparent about an unresolved validation challenge: "We are currently in the process of carrying out a detailed study of the end-to-end accuracy of our license identification pipeline." This means the precision and recall of the CC license detection are not yet quantified — a significant gap acknowledged openly.

The license coverage problem. A critical limitation the paper identifies is that "when filtering the Common Crawl, it is easy to determine that a website links to a CC-BY 4.0 license, and therefore contains some sort of CC-BY 4.0 statement. However, there is currently no automated way of determining which asset on the website that license covers. This leads to false positives if, for example, someone uses a third-party CC-BY 4.0 photograph as part of their non-CC article." This is the fundamental ambiguity at the heart of web-scale license detection: a CC license declaration on a page may apply to the entire page, to embedded media, to the site template, or to nothing at all — and current automated methods cannot disambiguate these cases.

Source-type-specific verification methods. The Common Pile case study describes verification approaches for multiple source categories, each with distinct challenges:

Academic papers (arXiv): "All papers are uploaded by the authors, who are required to identify the license under which the paper is shared. We include papers that are licensed under the permitted licenses, which is about 15% of all papers." This is a relatively clean case because arXiv enforces structured license metadata. The processing converts LaTeX source to plain text with math sections preserved.
BioDiversity Heritage Library: Contains "texts with a wide variety of licensing, including in-copyright texts that are included under restrictive terms by agreement with the copyright owners." The team includes only documents whose metadata explicitly lists an acceptable license, and explicitly excludes texts marked with a due diligence statement ("No known copyright restrictions as determined by scanning institution") pending further investigation — a cautious approach that privileges legal certainty over dataset size.
PubMed Central: Similar to BHL, "articles and article metadata in PubMed are sourced exclusively from the publishing journals." The paper does not specify what fraction of PMC articles are included, likely because this filtration is still in progress.
Code repositories (Stack v2): The team takes "the variant that they used for the training of StarCoder2 and filter it for our desired licenses," leveraging the license identification work already done by the BigCode project.
Government documents: "All U.S. Government-authored or produced data and information are considered to be in the U.S. Public Domain." This provides a large, legally unambiguous source — but is jurisdiction-specific (other governments have different rules).
YouTube transcripts: The team is "currently searching YouTube to identify videos based on uploader-specified licenses and then running the identified videos through a Whisper-based pipeline to obtain high-quality text transcripts." They note they "have not yet validated the correctness of the uploader-specified licenses," which is a significant gap — YouTube's CC-BY designation is self-reported by uploaders and may be inaccurate.

The Common Corpus approach to public domain determination. Pleias took a different approach from EleutherAI for Common Corpus: rather than attempting to determine copyright renewal status for 20th-century works, "the company mostly limited itself to content published before 1884 as a precaution." This conservative threshold eliminates the need for the complex renewal verification pipeline but substantially reduces the volume and recency of available content. The tradeoff is deliberate: legal certainty over dataset scale. Pleias "relied on a network of collaborators in the U.S. and across Europe, including the French Ministry of Culture, Occiglot, EleutherAI, HuggingFace, and Nomic AI" to identify public domain repositories, highlighting the community-coordination aspect of source identification.

Content Acquisition and Digitization Challenges

Even after sources are identified and their licenses verified, obtaining machine-readable text presents substantial obstacles. The paper identifies several categories of challenge:

The PDF extraction problem. Both Common Pile and Common Corpus face the difficulty that "Many books are only available as PDFs and not all of those PDFs have high quality optical character recognition (OCR), so extracting the plain text at scale is technically challenging." This is not a minor implementation detail — it is a fundamental bottleneck. The paper notes that "one of the challenges in using openly licensed or public domain content as training data remains extracting it from PDF" and recommends "investing in better tools for extracting openly licensed content from difficult formats like PDFs and providing these tools as open-source software."

Pleias deliberately "limited its data collection to PDFs with pre-existing OCR, but OCR quality was often low." Rather than re-OCRing from scratch (which would be computationally expensive and still error-prone), Pleias invested in "post-OCR correction to further improve the quality of Common Corpus and similar datasets." This is a domain where machine learning techniques could close the loop: using LLMs themselves to correct OCR errors in public domain texts that will then be used to train improved LLMs.

Format conversion for academic papers. For arXiv, "the LaTeX source is converted to plain text, with math sections left as-is." This preserves mathematical notation that would be lost in a simple text extraction, addressing a known failure mode in earlier datasets that stripped LaTeX markup and rendered mathematical content incomprehensible.

The access paradox: public domain works that cannot be accessed. The paper highlights a counterintuitive barrier: "Just because a document is in the public domain, it does not mean one can get a copy of it." The Google Books project scanned millions of books in partnership with libraries, and while some access is possible through the Google Books platform, "it is not possible to gain unrestricted bulk access to all books that even Google thinks are in the public domain." Even when access is granted, "organizations often need to sign agreements that limit their ability to use the content." This creates a situation where works are legally in the public domain — anyone has the right to use them — but practically inaccessible because the only digitized copies are controlled by entities that impose contractual restrictions on access.

This access paradox is identified as a policy problem: "This suggests there is a need for serious public support for open data commons — we cannot expect data will be available simply because they are not protected by copyright." The paper advocates for "requiring certain institutions or commercial entities to release data in a sanitized, structured way under an open license after a specific period" as a policy intervention.

YouTube transcript acquisition and quality. For the YouTube-Commons dataset, Pleias "created a dataset consisting of the video transcripts (including automatic transcripts and translations) and metadata provided by YouTube." This approach uses YouTube's built-in automatic transcription rather than a separate speech recognition pipeline — a pragmatic choice that trades transcription quality for scale and simplicity. The paper acknowledges this tradeoff: "this approach is not considered ideal because the quality of the transcripts and especially the automated translations varies, but it is viewed as a first step that a community could help improve and expand upon." The Common Pile YouTube component takes a different approach, "running the identified videos through a Whisper-based pipeline to obtain high-quality text transcripts" — achieving better transcription quality but at higher computational cost and pipeline complexity.

Metadata about videos is preserved in YouTube-Commons "for licensing reasons and to make YouTube-Commons useful for creating openly licensed multimodal training datasets in the future." This forward-looking design choice — preserving metadata even when it is not needed for the current text-only use case — reflects the principle that open datasets should enable downstream uses beyond those their creators envisioned.

Data Processing and Quality Filtering

Once raw content is acquired, it must be transformed into a format suitable for LLM training. The paper describes several processing approaches across different data types, with an emphasis on transparency and reproducibility.

The values-anchoring principle for filtering. The paper insists that filtering decisions should be explicitly connected to stated values: "Clearly and explicitly state the values and desired properties that shaped the way data was filtered or annotated." The term "high-quality data" is identified as problematic because "it is not a properly defined concept. It needs to be defined in relation to the dataset that is being curated." This is a meta-methodological point: rather than treating quality filtering as a purely technical optimization, the paper argues that filtering choices encode normative judgments about what content is desirable, and those judgments should be surfaced rather than hidden behind the label "quality."

Web data processing. For Common Crawl-derived content in Common Pile, the pipeline includes:

Text extraction: Content is "extracted into plain text using the Resiliparse library" after license identification.
Quality heuristics: After initial filtering for CC licenses, the team "further refine these subsets by applying heuristics that are commonly employed when curating text for language model pre-training to maximize the quality of the obtained text." The paper does not specify which heuristics — likely because these are still being selected — but references the filtering approaches "developed in Dolma, FineWeb, or MassiveText" as potential models.

The unintended consequences warning. The paper explicitly warns that filtering "introduces its own biases, and can cause harms if used without care. For example, simplistic word blocklist based filtering techniques might filter out non-toxic content, such as medical research articles discussing anatomy." This is a concrete example of how technical filtering decisions can have discriminatory effects: a filter designed to remove sexual content that operates on keyword matching will disproportionately exclude medical, biological, and health-related content — potentially degrading model performance on legitimate medical queries.

Personal information removal. Both Common Pile and Common Corpus address the presence of personal identifiers in training data. The paper cites FineWeb and Dolma as "demonstrating the effective removal of personal identifiers like email addresses and phone numbers." The recommendation is to "take steps upfront to avoid collecting sensitive data like phone numbers or health information" rather than relying entirely on post-hoc removal.

Format-specific processing across Common Pile components:

Stack Exchange data: "Each document in the datasets consists of a 'Question,' followed by each of the 'Answers,' sorted according to their votes at the time of the dump. If one of the answers is marked as the 'accepted answer,' it is always the first answer to appear. Comments on the question and answers are included, but can be removed during the data generation process." This structured formatting preserves the dialogic structure of Q&A content, which may be valuable for training models on conversational and instructional formats.
Case Law Access Project: "Each of the extracted texts was post-processed to fix encoding, normalization, repetition, redundancy, parsing, and formatting errors." A post-processed version with corrections is made available on the Hugging Face Hub, demonstrating the principle that processing improvements should benefit the broader community.
CourtListener: Processing includes "properly extract, normalize, and fix texts, correct OCR errors, remove boilerplate and repeating characters, and fix tags." The specific combination of corrections reflects the challenges of legal documents, which contain structured metadata (citations, dockets) mixed with free text.
Patent Office documents: The structured format of patents is identified as "particularly valuable for training models due to their long-context dependencies, where the claims must follow detailed descriptions and prior references within the document." This is a quality consideration that goes beyond simple heuristics — it reflects an understanding of what properties of text make it useful for training particular capabilities (long-range reasoning).

The reproducibility requirement. The paper emphasizes that processing must be documented sufficiently for replication: "Provide documentation that outlines the rationale for all steps in the filtering process. Share tools and code used to filter the data." This connects to the "replicable dataset" tier in the openness taxonomy — even if a dataset is released as openly licensed, the ability to verify and reproduce the processing pipeline is an independent dimension of transparency.

Metadata Preservation and Preference Encoding

A recurring theme throughout the paper is that metadata preservation is not merely documentation — it is the infrastructural prerequisite for downstream data governance. Without preserved provenance information, opt-outs, removal requests, and preference expression become impossible.

What metadata to preserve. The paper specifies: "For each data point, record the associated permissions and the metadata needed to determine them (such as url, crawl date, http headers and html metadata), if available, as well as methods used to determine them." This recommendation spans both the licensing information itself (what license applies) and the evidence chain that supports that determination (where the license was found, when it was observed, what method was used to extract it). This creates an auditable provenance trail.

Why machine-readable standards matter. The paper connects metadata practices to regulatory requirements: "Due to EU legislation, 'general-purpose AI' (or 'foundation model') model developers need to adopt, and ideally converge on, machine-readable opt-out standards by August 2025 in order to comply with the EU AI Act, which references the EU Copyright Directive's Text and Data Mining (TDM) exception." This is a concrete deadline that creates urgency for standardization efforts that might otherwise proceed slowly.

Existing and emerging standards. The paper catalogs several initiatives:

SPDX license identifiers: Recommended "for interoperability," these provide standardized identifiers for open source licenses that can be embedded in metadata.
International Standard Content Code (ISCC): An ISO standard for creating "unique digital identifiers that work regardless of medium (text, audio, image, video)." This enables content-based identification independent of URLs, which is important for opt-out mechanisms that need to identify content even when it appears at different locations.
Spawning's opt-out API: Provides "machine-readable opt-out methods, making it possible for copyright holders to express preferences" and is integrated with Hugging Face.
Creative Commons preference signaling: Working on "preference signaling for AI data that allows for more granular control than simply fully allowing or disallowing AI use cases."
BigCode opt-out process: A "manual option to exclude or remove repositories from the Stack dataset that relies on Github accounts to verify the user's identity and reasonably validate their data rights."

The distinction between network-level blocking and preference signals. The paper draws an important technical distinction between "mechanisms that work on the network blocking level (e.g., Cloudflare's 'One-click to block all AI bots') which effectively prevent scraping of websites" and "tools that focus on preference signals that rely on voluntary compliance by the crawling party, much like the decades-old robots.txt." Network-level blocking is binary and coarse — it prevents all access, including by non-commercial researchers and archival projects like Common Crawl. Preference signals are more granular but depend on the crawler choosing to respect them. The paper advocates for the latter as more compatible with maintaining an open web while respecting rights-holder preferences.

The tension with reproducibility. The paper acknowledges that opt-outs and removal mechanisms create a direct tension with the principle of reproducibility: implementing a rolling individual opt-out "contradicts the idea of working with a verifiably identical dataset over time." If a dataset changes every time someone opts out, two researchers using "the same" dataset at different times may actually have different data, making results incomparable. The paper does not resolve this tension but surfaces it as an open challenge.

Data Governance Models

The paper contrasts two governance philosophies illustrated by the case studies, each with different implications for community involvement, update cadence, and long-term sustainability.

The "stable default dataset" model (Common Pile). EleutherAI's approach is to create a dataset that "is meant to become a default dataset that should be reused for years to allow for better model comparison." Under this model, "there are currently no plans to update and change it after publication." The governance is centralized: EleutherAI makes decisions about what to include, processes the data, and releases a fixed artifact. Terms of use are minimal — "whatever license the content was originally published under applies to the subsets. EleutherAI is not adding any terms of use."

This model prioritizes scientific reproducibility and legal simplicity. A fixed dataset means experimental results are comparable across time. No additional terms of use means no new legal complexity beyond the underlying content licenses. However, it also means the dataset cannot improve with better processing, cannot respond to opt-out requests, and cannot incorporate newly available open content.

The "living community-maintained dataset" model (Pleias' vision). Pleias aims to "foster a community that constantly improves and expands Common Corpus and YouTube-Commons following a 'release often, release early' mindset." The aspiration is for a community "similar to those in the open science and open data space, with different stakeholders (both institutional and non-institutional) bringing stability and expertise, and ensuring continuity." Governance in this model is distributed — community members contribute improvements, identify new sources, and maintain the dataset collectively.

This model prioritizes growth, improvement, and community ownership. It can respond to technical advances (better OCR, better processing) and expand coverage over time. However, it sacrifices scientific reproducibility (the dataset is a moving target), requires sustained community engagement (which the paper acknowledges "does not yet exist"), and introduces coordination challenges around quality control and decision-making authority.

Post-release removal mechanisms. Regardless of governance model, the paper recommends that dataset builders "create models of redress and removal from a dataset if an issue is spotted. For example, from the start provide mechanisms for people to request removal of their data, and encourage downstream users of the dataset to only use the updated version." The practical challenge is that "this is only possible if sufficient content identifiers are available" — another reason metadata preservation is critical infrastructure, not optional documentation.

Platform choices for release. The paper notes that "dataset builders often release their datasets on multiple platforms like HuggingFace and their own websites" and recommends considering "where you release your datasets and how it influences your ability to control, maintain and update them consistently across platforms." Version fragmentation across platforms is identified as a practical governance challenge.

Engaging affected communities. The governance recommendations include "work with affected communities" as stakeholders in dataset development, citing examples like "data trusts for language communities, labor unions representing writers or artists." This moves beyond individual opt-outs toward collective governance models where communities make decisions about how their data is used, not just whether individual data points are included or excluded.

Release Practices and Terms of Use

The final pipeline stage concerns how datasets are packaged, documented, and legally framed for downstream users.

Documentation standards. The paper recommends that at minimum, dataset builders "follow established transparency best practices like data sheets or data cards." These are structured documentation formats that cover dataset composition, collection methodology, intended uses, limitations, and ethical considerations. The Croissant metadata standard is cited as an emerging format for ML-ready datasets, "adopted by platforms such as Hugging Face and Kaggle."

Terms of use philosophy. The paper articulates a principle that is not universally held in the open data community: "Terms of use should only be provided if there are clear pathways to enforce them." This reflects a pragmatic concern that unenforceable terms create a false sense of legal protection while adding complexity. Common Pile and Common Corpus both adopt the approach of not imposing additional terms beyond the underlying content licenses.

For datasets that do include terms of use, the paper recommends:

Standardization and modularization: "Developing systems for creating modular terms of use that are technically recognizable and easily adaptable." RAIL licenses are cited as an example of this approach, with "about 8000 datasets with RAIL licenses" available on Hugging Face.
Accessibility: "Design terms of use that are clear and centered around user needs and understanding by making terms of use accessible and understandable to non-legal professionals, reducing barriers to compliance."
No restrictive terms on public domain data: "Do not impose restrictive terms on public domain data... ensure that data that now belongs to the public are made maximally useful, and avoid adding restrictions to public domain works (e.g. via the misuse of Creative Commons licenses)." This is a specific warning against a known anti-pattern: applying CC licenses to public domain works in ways that suggest the work is subject to license restrictions when it is legally unrestricted.

Transparency for non-technical users. The paper recommends making it "easy for people without a technical background to check if their data is in a dataset." BigCode's tool for verifying repository inclusion in The Stack is cited as an example of this practice. This operationalizes the principle of accessible transparency — it is not sufficient for dataset documentation to be technically complete if it is not usable by the people most affected by data inclusion decisions.

Versioning and long-term preservation. The paper's principle of "preserving data for the long term" encompasses both technical preservation (ensuring data remains accessible as formats and platforms evolve) and version control (maintaining the ability to reference specific dataset versions for reproducibility). The Croissant standard and platform choices (Hugging Face, dedicated websites) are the primary mechanisms discussed for achieving this.

The synthetic data caveat. While not a major focus, the paper includes guidance on synthetic data: "When synthetic data is used, provide the full tooling for how this data was generated, including information on the source of the seed data, the prompts and model(s) that were used to generate synthetic data, as well as the implementation of privacy-preserving methods like differential privacy to protect individual data points, and anonymization of data to remove PII when fine-tuning models to generate synthetic data." The paper also warns that "synthetic data is no silver bullet; it comes with its own biases and problems, and should be used with care just like any other data."

4. Key Insights and Innovations

Innovation 1: A Ground-Level Systems View of Open Dataset Construction as a Socio-Technical Coordination Problem

The field's default framing for training data has largely been technical and reductionist: the problem is scraping enough web pages, filtering for quality, and training a model. When licensing is discussed, it is typically in the abstract—papers propose that future datasets "should" use open data, or policy documents argue that copyright law "should" be reformed, without grappling with what it actually takes to verify licensing status at scale. This paper makes a fundamental conceptual move by treating open dataset construction as a coordination problem across three deeply interdependent domains—legal, technical, and governance—that cannot be solved in isolation.

The insight is not that licensing matters (that is obvious) or that metadata is useful (equally obvious). The insight is the shape of the interdependency: a legal determination (is this book in the public domain?) depends on technical infrastructure (can we match OCR'd copyright renewal records at scale?), which in turn constrains governance mechanisms (can we honor opt-out requests if we do not preserve provenance metadata?), which feeds back into legal risk (if we cannot verify our license determinations, downstream users cannot rely on them). This is not a pipeline where one stage hands off cleanly to the next; it is a knot of mutual constraints where decisions in any domain cascade through the others.

Prior to this paper, discussions of training data openness tended to bifurcate into technical papers (describing filtering heuristics, dataset composition, deduplication methods—as in the FineWeb, Dolma, and RefinedWeb papers) or legal/policy analyses (articulating what the law should require, as in the various AI Act implementation proposals). The technical papers rarely discussed the legal reasoning behind their source selection, and the policy papers rarely engaged with the practical impossibility of implementing their proposals at scale. This paper bridges that gap not by proposing a unified theory but by documenting what actually happens when organizations try to do both simultaneously. The Common Pile copyright renewal pipeline (Figure 3, Appendix A) is the clearest illustration: it combines legal reasoning (US copyright renewal requirements for 1929–1964 works), software engineering (LLM-assisted metadata extraction from unstructured OCR), data quality analysis (error independence guarantees for false positive rates), and library science (locating digitized copies of identified books). None of these domains is optional; remove any one and the pipeline fails.

This framing reframes the problem statement itself. Rather than asking "how do we build an open dataset?" (a technical question), it asks "how do we coordinate the expertise, infrastructure, and community needed to make open dataset construction tractable?" (an organizational and institutional question). The paper's seven guiding principles and its emphasis on community-building, shared standards, and sustainable funding follow directly from this diagnosis. The innovation is not any single principle but the meta-level recognition that the unit of progress is not a better filtering heuristic but a functioning socio-technical community—analogous to what emerged for open source software in the 1990s and early 2000s.

This is a fundamental reframing, not an incremental refinement. It shifts the conversation from "what should happen" to "what must be true for it to happen," from principles to preconditions. The evidence for this insight is not a single table or figure but the entire structure of the paper: the detailed case studies expose how much manual, interdisciplinary, and collaborative work underlies each dataset component, making visible what is invisible in finished dataset documentation.

Innovation 2: The Tiered Openness Taxonomy as a Diagnostic Framework for Legal Risk

Prior discussions of dataset openness have tended toward binary categories: a dataset is either open or it is not, either downloadable or proprietary. This paper introduces a three-tier taxonomy—replicable, downloadable/open-access, openly licensed—that does not merely categorize datasets but diagnoses the specific legal risks associated with each tier and the verification work required to advance between them.

The innovation is not the observation that some datasets have ambiguous licensing (this was widely known) but the operationalization of the taxonomy as a decision-making tool. By distinguishing the license on the dataset compilation from the licenses on constituent data points—and noting that "the act of collecting, processing, and arranging existing data grants intellectual property interests in the arrangement" but "does not grant the compiler the right to change the licensing of the underlying data"—the paper identifies a failure mode that had caused substantial confusion in practice. The observation that "many people believe that AI2's Dolma dataset is openly licensed due to the ODC-by dataset license even though the constituent data is not openly licensed" is not merely a definitional clarification; it diagnoses a concrete risk: organizations using Dolma may believe they are legally protected when they are not, because the permissive compilation license does not extend to the copyrighted material within the dataset.

This is a diagnostic contribution, not a theoretical advance. It does not propose new licensing frameworks or legal theories; it provides a vocabulary that makes existing legal complexity legible to practitioners. The taxonomy can be applied to any dataset to determine what verification work remains: if a dataset is downloadable but not openly licensed, the gap is in constituent data licensing verification; if it is replicable but not downloadable, the gap is in distribution.

The comparison to prior work is instructive. Initiatives like the Open Source AI Definition and The Model Openness Framework have proposed multi-dimensional openness frameworks that include data, but these typically treat data as one dimension among many (model weights, training code, evaluation methodology). This paper's taxonomy is specifically designed for the data dimension and is driven by the legal complexity of copyright rather than general transparency concerns. The Creative Commons community has long distinguished between works and collections, but that distinction had not been systematically applied to LLM training datasets in a way that surfaced the specific verification burdens involved.

The significance of this innovation extends beyond the paper's immediate recommendations. By making legal risk legible and gradable, the taxonomy lowers the barrier to entry for new dataset builders who might otherwise be paralyzed by uncertainty. It also creates accountability: a dataset described as "open" can be evaluated against the taxonomy to determine whether that claim is substantiated. This is a modest conceptual contribution—it introduces no new mathematics, no new algorithms—but it addresses a genuine and consequential confusion in the field.

Innovation 3: Documenting the "Access Paradox" and Making It a First-Class Policy Concern

The paper identifies and names a phenomenon that has been widely experienced but rarely articulated as a coherent problem: the access paradox, where works that are legally in the public domain are practically inaccessible for LLM training because digitized copies are controlled by entities that impose contractual restrictions, or because digitization has simply never occurred. The paper's formulation is worth quoting directly: "Just because a document is in the public domain, it does not mean one can get a copy of it."

This observation may seem obvious in retrospect, but it has been largely absent from discussions of training data policy, which tend to assume that public domain status solves the access problem. The paper documents specific mechanisms by which access is blocked: the Google Books project scanned millions of public domain books but "it is not possible to gain unrestricted bulk access to all books that even Google thinks are in the public domain"; organizations that do get access "often need to sign agreements that limit their ability to use the content"; cultural heritage institutions "physically own" public domain works and can restrict access to their digitizations. The Common Pile team's experience with post-1929 unrenewed books is the empirical anchor: they have identified "approximately 500,000 books that we believe to be in the public domain, though it is currently unclear how many have accessible digitizations." Half a million books—a potentially enormous training data resource—exist in a legal limbo where they are free to use but impossible to obtain.

Prior discussions of training data availability have focused on copyright law as the primary barrier: the argument has been that if fair use exceptions were clarified or opt-out mechanisms implemented, the data access problem would be largely solved. This paper argues that this diagnosis is incomplete. Even in the absence of copyright barriers, there are structural barriers—the economics of digitization, the contractual restrictions imposed by platforms, the format problems of PDFs and OCR—that prevent public domain data from being practically usable. This reframes the policy conversation from "we need legal reform" to "we need public investment in digitization infrastructure and open access mandates."

The significance of this innovation is that it shifts the locus of policy intervention from copyright law (which is slow, contentious, and jurisdiction-specific) to public investment and infrastructure (which is more tractable and has precedents in library funding, open science initiatives, and cultural heritage digitization programs). The paper's recommendations follow from this diagnosis: "requiring certain institutions or commercial entities to release data in a sanitized, structured way under an open license after a specific period" and "investing in better tools for extracting openly licensed content from difficult formats like PDFs." These are infrastructure investments, not legal reforms.

This is a diagnostic advance: it identifies a specific failure mode in the current ecosystem (the gap between legal permission and practical access) that has been underappreciated, and it connects that diagnosis to actionable policy recommendations. The evidence is qualitative but compelling—the experiences of Common Pile and Common Corpus in trying and often failing to obtain machine-readable copies of public domain works, described in detail across the case studies and synthesized in Section 5.1.

Innovation 4: The Governance-Transparency Tension as a Design Constraint, Not an Oversight

A less ostentatious but intellectually significant contribution is the paper's explicit surfacing of the tension between data governance mechanisms (opt-outs, removal requests, community preference signals) and the principle of dataset reproducibility. The paper notes that implementing a rolling individual opt-out "contradicts the idea of working with a verifiably identical dataset over time."

This is not presented as a flaw to be fixed but as a genuine design tradeoff that dataset builders must navigate. The paper does not resolve the tension; it documents it and explores its implications. If a dataset changes whenever someone requests removal, two researchers using "the same" dataset at different times may have different training data, making experimental comparisons invalid. If a dataset refuses to implement removal mechanisms to preserve reproducibility, it denies data subjects the ability to correct errors or assert preferences—violating the principle of minimizing harms. There is no solution that satisfies both principles fully.

Prior discussions of dataset documentation and governance have tended to treat transparency and ethical considerations as aligned goals: more transparency enables more accountability, which enables better governance. This paper complicates that picture by showing that they can pull in opposite directions. The Common Pile team's decision to create a stable, immutable dataset—with no post-release updates—is a choice that prioritizes scientific reproducibility over ongoing governance responsiveness. Pleias' "release often, release early" model makes the opposite tradeoff. Neither choice is presented as definitively correct; the paper instead surfaces the tradeoff so that dataset builders can make it deliberately rather than accidentally.

This is a conceptual contribution about the structure of the problem space. It identifies an incommensurability that dataset builders must navigate and provides language for discussing it. The field's default assumption—that more openness, more transparency, and more governance are always mutually reinforcing—is shown to be oversimplified. The innovation is the honesty of the acknowledgment and the refusal to paper over the conflict.

The evidence for this tension is distributed throughout the paper: the discussion of opt-out mechanisms in Section 4.4, the contrasting governance models of Common Pile and Common Corpus, and the paper's own acknowledgment of unresolved open questions. It is a subtle but important corrective to the tendency in AI ethics and policy to treat all good things as going together.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper does not report a traditional experimental evaluation with quantitative metrics, test sets, or model training runs. This is not an oversight — it is a deliberate consequence of the paper's nature as a community consensus and best-practices document. The paper's "results" are qualitative: the documented experiences of organizations building open datasets, the identification of common challenges, the articulation of shared principles, and the codification of emerging best practices. The case studies in Appendices A and B serve as the primary evidentiary base.

Base model(s). No base models are trained, fine-tuned, or evaluated. The paper does reference several existing models in passing — Whisper (for YouTube transcript generation in the Common Pile pipeline), a language model trained by Nous Research (for extracting copyright metadata from unstructured OCR in the copyright renewal verification pipeline), and StarCoder2 (whose training data variant is used as a starting point for the Stack v2 code subset) — but these are tools within pipeline descriptions, not objects of experimental evaluation.

Metrics. The paper employs no quantitative metrics in the conventional ML sense (accuracy, F1, BLEU, perplexity). The "metrics" are qualitative assessments of dataset properties: licensing status verification coverage, OCR quality, metadata completeness, community engagement depth, and governance mechanism robustness. Where numbers do appear — e.g., "259,728,610 pages comprising 221,715,271,483 words" for the Common Crawl CC subset, "96.58% accuracy for titles and 97.20% for registration numbers" for LLM-based copyright metadata extraction, "approximately 500,000 books that we believe to be in the public domain" — they are descriptive statistics about dataset composition or pipeline component performance, not evaluation metrics against ground truth.

Baselines. The paper does not compare against alternative approaches in a controlled experimental framework. The "baselines" are implicit: the status quo of proprietary training datasets (characterized by opacity, legal risk concentration, and community exclusion), and the prior generation of downloadable-but-not-openly-licensed datasets (the original Pile, Dolma, C4). The paper's argument is that these baselines are insufficient for the ecosystem's needs, and the case studies demonstrate that alternatives are technically feasible.

Generation budget / compute accounting. No compute budget is specified or compared in the conventional sense. The paper does not report FLOP counts, GPU hours, training costs, or inference budgets. The computational costs discussed are incidental to pipeline descriptions: OCR processing for scanned books, Whisper-based transcription for YouTube videos, LLM inference for copyright metadata extraction. None of these are framed as experimental variables to be optimized or compared.

Cross-validation / statistical protocol. No statistical protocol — cross-validation, train/test splits, confidence intervals, significance testing — is employed. The LLM-based copyright metadata extraction accuracy numbers (96.58%, 97.20%, 90.25%, 87.66%) are reported without information about sample sizes, validation set construction, or confidence intervals. The error independence analysis for false positive rate (0.09%) is a probabilistic calculation rather than an empirical measurement. For a document of this type, this is appropriate — the paper does not claim experimental rigor in these metrics, and they serve illustrative rather than evidentiary purposes.

Main Quantitative Results

There are no main quantitative results in the sense that a machine learning paper would report them. The paper's "results" are the documented practices, identified challenges, and articulated principles themselves.

What the paper does provide, across the case studies and best practices sections, are descriptive quantitative characterizations of the datasets being built. These serve to establish that openly licensed dataset construction at scale is not merely aspirational but is being actively realized. The key numbers, organized by pipeline stage:

Source Identification and License Verification

The Common Pile copyright renewal verification pipeline (Appendix A, Figure 3) reports:

424,059 works analyzed through the registration-renewal matching process
0.09% estimated false positive rate for public domain classification (approximately 380 books out of 424,059) due to independent errors in title and registration number extraction
96.58% accuracy for title extraction and 97.20% accuracy for registration number extraction by the LLM-based metadata extractor
98.6% accuracy for LLM-based ambiguous match resolution in preliminary experiments
Approximately 500,000 books identified as likely in the public domain, with unknown digitization availability

The Common Crawl CC license extraction pipeline reports:

52 Common Crawl snapshots processed
259,728,610 pages retained after license filtering
221,715,271,483 words total
License identification accuracy is explicitly noted as unvalidated: "We are currently in the process of carrying out a detailed study of the end-to-end accuracy of our license identification pipeline"

The arXiv subset reports that approximately 15% of all papers are under permitted licenses.

Content Acquisition

For the post-1929 unrenewed books component, the paper reports approximately 500,000 books identified as likely public domain but notes it is "currently unclear how many have accessible digitizations" — a quantitative characterization of the access paradox.

The YouTube-Commons dataset used automatic transcripts and translations from YouTube's built-in system, trading transcription quality for scale. The Common Pile YouTube component uses a Whisper-based pipeline for higher-quality transcription, but no quantitative comparison of transcription quality is reported.

Dataset Scale and Composition

The Common Pile case study (Appendix A) enumerates the following components, though not all have final size numbers reported:

Academic texts: arXiv (subset of ~15% of papers under permitted licenses), BioDiversity Heritage Library (filtered by license metadata), Case Law Access Project (CC0/public domain, 40 million pages, 6.7 million cases), CourtListener (CC0/public domain), PubMed Central (filtered by license)
Code data: Stack v2 (filtered from the StarCoder2 training variant), Stack Exchange (CC BY-SA, question-answer structured)
Public domain books: Project Gutenberg (~70,000 books including PG19 collection), Library of Congress Books (>130,000 English books from Selected Digitized Books collection), Pre-1929 Books (all US publications from 1929 and earlier, partially available through multiple repositories), Post-1929 Unrenewed Books (~500,000 identified, digitization availability unknown)
News: Permissively licensed news-wire text (no specific size reported)
Web data: Common Crawl CC subset (259,728,610 pages, 221,715,271,483 words), Data Provenance Collection (permissively licensed datasets filtered from the Data Provenance Explorer), Foodista (CC BY/3.0), Public Domain Review (CC BY-SA/3.0), Ubuntu IRC (public domain chat logs)
Wikis: Wikimedia Foundation wikis (Wikipedia, Wiktionary, etc.), WikiTeam archives from WikiFarms, including "talk" pages
YouTube transcripts: Videos with uploader-specified permissive licenses, transcribed via Whisper
Government documents: USPTO patent applications (~20 million applications), Government Publishing Office documents (public domain)

The Common Corpus case study (Appendix B) emphasizes that the first release focused on pre-1884 public domain content (a conservative threshold for cross-jurisdictional certainty) with subsequent expansions to include permissively licensed text. Specific size numbers for Common Corpus are not reported in this paper.

Ablation Studies and Robustness Checks

The paper does not conduct ablation studies in the conventional ML sense, because there is no model being trained, no algorithm being optimized, and no controlled experimental setup. However, the paper does engage in something functionally analogous: it documents design choices, tradeoffs, and failure modes across different approaches to the same pipeline stage, effectively serving as a qualitative ablation analysis of open dataset construction strategies.

Stable immutable dataset (Common Pile) vs. living community-maintained dataset (Pleias' vision): The paper presents these as alternative governance models with different tradeoffs rather than as competing approaches with a clear winner. The Common Pile model prioritizes scientific reproducibility (fixed dataset enables comparable experiments over time) at the cost of responsiveness to community feedback and technical improvements. Pleias' model prioritizes continuous improvement and community ownership at the cost of reproducibility ("release often, release early" means the dataset is a moving target). The paper presents this as a genuine design choice, not a question with a correct answer, and notes that Pleias' envisioned community "does not yet exist."

Pre-1884 threshold (Common Corpus) vs. copyright renewal verification pipeline (Common Pile) for public domain book identification: These represent fundamentally different approaches to the same problem. Pleias' conservative pre-1884 threshold eliminates the need for complex legal analysis but "substantially reduces the volume and recency of available content." Common Pile's sophisticated pipeline (copyright registration-renewal matching, LLM-based metadata extraction, ambiguous match resolution) enables inclusion of 1929–1964 works but requires substantial legal expertise, software engineering, and validation effort. The paper provides enough detail about both approaches to make the tradeoff legible without declaring either superior.

YouTube automatic transcription (YouTube-Commons) vs. Whisper-based transcription (Common Pile): For extracting text from video content, Pleias used YouTube's built-in automatic transcription, acknowledging that "this approach is not considered ideal because the quality of the transcripts and especially the automated translations varies." The Common Pile YouTube component used a Whisper-based pipeline to obtain higher-quality transcripts. The paper presents this as a quality-vs-simplicity tradeoff without reporting quantitative comparison data. Pleias' approach is explicitly framed as "a first step that a community could help improve and expand upon."

License identification scope: CC-only regex matching vs. broader Blue Oak Council Bronze+ detection: The Common Pile team limited their web data license detection to Creative Commons licenses specifically because "we found automatically detecting all Blue Oak Council Bronze or higher licenses to be infeasible due to the wide variety of ways people describe their licensing terms." This is a deliberate scope restriction — high precision for CC licenses at the cost of missing other permissive licenses. The paper acknowledges this tradeoff explicitly. The end-to-end accuracy of even the restricted CC pipeline is noted as still under study.

ReSTᴱᴹ fine-tuning for revision models: catastrophic degradation: If we extend to the reference example's style where negative results are treated as important findings, the paper documents a notable negative result regarding data processing approaches. While this appears in the prior work rather than the current paper, the paper does surface a cautionary finding about synthetic data: "synthetic data is no silver bullet; it comes with its own biases and problems, and should be used with care just like any other data." The recommendation to "use quality metrics to ensure synthetic data is accurate, consistent, and representative of the original dataset" and to "regularly inspect the synthetic data to maintain ethics, privacy, and quality standards" reflects awareness that synthetic data generation can degrade rather than improve dataset quality if applied naively.

The license coverage ambiguity problem (site-level CC declaration vs. per-asset license assignment): The paper identifies a fundamental limitation of web-scale license detection: "there is currently no automated way of determining which asset on the website that license covers." This leads to false positives where a CC-BY image embedded in a non-CC article causes the entire page to be included. The paper does not claim to have solved this problem; it documents it as an open challenge and explicitly acknowledges that the CC filtering pipeline's accuracy is still being evaluated. This functions as a negative result for the naive approach of treating site-level license declarations as applying to all page content.

No additional terms of use (Common Pile and Common Corpus) vs. RAIL-based terms: The paper notes that both Common Pile and Common Corpus decline to impose additional terms beyond the underlying content licenses — "whatever license the content was originally published under applies." This contrasts with datasets that use RAIL licenses to restrict particular uses. The paper presents this as a design choice reflecting the principle that "Terms of use should only be provided if there are clear pathways to enforce them," but does not present empirical evidence about which approach produces better outcomes.

Critical Assessment

This section must address a fundamental challenge: the paper does not make claims that can be evaluated through the lens of experimental evidence in the conventional sense. It makes claims about what is possible (building openly licensed datasets at scale), what challenges exist, what principles should guide practice, and what practices are emerging. These are claims about the state of the world and the shape of the problem space, not claims about the performance of a particular method.

The appropriate critical assessment therefore examines whether the paper's evidence supports its central assertions — and, perhaps more importantly, identifies what kind of evidence would be needed to fully substantiate them but is not provided.

Claim 1: Building openly licensed training datasets at meaningful scale is technically feasible

This is the bedrock claim. The paper supports it through existence proofs: Common Pile is being built, Common Corpus has been released, YouTube-Commons exists. The detailed component-by-component documentation in Appendix A serves as evidence that real work has been done — specific sources have been identified, specific processing pipelines have been implemented, specific license verification procedures have been executed, and real text has been extracted.

What the evidence actually demonstrates: That motivated, well-resourced teams can assemble large collections of openly licensed text. The Common Crawl CC subset (259 million pages, 221 billion words) demonstrates web-scale license filtering. The 500,000 identified public domain books demonstrate systematic copyright research at scale.

What the evidence does not demonstrate: (1) That these datasets are sufficient to train competitive models. The paper asserts that Common Pile is meant to disprove "claims by some leading AI companies that training performant LLMs without copyrighted material is impossible," but no model trained on Common Pile is evaluated. This is the most significant gap in the paper. Without evidence that models trained on these openly licensed datasets achieve competitive performance, the feasibility claim remains partially substantiated — the data exists, but its fitness for purpose is unproven. (2) That the pipeline is reproducible by others. The paper advocates for reproducibility but does not report whether an independent team has successfully replicated the Common Pile or Common Corpus construction process. (3) That the license verification is accurate. The Common Crawl CC pipeline's end-to-end accuracy is explicitly unvalidated. The copyright renewal pipeline's false positive rate calculation is based on a probabilistic model with assumptions (independence of errors) that may not hold in practice. The YouTube uploader-specified licenses are explicitly unvalidated. This means the "openly licensed" designation for significant portions of these datasets rests on unverified automated methods.

Missing experiment: A systematic validation study of license identification accuracy — sampling documents from the CC pipeline, having legal experts manually verify their licensing status, and reporting precision and recall. Without this, users of the datasets cannot assess their legal risk exposure.

Claim 2: The primary bottlenecks are legal, technical, and governance challenges that require coordinated interdisciplinary effort

This claim is strongly supported by the case study detail. The copyright renewal verification pipeline (Figure 3) is the paper's strongest piece of evidence: it requires copyright law expertise (to know which failure modes matter), software engineering (to build the matching system), data quality assessment (to understand OCR error propagation), machine learning (to apply LLMs to unstructured metadata extraction), and library science (to locate physical copies). No single discipline could build this pipeline.

The access paradox — public domain works that cannot be accessed due to digitization gaps or contractual restrictions — is similarly well-documented through the Common Pile team's experience with finding digitized copies of identified public domain books. The paper's honesty about what remains unknown ("it is currently unclear how many have accessible digitizations") strengthens rather than weakens this claim by demonstrating that even after substantial investment, fundamental access problems remain.

What this evidence does not demonstrate: Whether the coordination model described (convenings, shared principles, open-source tooling) is sufficient to sustain the work at scale. Pleias' envisioned community "does not yet exist" — the Common Corpus work is still done by "only a handful of people." The paper documents what coordination is needed but cannot demonstrate that the proposed mechanisms will produce it. This is not a failure of the paper's analysis; it is a limitation of what a convening-based document can establish.

Claim 3: The three-tier openness taxonomy resolves confusion and enables better decision-making

This claim is supported primarily through analytical clarity rather than empirical evidence. The paper's identification of the Dolma licensing confusion ("many people believe that AI2's Dolma dataset is openly licensed due to the ODC-by dataset license even though the constituent data is not openly licensed") is a concrete example of the confusion the taxonomy addresses. The taxonomy's operational value is demonstrated implicitly through the case studies: the distinction between dataset-level and constituent-level licensing structures the entire Common Pile approach of verifying every component's license individually.

What the evidence does not demonstrate: Whether the taxonomy has been adopted, whether it has reduced confusion in practice, or whether it is sufficient to capture all relevant dimensions of data openness. The taxonomy focuses on licensing but does not address data quality, representativeness, documentation completeness, or other dimensions that users might reasonably consider part of "openness." There is no evidence presented about whether dataset users find the taxonomy useful.

Claim 4: Current opt-out mechanisms are crude and risk shrinking the open web

This claim has strong empirical support from external evidence (the paper cites Longpre et al.'s "Consent in Crisis" documenting the decline of the AI data commons) but the paper's own evidence is limited to qualitative observation. The distinction between network-level blocking and preference signals is analytically clear, and the paper provides concrete examples of both (Cloudflare's one-click blocking vs. Spawning's API-based opt-outs), but no data is presented about the relative prevalence or effectiveness of each approach.

The paper's warning that blanket blocking "leads many website owners to completely block non-profit archives like Common Crawl that are used for non-commercial purposes" is a significant claim about ecosystem dynamics. It is not directly evidenced in the paper — the reader must either accept it based on the expert consensus of the convening participants or seek external confirmation.

Claim 5: Sustainable funding models for open datasets are needed but do not yet exist

The paper is candid about this being an open question rather than a claim with supporting evidence. Section 5.4 asks whether income models like Wikimedia Enterprise or Spawning's Source.Plus could provide needed revenue, and whether "we need to change the way we think about openness and introduce access limitations to prevent open data exploitation." These are framed explicitly as open questions. The paper's contribution is identifying the problem and cataloging potential approaches, not demonstrating that any particular approach works.

Structural Limitations

Single organizational perspective — or a narrow coalition of aligned organizations: The convening brought together 30 participants from "prominent open-source AI startups, nonprofit AI labs, and civil society organizations working on open access and openly licensed datasets." This is a specific and like-minded group. The paper does not incorporate perspectives from: large AI companies (who might argue that copyright-safe training data is impractical at scale), content creators who oppose all AI training regardless of licensing (who might argue that even public domain training is extractive), or organizations from jurisdictions with fundamentally different copyright frameworks (the case studies focus heavily on US copyright law, with some EU considerations in the Common Corpus case study). The consensus documented is real but partial.

No evaluation of downstream model quality: The most significant gap is the absence of any model trained on the described datasets. The paper's core strategic claim — that "training performant LLMs without copyrighted material" is possible — is asserted but not demonstrated. Common Pile was still under construction at the time of writing ("a work-in-progress dataset"). Common Corpus had a first release, and Pleias "later expanded Common Corpus to include permissively licensed text, using the updated dataset to train and fine-tune models" — but no results from those models are reported. This is not a flaw in the paper's stated scope, but it means the paper cannot validate its most consequential assertion.

The cost question is unaddressed: Building Common Pile and Common Corpus required enormous manual effort — copyright renewal record analysis, bibliographic metadata reconciliation, PDF OCR correction, license verification across jurisdictions. The paper does not estimate the total person-hours or financial cost of these efforts, nor does it compare these costs to alternatives (e.g., licensing commercial data, paying copyright holders). Without cost estimates, the claim that open datasets are a viable alternative to proprietary approaches is incomplete — feasibility includes economic feasibility.

The temporal dimension is unexplored: The paper discusses preserving data "for the long term" but does not address the temporal dimension of the pipeline itself. Copyright terms expire over time — works that are not public domain today will become public domain in the future. License designations on web pages change. Opt-out preferences evolve. The static vs. living dataset governance models address some of this, but the paper does not discuss how the pipeline accounts for temporal dynamics in licensing status.

The representativeness question is unquantified: The paper advocates for "improving diversity" in training data (Principle 4) and acknowledges that "voice and text datasets powering AI dramatically under-represent 99%+ of global languages." But the case studies do not report demographic, linguistic, or geographic diversity statistics for the datasets being built. The Common Pile components are overwhelmingly English-language and US-centric (US government documents, US case law, US patent office, English Wikipedia, English books from Library of Congress, English Stack Exchange). The Common Corpus began with French subsets and expanded, but no language distribution is reported. Without diversity statistics, it is impossible to assess whether the openly licensed datasets being built actually advance the diversity principle or reproduce the same English-and-Western-centric biases as proprietary datasets — just with better licensing.

The governance mechanisms are aspirational rather than operational: The paper recommends community engagement, data trusts, collective governance, and preference signaling infrastructure. The actual governance mechanisms described in the case studies are minimal: Common Pile has no post-release update mechanism, no community governance structure, no opt-out process. Common Corpus' community governance "does not yet exist." BigCode's opt-out process for The Stack is cited as an example of a working mechanism, but it is relatively lightweight (a GitHub-based verification process) and its effectiveness is not evaluated. The gap between the governance recommendations and the governance actually implemented is substantial, and the paper does not fully acknowledge this.

In summary: the paper's evidence is strongest for its diagnostic claims (what the challenges are, why they are interconnected, why coordination is needed) and weakest for its predictive and normative claims (that openly licensed training data will produce competitive models, that the proposed governance mechanisms will work at scale, that the community coordination model is sustainable). This is consistent with the paper's nature as a community consensus document rather than a research contribution — but readers should not mistake the detailed case studies for experimental validation of the paper's strategic assertions.

6. Limitations and Trade-offs

The Unvalidated Core Premise: No Evidence That These Datasets Produce Competitive Models

The assumption or constraint. The paper's central strategic claim — articulated most directly in the Common Pile case study — is that building openly licensed datasets at scale serves as "a statement against claims by some leading AI companies that training performant LLMs without copyrighted material is impossible." Every principle, every best practice, and every pipeline design choice in the paper is predicated on the assumption that openly licensed data, responsibly curated, can produce models competitive with those trained on proprietary or unverified web data. The paper does not test this assumption.

The consequence. Without experimental validation, the paper cannot distinguish between two very different scenarios: (1) openly licensed datasets are genuinely sufficient to train competitive models, in which case the barrier to adoption is primarily coordination and investment; or (2) openly licensed datasets systematically lack properties that make web-scale uncurated data effective for LLM training — perhaps because the open subset of the web has different genre distributions, different information density, lower coverage of specialized domains, or lacks the "long tail" of diverse examples that contributes to generalization. If scenario (2) is true, the paper's recommendations, while normatively sound, would not produce the competitive ecosystem it envisions.

The consequence is not merely academic. Organizations investing in open dataset construction based on this paper's guidance would be betting substantial resources on an unverified hypothesis. The paper asserts that Common Pile is "meant to become a default dataset that should be reused for years to allow for better model comparison," but if models trained on Common Pile significantly underperform those trained on larger uncurated datasets, it will not become a default — researchers will revert to higher-performing but legally riskier alternatives.

What evidence exists in the paper. Common Pile is described as "a work-in-progress dataset" at the time of writing. Common Corpus had a first release, and the paper notes that "Pleias later expanded Common Corpus to include permissively licenced text, using the updated dataset to train and fine-tune models" — but no training results, benchmark scores, or performance comparisons are reported. Neither case study includes model evaluations. This is the most significant evidentiary gap in the paper. The sole performance-related claim — that openly licensed data can produce competitive models — remains entirely unsubstantiated.

Mitigation status. The paper does not acknowledge this gap as a limitation. The framing throughout assumes that the primary challenge is assembling the data, not demonstrating its fitness for purpose. Section 6 briefly mentions that "the openness of datasets alone doesn't guarantee positive societal impact or prevent possible harms," but this refers to downstream governance, not model performance. The absence of any training results means the paper's most consequential claim is, at present, an article of faith rather than an empirically grounded assertion.

License Verification Accuracy Is Unknown for the Largest Data Component

The assumption or constraint. The paper describes a pipeline for extracting Creative Commons-licensed content from Common Crawl that produced 259 million pages comprising 221 billion words — the largest single component of Common Pile by volume. The pipeline relies on regex-based identification of CC license identifiers in HTML metadata. The paper's assumption is that this automated approach accurately separates permissively licensed content from restrictively licensed or unlicensed content. This assumption is central to the claim that Common Pile will be an openly licensed dataset.

The consequence. If the license identification pipeline has significant false positive rates — incorrectly classifying restrictively licensed or unlicensed pages as CC-licensed — then the resulting dataset would contain copyrighted material whose inclusion defeats the purpose of building an openly licensed dataset. Users who rely on Common Pile's claimed licensing status for legal risk management would be exposed to precisely the liability the dataset was designed to avoid.

The paper itself identifies the mechanism by which false positives occur: a website may link to a CC license for a specific embedded asset (e.g., a CC-BY photograph) while the surrounding article text is not CC-licensed. The pipeline detects the site-level CC declaration and includes all page content, not just the CC-licensed asset. At 259 million pages, even a low false positive rate would translate to millions of incorrectly included documents. The paper explicitly states that "there is currently no automated way of determining which asset on the website that license covers" — meaning this error mode is inherent to the approach, not a bug that can be fixed with better regex.

What evidence exists in the paper. The paper is transparent about this limitation in a way that is worth quoting directly: "We are currently in the process of carrying out a detailed study of the end-to-end accuracy of our license identification pipeline." The study had not been completed at the time of writing. The false positive rate, false negative rate, precision, and recall of the license detection pipeline are all unknown. The paper also notes that detecting non-CC permissive licenses was "found to be infeasible due to the wide variety of ways people describe their licensing terms," meaning many genuinely permissive pages are systematically excluded (false negatives) with unknown impact on dataset coverage and quality.

Mitigation status. The paper acknowledges the pipeline accuracy is unvalidated but does not treat this as a limitation that threatens the dataset's fitness for purpose. The planned accuracy study is mentioned but its design, methodology, and expected timeline are not described. Until this validation is completed and published, the licensing status of the single largest Common Pile component must be treated as uncertain — and with it, the status of the entire dataset as "openly licensed" under the paper's own Tier 3 definition, which requires that "every individual data component be under permissive licenses or in the public domain."

The Access Paradox Is Documented but Has No Operational Solution

The assumption or constraint. The paper identifies what it calls the "access paradox": "Just because a document is in the public domain, it does not mean one can get a copy of it." The authors document specific barriers — Google Books digitizations that are technically public domain but access-restricted by contractual agreements, cultural heritage institutions that "physically own" public domain works and restrict access to their digitizations, and the fundamental problem that many public domain books have never been digitized at all. The Common Pile team identified approximately 500,000 books believed to be in the public domain through the copyright renewal verification pipeline, but notes "it is currently unclear how many have accessible digitizations." The implicit assumption in the paper's vision — that openly licensed datasets at meaningful scale can be assembled — is that these access barriers can be overcome.

The consequence. The access paradox means that legal permission does not imply practical availability. The paper's most ambitious pipeline component — identifying post-1929 unrenewed public domain books through copyright record analysis — may produce a list of half a million works that are legally free to use but practically impossible to obtain in machine-readable form. This is not a temporary inconvenience that better tools will resolve; it reflects structural facts about who controls digitized copies of books and under what terms they are made available.

The consequence for the open dataset vision is severe: if the most legally unambiguous and highest-quality sources (published books, which have been professionally edited and represent sustained intellectual work) are largely inaccessible in practice, open dataset builders are forced to rely disproportionately on web data (with its license verification problems, as discussed above) and government documents (which are public domain but have limited topical and genre diversity). The resulting dataset may be legally clean but substantively impoverished — missing the long-form, carefully composed, domain-diverse content that makes books corpora valuable for LLM training. The paper's own discussion of why books matter for training data underscores this: "Books are an essential source of long context, semantically complex, text. Most LLM trainers view large books corpora as an essential source of training data." If the books cannot be obtained, the essential source is missing.

What evidence exists in the paper. The paper documents the access paradox through the Common Pile team's direct experience: the copyright renewal verification pipeline identified works, but locating digitized copies required "a range of resources to obtain cross-reference ID numbers and identify libraries or other sources that possess these texts," with no guarantee of success. The Common Corpus team took a different approach — limiting to pre-1884 works — but this "substantially reduces the volume and recency of available content." Both case studies provide qualitative evidence that access barriers are real and consequential, but neither quantifies the scale of the problem (what percentage of identified public domain books are actually obtainable in usable form?).

Mitigation status. The paper identifies this as a policy problem ("there is a need for serious public support for open data commons") and recommends interventions including "requiring certain institutions or commercial entities to release data in a sanitized, structured way under an open license after a specific period" and "investing in better tools for extracting openly licensed content from difficult formats like PDFs." These are sensible long-term recommendations, but they are external to the dataset construction pipeline — dataset builders cannot control whether Google Books changes its access policies or whether libraries invest in digitization. The paper offers no operational strategy for dataset builders to work around the access paradox in the near term, beyond persevering with ad-hoc efforts to locate digitized copies from whatever sources are available. The gap between the policy recommendation and the operational reality is substantial and unacknowledged.

Governance Mechanisms Are Aspirational Rather Than Implemented

The assumption or constraint. The paper articulates ambitious governance principles: dataset builders should "work with affected communities" as stakeholders, "create models of redress and removal," "make it easy for people without a technical background to check if their data is in a dataset," and implement community-based governance through mechanisms like data trusts. These recommendations assume that community governance structures can be built and sustained, that affected communities have the capacity and desire to engage, and that the resulting mechanisms will be effective at identifying and addressing harms.

The consequence. The gap between the paper's governance recommendations and the governance actually implemented in the case study datasets is large and unacknowledged:

Common Pile "is meant to become a default dataset that should be reused for years" with "currently no plans to update and change it after publication." EleutherAI "is not adding any terms of use." There is no opt-out mechanism, no post-release removal process, no community governance structure, and no way for individuals to check whether their data is included. The dataset is fixed and immutable — which serves scientific reproducibility but provides none of the responsive governance the paper advocates. If someone discovers that their copyrighted work was incorrectly included due to a license detection error, there is no specified recourse.
Common Corpus aims for a community governance model similar to Wikipedia's, but the paper acknowledges "this imagined community does not yet exist, and most of the work is still done by only a handful of people." There is no mechanism described for how community governance would operate, how decisions would be made, or how disputes would be resolved. The dataset is released without additional terms of use and without curation of content (the YouTube-Commons component is described as having "no further curation of the contents of the data, meaning that no YouTube videos under CC-BY license were excluded due to their contents").

The consequence is that the paper advocates for governance practices that its own case study datasets do not implement. This does not invalidate the recommendations — it may simply reflect that governance infrastructure takes time to build and that these are early-stage projects. But it means the paper provides no evidence that the recommended governance mechanisms are practically achievable or effective. A practitioner reading the paper for guidance on how to implement governance for their own dataset would find principles but no worked examples of those principles in operation.

What evidence exists in the paper. The case studies are transparent about their current governance states. Common Pile's fixed-release, no-update model is clearly described. Common Corpus's aspirational community governance is explicitly noted as not yet existing. The BigCode opt-out process for The Stack is cited as an example of an implemented mechanism — it provides GitHub-verified opt-out requests — but this is described in the general best practices section, not as part of Common Pile or Common Corpus governance. The Spawning API for dataset opt-outs is mentioned but not described as integrated into either case study dataset.

Mitigation status. The paper presents the governance principles as emerging best practices rather than as fully realized implementations. The tension is partially acknowledged: the document notes that removal processes "could limit both reproducibility and transparency, if not well documented," and that "not all potential harms and risks can be mitigated directly via interventions in the dataset." But the paper does not address the fundamental question of whether the stable-default-dataset model (which Common Pile has adopted) and responsive governance (which the paper recommends) are compatible at all. If a dataset is intended to be immutable for scientific reproducibility, post-release removal is impossible by definition. The paper identifies this tension but does not resolve it or provide guidance on how dataset builders should navigate it.

Cost, Labor, and Sustainability Are Not Quantified

The assumption or constraint. The paper describes in detail the manual, expert-intensive work required to build openly licensed datasets: copyright renewal record matching across hundreds of thousands of records, bibliographic metadata reconciliation across library catalogs, PDF OCR correction, jurisdiction-specific public domain determinations requiring legal expertise, and community relationship building with cultural heritage institutions. The implicit assumption is that this work, while substantial, is sustainable — that the ecosystem can support the ongoing labor required to build and maintain these datasets.

The consequence. Without cost quantification, the paper's recommendations are difficult for practitioners to evaluate. Building Common Pile required: matching 424,059 copyright records, processing 52 Common Crawl snapshots, extracting and correcting OCR from millions of scanned pages, verifying licenses across dozens of source types, and coordinating with multiple institutions (libraries, government agencies, research organizations). The paper reports no estimate of person-hours, financial cost, legal fees, or compute resources consumed. This matters because the feasibility of open dataset construction is not merely technical — it is economic. If building a dataset like Common Pile requires multiple person-years of specialized labor (copyright lawyers, software engineers, library scientists, community organizers) at a cost comparable to licensing commercial data, the economic case for the open approach weakens considerably.

The sustainability question is particularly acute given that both case study projects are grant-funded or volunteer-driven. Pleias' community governance model "does not yet exist." EleutherAI is a nonprofit research organization. The paper identifies sustainable funding as an open question (Section 5.4) but does not provide data on what it would cost to maintain and expand these datasets over time, or what funding models would be sufficient. The recommendations for policy interventions (public funding, mandatory data release) are external to the dataset builder's control and may or may not materialize.

What evidence exists in the paper. The paper provides qualitative descriptions of labor intensity — the Common Pile copyright pipeline is described in enough detail to convey its complexity — but no quantitative cost data. Section 5.4 on sustainable funding lists open questions ("What are viable routes to make open datasets sustainable long-term?") without providing estimates of what sustainability would cost. The paper does not compare the cost of building openly licensed datasets to alternatives (licensing commercial data, using uncurated web data with legal risk, or funding copyright litigation defense).

Mitigation status. The paper acknowledges the funding challenge as an open question but does not treat the absence of cost data as a limitation of its current analysis. The recommendations to policymakers (public funding for data commons, requiring institutions to release data under open licenses) implicitly recognize that current funding models may be insufficient, but the paper provides no analysis of the funding gap — how much money is needed, from what sources, over what time horizon — that would make those recommendations actionable. For a practitioner deciding whether to invest in building an openly licensed dataset, the paper provides principles and practices but no framework for assessing whether the investment is economically viable.

Linguistic, Cultural, and Geographic Diversity Remain Aspirational

The assumption or constraint. Principle 4 states that dataset builders should "support and improve diversity," noting that "voice and text datasets powering AI dramatically under-represent 99%+ of global languages, variants and dialects, as well as Black, Indigenous and People of Color and gender-diverse communities." The paper assumes that openly licensed dataset construction can advance this principle — that the open data ecosystem, if properly guided, will produce more diverse and representative training data than proprietary alternatives.

The consequence. The datasets described in the case studies are, by the paper's own description, overwhelmingly English-language and US-centric. The Common Pile components include: US government documents (Patent Office, Government Publishing Office), US case law (Case Law Access Project, CourtListener), US-published books (Library of Congress, Project Gutenberg), English-language wikis (Wikipedia, Wiktionary), English Stack Exchange, English Ubuntu IRC chat logs, and English web data from Common Crawl. The arXiv subset includes multilingual papers but is dominated by English-language scientific writing. The YouTube transcripts component is in-progress and language distribution is not reported. The only explicitly multilingual component is the Data Provenance Collection, where English is still a selection criterion ("contains English or code data, is not model generated, and has a permissive license").

The Common Corpus case study notes that Pleias "started with the French subsets only and slowly expanded to subsets in other languages created with local partners" and expresses a long-term hope that "localized, community-driven contributions worldwide" would improve linguistic diversity. But no language distribution statistics are reported for any Common Corpus release.

The consequence is that the paper advocates for diversity while describing datasets that reproduce the English-and-Western-centric biases of the proprietary datasets they are meant to replace — simply with better licensing. This is not a contradiction of the paper's principles (diversity is presented as a goal, not an achieved state), but it means the paper provides no evidence that the open dataset construction methodology is capable of producing diverse datasets. If the openly licensed subset of available text is systematically skewed toward English, Western, and institutionally-produced content (because these are the sources most likely to have clear licensing metadata and to have been digitized), then the open dataset approach may be structurally incapable of achieving the diversity the paper advocates — regardless of how well-intentioned the builders are.

The paper's warning against using automated translations to increase language coverage is well-taken ("their quality is bad and ignores culturally specific aspects, especially for minoritized languages and low-resource languages"), but it offers no alternative operational strategy. If automated translation is harmful and community-driven data collection for underrepresented languages is resource-intensive and slow, the path to linguistic diversity in openly licensed datasets is unclear.

What evidence exists in the paper. The case studies provide component-level descriptions that allow the reader to infer the English-and-US-centric composition, but the paper itself does not compute or report diversity statistics. No language distribution, geographic origin breakdown, or demographic representation analysis is presented for any dataset. This is a significant documentation gap given that the paper advocates for documentation standards and transparency.

Mitigation status. The paper identifies diversity as a principle and acknowledges current underrepresentation, but provides no concrete plan for how open dataset construction methodology can overcome the structural biases in openly licensed data availability. The recommendation to "involve local communities to identify relevant data sources" is sensible but underspecified: it does not address who funds this community engagement, how communities are identified and approached, what incentives they have to participate, or how the resulting data is integrated into existing pipelines. The paper's own case studies demonstrate that even well-resourced, motivated teams produce English-centric datasets — suggesting the barriers to diverse data are not merely a matter of effort or awareness, but reflect deeper structural properties of what openly licensed text is available. Until these structural barriers are addressed or alternative approaches are demonstrated, the diversity principle remains an aspiration without an operational path to realization.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new algorithm, model, or quantitative result. Its contribution is of a different kind: it attempts to create the shared vocabulary, normative framework, and documented practices necessary for a community to form around openly licensed LLM training datasets—and in doing so, it changes the conversation from "should open datasets exist?" to "what must be true for open datasets to be built at scale?"

The magnitude of this shift is best understood as a reframing of the problem space, not a paradigm shift. The paper does not overturn established technical approaches (filtering heuristics, deduplication methods, web scraping) or propose novel licensing frameworks. What it does change is the unit of analysis. Prior to this work, discussions of LLM training data have largely operated at two disconnected levels: the technical level (how to scrape, filter, and process web text—as in the FineWeb, Dolma, and RefinedWeb papers) and the policy level (what copyright law should require, whether fair use applies, how opt-outs should work). This paper connects them by documenting what happens when organizations attempt to simultaneously satisfy technical quality requirements, legal verifiability standards, and ethical governance principles—and it shows that these domains are not separable concerns that can be addressed in sequence, but mutually constraining design dimensions that must be navigated together.

The paper's most consequential reframing is the elevation of metadata preservation from an afterthought to a first-class infrastructure requirement. The argument, threaded through every section, is that preserving URLs, crawl timestamps, HTTP headers, license identifiers, and content codes is not merely good documentation practice—it is the precondition for every governance mechanism the field wants to build. Opt-outs cannot function without content identification. Removal requests cannot be honored without provenance records. License verification cannot be audited without evidence chains. This is not a new technical insight (archivists and librarians have understood it for decades), but it is a new priority ordering for the ML community, which has historically treated metadata as optional and fragile. The paper's insistence on metadata preservation at every pipeline stage—sourcing, processing, release—implies a different engineering discipline than what has been standard, one where the cost of losing metadata is recognized as equal to the cost of losing data.

A second reframing concerns the nature of the "access" problem. The paper introduces the term "access paradox" to describe a situation where works legally in the public domain are practically inaccessible because digitized copies are controlled by entities that impose contractual restrictions, or because digitization has never occurred. This shifts the policy conversation from copyright reform (which is slow, contentious, and jurisdiction-specific) to public investment in digitization infrastructure (which has precedents in library funding, open science, and cultural heritage preservation). If the paper's diagnosis is correct—that the primary bottleneck is not copyright law but the economics of digitization and the contractual control of digital copies—then the most impactful policy interventions are not about expanding fair use or streamlining opt-outs, but about funding large-scale digitization of public domain works and mandating open access to publicly funded digitization efforts. This is a more tractable target than copyright reform and one where the open dataset community can build coalitions with libraries, archives, and cultural heritage organizations whose interests align.

Third, the paper provides a framework for reconciling apparently conflicting positions in the training data debate. The disagreement between those who argue that training on copyrighted data is legally permissible (relying on fair use or TDM exceptions) and those who argue it is ethically indefensible (because it uses creative works without consent) has often been framed as a binary conflict with no middle ground. The paper's distinction between Tier 2 (downloadable but not openly licensed) and Tier 3 (verified openly licensed) datasets provides a more nuanced vocabulary: it acknowledges that Tier 2 datasets may be legally defensible in some jurisdictions but are not "open" in the sense that matters for risk-averse downstream users, while Tier 3 datasets represent a higher bar that satisfies both legal and ethical requirements for consent-based data use. This does not resolve the underlying disagreement, but it creates a framework where different positions can be located relative to each other rather than treated as incommensurable.

The paper also shifts the research agenda in several concrete ways. Prior to this work, a PhD student interested in training data might focus on improving filtering heuristics, developing better deduplication methods, or optimizing data mixtures for downstream performance. This paper makes newly visible an entirely different set of research problems: automated license verification at web scale (how to determine which asset on a page a license applies to), post-OCR correction for public domain texts (as a domain-specific ML problem with unique error modes), cross-jurisdictional public domain determination (as a legal informatics challenge), and metadata standardization for preference signaling (as a socio-technical coordination problem). These are not incremental improvements to existing ML pipelines; they are foundational problems that must be solved for the open dataset vision to be realized, and they require skills (legal reasoning, library science, archival practice) that are not typically part of ML training. The paper implicitly argues that these problems are as intellectually significant and practically consequential as improving perplexity on language modeling benchmarks—a position that, if adopted, would change what kinds of work are considered "core" ML research.

Finally, the paper provides a strategic rebuttal to a claim that has shaped the industry: that training competitive models without copyrighted material is impossible. This claim, advanced by major AI companies as a justification for their data practices, has had a chilling effect on the open dataset ecosystem—if it is true, then building openly licensed datasets is a noble but ultimately futile exercise. The paper does not disprove this claim (it presents no model trained on Common Pile or Common Corpus), but it does the preparatory work for a disproof by demonstrating that the assembly of a large, diverse, openly licensed corpus is technically feasible. If subsequent work demonstrates that models trained on such corpora achieve competitive performance, the strategic landscape shifts dramatically: companies can no longer claim necessity as a defense for using unlicensed data, and the open dataset approach becomes a viable path for organizations that cannot or will not accept the legal risk of training on unverified web data. The paper's contribution is to make this empirical test possible by building the dataset infrastructure, even if it does not perform the test itself.

Follow-Up Research This Work Enables

A systematic validation study of web-scale license identification accuracy. The paper acknowledges that the end-to-end accuracy of the Common Pile Common Crawl CC license detection pipeline is unvalidated—"We are currently in the process of carrying out a detailed study." A rigorous follow-up would sample, say, 2,000 pages from the 259 million pages in the CC-filtered subset, stratified by license type (CC-BY, CC-BY-SA, CC0, etc.) and domain category, have multiple legal experts independently classify each page's licensing status (with inter-annotator agreement metrics), and report precision, recall, and F1 for the automated pipeline broken down by license type and by the specific failure mode identified in the paper—cases where a site-level CC declaration applies only to an embedded asset, not the page content. The study should also report false negative rates by sampling pages from Common Crawl that were NOT included by the pipeline but contain permissive license signals (to estimate how much openly licensed content is being missed). This would transform the pipeline from a plausibly-useful heuristic to a validated tool with quantified error rates, enabling downstream users to assess legal risk exposure concretely. Given the paper's identification of the asset-vs-page license ambiguity as the fundamental limitation, a negative result—showing that false positive rates are unacceptably high for the largest license categories—would redirect effort toward more sophisticated per-asset license identification methods, potentially using visual layout analysis combined with HTML structure.

Training and evaluating a model on Common Pile compared to a model trained on The Pile, matched for compute and architecture. This is the most obvious and most necessary follow-up, addressing the paper's central unsubstantiated claim. A strong experiment would train two models with identical architecture (say, a 7B parameter decoder-only transformer), identical training recipe (learning rate schedule, batch size, sequence length), and identical compute budget, differing only in training data: one on the completed Common Pile (openly licensed), one on the original Pile (which contains unlicensed copyrighted material). Evaluation would cover standard benchmarks (MMLU, HellaSwag, ARC, GSM8K, HumanEval) plus a targeted set of tasks that might differentially suffer from the removal of copyrighted content—book summarization, literary analysis, domain-specific question answering in fields where the training corpus composition differs substantially. The critical measurement is the performance gap at matched compute. If the gap is small or nonexistent, the paper's strategic claim is validated and the open dataset approach becomes empirically grounded. If the gap is large, it quantifies the "openness tax"—the performance cost of licensing purity—and enables an informed societal conversation about whether that cost is worth paying. A nuanced version would also measure performance as a function of model size, since scaling laws may interact with dataset composition (the gap might be larger at smaller scales and narrow with scale, or vice versa).

Cross-jurisdictional public domain determination as a legal informatics benchmark. The paper's copyright renewal verification pipeline (Figure 3 in Appendix A) operates entirely within US copyright law. A natural extension is to build analogous pipelines for other jurisdictions—EU countries where public domain entry is based on author death date plus 70 years, countries with different rules for government works, jurisdictions with complex post-colonial copyright transitions—and to evaluate how much additional text becomes available. The research contribution would be both practical (more data) and methodological: the development of a standardized evaluation framework for automated public domain determination, with ground-truth labels produced by legal experts for a sample of works from each jurisdiction. This would enable systematic comparison of different approaches (rule-based vs. LLM-assisted, jurisdiction-specific vs. general) and the measurement of how accuracy degrades as pipelines are adapted across legal systems. The paper's existing pipeline—with its LLM-based metadata extraction (96.58% title accuracy, 97.20% registration number accuracy) and its probabilistic error independence analysis (0.09% false positive rate)—provides a strong baseline and evaluation methodology that can be replicated and extended. A negative finding—that accuracy drops substantially when moving beyond US copyright law—would indicate that open dataset construction is more jurisdictionally constrained than the paper implies.

Post-OCR correction models specialized for public domain texts, evaluated on downstream LLM training quality. Both Common Pile and Common Corpus identify PDF OCR quality as a major bottleneck. A targeted ML contribution would train post-OCR correction models specifically on the error distribution found in public domain book digitizations—which differs from generic OCR errors because of historical typography, degraded page quality in older works, and the idiosyncrasies of specific scanning technologies. The evaluation should not be limited to character error rate or word error rate on held-out text; it should measure the impact on downstream LLM training quality. The experimental design: take a set of public domain books with both automatically OCR'd text and manually corrected gold-standard text, train small language models on each version, and measure the performance difference on relevant benchmarks. This establishes the "cost" of uncorrected OCR in terms of model quality degradation, which directly informs investment decisions about whether to fund post-OCR correction efforts. If the performance degradation from uncorrected OCR is small (say, less than 1% on standard benchmarks), then OCR correction should be deprioritized relative to other bottlenecks like license verification or digitization access. If it is large, it justifies the development of specialized correction pipelines.

A field study of community-based data governance for an openly licensed dataset. The paper advocates for community governance but acknowledges that Pleias' envisioned community "does not yet exist" and that Common Pile has no governance mechanism beyond immutability. A concrete follow-up would partner with a specific affected community—for example, a language community whose texts are being included in an open dataset, or a creative professional organization whose members' public domain works are being digitized—to co-design and implement a governance mechanism over a 12-month period, and then publish an ethnographic or mixed-methods evaluation of what worked, what failed, and what was unexpectedly difficult. The study would operationalize the paper's abstract governance principles into testable propositions: Are community members willing to invest time in dataset governance? What incentives or compensation structures are necessary? Do opt-out mechanisms actually get used, and by whom? Does community involvement change what data is included or excluded in ways that measurably affect dataset diversity? This research is essential because the paper's governance recommendations are entirely unvalidated—no evidence is presented that any of the proposed mechanisms (data trusts, community-based governance, preference signaling infrastructure) produce better outcomes than simpler approaches (fixed datasets with no governance, or centralized decision-making by dataset builders). A negative result—showing that community governance imposes substantial coordination costs without changing outcomes—would redirect effort toward lighter-weight governance mechanisms.

Quantifying the "access paradox": what fraction of public domain works have accessible, machine-readable digitizations? The Common Pile team identified approximately 500,000 books believed to be in the public domain but noted "it is currently unclear how many have accessible digitizations." A systematic study would take a random sample of, say, 5,000 titles from the identified public domain list and attempt to locate machine-readable digitized copies through all available channels—HathiTrust, Internet Archive, Google Books, library catalogs, national digitization programs—documenting for each title whether a digital copy exists, what format it is in (plain text, PDF with OCR, PDF without OCR, image-only), what access restrictions apply (open access, institutional subscription, contractual restrictions, no bulk download), and what license or terms of use are asserted (if any). The output would be an estimate of the "accessible fraction" with confidence intervals, broken down by publication decade, genre, and country of publication. This would quantify the gap between legal permissibility and practical availability, providing the evidentiary basis for policy advocacy around digitization funding. If, as the paper suspects, a large fraction of public domain books are practically inaccessible, the study would also identify which access barriers are most common (never digitized? digitized but restricted? digitized but in non-machine-readable format?) and thus where policy intervention would be most impactful.

Practical Applications and Downstream Use Cases

Risk-mitigated training for startups and academic labs with limited legal resources. The paper's central practical value proposition is enabling organizations that cannot afford the legal uncertainty of training on unverified web data to build LLMs anyway. A startup with 10 employees and no in-house counsel currently faces a difficult choice: use publicly available datasets like the original Pile or Dolma (accepting unknown legal risk), license proprietary data (expensive and often unavailable to small entities), or not build a model at all (ceding the market to incumbents). Common Pile, once completed and validated, provides a fourth option: train on a dataset where every component has been individually verified as openly licensed, reducing (though not eliminating) copyright liability exposure. The practical benefit is not merely legal—it is ecosystem competitive. If open datasets enable 50 startups to train models where only 5 could previously afford the legal risk, the market becomes more diverse and innovative. The paper's case studies provide enough implementation detail (component-by-component source descriptions, processing pipelines, license verification methods) that a new organization could replicate the approach for their own domain-specific dataset, adapting the Common Pile methodology rather than starting from scratch. The documentation of what NOT to do—the CC license coverage ambiguity, the access paradox, the synthetic data caveat—is arguably as valuable as the positive recommendations, since it helps new builders avoid known failure modes that took the Common Pile and Common Corpus teams substantial effort to discover.

Public-sector AI procurement with verifiable data provenance requirements. Governments and public institutions increasingly face mandates to ensure that AI systems they deploy meet transparency and accountability standards. The EU AI Act's training data transparency requirements for general-purpose AI, the French public sector's interest in "local technological sovereignty" cited in the Common Corpus case study, and similar emerging regulations in other jurisdictions create demand for models whose training data provenance can be audited. The paper's three-tier openness taxonomy provides a framework for procurement specifications: a government agency could require that any LLM it purchases be trained on a Tier 3 (openly licensed and verified) dataset, or at minimum on a Tier 1 (replicable) dataset with documented processing. Common Pile and Common Corpus serve as existence proofs that such datasets can be built, and the paper's detailed component documentation (source-by-source license verification methods, processing decisions, metadata preservation practices) provides a template for what adequate documentation looks like—essentially, a standard that can be referenced in contracts and regulatory guidance. The practical benefit is that public institutions gain leverage: rather than accepting vendor claims about data practices, they can point to a concrete, implemented standard and require comparable transparency.

Data commons infrastructure for community-driven AI development. The paper envisions—particularly through the Pleias case study—a future where communities, cultural institutions, and domain experts collectively maintain and improve open training datasets, analogous to how Wikipedia's community maintains a shared knowledge resource. The practical application is not a single dataset but a model for sustained collaborative data curation. Libraries digitizing public domain collections, scientific societies maintaining open-access publication corpora, language communities building resources for underrepresented languages, and open-source AI developers all need training data but individually lack the resources to build comprehensive datasets. The paper's emphasis on shared standards (SPDX identifiers for licenses, Croissant for metadata, ISCC for content identification, RAIL for terms of use modularization) makes it possible for these diverse actors to contribute data that interoperates—a library's digitized public domain books, formatted according to shared metadata standards, can be incorporated into Common Pile or a similar dataset without ad-hoc integration work. The practical benefit is reducing the coordination cost of data pooling: if every organization follows compatible metadata and documentation practices, the composability of open data increases dramatically, enabling a "Lego block" approach to dataset construction where components from different sources can be combined with minimal friction.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Training Data Transparency Is Collapsing Just When We Need It Most

Why This Problem Matters: Four Distinct Harms of Dataset Opacity

Prior Approaches and Where They Fall Short

How This Paper Positions Itself

3. Technical Approach

3.1 Reader orientation

3.2 Big-picture architecture (diagram in words)

3.3 Roadmap for the deep dive

3.4 Detailed, sentence-based technical breakdown

The Three-Tier Openness Taxonomy

Source Identification and License Verification

Content Acquisition and Digitization Challenges

Data Processing and Quality Filtering

Metadata Preservation and Preference Encoding

Data Governance Models

Release Practices and Terms of Use

4. Key Insights and Innovations

Innovation 1: A Ground-Level Systems View of Open Dataset Construction as a Socio-Technical Coordination Problem

Innovation 2: The Tiered Openness Taxonomy as a Diagnostic Framework for Legal Risk

Innovation 3: Documenting the "Access Paradox" and Making It a First-Class Policy Concern

Innovation 4: The Governance-Transparency Tension as a Design Constraint, Not an Oversight

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

Source Identification and License Verification

Content Acquisition

Dataset Scale and Composition

Ablation Studies and Robustness Checks

Critical Assessment

Claim 1: Building openly licensed training datasets at meaningful scale is technically feasible

Claim 2: The primary bottlenecks are legal, technical, and governance challenges that require coordinated interdisciplinary effort

Claim 3: The three-tier openness taxonomy resolves confusion and enables better decision-making

Claim 4: Current opt-out mechanisms are crude and risk shrinking the open web

Claim 5: Sustainable funding models for open datasets are needed but do not yet exist

Structural Limitations

6. Limitations and Trade-offs

The Unvalidated Core Premise: No Evidence That These Datasets Produce Competitive Models

License Verification Accuracy Is Unknown for the Largest Data Component

The Access Paradox Is Documented but Has No Operational Solution

Governance Mechanisms Are Aspirational Rather Than Implemented

Cost, Labor, and Sustainability Are Not Quantified

Linguistic, Cultural, and Geographic Diversity Remain Aspirational

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining

People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text