OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text

ArXiv: 2310.06786

🎯 Pitch

A 1.4B-parameter model trained on just 14.7B tokens of carefully filtered mathematical web pages matches the performance of one trained on over 20 times as much general text, demonstrating that per-token quality trumps quantity for mathematical reasoning. This dataset, OpenWebMath, provides the first open reproduction of the proprietary data behind Minerva, using a novel pipeline that preserves LaTeX from raw HTML to capture the dense logical structure missing from standard web crawls.

1. Executive Summary

This paper introduces OpenWebMath, an open dataset of 14.7B tokens of high-quality mathematical web text extracted from Common Crawl, designed to serve as an open alternative to the proprietary Math Web Pages dataset used to train Minerva. The core contribution is a carefully documented pipeline for extracting text and LaTeX content from HTML documents—including a novel MathJax delimiter-based extraction method that handles the diverse ways mathematics is encoded on the web—combined with multi-stage filtering (language identification, a MathScore classifier trained to detect mathematical documents from surrounding text alone, perplexity-based quality filtering) and SimHash deduplication. In small-scale experiments training 1.4B parameter models, OpenWebMath yields models that outperform those trained on over 20× the amount of general language data from The Pile on mathematical benchmarks, and a 50/50 mixture with ProofPile performs best overall, establishing that per-token mathematical quality matters more than raw token quantity for reasoning benchmarks—though the paper is careful to characterize this as a dataset release rather than a claim about optimal pretraining recipes.

2. Context and Motivation

The Core Gap: Mathematical Web Data for LLMs Is Proprietary and Unavailable

The central problem this paper addresses is deceptively simple: there is no publicly available large-scale dataset of high-quality mathematical web text suitable for training language models. While this may sound like a narrow infrastructure gap, it is in fact a significant barrier to progress in a burgeoning area of LLM research. The paper is motivated by a specific finding from the literature that has not been reproducible in the open research community: the observation that training on carefully curated mathematical web pages can dramatically improve quantitative reasoning capabilities.

The landmark result that sets up this paper is the Minerva model (Lewkowycz et al., 2022). Minerva was created by fine-tuning PaLM (a proprietary 540B parameter model) on what the authors call the Math Web Pages dataset—17.5B tokens of mathematical content sourced from arXiv and the web, processed to preserve LaTeX notation. Minerva achieved state-of-the-art results on quantitative reasoning benchmarks like MATH (Hendrycks et al., 2021) and MMLU-STEM, demonstrating that domain-specific pretraining on high-quality technical tokens could unlock reasoning capabilities that general web text alone could not provide. This was a conceptually important result because it suggested that the nature of the training data—not just its scale—matters critically for reasoning tasks. Mathematical text contains dense logical structure, formal notation, and step-by-step derivations that are qualitatively different from the conversational, narrative, or encyclopedic text that dominates general web corpora. Minerva's success implied that exposing models to this structure during training might help them internalize patterns of deductive reasoning.

However, the Minerva paper did not release either the model or the Math Web Pages dataset. The current state of open-source mathematical reasoning models reflects this gap acutely. Open-source models like LLaMA (Touvron et al., 2023a;b;c), Pythia (Biderman et al., 2023), and OpenLLaMA (Geng & Liu, 2023)—while impressive in many respects—lag significantly behind Minerva's reported performance on quantitative reasoning benchmarks. The authors of OpenWebMath argue that this is not merely a model scale issue but a data availability issue:

"Minerva and the datasets used for its training were not released publicly and the current capabilities of open-source models in quantitative reasoning lags behind. We believe that there are important research directions that can only be enabled through open-access to such models and datasets" (Section 1).

The key phrase here is "important research directions that can only be enabled through open-access." The paper is not merely arguing for the practical utility of having such a dataset—it's arguing that certain fundamental research questions are impossible to investigate without it.

Why This Gap Matters: Research Questions That Require Open Data

The paper enumerates specific research directions that are blocked by the absence of an open mathematical web dataset (Section 1):

Memorization and generalization studies. There is growing concern in the LLM community about whether models truly "reason" about mathematics or merely memorize problem-solution pairs seen during training. Investigating this question requires (a) knowing exactly what data the model was trained on and (b) being able to create controlled training sets with known levels of contamination. Proprietary datasets make both impossible. An open dataset like OpenWebMath would allow researchers to carefully audit what mathematical content models have seen, construct counterfactual training runs that exclude specific problem types, and measure whether models generalize to unseen problem structures or simply recall memorized solutions.

Reinforcement learning for mathematical reasoning. Several promising directions involve using reinforcement learning to improve mathematical reasoning—for example, training process reward models (PRMs) that score intermediate reasoning steps (Lightman et al., 2023), or using RL to optimize the model's own chain-of-thought trajectories. These methods require not just a capable base model but also access to the training data distribution so that researchers can understand the model's initial capabilities, generate appropriate training rollouts, and avoid distribution shift. When the base model is trained on proprietary data, the research community can only treat it as a black box, severely limiting the types of RL experiments that are possible.

Development of new reasoning benchmarks. The creation of challenging reasoning benchmarks typically requires understanding what capabilities current models lack. But if the training data is unknown, benchmark creators cannot control for data contamination or design problems that specifically probe generalization. OpenWebMath could serve as a "standard" mathematical pretraining corpus against which new benchmarks are calibrated—researchers could train models on known data distributions and then test for genuine out-of-distribution reasoning.

Advancing the reasoning capabilities of language models more broadly. The paper positions mathematical reasoning as a testbed for reasoning in general. Mathematics provides clean correctness signals (answers are unambiguous), requires multi-step logical deduction, and admits of varying difficulty levels. An open mathematical pretraining dataset would lower the barrier to entry for researchers wanting to work on reasoning, enabling the kind of rapid iteration and controlled experimentation that has driven progress in other areas of LLM research (e.g., instruction tuning, RLHF) where open datasets are available.

The paper doesn't frame this merely as a "release the dataset" argument. It's making a stronger claim: the closed nature of existing mathematical training data is a structural obstacle to scientific progress in understanding LLM reasoning, not just a practical inconvenience. This elevates the dataset release from an engineering contribution to a research-enabling contribution.

Prior Mathematical Datasets: What Exists and Why It's Insufficient

The paper situates OpenWebMath within an existing landscape of mathematical datasets (Section 2.1), each of which falls short in specific ways:

AMPS (Hendrycks et al., 2021): Contains over 100,000 Khan Academy problems with solutions and over 5 million synthetically generated problems using Mathematica scripts (~23GB total). The limitation is scope: AMPS consists almost entirely of problems and solutions—it's a collection of exercises, not a broad corpus of mathematical exposition. Training on AMPS teaches a model to solve specific problem types but doesn't expose it to mathematical definitions, theorems, proofs, or expository text that might build deeper conceptual understanding.

NaturalProofs (Welleck et al., 2021): A curated collection of 32,000 theorem statements and proofs, 14,000 definitions, and related content from ProofWiki, the Stacks project, and textbooks. This is high-quality but very small by LLM pretraining standards—suitable for fine-tuning but not for imparting broad mathematical knowledge during pretraining. It also skews heavily toward formal proof content rather than the informal mathematical reasoning that dominates most benchmarks.

Proof-Pile (Azerbayev et al., 2023): The closest existing open alternative to OpenWebMath, containing over 14.5GB of mathematical text from arXiv, Stack Exchange, ProofWiki, Wikipedia, openly licensed books, and the MATH dataset. This is a significant resource and the paper uses it as a comparison point in experiments. However, Proof-Pile is sourced from a limited set of known, curated domains—it does not include the "long tail" of mathematical content scattered across the broader web in forum posts, blog entries, educational websites, and reference pages. The paper explicitly notes (Section 4) that "there is very little overlap between the two datasets," indicating that OpenWebMath captures different and complementary sources of mathematical text.

Proprietary datasets (WebMath, MathMix, Math Web Pages): Several companies have built large-scale mathematical pretraining corpora. WebMath (Polu & Sutskever, 2020) contains 35B tokens from GitHub, arXiv, and Math StackExchange. MathMix (Lightman et al., 2023) contains roughly 1B high-quality mathematical tokens of both natural and synthetic data, used to fine-tune GPT-4. The Math Web Pages dataset used for Minerva (Lewkowycz et al., 2022) contains 17.5B tokens from web pages containing LaTeX. None of these are publicly available. They establish that large-scale mathematical web data is valuable, but they don't enable the broader research community to build on that finding.

The fundamental gap across all prior work is the absence of a large-scale, openly accessible, web-sourced mathematical corpus that preserves mathematical notation. Proof-Pile comes closest but is restricted to curated domains rather than the open web. The proprietary datasets demonstrate what's possible but are inaccessible. OpenWebMath positions itself as filling exactly this gap: providing a web-sourced dataset of comparable scale to Minerva's Math Web Pages (14.7B vs. 17.5B tokens) with the explicit goal of enabling the kind of research that proprietary datasets preclude.

The Technical Challenge: Why Extracting Mathematical Web Text Is Harder Than It Sounds

A reader unfamiliar with web data processing might assume that creating a mathematical web corpus is simply a matter of filtering Common Crawl for pages that mention mathematical terms. The paper makes clear that the actual challenge is far more subtle. Understanding this challenge is essential to appreciating the contribution, because OpenWebMath's pipeline is not just a routine data processing effort—it addresses genuinely difficult technical problems that prior open datasets had not solved.

The LaTeX extraction problem. Mathematical notation on the web is predominantly encoded using LaTeX, and LaTeX appears in HTML documents in many different forms (Section 3.4, Figure 3). The paper identifies at least seven distinct encoding methods: equation and align environments, the alttext of elements with special classes, image URLs containing encoded equations (e.g., from latex.codecogs.com), WordPress plugins, <math> tags with annotation blocks, MathML content (which requires conversion to LaTeX via stylesheets), and MathJax-delimited inline text. The relative frequencies in Table 6 show that MathJax with delimiters dominates: 50.27% of documents containing math use inline MathJax delimiters, and 23.37% use display MathJax delimiters. This is the central technical challenge.

Here's why: MathJax uses user-configurable delimiters—by default, $...$ for inline math and $$...$$ for display math, but website owners can customize these to any pair of strings. The Common Crawl HTML snapshots do not execute JavaScript, so the rendered MathJax tags that would make equations easy to identify are not present. Instead, the raw HTML contains only the original delimiter-enclosed LaTeX, and most dollar signs on the web are not LaTeX delimiters—they're currency symbols, prices, or other financial notation. The paper describes the heuristic approach they developed (Section 3.4, LaTeX Extraction, and Appendix B):

1. Detect whether the page imports the MathJax script.
2. Search for calls to the MathJax configuration function to extract the user's chosen delimiters.
3. Add these to an extensive default delimiter list.
4. Detect common LaTeX commands between dollar signs to confirm they're math.
5. Use the MathScore classifier (described below) to determine whether the page context is mathematical, and if so, treat dollar signs as LaTeX.

This is a multi-layered heuristic because no single rule works. The paper notes that "the MathJax javascript code may be nested inside of another import or named differently depending on the website" (Appendix B), making detection unreliable. The MathScore classifier serves as a fallback for cases where explicit MathJax detection fails.

Comparison with Minerva's approach. The paper explicitly speculates (Appendix B) that Minerva likely had an easier time with this problem because they had access to HTML documents after JavaScript execution—within Google's infrastructure, pages could be rendered fully, converting MathJax delimiters into explicit HTML tags that are trivial to detect. This is a significant advantage of proprietary datasets that open efforts cannot replicate: running a full browser engine on billions of web pages to execute JavaScript is computationally prohibitive for academic-scale projects. OpenWebMath's LaTeX extraction pipeline is thus a creative workaround for a constraint that proprietary efforts didn't face.

The boilerplate removal problem. Web pages contain vast amounts of non-content text: navigation menus, advertisements, comment sections, related-article links, and site footers. This boilerplate is particularly problematic for mathematical documents because it can dilute the signal from mathematical content. Even worse, boilerplate removal tools that work well on general web pages may aggressively remove mathematical content because it looks structurally unusual (equations have different line lengths, contain special characters, and are interspersed with normal text in ways that confuse standard text extraction heuristics). The paper found that jusText (used in The Pile) "removed boilerplate too aggressively, leading to sections containing math to be discarded" (Appendix B).

The solution—building on Resiliparse (Bevendorff et al., 2018; 2021), a Cython-based library with "balanced boilerplate removal"—is unusual. Most comparable datasets use WET files (pre-extracted plain text from Common Crawl) or standard libraries like jusText. Resiliparse is chosen primarily for speed (Table 5 shows it's roughly 8× faster than jusText and 16× faster than Trafilatura on a 1K document benchmark) and for its customizable DOM processing pipeline that allows step-by-step intervention (removing invisible elements, annotating code blocks, handling edge cases class-by-class).

The Interplay Between Extraction and Filtering: A Subtle but Critical Point

The paper makes an observation in Appendix C that reveals a sophisticated understanding of data pipeline design that is easy to miss on a casual reading:

"if a text extractor fails to extract the main text, gets the formatting wrong, or includes too much boilerplate in the extraction, then both the classification and perplexity filters can filter out such examples. This can lead to subtle biases in the dataset, where specific poorly-extracted websites are excluded entirely even though they do contain high quality content."

This is important because it exposes a hidden feedback loop in data pipelines. When researchers design a pipeline with text extraction followed by quality filtering, they typically test whether the final output looks good. If it does, they assume the extraction step is working. But what might actually be happening is that the extraction step is failing silently on certain types of documents, and the quality filter is simply discarding those documents—leading to a dataset that looks clean but systematically excludes certain formats, domains, or content types. In the mathematical domain specifically, failure to properly handle LaTeX can cause extracted text to appear garbled (raw LaTeX commands mixed with text, equations broken across lines), which drives up perplexity scores, which causes the document to be filtered out. The result: a dataset that appears to contain mathematical content but is actually biased toward sources that encode math in extraction-friendly ways, potentially excluding entire categories of valuable content (e.g., certain forum platforms, older content, or niche technical blogs with non-standard formatting).

OpenWebMath's approach—tuning the text extraction pipeline on a diverse set of documents before applying filtering—aims to minimize this bias by ensuring that as many document types as possible survive the extraction step with their mathematical content intact.

The MathScore Classifier: A Clever Solution to a Subtle Problem

The MathScore classifier (Section 3.5.1, Figure 4) deserves special attention because it represents a non-obvious solution to a recurring problem in domain-specific data filtering. The challenge is: how do you identify whether a web page contains mathematical content when the mathematical notation itself might not have been successfully extracted?

The paper's approach is to train a classifier on the surrounding text alone, without LaTeX. The procedure is:

1. From an early version of the pipeline, gather documents that do contain extractable LaTeX (specifically, documents containing one of the top-100 most common LaTeX commands).
2. Remove all LaTeX code from those documents.
3. Train a FastText classifier to predict whether the original (pre-removal) document contained common LaTeX commands based only on the non-LaTeX text.

The trained classifier therefore learns to recognize the linguistic patterns that co-occur with mathematical content—words like "theorem," "proof," "definition," "integral," "equation," "derivative," as well as more subtle stylistic markers of mathematical writing (e.g., the presence of "let" statements, the phrase "we have," or the structure of question-and-answer pairs). This means it can identify pages as mathematical even when the equations themselves are encoded in ways the pipeline failed to extract (e.g., as images, in non-standard formats, or within JavaScript that wasn't executed). Documents with a MathScore below 0.2 are judged "very unlikely to contain useful mathematical content" based on manual inspection.

This is an example of solving a representation learning problem rather than a rule-based filtering problem. Instead of trying to enumerate all possible ways math could be encoded (which would be an endless game of whack-a-mole), the classifier learns the textual context that surrounds mathematics, which is a more robust signal and generalizes to encoding formats the pipeline designers never encountered.

How OpenWebMath Positions Itself

OpenWebMath is not presented as a novel machine learning method but as an infrastructure contribution that enables novel methods to be developed. The paper explicitly draws the connection to Minerva's Math Web Pages dataset (Section 1):

"In our work, we produce an open alternative to the Math Web Pages dataset used to train Minerva (Lewkowycz et al., 2022)."

The positioning is careful: OpenWebMath is not claiming to be identical to Minerva's dataset (the extraction pipelines differ, the filtering choices differ, and the HTML sources likely differ due to the JavaScript execution issue discussed above). Rather, it claims to serve the same role in the research ecosystem—a large-scale, LLM-ready corpus of mathematical web text—while being fully open and documented. The extensive pipeline documentation (Section 3) and open-source code are essential to this positioning: they make the dataset reproducible, auditable, and adaptable, enabling the kinds of research (on memorization, contamination, and generalization) that proprietary datasets preclude.

The small-scale training experiments (Section 4, Tables 1–2) serve a specific rhetorical purpose: they are not intended to establish new state-of-the-art results but to validate that the dataset is useful. The finding that 14.7B tokens of OpenWebMath outperforms 300B tokens of The Pile (Pythia 1.4B) on mathematical perplexity benchmarks—despite being 20× smaller—is presented as evidence that the filtering pipeline succeeded in selecting genuinely high-quality mathematical content. This is not a claim about optimal pretraining recipes; it's a "smoke test" confirming that the dataset contains the kind of mathematical signal that improves reasoning, analogous to what Minerva demonstrated with proprietary data.

3. Technical Approach

3.1 Reader Orientation

OpenWebMath is a data processing pipeline—not a machine learning model—that extracts, filters, and deduplicates mathematical web pages from Common Crawl to produce a clean 14.7B-token dataset of LaTeX-preserving mathematical text. The problem it solves is that no open dataset of mathematical web text exists at a scale suitable for LLM pretraining (the closest analogue, Minerva's Math Web Pages, is proprietary), and the solution is an extensively documented, open-source pipeline that addresses the core technical challenge of the domain: faithfully extracting mathematical notation from HTML pages where it appears in at least seven different encoding formats, none of which are handled by standard text extraction tools.

3.2 Big-Picture Architecture (Diagram in Words)

The pipeline has five sequential stages, each narrowing the dataset from the vastness of Common Crawl down to a focused collection of high-quality mathematical documents:

1. Prefilter — rapidly scans all ~237 billion HTML documents in Common Crawl using lightweight heuristics (keyword matching, LaTeX symbol detection, and a fast MathScore classifier) to identify candidate mathematical pages, reducing the workload by eliminating the vast majority of non-mathematical content before expensive processing begins.

2. Text Extraction — takes the surviving HTML documents and extracts the main content text while preserving mathematical notation. This involves: (a) detecting how each page encodes LaTeX (MathJax delimiters, image URLs, <math> tags, MathML, etc.), (b) extracting those equations into inline LaTeX notation, (c) processing the DOM tree to remove navigation, advertising, and other boilerplate elements, (d) extracting plain text from the cleaned DOM using Resiliparse, and (e) applying line-level boilerplate removal.

3. Filtering — applies three sequential quality gates: (a) FastText language identification to keep only English documents, (b) the MathScore classifier (a model trained to predict the presence of LaTeX from surrounding text alone) to keep mathematical documents, and (c) a KenLM perplexity filter (trained on ProofPile, threshold = 15,000) to remove low-quality text.

4. Deduplication — applies SimHash near-deduplication (threshold = 0.7) to remove duplicate documents, which are pervasive in Common Crawl due to its periodic crawling of the same pages.

5. Manual Inspection — human review of the top domains by document count and character count, plus the longest documents, to blacklist non-mathematical or low-quality domains that survived automated filtering.

The output is 6.3 million documents containing 14.7B tokens.

3.3 Roadmap for the Deep Dive

• First, the prefiltering stage — because it determines which documents survive to the expensive processing steps and shapes everything downstream. Understanding its design (hierarchical, recall-optimized) explains why the pipeline can process Common Crawl at all.
• Second, the LaTeX extraction pipeline — because it is the central technical innovation of the paper and the hardest problem in mathematical web data processing. This covers the seven encoding formats, the MathJax delimiter detection heuristic, and the edge cases that required iterative refinement.
• Third, the DOM processing and text extraction — because once LaTeX is extracted, the remaining text must be separated from boilerplate without accidentally discarding mathematical content. This covers Resiliparse, the DOM preprocessing steps, and the line-level boilerplate removal.
• Fourth, the MathScore classifier — because it serves a dual role in the prefilter (as a quick mathematical detector) and the main filtering stage (as a quality gate), and its training methodology (removing LaTeX from training documents and learning from context alone) is a non-obvious design choice that merits detailed explanation.
• Fifth, the remaining filtering and deduplication stages — language identification, perplexity filtering, SimHash deduplication, and manual inspection — including the specific thresholds, models used, and the rationale for each.
• Sixth, the interplay between extraction and filtering — because the paper explicitly warns that extraction failures can cause systematic biases that are invisible if filtering masks them, a subtle point that is important for anyone building similar pipelines.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data engineering paper whose core idea is that creating a high-quality open mathematical web corpus requires solving a specific extraction problem (recovering LaTeX from raw HTML where JavaScript hasn't executed) that prior open pipelines failed to address, and that once solved, the resulting dataset enables models to achieve mathematical reasoning performance on par with or exceeding what much larger general-domain corpora provide.

Prefiltering: Reducing 237 Billion Documents to ~1 Billion Candidates

The prefiltering stage (Section 3.3) is the computational gatekeeper of the entire pipeline. Common Crawl contains over 237 billion HTML documents, and applying the full extraction pipeline (DOM parsing, LaTeX detection, text extraction, classification) to every document would be prohibitively expensive. The prefilter must therefore be fast enough to process 237 billion documents while maintaining high recall for mathematical content—missing a mathematical page here means it is lost forever from the dataset.

The paper adopts a hierarchical filter stack optimized for recall, meaning each layer is progressively more expensive but more accurate, and documents pass through only as many layers as necessary:

Layer 1: Keyword matching (cheapest, broadest). The system checks the raw HTML for a set of mathematical keywords, following Lewkowycz et al. (2022). The full keyword list is provided in Table 8 and includes: MathJax, mathjax, <math, math-container, katex.min.css, latex.php, codecogs, tex.cgi, class="tex", and class='tex'. These are terms that strongly indicate the presence of mathematical content—MathJax is the dominant JavaScript library for rendering LaTeX on the web, <math> tags indicate MathML content, and latex.php/codecogs/tex.cgi are server-side equation renderers commonly used on technical websites. Any page matching any of these terms passes the prefilter immediately without further checking. The computation involved is simple substring matching, which can be performed at the throughput needed for 237 billion documents.

Layer 2: LaTeX symbol detection (medium cost). If the keyword check fails, the system searches for the presence of common LaTeX symbols. This is done in two steps: first, a simple regular expression checks whether the document contains any backslash command (the \ character followed by letters, which is the universal LaTeX command syntax). If no backslash commands are found, the document is discarded. If backslash commands ARE found, the system searches specifically for the top 100 most-popular LaTeX symbols in the plain text extracted from the HTML. The paper doesn't enumerate these 100 symbols, but typical candidates would include \frac, \sum, \int, \sqrt, \alpha, \beta, \geq, \leq, \cdot, \times, \partial, \infty, etc. If any of these common symbols appear, the document passes to the next stage.

Why top-100 symbols rather than all LaTeX commands? The set of all possible LaTeX commands is enormous (thousands of symbols across hundreds of packages). Searching for all of them would be slow and would produce many false positives from non-mathematical uses of backslash (e.g., Windows file paths, escape sequences in code). By restricting to the 100 most common mathematical commands, the filter achieves both speed and precision: these commands almost never appear in non-mathematical contexts, so their presence is a strong signal.

Layer 3: MathScore classifier (most expensive prefilter, highest precision). If neither keyword matching nor LaTeX symbol detection triggers, the system runs the plain text through the MathScore classifier (described in detail in Section 3.5.1) and keeps documents that exceed a confidence threshold of 0.8. The MathScore classifier is a FastText model trained to predict whether a document contains LaTeX based only on the surrounding text without the LaTeX itself. A score above 0.8 means the classifier is very confident that the document is mathematical even though no LaTeX commands were found. This catches pages where the mathematical content is encoded in ways the previous filters couldn't detect (e.g., as images with descriptive alt text, in non-standard notation, or in formats where equation delimiters failed to match).

Why hierarchical? The paper explicitly states that this stacking "tun[es] these filters and us[es] hierarchical layers of progressively more accurate but more expensive filters" (Section 3.3) to reduce compute by "several times while retaining a high recall of relevant documents." The key insight is that most of the 237 billion documents are unambiguously non-mathematical and can be eliminated by the cheap first layer alone. The more expensive layers only run on the small fraction of documents that survive the earlier stages. This is standard engineering practice (a "filter cascade") but is especially important here because the downstream extraction pipeline is orders of magnitude more expensive per document than any prefiltering step.

Recall optimization. The paper emphasizes that prefilter thresholds were tuned "to ensure that we do not remove too many relevant documents, optimizing for high recall whenever possible" (Section 3.1). This means accepting some false positives (non-mathematical pages that pass the prefilter) to avoid false negatives (mathematical pages that are incorrectly discarded). False positives are handled by later filtering stages; false negatives are irreversible.

LaTeX Extraction: The Central Technical Challenge

The LaTeX extraction pipeline (Section 3.4, Appendix B) is where the paper makes its most significant technical contribution. The fundamental problem is that mathematical notation on the web is encoded in numerous incompatible formats, and Common Crawl's HTML snapshots—which do not execute JavaScript—lack the rendered MathJax tags that would make equation detection straightforward.

The JavaScript execution gap. The paper explicitly identifies a critical difference between their data source and what Minerva likely used (Appendix B):

"We suspect that this is due to a difference between the HTML files available within Google (Lewkowycz et al., 2022) and those available on Common Crawl. The majority of the LaTeX on the internet is written using MathJax, where developers write equations delimited by dollar signs or other delimiters in their HTML pages and then the included javascript code replaces these equations with properly rendered LaTeX equations within the above script tags when the page is loaded. HTML documents on Common Crawl do not include the changes to the HTML that result from running javascript, requiring that we instead extract the LaTeX equations by finding delimiters ourselves."

To understand this: when a web browser loads a page using MathJax, the JavaScript code runs and transforms inline delimited text like $x^2 + y^2 = z^2$ into a rendered equation displayed as formatted mathematics, with the original LaTeX stored in clearly marked <script type="math/latex"> tags in the DOM. If you have access to the post-JavaScript DOM, extracting equations is trivial: find all <script type="math/latex"> tags and read their content. But Common Crawl stores the raw HTML before JavaScript execution. In that raw HTML, the equation is just $x^2 + y^2 = z^2$ sitting among regular text, and the vast majority of dollar signs on the web denote US currency, not LaTeX equations.

The seven encoding formats. Through iterative refinement—filtering small portions of Common Crawl for documents containing \frac (one of the most common LaTeX commands) and manually inspecting pages where their extraction code found nothing—the authors discovered at least seven distinct ways that LaTeX is encoded on the web. Table 6 provides the relative frequencies:

Let me walk through each format and how the pipeline handles it:

1. equation and align environments (1.72% + 1.18%): These are LaTeX display environments typically used in blog posts and articles where authors write raw LaTeX in their HTML. They appear as \begin{equation} ... \end{equation} and \begin{align} ... \end{align} blocks. Since these environments use standard LaTeX syntax with explicit begin/end markers, they are straightforward to detect with regular expressions. The pipeline extracts the content between \begin{equation}/\begin{align} and their corresponding \end tags.

2. The alttext of elements with special classes like tex (0.24%): Some websites, particularly older WordPress installations, use plugins that render LaTeX equations as images and store the original LaTeX source in the alt attribute of <img> tags with CSS classes like tex. The pipeline searches for elements where the class attribute contains "tex" and extracts the LaTeX from their alttext property.

3. Images from latex.codecogs.com and similar services (6.96%): This is the second-most-common format after MathJax delimiters, occurring in nearly 7% of mathematical documents. CodeCogs is a service that dynamically generates equation images from LaTeX code embedded in image URLs. A typical CodeCogs URL looks like:

<img src="https://latex.codecogs.com/gif.latex?\frac{d}{dx}f(x)">

The LaTeX code is URL-encoded in the src attribute. The pipeline uses regular expressions to detect URLs from known equation rendering services (CodeCogs, latex.php scripts, tex.cgi scripts, and similar patterns from the keyword list in Table 8) and URL-decodes the LaTeX content.

4. Special WordPress plugins (not separately counted; part of image equations and other categories): WordPress has a variety of math plugins, each with its own encoding conventions. The paper mentions these as a distinct category discovered during iterative refinement but doesn't break out specific statistics. These plugins typically add custom CSS classes or shortcodes that the pipeline detects via regular expressions on class names and tag attributes.

5. <math> tags with <annotation encoding="application/x-tex"> blocks (2.35%): This is the MathML standard way of including LaTeX source alongside rendered MathML. A MathML equation looks like:

<math>
  <semantics>
    <mrow>...</mrow>
    <annotation encoding="application/x-tex">\frac{a}{b}</annotation>
  </semantics>
</math>

The pipeline locates <math> elements, searches for child <annotation> elements with the encoding application/x-tex, and extracts their text content as LaTeX. This is the format Minerva's pipeline primarily relied on (via <script type="math/latex"> and similar), but it accounts for only 2.35% of documents—explaining why a simpler pipeline modeled directly on Lewkowycz et al. (2022) would miss over 97% of the available mathematical content.

6. <math> tags with MathML content (3.28%): Some pages contain MathML equations without the convenience of an annotation block containing LaTeX source. MathML is an XML-based markup language for mathematics, not LaTeX. The pipeline uses an XSLT stylesheet to convert MathML markup into LaTeX notation. This is a lossy conversion in some cases (MathML can represent some things that don't have a clean LaTeX equivalent), but the paper doesn't report specific issues. The presence of MathML without annotation tags likely represents pages where LaTeX source was never available in the HTML—the MathML was generated server-side or directly authored.

7. MathJax equations encoded in the text of the page (50.27% + 23.37%): This is the dominant format, accounting for nearly 74% of all mathematical documents. The challenge is described in detail in Appendix B. MathJax is a JavaScript library that page authors include to render LaTeX equations. When writing their HTML, authors embed LaTeX inside configurable delimiters—by default $...$ for inline math and $$...$$ for display math, but any pair of strings can be configured. When the page loads in a browser, the MathJax JavaScript executes and replaces each delimited expression with a rendered equation. In Common Crawl's raw HTML, the delimiters are present but nothing distinguishes a mathematical dollar sign from a currency dollar sign.

The pipeline's approach to MathJax extraction involves three cascading heuristics:

Heuristic A — Explicit script detection: The system searches the HTML for the word "MathJax" (case-insensitive). If found, it confirms that the page uses MathJax. It then searches for JavaScript code that calls the MathJax configuration function to extract the user's custom delimiters. A typical MathJax configuration looks like:

MathJax.Hub.Config({
  tex2jax: {
    inlineMath: [['$','$'], ['\\(','\\)']],
    displayMath: [['$$','$$'], ['\\[','\\]']]
  }
});

Regular expressions extract the delimiter pairs from these configuration blocks. The extracted delimiters are added to an extensive default list (which includes $...$, $$...$$, \(...\), \[...\], and other common conventions). Content between any matched delimiter pair is treated as LaTeX.

Heuristic B — LaTeX command detection: If no MathJax script is found, the system falls back to detecting common LaTeX commands between dollar signs. If the text between a pair of dollar signs contains recognized LaTeX commands (from the same top-100 list used in prefiltering), those dollar signs are treated as math delimiters. This heuristic works because it's extremely unlikely for a price like "$5.00" to contain \frac or \sum.

Heuristic C — MathScore classifier: As a final fallback, the MathScore classifier determines whether the broader document context is mathematical. If the classifier is confident the page is about mathematics (score > 0.8), all dollar sign pairs are treated as potential LaTeX delimiters. This catches cases where the page uses MathJax but the script tag is nested, renamed, or loaded asynchronously in ways that evade heuristic A.

The delimiter problem in detail. Appendix B explains why dollar sign detection is fundamentally ambiguous:

"The majority of MathJax documents use dollar sign delimiters, but most dollar signs on the web do not delimit LaTeX equations."

Consider a page that contains both "The price is $10.50" and "The equation$x^2 + y^2 = z^2$." Without context, every `$` character is ambiguous. The three heuristics above provide progressively weaker but still useful signals: explicit MathJax configuration (strongest, but rare), LaTeX content between delimiters (strong, but misses equation-free math contexts like simple formulas), and document-level topic (weakest, but catches edge cases). No single heuristic solves the problem; the combination aims for high recall at the cost of some false positives (misinterpreting currency as math), since the filtering stage can later remove documents that were incorrectly flagged.

The iterative refinement methodology. The paper describes how the LaTeX extraction rules were developed (Appendix B):

"We employed an iterative process to refine our extraction rules. First, we filtered shards of Common Crawl for documents that contain the string \frac. Then, we filtered those documents to find those which our extraction code found no extractable LaTeX. Then, we refined our code to include additional sources of math until we were confident that we had reasonable support for all formats of LaTeX in HTML documents."

This is a practical engineering methodology: identify a high-recall signal (\frac appears in almost all mathematical LaTeX), find false negatives (pages with \frac where the pipeline extracted nothing), inspect those pages manually, and add extraction rules for the newly discovered format. The authors report doing this "until we were confident that we had reasonable support for all formats"—an inherently subjective endpoint, but the Table 6 statistics suggest they achieved broad coverage.

DOM Processing and Text Extraction

After LaTeX equations have been extracted and marked, the pipeline processes the HTML Document Object Model (DOM) to isolate the main content from boilerplate. This stage has four sub-stages (Section 3.4):

DOM preprocessing. Before extracting text, the pipeline performs several operations on the DOM tree:

1. Remove invisible elements: Elements that are hidden via CSS (display: none, visibility: hidden, zero dimensions) are removed from the DOM tree before text extraction. These often contain tracking scripts, hidden SEO text, or collapsed navigation menus that would otherwise appear as content.

2. Remove buttons and link clusters: Navigation elements (menus, sidebars, footers full of links) are identified by their HTML structure (lists of <a> tags, elements with button-like CSS classes) and removed. This is done because common text extractors struggle to distinguish content links (e.g., a citation in a paper) from navigation links (e.g., "Home / About / Contact").

3. Annotate code, tables, and headers: These elements are marked with special annotations so that the text extractor can handle them appropriately—preserving code blocks verbatim, maintaining table structure, and keeping header hierarchy.

4. Remove known problematic elements by class or ID: The authors maintain a blacklist of CSS classes and element IDs that are associated with boilerplate across mathematical websites. These were discovered through manual inspection of extraction failures and include things like sidebar, footer, comment-section, related-posts, and ad-related identifiers.

Text extraction via Resiliparse. The processed DOM is then passed to the extract_plain_text(main_content=True) method of the Resiliparse library (Bevendorff et al., 2018). Resiliparse was chosen over alternatives after a performance benchmark on 1,000 Common Crawl documents (Table 5):

Resiliparse is approximately 8× faster than jusText (used in The Pile) and 16× faster than Trafilatura (used in RefinedWeb). But speed alone wasn't the deciding factor—the paper tested extraction quality and found that jusText "removed boilerplate too aggressively, leading to sections containing math to be discarded" (Appendix B). Trafilatura had "poor efficiency" but the paper doesn't report whether it also had quality issues. Resiliparse provided a balance of speed and "balanced boilerplate removal" that preserved mathematical content.

Resiliparse's main_content=True mode uses heuristics to identify the primary content area of a web page based on text density, HTML structure, and element positioning. It's designed to remove headers, footers, sidebars, and navigation while preserving the main article body.

Why not use Common Crawl's pre-extracted WET files? Many Common Crawl-based datasets (C4, CCNet) use the WET files provided by Common Crawl, which contain pre-extracted plain text. The paper states:

"The most commonly used option, using WET files extracted by Common Crawl, was not an option since they do not deal with LaTeX correctly and offer no customization."

The WET extraction strips all HTML tags, including MathJax delimiters, script tags, and annotation blocks that contain the LaTeX equations. The resulting text would have LaTeX commands garbled among content, if they survived at all—<annotation encoding="application/x-tex">\frac{a}{b}</annotation> would become just the text "application/x-tex" followed by "fracab" with no delimiters to indicate it's an equation. This is precisely the failure mode the paper identifies as causing "subtle biases" in datasets (Appendix C).

Line-level processing. After Resiliparse extracts the plain text, the pipeline applies its own line-by-line post-processing:

1. Remove boilerplate lines: An iteratively-refined set of boilerplate phrases is matched against each line. Lines consisting entirely of boilerplate (e.g., "Click here to read more," "Share this article," "Posted by," "Copyright ©," navigation text, search prompts) are removed. The paper doesn't enumerate the full phrase list, but notes it was built through iterative refinement—running the pipeline, inspecting output, and adding new boilerplate patterns as they were discovered.

2. Remove empty headers: Headers (lines consisting of header markup like ### or <h1>) that are followed by empty content are removed.

3. Escape non-LaTeX dollar signs: This is the flip side of the MathJax detection problem. After LaTeX extraction, the remaining text still contains dollar signs that are NOT LaTeX delimiters (prices, informal notation, etc.). The pipeline attempts to escape these by identifying dollar signs that don't follow LaTeX delimiters patterns—if a $ appears without a matching closing $, or if no LaTeX commands appear between a $ pair, the dollar signs are escaped (converted to \$) to prevent downstream systems from misinterpreting them.

Randomization for diversity. A notable design choice: the pipeline randomizes certain extraction parameters to add diversity to the dataset (Section 3.4):

"This includes randomizing whether we use a plain text or Markdown format for the documents and randomizing the amount of boilerplate terms required to trigger a line being removed."

Randomizing between plain text and Markdown format means the same webpage could produce different output representations if crawled multiple times or if the pipeline is rerun. Markdown output preserves some structural information (headers, lists, emphasis) while plain text strips all formatting. The randomized boilerplate threshold means some documents retain more peripheral text than others. This diversity is valuable for language model training because it prevents the model from overfitting to a single, consistent output format—models trained on data with variable formatting are more robust to differently-formatted inputs at inference time.

The MathScore Classifier

The MathScore classifier (Section 3.5.1, Figure 4) is used in both the prefiltering stage (Layer 3, threshold = 0.8) and the main filtering stage (threshold = 0.17 for documents with extracted LaTeX, 0.8 for documents without). Its training methodology is non-obvious and clever:

Training data construction. From an early version of the pipeline (before final filtering decisions were made), the authors collected "hundreds of thousands" of extracted documents. Each document was automatically labeled as "mathematical" or "non-mathematical" based on a simple criterion: does the document contain at least one of the top-100 most common LaTeX commands? This is a noisy but high-recall labeling strategy—documents containing \frac, \sum, \int, etc. are almost certainly mathematical, though documents missing these commands might still be mathematical (e.g., a purely prose discussion of mathematics without equations).

The critical preprocessing step. Before training the classifier, all LaTeX code is removed from all documents in the training set—both positive examples (documents that originally contained LaTeX) and negative examples (documents that didn't). This is the key insight. After removal, the positive examples consist only of the English prose surrounding where equations used to be; the negative examples consist of their original prose (which never had equations to begin with). The classifier is trained to predict the original label (did the document contain LaTeX?) from the stripped text.

What the classifier learns. Because LaTeX itself is unavailable to the classifier, it must learn the linguistic patterns that co-occur with mathematical writing. These include:

• Mathematical vocabulary: "theorem," "proof," "definition," "lemma," "integral," "derivative," "equation," "function"
• Stylistic conventions: "Let x be...," "We have...," "It follows that...," "Suppose..."
• Structural patterns: question-answer pairs ("Q: ... A: ..."), step-by-step enumeration ("Step 1: ... Step 2: ..."), formal variable introductions
• Domain vocabulary from fields that use mathematics: "momentum," "Lagrangian," "eigenvalue," "algorithm," "complexity," "thermodynamic"

The classifier "needs to learn the words and phrases most commonly associated with LaTeX content" (Section 3.5.1) without being able to see the LaTeX itself.

Model architecture and training. The classifier is a FastText model (Joulin et al., 2016). FastText is chosen for its speed and efficiency—it can train on hundreds of thousands of documents quickly and produce predictions with minimal latency, making it suitable for use as a prefilter that must process billions of documents. FastText represents documents as bags of character n-grams, which gives it some robustness to out-of-vocabulary words and spelling variations (important for web text).

Threshold calibration. The authors report that "based on manual inspection that content with a score of under 0.2 is very unlikely to contain useful mathematical content" (Section 3.5.1). This manual calibration—inspecting documents at various score levels to find the threshold where mathematical content becomes rare—determined the filtering thresholds. In the main filtering stage, two different thresholds are used:

• Documents with extracted LaTeX: keep if MathScore > 0.17 (a lenient threshold, since the presence of extractable LaTeX already strongly indicates mathematical content)
• Documents without extracted LaTeX: keep if MathScore > 0.8 (a strict threshold, since the only evidence of mathematical content is the surrounding prose)

The asymmetry reflects the different information available: if LaTeX was successfully extracted, we know the document is mathematical, so the classifier threshold is set low to avoid discarding correctly-extracted documents with unusual prose patterns. If no LaTeX was found, the classifier is the only signal, so the threshold is set high to ensure precision.

Dual role in the pipeline. The MathScore classifier serves two distinct functions:

1. In the prefilter (Layer 3): It catches mathematical documents that the keyword and LaTeX symbol filters missed—documents where LaTeX is encoded in truly exotic ways, where the encoding is broken, or where mathematical content is purely prose. The threshold here is 0.8 (strict), prioritizing precision to avoid flooding the expensive extraction stage with false positives.

2. In the main filtering stage: It removes non-mathematical documents that survived the prefilter and went through extraction without yielding LaTeX. The threshold is again 0.8 for documents without extracted LaTeX, maintaining the same precision priority. For documents WITH extracted LaTeX, the threshold drops to 0.17, serving as a minimal sanity check rather than a primary filter.

Language Identification, Perplexity Filtering, and Deduplication

After text extraction and MathScore classification, the pipeline applies three more quality gates (Section 3.5) followed by deduplication (Section 3.6).

Language identification. A FastText language identification model (Joulin et al., 2016) classifies the language of each document. Only documents classified as English are kept. This is not because mathematical content in other languages is lower quality—it's because "our filters pipeline is optimized for English" (Section 3.5). Specifically, the MathScore classifier was trained on English documents, the perplexity model was trained on English mathematical text, and the boilerplate phrase list was developed for English. Documents in other languages would likely be incorrectly filtered by these English-tuned components. The paper acknowledges this limitation in Appendix A: "OpenWebMath is an English-only dataset, which limits its applications for researchers and users who speak other languages."

FastText language identification works by extracting character n-gram features and classifying them into 176 languages using a pretrained model. It is fast and accurate on documents of sufficient length, though very short documents can be misclassified.

Perplexity-based quality filtering. A KenLM language model (Heafield, 2011) trained on ProofPile (Azerbayev et al., 2023) is used to score each document's quality. Specifically:

• Training data for the LM: ProofPile, a curated dataset of mathematical text from arXiv, Stack Exchange, ProofWiki, Wikipedia, openly licensed books, and the MATH dataset. This model captures the distribution of "high-quality mathematical English"—it assigns low perplexity to well-formed mathematical prose and high perplexity to garbled, boilerplate-heavy, or nonsensical text.
• Model type: A 5-gram Kneser-Ney language model implemented in the KenLM library. This is the same approach used by CCNet (Wenzek et al., 2019) and is chosen for speed—KenLM models can score millions of documents quickly, which is essential for a pipeline processing billions of tokens.
• Threshold: Documents with a perplexity score greater than 15,000 are removed.

What does a perplexity of 15,000 mean operationally? Perplexity is the exponential of the average negative log-likelihood per token. A perplexity of 15,000 means that, on average, the model is as uncertain about each next token as if it were choosing uniformly among 15,000 options. For a well-trained mathematical language model, well-formed mathematical text should have perplexity in the low hundreds or even tens. A perplexity of 15,000 indicates text that is extremely unlike the ProofPile distribution—garbled extraction output, pages full of raw code or binary data, massive boilerplate dumps, or completely non-mathematical content that survived the MathScore filter.

Why 15,000? The paper doesn't explain how this specific threshold was chosen, but the methodology is presumably similar to the MathScore calibration: manual inspection of documents at various perplexity levels to find the threshold where quality becomes unacceptable. The high threshold (15,000 is quite lenient) reflects the paper's stated preference for recall over precision: "we prefer having a small percentage of non-mathematical but high quality documents in the dataset rather than removing them and potentially losing relevant mathematical content" (Section 3.1).

SimHash deduplication. Near-duplicate documents are pervasive in Common Crawl because many pages are crawled multiple times across different snapshots, and because content is frequently syndicated across multiple domains. The pipeline uses SimHash (Manku et al., 2007) via the text-dedup library (Mou et al., 2023) with a threshold of 0.7.

SimHash works by:

1. Converting each document into a set of features (typically character n-grams or word n-grams).
2. Hashing each feature into an f-bit fingerprint (typically f = 64).
3. Computing the weighted sum of these fingerprints, where each bit position is weighted by the TF-IDF weight of the corresponding feature.
4. Binarizing the result to produce a single f-bit hash for the document.

Documents with similar SimHash values (Hamming distance below a threshold) are considered near-duplicates. The threshold of 0.7 on a 64-bit SimHash corresponds to documents that are approximately 30% different in their feature sets—close enough to be variants of the same content but allowing for minor differences like timestamps, navigation changes, or different ad insertions.

The paper notes: "We find that this threshold is high enough to remove most duplicate documents even if they have slight differences in their texts" (Section 3.6). This is important because Common Crawl snapshots of the same page can differ in boilerplate (dynamic ads, timestamp updates) while having identical main content. A too-strict threshold would fail to merge these; a too-lenient threshold would merge genuinely different pages.

The deduplication stage processes documents extracted from all Common Crawl snapshots before May 2023.

Manual Inspection and Domain Blacklisting

The final stage of the pipeline involves human review (Section 3.7). The authors manually inspect:

1. Top domains by document count: Domains that contribute the largest number of documents to the dataset. If a domain is clearly non-mathematical or low-quality (e.g., a general-interest blog that happens to use LaTeX once, a site that scrapes and republishes content), it is added to a blacklist.

2. Top domains by character count: Domains contributing the most total characters. This catches sites with very long individual pages (e.g., PDF archives, raw data dumps) that might not rank high by document count.

3. The longest documents in the dataset: Very long documents are suspicious—they might be concatenations of multiple articles, raw code dumps, or extraction failures where boilerplate was incorrectly included as content.

Additional domain filters include:

• Removing user profile pages: Forum platforms often have user profile pages that contain mathematical content (because the user discussed math in comments) but are not themselves useful mathematical exposition.
• Removing abstract-hosting websites: Following Lewkowycz et al. (2022), sites that only host abstracts or summaries of papers rather than full content are removed. An abstract may mention mathematical concepts but doesn't provide the kind of detailed mathematical exposition useful for training.
• Removing search result pages: Pages that are lists of search results for mathematical queries contain snippets of mathematical text but are not coherent mathematical documents.

The blacklist is domain-based, meaning all pages from a blacklisted domain are removed regardless of their individual content. This is a coarse but efficient approach—manual review cannot inspect all 6.3 million documents, but identifying problematic sources eliminates large numbers of low-quality documents at once.

The Interplay Between Extraction and Filtering

Appendix C contains a methodological observation that is important for understanding the design philosophy behind OpenWebMath and for anyone building similar pipelines:

"if a text extractor fails to extract the main text, gets the formatting wrong, or includes too much boilerplate in the extraction, then both the classification and perplexity filters can filter out such examples. This can lead to subtle biases in the dataset, where specific poorly-extracted websites are excluded entirely even though they do contain high quality content."

This insight explains several of the paper's design choices that might otherwise seem arbitrary:

Why custom text extraction instead of using mature, tested libraries? Because the downstream filters (perplexity, MathScore) would silently discard any content that the extractor handled poorly. A dataset built on WET files or jusText extraction would appear high-quality after filtering, but actually be systematically missing documents from domains where those extractors performed poorly. In the mathematical domain, this would disproportionately affect pages with heavy LaTeX usage (since LaTeX confuses standard extractors), which is exactly the content you want to preserve.

Why manual inspection of the pipeline output rather than automated quality metrics? Because automated metrics (perplexity, MathScore) measure conformance to a distribution, not absolute quality. If the extraction pipeline introduces a formatting bias—say, it consistently garbles equations from a particular forum platform—then the filtering stage will remove those documents. The resulting dataset will have high average quality (per the metrics) but will be missing an entire category of mathematical content. Only manual inspection—looking at what's missing as well as what's present—can detect this.

The recommendation for practitioners: "We suggest practitioners tune their text extraction pipeline on a diverse set of documents before applying filtering to avoid this bias." In other words: validate that extraction works correctly across a representative sample of your target content types before you run quality filters, because quality filters will hide extraction failures.

Summary of Design Choices and Their Justifications

• Hierarchical prefiltering (keyword → LaTeX symbol → MathScore): maximizes recall while minimizing computation by eliminating obvious non-mathematical documents with cheap checks before running expensive ones on the remainder.
• Custom LaTeX extraction with seven encoding formats rather than relying on MathML annotation tags alone: proprietary efforts like Minerva likely had access to post-JavaScript DOMs, making tag-based extraction sufficient; Common Crawl's pre-JavaScript HTML requires broader extraction to achieve comparable coverage.
• MathJax delimiter detection via three cascading heuristics (explicit script detection, LaTeX command detection, document-level MathScore): no single heuristic works for all pages, and the cascading design trades off between precision and recall—using strong signals when available and falling back to weaker signals when necessary.
• Resiliparse for text extraction over jusText (too aggressive with mathematical content), Trafilatura (too slow), or WET files (cannot handle LaTeX at all): chosen for balanced boilerplate removal that preserves mathematical text at an acceptable speed (roughly 8× faster than jusText).
• MathScore trained on LaTeX-stripped documents rather than raw documents: enables detection of mathematical content from surrounding prose patterns alone, making the classifier robust to encoding formats that the extraction pipeline fails to handle.
• FastText for both language identification and MathScore rather than more accurate but slower models: the scale of Common Crawl (billions of documents) means inference speed is a binding constraint; FastText provides adequate accuracy at the required throughput.
• KenLM 5-gram perplexity filter trained on ProofPile rather than on general English: the model captures the distribution of high-quality mathematical English specifically, making it sensitive to formatting issues that a general English LM would accept (e.g., garbled LaTeX commands look like nonsense to a mathematical LM but might look like code or foreign language to a general LM).
• SimHash deduplication at threshold 0.7 rather than exact matching or paragraph-level deduplication: document-level near-duplicate detection catches boilerplate-variant copies while being efficient enough to run on the full dataset.
• Manual inspection and domain blacklisting rather than purely automated quality control: catches systematic biases (e.g., entire domains of low-quality content, extraction failures on specific platforms) that automated metrics would miss because they measure conformance to a distribution, not absolute quality.

4. Key Insights and Innovations

Innovation 1: The "Invisible Extraction Bias" Framework for Web Data Pipelines

The paper's most conceptually significant contribution is not the LaTeX extraction pipeline itself—that's an engineering solution—but the diagnostic insight that standard web data pipelines contain a hidden, self-reinforcing bias when applied to domain-specific content. This is articulated in Appendix C, a section that could easily be overlooked but which represents the kind of methodological contribution that changes how researchers think about data pipeline design.

The core observation: when a text extraction step fails silently on certain types of documents—rendering them garbled, fragmented, or boilerplate-contaminated—the subsequent quality filtering stages (perplexity, classifier-based) will naturally discard those documents for being "low quality." The final dataset appears clean, and automated quality metrics look good, but the dataset is systematically missing content from sources that the extractor handled poorly. The filtering stage masks the extraction failure. The authors state this with precision:

"if a text extractor fails to extract the main text, gets the formatting wrong, or includes too much boilerplate in the extraction, then both the classification and perplexity filters can filter out such examples. This can lead to subtle biases in the dataset, where specific poorly-extracted websites are excluded entirely even though they do contain high quality content."

What makes this an insight rather than an observation is that it identifies a causal structure that is invisible to standard evaluation. Prior work on web data pipelines (C4, RefinedWeb, CCNet, The Pile) focused on the final dataset quality as measured by downstream perplexity or task performance. The pipeline was treated as a black box: if the output dataset produces good models, the pipeline works. But this black-box evaluation creates a dangerous feedback loop: an extractor that fails on category X of documents will produce a dataset where category X is underrepresented, but the quality metrics will improve (because garbled category-X documents were removed), and researchers will conclude the pipeline is working well.

For mathematical web data specifically, this dynamic is lethal because LaTeX content is exactly the kind of content that confuses standard extractors. Equations have unusual line lengths, contain special characters that look like noise to a general-domain language model, and are interspersed with prose in patterns that break boilerplate detection heuristics. A standard pipeline (WET extraction + perplexity filtering) would systematically discard mathematical pages because the extracted text looks garbled. The resulting dataset would have excellent perplexity scores—all the "messy" math documents were removed—but would be useless for training mathematical reasoning.

This framing changes how the community should think about domain-specific dataset construction. The paper's recommendation—"tune your text extraction pipeline on a diverse set of documents before applying filtering to avoid this bias"—is a methodological prescription that inverts the standard workflow. Instead of extracting, filtering, and then evaluating the output, practitioners should validate extraction quality across a representative sample of target content types before letting filters mask extraction failures. This is analogous to the principle in causal inference that conditioning on a collider can induce spurious correlations: filtering on quality (a collider between extraction quality and true content quality) can hide extraction failures while making the dataset appear better.

Comparison to prior work: Prior dataset papers (Raffel et al., 2020; Gao et al., 2020; Penedo et al., 2023) describe extraction and filtering as sequential stages without discussing their interaction. CCNet (Wenzek et al., 2019) explicitly uses perplexity filtering as a quality measure and does not discuss whether certain content types are systematically filtered due to extraction failures. The OpenWebMath paper is the first to explicitly name this interaction as a bias source and to make it a first-class design consideration.

Significance: This is a fundamental methodological contribution that extends beyond mathematics to any domain where text has unusual formatting (code, chemistry, legal documents, medical records, multilingual text, historical documents). It identifies a failure mode that likely affects many existing domain-specific datasets without their creators being aware of it. The insight is not performance-enhancing—you can't point to a table showing improved numbers because of it—but it is correctness-preserving: it prevents the dataset from silently misrepresenting the distribution of mathematical content on the web.

Evidence anchor: The paper provides indirect evidence through its own pipeline design. The decision to build custom extraction on top of Resiliparse rather than using WET files or jusText is explicitly justified by the need to preserve LaTeX content that standard extractors mishandle. Table 5's efficiency benchmark is secondary; the primary claim is about coverage, not speed, and is validated by the iterative refinement methodology described in Appendix B, where the authors deliberately searched for documents where their extraction failed and kept adding support for new formats until failures became rare.

Innovation 2: The MathScore Classifier as a "Content-Aware" Filter That Operates Without Seeing the Content

The MathScore classifier (Section 3.5.1, Figure 4) represents a non-obvious inversion of standard classification for data filtering that has implications for any domain where the signal you want to detect might not be reliably extractable.

Standard practice in domain-specific data filtering is to train a classifier on documents labeled as in-domain vs. out-of-domain, using the full document text as features. For mathematical content, the natural approach would be: collect examples of mathematical web pages (containing LaTeX), collect examples of non-mathematical web pages, train a classifier on the raw text, and filter. This works, but it learns to detect LaTeX commands as the primary signal—\frac, \sum, \int, dollar signs, \begin{equation}—because those are the most discriminative features. The classifier becomes a LaTeX detector, not a mathematical content detector.

The problem with this is precisely what OpenWebMath's LaTeX extraction pipeline is designed to address: LaTeX is encoded in many formats, some of which will not be extractable by any reasonable extraction pipeline. A LaTeX-detector classifier would systematically miss pages where LaTeX is encoded as images, in non-standard delimiters, inside unexecuted JavaScript, or in other exotic formats. The classifier would be blind to exactly the edge cases the pipeline struggles with.

The MathScore classifier solves this by a simple but clever intervention: remove all LaTeX from the training documents before training. The classifier is forced to predict the presence of mathematics from the surrounding prose alone—the vocabulary, sentence structures, and discourse patterns of mathematical writing. It learns that "theorem," "proof," "Let x be...," "it follows that," "derivative," and similar patterns indicate mathematical content even when the equations themselves are invisible. This is a content-aware filter that operates on the linguistic context of mathematics rather than on the mathematical notation itself.

Why this is conceptually distinctive: It reframes the problem from "detect documents containing LaTeX" to "detect documents that are about mathematics." These are not the same thing. A page could be about mathematics without containing any extractable LaTeX (e.g., a purely textual discussion, an image-heavy tutorial, a page where equations failed to extract). A page could contain LaTeX without being meaningfully about mathematics (e.g., a web development tutorial showing how to configure MathJax). The MathScore classifier targets the aboutness of the content rather than the format of the notation, which makes it robust to the very format diversity that makes LaTeX extraction difficult.

Comparison to prior work: The closest analogue in prior work is the classifier-based filtering used in GPT-3's training data (Brown et al., 2020), where a classifier was trained to distinguish high-quality documents (Wikipedia) from random Common Crawl documents. But that classifier was trained on full document text and used for general quality assessment, not domain-specific content detection. CCNet (Wenzek et al., 2019) uses perplexity under a language model as a quality filter—this implicitly captures some domain information (a mathematical LM would assign lower perplexity to mathematical text) but doesn't specifically target aboutness. The MathScore classifier's training methodology (removing the target signal from training data to force the model to learn contextual markers) is, to the best of my knowledge, novel in the data filtering literature.

Dual-use architecture: The paper deploys MathScore in two different roles at two different thresholds:

• Prefilter (threshold = 0.8): High-precision gate to catch mathematical documents that the keyword and symbol filters missed, without letting through too many false positives that would waste downstream extraction compute.
• Post-extraction filter (threshold = 0.17 for documents WITH LaTeX, 0.8 for documents WITHOUT): Lenient sanity check for documents where LaTeX extraction succeeded (we already know they're mathematical), strict gate for documents where extraction failed (the classifier is our only signal).

This dual use is a design pattern worth highlighting: the same model provides coarse filtering before expensive processing and fine filtering after processing, with different operating points tuned to the different cost-benefit tradeoffs at each stage.

Significance: This is an incremental methodological innovation with potentially broad applicability. Any domain where the target content type is encoded in format-diverse, extraction-resistant ways could benefit from training a classifier on the context markers of that content rather than on the content itself. Examples include: detecting legal documents (where formatting conventions vary widely but vocabulary is distinctive), identifying medical text (where terminology is a strong signal even if tables and structured data are garbled), or filtering for multilingual content in specific scripts. The MathScore approach is simple to implement (train a FastText classifier on label-stripped documents) and computationally cheap enough to run at Common Crawl scale, lowering the barrier to adoption.

Evidence anchor: The paper reports that manual inspection found content with MathScore < 0.2 to be "very unlikely to contain useful mathematical content" (Section 3.5.1). The rigorous evidence is indirect: the downstream model training results (Tables 1–2) show that models trained on OpenWebMath substantially outperform models trained on general-domain data, suggesting the filtering pipeline—of which MathScore is a key component—successfully selected mathematical documents. However, the paper does not report an ablation study comparing MathScore filtering to a baseline LaTeX-detection classifier, so the specific contribution of the LaTeX-removal training trick is not quantified.

Innovation 3: Per-Token Mathematical Quality Dominates Raw Token Quantity, with Evidence at 20× Scale

The training experiments in Section 4 (Tables 1–2) are not merely a "smoke test" that OpenWebMath contains useful data. They provide direct quantitative evidence for a claim that the field has long assumed but rarely demonstrated at this scale ratio: that carefully curated domain-specific tokens can be worth an order of magnitude more than general-domain tokens for domain-specific downstream tasks.

The key result: a 1.4B parameter model trained on 14.7B tokens of OpenWebMath outperforms Pythia 1.4B trained on 300B tokens of The Pile—over 20× more data—on mathematical perplexity benchmarks (Table 1). The numbers:

Lower perplexity indicates better modeling of the mathematical text distribution. The OpenWebMath model achieves substantially lower perplexity on all MATH subsets despite being trained on 1/20th the data. On MATH Algebra-Easy accuracy (Table 2), the gap is stark: 5.62% for OpenWebMath vs. 3.93% for Pythia—a 43% relative improvement. On LILA-multiarith (testing coding ability on math problems): 16.67% vs. 21.80%, actually lower, but this likely reflects that The Pile contains code that transfers to this task. The 50/50 mixture of ProofPile and OpenWebMath generally performs best across benchmarks, confirming that mathematical data from different sources is complementary.

Why this is a contribution, not just a validation: Prior work demonstrated the value of domain-specific pretraining qualitatively (Minerva's improvements on quantitative reasoning benchmarks) but did not provide a clean comparison isolating data quality from model scale, architecture, and training procedure. Minerva fine-tuned PaLM 540B on 17.5B math tokens; the improvement could be attributed to the interaction between the base model's capabilities and the specialized data. OpenWebMath provides a controlled, same-architecture, same-hyperparameter comparison at a scale (1.4B parameters) where the cost is low enough to be independently replicated. The 20× efficiency multiplier is not claimed as a universal constant—it's specific to these benchmarks, this model size, and this data comparison—but it establishes the existence of a regime where data quality dominates data quantity by a large margin.

Comparison to prior work: The Chinchilla scaling laws (Hoffmann et al., 2022) established relationships between model size, data quantity, and loss, but they treated data as homogeneous. GPT-3 (Brown et al., 2020) emphasized data quality filtering but didn't quantify the tradeoff between domain-specific quality and general-domain quantity. Proof-Pile (Azerbayev et al., 2023) demonstrated that mathematical data improves reasoning but didn't compare against a 20× larger general corpus. This paper provides the first clean, replicable quantification of the domain-data-vs-general-data tradeoff for mathematical reasoning at a scale where the ratio can be measured precisely.

The mixture result is equally important: The finding that a 50/50 mixture of OpenWebMath and ProofPile outperforms either dataset alone (Tables 1–2) suggests that mathematical data from web sources and from curated sources are complementary, not redundant. The paper notes "very little overlap between the two datasets" (Section 4). This is a practically important result for practitioners: it implies that adding OpenWebMath to an existing mathematical pretraining corpus (like Proof-Pile) is likely to improve performance even if the existing corpus is already high-quality. The diversification benefit—seeing mathematical content in different styles, formats, and difficulty levels—outweighs any dilution from potentially lower-quality web content.

Significance: This is an incremental but practically important empirical contribution. It doesn't introduce a new concept or method, but it provides a concrete data point that changes how practitioners should allocate their pretraining compute budgets. If 1 token of curated mathematical web text is worth ~20 tokens of general web text for mathematical reasoning, then the optimal pretraining data mix for a math-capable model should be heavily skewed toward mathematical sources—far more than intuition might suggest. The paper doesn't claim this 20× ratio generalizes (it almost certainly depends on the specific benchmarks, model size, and data sources), but it establishes that the ratio is large enough to matter for practical decisions.

Evidence anchor: Tables 1 and 2 are the primary evidence. The paper does not overclaim: it explicitly notes that "future work could explore filtering OpenWebMath more aggressively to further improve its quality" (Section 4), acknowledging that the current dataset is not necessarily at the quality ceiling. The controlled comparison (same architecture, same hyperparameters, same token count for the 14.7B runs) makes the comparison fair, and the inclusion of Pythia 1.4B as an additional baseline at 300B tokens—with the same architecture—removes confounds about model design.

Innovation 4: The "JavaScript Execution Gap" as a Recognized Structural Disadvantage in Open Web Data

The paper identifies and explicitly names what might be called the JavaScript execution gap: the systematic difference between HTML documents available in proprietary web crawls (where JavaScript may have been executed, rendering dynamic content into static tags) and those available in Common Crawl (where JavaScript is not executed, leaving dynamic content in its pre-rendered form). This is not a new discovery—researchers working with Common Crawl are aware that JavaScript isn't executed—but the paper is the first to recognize this as the primary structural barrier to creating an open mathematical web dataset and to document the specific consequences for LaTeX extraction.

The paper states the gap clearly (Appendix B):

"We suspect that Minerva (Lewkowycz et al., 2022) gets around this issue by using HTML documents where javascript code has already been executed, in which case MathJax is converted from delimited text to explicit HTML tags that are easy to detect."

This is a specific hypothesis about why proprietary datasets achieved easier LaTeX extraction, and it has implications beyond mathematics. The JavaScript execution gap means that any web content rendered dynamically—interactive visualizations, comment systems, equation rendering, syntax-highlighted code, dynamically-loaded article text—will appear in fundamentally different forms in proprietary vs. open web crawls. Open efforts must solve extraction problems that proprietary efforts never face, because proprietary crawls can afford the computation to render pages fully.

Why this is a contribution: It reframes the "open vs. proprietary" data gap from a question of access (we don't have the dataset) to a question of structural disadvantage (we can't build the same dataset using the same methods because the underlying HTML sources are fundamentally different). This is more than a practical complaint—it's a diagnostic insight that explains why certain data types are harder to open-source than others. Mathematics, with its heavy reliance on client-side rendering (MathJax), is a worst-case scenario for this gap. Code documentation with syntax highlighting, interactive data journalism, and pages with dynamic article loading are similarly affected.

Comparison to prior work: No prior dataset paper that I'm aware of explicitly names this gap or discusses its consequences. Papers releasing Common Crawl-derived datasets (C4, RefinedWeb, The Pile, CCNet) note that they use raw HTML, but they don't analyze what content is systematically lost due to unexecuted JavaScript. The OpenWebMath paper makes this gap a central part of its narrative: the entire LaTeX extraction pipeline is a workaround for it. The paper's repeated emphasis on iterative refinement and manual inspection reflects the reality that no automated solution fully bridges the gap—you can only approximate what a rendered DOM would have provided.

Significance: This is a diagnostic contribution with implications for the open data community. It identifies a specific mechanism (JavaScript-dependent rendering) that creates systematic quality differences between proprietary and open datasets, and it provides a case study in how to work around it. The broader message is that open data efforts in domains that rely on client-side rendering need fundamentally different extraction approaches than general web datasets—approaches that are more expensive, more manual, and less scalable. This is a sobering message for the open-source community, but it's better to understand the structural disadvantage than to be blindsided by it.

Evidence anchor: The relative frequencies in Table 6 provide indirect evidence. MathJax with delimiters (the pre-JavaScript format) accounts for 73.6% of documents containing math, while <annotation> tags within <math> blocks (the post-JavaScript format Minerva likely targeted) account for only 2.35%. If Minerva had access to post-execution DOMs, their pipeline could ignore the 73.6% of cases that OpenWebMath's pipeline is designed to handle, because those cases would have been converted to the 2.35% format during rendering. This 30× difference in the prevalence of "easy" vs. "hard" extraction formats is the quantitative manifestation of the JavaScript execution gap.

Meta-Level Observation: What This Paper Contributes to the "Datasets as Research Contributions" Genre

OpenWebMath sits in a tradition of dataset papers (C4, The Pile, RefinedWeb, Proof-Pile) that contribute infrastructure rather than algorithms. What distinguishes it within this genre is the depth of its pipeline documentation and methodological reflection. The paper doesn't just describe what was done—it explains why design choices were made, documents failures of alternative approaches (jusText was too aggressive, Trafilatura was too slow, WET files can't handle LaTeX, Minerva's tag-based approach misses 97% of content), and explicitly discusses interactions between pipeline stages that are normally treated as independent (the extraction-filtering bias in Appendix C).

This is significant because dataset papers are the primary mechanism for transferring data engineering knowledge in the NLP community. A dataset that is released with extensive documentation enables other researchers to understand its biases, adapt its methods to new domains, and avoid repeating its mistakes. A dataset released with minimal documentation—even if high-quality—is a black box. OpenWebMath's contribution is as much about showing the community how to build mathematical web datasets as it is about providing the finished dataset itself. The open-source code and pipeline documentation make it a template for domain-specific web data extraction that other researchers can modify for their own domains.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All training experiments use the datasets being compared: OpenWebMath (14.7B tokens across 6.3M documents), The Pile (Gao et al., 2020), ProofPile (Azerbayev et al., 2023), and a 50/50 mixture of OpenWebMath and ProofPile. Each dataset is trained for exactly 14.7B tokens (one epoch for OpenWebMath, truncated for The Pile). The evaluation benchmarks are GSM8k (Cobbe et al., 2021), MATH (Hendrycks et al., 2021), and LILA-multiarith (Mishra et al., 2022). For MATH accuracy, the paper specifically uses "the subset of level-1 algebra questions from MATH" (Section 4). For perplexity evaluations, multiple MATH subdomains are used: Prealgebra, Algebra, Intermediate Algebra, Counting & Probability, Number Theory, Precalculus, and Geometry.

• Base model(s). All models are 1.4B parameter transformers using "the same architecture as Pythia 1.4B" (Biderman et al., 2023): 24 layers, model dimension 2048, 16 attention heads. The tokenizer is LLaMA's (Touvron et al., 2023c). The models are "randomly initialized" and trained from scratch—this is not fine-tuning of an existing model. Hyperparameters follow Pythia exactly except for batch size (1M tokens; full hyperparameter table in Appendix D, Table 7). The paper states the model size choice is because these benchmarks "show signal on models of this size" (Section 4). Training uses the GPT-NeoX library (Andonian et al., 2023) on 8 A100 80GB GPUs.

• Metrics. Three types of metrics are reported: (1) Perplexity on held-out evaluation sets from GSM8k and seven MATH subdomains—lower perplexity indicates better modeling of mathematical text distributions. Perplexity is standard autoregressive crossover entropy exponentiated. (2) Accuracy on MATH Algebra-Easy (the subset of level-1 algebra problems) reported as both pass@1 (single greedy generation) and maj@16 (majority voting over 16 samples at non-zero temperature). (3) Accuracy on LILA-multiarith, which tests the model's ability to write Python code that solves arithmetic word problems, reported as execution accuracy. The paper chooses these specific evaluations because they "show signal on models of this size" (Section 4) whereas harder MATH subsets or full MATH might produce near-zero accuracy that doesn't differentiate between datasets.

• Baselines. Four training data baselines are compared, all with identical model architecture and training hyperparameters at 14.7B tokens: (1) The Pile (14.7B tokens) — a general-domain dataset (Gao et al., 2020) truncated to 14.7B tokens for direct comparison. (2) ProofPile (14.7B tokens) — a curated mathematical dataset (Azerbayev et al., 2023) containing arXiv, Stack Exchange, ProofWiki, Wikipedia, and openly licensed books, also trained for one epoch. (3) OpenWebMath (14.7B tokens) — the dataset introduced by this paper. (4) Mixture (14.7B tokens) — a 50/50 mixture of ProofPile and OpenWebMath tokens, interleaved to the same total token budget. Additionally, the paper compares against Pythia 1.4B trained on 300B tokens of The Pile (Biderman et al., 2023), which uses the identical model architecture but with over 20× more training data.

• Generation budget / compute accounting. All models at 14.7B tokens are trained on exactly the same number of tokens, making the comparison "per-token" fair. The paper measures "generation budget" implicitly through the token count—each dataset gets 14.7B training tokens regardless of the source. This is the key experimental design choice: hold the total training FLOPs constant across datasets and measure the quality of the resulting model. The Pythia 1.4B comparison at 300B tokens is not FLOPs-matched; it is included to show that 14.7B tokens of mathematical data can outperform 300B tokens of general data despite using far less compute.

• Cross-validation / statistical protocol. The paper does not report cross-validation, statistical significance testing, confidence intervals, or multiple training runs with different random seeds. The results in Tables 1 and 2 appear to come from a single training run per data configuration. The perplexity numbers are reported to 4 decimal places, but no standard deviations or standard errors are given. For accuracy on MATH Algebra-Easy and LILA-multiarith, the numbers are reported to 2 decimal places as percentages. The paper does not discuss whether differences between configurations are statistically significant or within the range of training noise at this model scale.

Main Quantitative Results

The experimental results are presented in two tables (Tables 1 and 2) and a brief narrative in Section 4. The experiments test a single core hypothesis: that OpenWebMath contains higher-quality mathematical training data per token than either general-domain data (The Pile) or existing curated mathematical data (ProofPile), and that mixing OpenWebMath with other mathematical datasets is beneficial. There are no separate axes of investigation—no ablation of pipeline components, no scaling curves at multiple model sizes, no comparison of different filtering thresholds—so the results are presented here as a single group.

Headline finding: OpenWebMath outperforms general-domain data by a large margin on mathematical metrics despite 20× less data.

The central comparison in Table 1 pits the OpenWebMath-trained model (14.7B tokens) against Pythia 1.4B (300B tokens of The Pile). Across all seven MATH subdomain perplexity evaluations, the OpenWebMath model achieves substantially lower (better) perplexity:

Note: The paper's Table 1 reports OpenWebMath Geometry perplexity as 1.5748, but this is inconsistent with the column labeling (the last column should be Geometry based on the Pythia baseline ordering; the table layout is slightly ambiguous). The key point stands regardless: OpenWebMath perplexity is lower than Pythia on every subdomain.

On GSM8k perplexity, OpenWebMath achieves 1.9075 vs. Pythia's 2.2032—a difference of approximately 0.3, which is substantial at this perplexity range. On MATH Algebra-Easy accuracy (Table 2), OpenWebMath achieves 5.62% pass@1 vs. Pythia's 3.93% pass@1 (a 43% relative improvement) and 9.55% maj@16 vs. Pythia's 5.62% maj@16 (a 70% relative improvement).

These results demonstrate that 14.7B tokens of mathematical web text produce a model that better captures mathematical language distributions than 300B tokens of general web text—by a substantial margin, despite using 1/20th the training compute.

OpenWebMath vs. ProofPile at equal token counts: OpenWebMath wins on most metrics.

Comparing the two mathematical datasets at 14.7B tokens each (Table 1), OpenWebMath achieves lower perplexity than ProofPile on every MATH subdomain. The margins are moderate but consistent: Prealgebra (1.6503 vs. 1.7214), Algebra (1.5949 vs. 1.5739—here ProofPile is slightly better; this appears to be an anomaly as the trend reverses elsewhere), Intermediate Algebra (1.6002 vs. 1.6462), Counting & Probability (1.6894 vs. 1.7291), Number Theory (1.4542 vs. 1.4838), Precalculus (1.5748 vs. 1.7229). On GSM8k perplexity, OpenWebMath achieves 1.9075 vs. ProofPile's 2.2350.

On MATH Algebra-Easy accuracy (Table 2), OpenWebMath substantially outperforms ProofPile: 5.62% vs. 2.81% pass@1 and 9.55% vs. 3.93% maj@16. This is a 2× improvement in pass@1 accuracy, suggesting that web-sourced mathematical content provides training signal that curated academic sources like arXiv and Stack Exchange do not fully capture.

On LILA-multiarith coding accuracy, OpenWebMath achieves 16.67% vs. ProofPile's 8.04%—again roughly 2× improvement. This is notable because LILA tests Python code generation for math problems, a skill that likely benefits from the diverse formats (code blocks, tutorials, informal explanations) found in web data but less so in formal academic text.

The 50/50 mixture of OpenWebMath and ProofPile performs best overall.

The most consistent winner across benchmarks is the Mixture configuration (Tables 1–2). On perplexity, the Mixture achieves the lowest (best) score on 5 out of 7 MATH subdomains: Prealgebra (1.6190), Algebra (1.5301—substantially better than either dataset alone), Intermediate Algebra (1.5719), Counting & Probability (1.6607), Precalculus (1.5599). On GSM8k, the Mixture achieves 1.8968, edging out OpenWebMath alone (1.9075).

On MATH Algebra-Easy accuracy, the Mixture achieves 5.06% pass@1 and 10.11% maj@16—slightly lower pass@1 than OpenWebMath alone (5.62%) but higher maj@16 (10.11% vs. 9.55%). On LILA-multiarith, the Mixture achieves 13.22%, between OpenWebMath (16.67%) and ProofPile (8.04%).

The finding that mixing two complementary mathematical datasets outperforms either alone is practically significant: it suggests that the diversity of mathematical content (web forums, blogs, educational sites from OpenWebMath + academic papers, formal proofs, textbook excerpts from ProofPile) is more important than simply having more tokens from a single source. The paper explicitly notes this complementarity: "there is very little overlap between the two datasets" (Section 4).

What the 14.7B-token baselines show about The Pile and ProofPile at equal scale.

When all three datasets are compared at the same 14.7B token budget (Table 1), the ranking is clear: Mixture > OpenWebMath > ProofPile > The Pile for most benchmarks. The Pile at 14.7B tokens is the weakest across all mathematical metrics, as expected for general-domain data. ProofPile at 14.7B tokens outperforms The Pile at 14.7B tokens but underperforms OpenWebMath at 14.7B tokens—confirming that the web-sourced mathematical content in OpenWebMath provides additional value beyond what curated academic sources offer.

The fact that The Pile at 300B tokens (Pythia) outperforms The Pile at 14.7B tokens on all metrics confirms that scale helps even for general-domain data—but the fact that 14.7B tokens of mathematical data beats 300B tokens of general data on mathematical metrics demonstrates that domain relevance can outweigh scale by a factor of 20× or more in this regime.

Ablation Studies and Robustness Checks

The paper does not report ablations of the filtering pipeline, the text extraction components, or the training hyperparameters. There are no comparisons of different MathScore thresholds, no tests of different perplexity filter cutoffs, no experiments with and without deduplication, no comparisons of different LaTeX extraction strategies, and no evaluation of how dataset size affects downstream performance (e.g., training on 1B, 3B, 7B, and 14.7B tokens to show scaling trends). The only "ablation" in the experimental section is implicit: comparing OpenWebMath alone vs. Mixture to test whether combining datasets is beneficial. This is not an ablation in the usual sense (removing a component to measure its contribution) but rather a data composition experiment.

Mixture vs. single dataset: As discussed above, the 50/50 mixture of OpenWebMath and ProofPile generally outperforms either dataset alone (Tables 1–2). This suggests complementarity rather than redundancy. However, without experiments at different mixture ratios (e.g., 25/75, 75/25), it's impossible to know whether 50/50 is optimal or whether the improvement comes primarily from one dataset with the other providing marginal benefit.

ProofPile as a mathematical data baseline: Including ProofPile (Azerbayev et al., 2023) as a comparison point is valuable because it demonstrates that OpenWebMath's value is not simply "mathematical data is better than general data"—it's specifically web-sourced mathematical data that provides additional benefits over curated academic sources. The consistent gap between OpenWebMath and ProofPile at equal token counts (Table 1) supports this claim.

The Pile at two different scales: Comparing The Pile at 14.7B vs. 300B tokens (Pythia) is not a formal scaling law experiment, but it provides a reference point for interpreting the magnitude of the domain-specific benefit. This is more of a sanity check than an ablation: it confirms that The Pile models improve with more data, which establishes that the 20× advantage of OpenWebMath is due to data quality rather than some pathology in the 14.7B-token The Pile baseline.

What is missing: The most obvious missing ablation is a comparison of OpenWebMath at different sizes or trained with different filtering thresholds to show that the filtering choices were well-calibrated. For example: training models on OpenWebMath with more aggressive filtering (e.g., MathScore > 0.5 instead of 0.17 for LaTeX-containing documents) and showing whether performance improves or degrades. The paper acknowledges this gap directly in Appendix A: "due to the high cost of extracting data from all shards on Common Crawl, we were only able to run our pipeline once. Therefore, many of our choices are without empirical justification and we provide no ablation study." This is an honest limitation but also a significant one—without ablations, the reader cannot distinguish between pipeline choices that were essential for quality and those that were incidental.

Critical Assessment

The experiments demonstrate a narrower claim than the paper's framing suggests. The paper's introduction positions OpenWebMath as an open alternative to Minerva's Math Web Pages—a dataset that, when used to fine-tune PaLM 540B, produced dramatic improvements on quantitative reasoning benchmarks like MATH and MMLU-STEM. The experiments in this paper, however, train 1.4B parameter models from scratch on 14.7B tokens and evaluate on a limited subset of benchmarks: GSM8k perplexity, MATH subdomain perplexities, the level-1 algebra subset of MATH, and LILA-multiarith. These are reasonable evaluations for demonstrating that the dataset contains useful mathematical signal, but they do not demonstrate that OpenWebMath would produce Minerva-like improvements if used for large-scale fine-tuning of a frontier model. The gap between "this dataset improves mathematical perplexity on small models" and "this dataset enables state-of-the-art quantitative reasoning when used at scale" is substantial and unbridged by the experiments presented.

Specifically, the paper does not show: (1) training at a scale comparable to Minerva (billions of parameters, hundreds of billions of tokens), (2) evaluation on the same benchmarks Minerva excelled at (full MATH, MMLU-STEM, OCWCourses), or (3) fine-tuning a pretrained model on OpenWebMath rather than training from scratch. The experimental setup tests whether OpenWebMath is better than general-domain data for teaching a small model the language of mathematics (as measured by perplexity), not whether it teaches reasoning in a way that transfers to novel problem-solving. The distinction matters: one could imagine a dataset of mathematical text that produces excellent perplexity (because it contains many well-formed equations and proofs) but doesn't improve problem-solving accuracy (because it lacks the kind of step-by-step solution demonstrations that teach reasoning procedures). The paper's accuracy results on MATH Algebra-Easy and LILA provide some evidence of reasoning transfer, but at very low absolute accuracy levels (5–10% on algebra, 13–17% on LILA) where the difference between "memorizing surface patterns" and "learning to reason" is difficult to disentangle.

The 20× efficiency claim is specific to this model size, these benchmarks, and this comparison. The paper states that "models trained on 14.7B tokens of our dataset surpass the performance of models trained on over 20x the amount of general language data." This is technically true for the specific comparisons in Tables 1–2, but the claim generalizes to "over 20×" whereas the actual comparison is against exactly one general-domain dataset (The Pile) at exactly one model size (1.4B parameters) on exactly the benchmarks listed. The Pile itself is a particular general-domain dataset with its own composition biases; comparing against a different general-domain corpus (e.g., C4, RefinedWeb, or a different snapshot of Common Crawl) might yield a different efficiency ratio. The efficiency ratio also depends on model size—whether a 7B or 70B model trained on 14.7B tokens of OpenWebMath would outperform a 300B-token The Pile model at those scales is unknown. The 20× number is best understood as an existence proof that a large efficiency gap can exist, not as a measured constant that holds across settings.

The baselines are appropriate but incomplete. The comparison against Pythia 1.4B at 300B tokens is the strongest piece of evidence in the paper's favor because it uses an independently developed, publicly available model with identical architecture. The comparison against ProofPile at 14.7B tokens is also valuable for isolating the web-sourced vs. curated-source distinction. However, two baselines are notably missing:

First, there is no comparison against a model trained on a random sample of Common Crawl without the mathematical filtering pipeline. This would directly measure the value added by the filtering: does the MathScore + perplexity + deduplication pipeline produce substantially better mathematical performance than simply taking 14.7B tokens of unfiltered Common Crawl text? Without this, we can't distinguish between "mathematical data is better than general data" (which we already knew from Minerva) and "the OpenWebMath filtering pipeline successfully identifies high-quality mathematical data" (which is the paper's claimed contribution). It's possible that simply filtering Common Crawl for documents containing \frac and extracting them with a standard pipeline would yield similar downstream performance—the paper provides no evidence against this simpler baseline.

Second, there is no comparison against a model trained on a larger amount of ProofPile (e.g., 300B tokens if it exists, or repeated epochs). The Mixture result shows ProofPile and OpenWebMath are complementary, but if the resource constraint is total FLOPs, one could simply train on ProofPile for more epochs rather than collecting new web data. Whether new web data beats more epochs of existing curated data is an open question the experiments don't address.

The statistical rigor is below the standard expected for empirical claims. The paper reports results from single training runs with no error bars, no standard deviations, and no discussion of training variance. At the 1.4B parameter scale with 14.7B tokens of data, training runs can have non-trivial variance depending on random seed—the ordering of datasets with perplexity differences of 0.05–0.1 might not be statistically reliable. The accuracy numbers on MATH Algebra-Easy (2.81% vs. 5.62% vs. 5.06%) are based on a small number of questions getting answered correctly at these low accuracy rates—the difference between 2.81% and 5.62% might represent just a handful of additional correct answers. The paper provides no information about the size of the MATH Algebra-Easy subset (how many questions? what's the standard error on a proportion at this sample size?), making it impossible for the reader to assess whether observed differences are meaningful.

The evaluation benchmarks are a subset of what would be needed to validate the dataset's utility. The paper evaluates on GSM8k perplexity, MATH subdomain perplexities, MATH Algebra-Easy accuracy, and LILA-multiarith accuracy. These are reasonable choices for models at this scale, but they don't test the full range of mathematical reasoning that Minerva demonstrated. Specifically: there is no evaluation on full MATH (where non-algebra questions test geometry, number theory, and precalculus reasoning), no evaluation on MMLU-STEM (which tests scientific and engineering knowledge), no evaluation on OCWCourses (which tests undergraduate-level problem-solving), and no evaluation on GSM8k accuracy (only perplexity). The MATH Algebra-Easy subset is specifically the easiest questions from one topic area—it's unclear whether improvements there would translate to harder questions or different mathematical domains. The LILA-multiarith evaluation tests Python code generation for arithmetic word problems, which is a specific skill that may benefit from the code blocks found in web data but doesn't directly measure mathematical reasoning in natural language.

What would strengthen the paper: Several experiments would substantially increase confidence in the dataset's value:

• Training 1.4B models on OpenWebMath at multiple sizes (e.g., 100M, 500M, 1.4B, 3B tokens) to show that improvements scale with data quantity within the dataset.
• Comparing against an un-filtered Common Crawl sample matched for token count to isolate the pipeline's contribution.
• Evaluating on a broader set of mathematical reasoning benchmarks, even at low accuracy, to see if improvements transfer across mathematical domains.
• Fine-tuning an existing pretrained model (e.g., Pythia 1.4B) on OpenWebMath vs. The Pile to test transfer learning rather than from-scratch training.
• Reporting results with multiple random seeds to establish whether observed differences exceed training variance.

What the experiments do demonstrate convincingly: Despite these limitations, the experiments establish a clear and valuable finding: 14.7B tokens of OpenWebMath produce substantially better mathematical language modeling (as measured by perplexity on MATH and GSM8k) than the same amount of ProofPile or general-domain data, and the improvement over general-domain data persists even when the general-domain model is trained on 20× more tokens. This is sufficient to validate that the dataset contains high-quality, on-domain mathematical text that was not previously available in open-source form. The Mixture result further demonstrates that OpenWebMath is complementary to existing mathematical datasets, which is actionable guidance for practitioners. The paper's explicit acknowledgment that it provides "no ablation study" and that "many of our choices are without empirical justification" (Appendix A) is honest and tempers the strength of the experimental claims appropriately.

The experiments should be understood as a dataset validation rather than a comprehensive empirical study of mathematical pretraining. They answer the question "does this dataset contain useful mathematical signal?" with a qualified yes—qualified because the signal is measured primarily through perplexity on in-domain text rather than through reasoning accuracy on challenging out-of-distribution problems. The gap between perplexity improvements and reasoning improvements is where the most important future work lies, and the paper's experiments don't bridge it—but they provide the dataset and baseline models needed for others to do so.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in the Headline Efficiency Gains

The assumption or constraint. The paper's training experiments—and the central claim that models trained on OpenWebMath outperform models trained on 20× more general-domain data—are based on a single training run per data configuration using the entire 14.7B-token dataset. However, the paper is explicitly transparent that the pipeline itself required enormous computation that was only run once, and that the cost of this extraction fundamentally limits what can be empirically validated about the filtering choices. The authors state this directly in Appendix A:

"due to the high cost of extracting data from all shards on Common Crawl, we were only able to run our pipeline once. Therefore, many of our choices are without empirical justification and we provide no ablation study."

This is not a minor caveat—it is an admission that the central design decisions of the pipeline (choice of Resiliparse over jusText, MathScore thresholds of 0.17 and 0.8, perplexity cutoff of 15,000, which LaTeX encoding formats to support, which domains to blacklist) were made based on manual inspection and engineering judgment rather than quantitative comparison of alternatives.

The consequence. The downstream experiments (Tables 1–2) validate the final dataset as a whole but cannot validate the individual design choices that produced it. A practitioner attempting to replicate or adapt this pipeline for a different domain—say, extracting chemical notation from the web, or legal documents with specialized markup—cannot know from the paper's evidence whether the specific thresholds, tools, and heuristics are essential for quality or whether they are incidental choices that could be changed without consequence. The failure mode is not that OpenWebMath is low-quality (the experiments suggest it is high-quality) but that the paper cannot provide evidence-based guidance for which pipeline components matter and by how much. This makes the pipeline harder to adapt than a more extensively ablated approach would be.

The extraction cost itself also limits generalizability. The paper processed Common Crawl shards from before May 2023. A researcher wanting to create an updated OpenWebMath with more recent Common Crawl data would need to rerun the full pipeline—including manual inspection of top domains, iterative refinement of LaTeX extraction rules, and retraining of the MathScore classifier on current web text. The paper does not estimate this cost (in compute-hours or dollars), but the fact that it was only run once despite being the paper's core contribution suggests it is substantial enough to be a barrier to reproduction or extension.

What evidence exists in the paper. Appendix A is the primary source. The paper explicitly acknowledges: "many of our choices are without empirical justification." There is no ablation table showing, for example, model perplexity when trained on datasets produced with different MathScore thresholds, different perplexity cutoffs, different deduplication parameters, or different text extraction libraries. The LaTeX extraction methods are documented (Table 6 shows relative frequencies) but there is no experiment showing that adding each encoding format (MathML conversion, CodeCogs URL extraction, WordPress plugin handling) progressively improved downstream model performance. The efficiency comparison in Table 5 (runtime of different extraction libraries) establishes that Resiliparse is fastest but does not establish that it produces the best-quality mathematical text for downstream training.

Mitigation status. The paper does not attempt to address this limitation within the current work. It suggests in Appendix A that "future work should focus on finding empirical answers to the questions of what constitutes good data, creating new, efficient filtering methodologies, and extracting images inline with math text." This is a general call for more systematic data ablation research rather than a specific plan to ablate OpenWebMath's pipeline. The open-source release of the pipeline code partially mitigates the replication barrier (others can run the pipeline, even if they cannot assess its components independently), but does not provide the empirical justification that would help practitioners make informed modifications.

The Dataset Is English-Only, Which Excludes a Substantial Fraction of Mathematical Web Content

The assumption or constraint. The filtering pipeline includes a FastText language identification step that removes all non-English documents (Section 3.5). The authors are explicit about this:

"We use a FastText language identification model to remove documents that are not in English."

And in Appendix A, they acknowledge the scope of this constraint:

"OpenWebMath is an English-only dataset, which limits its applications for researchers and users who speak other languages."

The authors explain that this is necessary because "our filters pipeline is optimized for English" (Section 3.5)—the MathScore classifier was trained on English documents, the perplexity model was trained on English mathematical text from ProofPile, and the boilerplate phrase list was developed for English. Documents in other languages would likely be incorrectly filtered or scored by these English-tuned components.

The consequence. Mathematics is an international discipline with substantial mathematical content written in languages other than English—French, German, Russian, Chinese, Japanese, and others have significant mathematical literatures on the web. By restricting to English, OpenWebMath excludes potentially valuable mathematical content that could improve models' mathematical reasoning in general (mathematical notation itself is language-independent; the surrounding prose provides context and explanation). More importantly, the English-only restriction means that models trained on OpenWebMath are likely to perform worse on mathematical reasoning tasks presented in other languages—the model may understand the equations but not the framing, problem statements, or expository context.

The restriction also limits the dataset's utility for multilingual mathematical applications. A researcher studying how mathematical reasoning transfers across languages, or building a multilingual mathematical assistant, cannot use OpenWebMath alone—the English bias in training data would confound any cross-lingual analysis. As the authors note, this "limits its applications for researchers and users who speak other languages."

The magnitude of this exclusion is unknown. The paper does not report what fraction of mathematical documents in Common Crawl are in non-English languages, so the reader cannot assess whether this is a 5% loss or a 50% loss of potential mathematical content.

What evidence exists in the paper. The limitation is acknowledged but not quantified. The language filtering step is described in Section 3.5, and Appendix A lists it as a limitation. Table 3 (top domains by document count) and Table 4 (top domains by character count) show primarily English-language domains (stackexchange.com, physicsforums.com, nature.com, wikipedia.org), which is consistent with English filtering but does not reveal what was excluded. The figures in Section 4 (document types and subjects) are based on analysis of the filtered dataset and thus provide no information about non-English content that was removed.

Mitigation status. None. The paper does not suggest methods for extending the pipeline to other languages, does not estimate the cost of doing so, and does not recommend that future work prioritize multilingual expansion. The exclusion is presented as a practical necessity given the current design of the filtering components rather than as a problem to be solved.

The Small-Scale Training Experiments Do Not Demonstrate Reasoning Transfer at a Level Comparable to Minerva's Reported Results

The assumption or constraint. The paper's experimental validation trains 1.4B parameter models from scratch on 14.7B tokens and evaluates on a limited set of mathematical benchmarks. The authors state their model size choice explicitly: these evaluations "show signal on models of this size" (Section 4). The largest and most successful mathematical pretraining result the paper positions itself against—Minerva (Lewkowycz et al., 2022)—involved fine-tuning a 540B parameter model that had been pretrained on hundreds of billions of tokens, and evaluated on the full MATH benchmark, MMLU-STEM, and OCWCourses. The gap between these experimental setups is enormous: roughly 400× fewer parameters, a fraction of the training tokens, and evaluation on only the easiest subset of one benchmark rather than the full suite.

The consequence. The experiments validate that OpenWebMath contains high-quality mathematical text that improves mathematical language modeling (as measured by perplexity on in-domain text). They do not validate that training—or fine-tuning—on OpenWebMath produces the kind of dramatic improvements on quantitative reasoning benchmarks that Minerva demonstrated. A practitioner reading this paper to decide whether to include OpenWebMath in a large-scale training run (e.g., for a 70B or 540B model) cannot infer from the 1.4B results whether the benefit scales with model size, whether it transfers to harder reasoning problems, or whether fine-tuning a pretrained model on OpenWebMath (as Minerva did) would outperform training from scratch on a mixture including OpenWebMath.

The specific concern is that mathematical perplexity improvements and reasoning accuracy improvements may not scale together. A dataset full of well-formatted mathematical prose—theorems, definitions, proofs written in clear LaTeX—will produce excellent perplexity on mathematical text. But reasoning requires more than being able to predict the next token in a proof: it requires the ability to solve novel problems by composing known concepts in new ways. If OpenWebMath is dominated by expository content (theorem statements, proof sketches, definitions) and lacks diverse step-by-step problem solutions with correct answers, a model trained on it might become fluent in mathematical language without becoming substantially better at solving problems. The paper's accuracy results on MATH Algebra-Easy (5.62% pass@1) and LILA-multiarith (16.67%) provide some evidence of reasoning transfer, but these numbers are very low in absolute terms—the improvement over baselines (e.g., 5.62% vs. 3.93% on MATH Algebra-Easy) represents a small number of additional correct answers that may not generalize to harder problems or larger models.

What evidence exists in the paper. The limitation is evident in the experimental design described in Section 4 and detailed in Appendix D. The paper reports accuracy on only two benchmarks that test problem-solving: MATH Algebra-Easy (the simplest subset of level-1 algebra questions from MATH) and LILA-multiarith (Python code generation for arithmetic word problems). It does not evaluate on full MATH, GSM8k accuracy (only perplexity), MMLU-STEM, OCWCourses, or any other quantitative reasoning benchmark where Minerva showed gains. The absolute accuracy numbers are low (2.81%–5.62% on MATH Algebra-Easy, 8.04%–16.67% on LILA), and the paper does not report whether the improvements over baselines are statistically significant at this sample size.

The paper does not claim to have demonstrated Minerva-level reasoning improvements—it presents the experiments as showing that "per-token, [OpenWebMath] is more effective than existing mathematical pretraining datasets" (abstract) and that "models trained on OpenWebMath outperform models trained on 20x more general-domain tokens in mathematics" (Section 5). These claims are supported by the perplexity and accuracy results presented. But the implicit framing throughout the paper—that OpenWebMath is an open alternative to the Math Web Pages dataset that enabled Minerva's reasoning improvements—implies a connection between the dataset and reasoning capability that the experiments do not establish.

Mitigation status. The paper does not explicitly address this gap. It does not discuss whether 1.4B parameter models are expected to show reasoning improvements comparable to 540B models, does not suggest that future work should evaluate OpenWebMath at larger scale, and does not qualify its claims about Minerva comparisons with the caveat that the experimental validation is at a radically different scale. The honest framing would be: "We demonstrate that OpenWebMath improves mathematical language modeling on small models; whether it enables Minerva-like reasoning improvements at scale remains to be tested." The paper comes close to this in its conclusion—"We hope that OpenWebMath can lead to the creation of language models with improved mathematical reasoning capabilities" (Section 5)—but does not explicitly state that the current experiments don't demonstrate this yet.

The Dataset Contains No Images, Despite the Mathematical Importance of Figures, Diagrams, and Rendered Equations

The assumption or constraint. OpenWebMath consists exclusively of extracted text and LaTeX code. All images, figures, diagrams, plots, and rendered visual content from the original web pages are discarded during text extraction. The paper states this explicitly in Appendix A:

"OpenWebMath only contains the text from math on the web, not associated figures, which can be important for solving mathematical problems (OpenAI, 2023)."

The reference to OpenAI's GPT-4 technical report is significant: GPT-4 is a multimodal model that can process images, and the report noted the importance of figures in mathematical reasoning. Many mathematical documents on the web—tutorials, textbook excerpts, forum answers—rely heavily on diagrams (geometry figures, function plots, commutative diagrams, graph illustrations) to convey information that is difficult or impossible to express in text alone. Even when the textual description exists alongside the image, removing the image loses the complementary visual information.

An additional, more subtle consequence: some of the LaTeX extraction methods the paper describes involve extracting equations from image URLs and alt text (e.g., CodeCogs URLs, WordPress plugin image alt text). When the pipeline does this, it extracts the LaTeX source from the image metadata but discards the rendered image that a human reader would see. For simple equations, this is fine—the LaTeX fully captures the mathematical content. But for complex commutative diagrams, annotated figures, or equations with non-standard notation that relies on visual layout, the LaTeX source may be incomplete (the image may have been created by a tool that doesn't produce clean LaTeX source, or the alt text may be abbreviated). In these cases, the pipeline extracts whatever LaTeX-like text it can find but loses the visual information that would have disambiguated the notation.

The consequence. A language model trained on OpenWebMath learns mathematics from text and LaTeX alone. It has no exposure to the visual representations of mathematical concepts that are ubiquitous in human mathematical communication. For a text-only language model, this is not necessarily a limitation—the model wouldn't be able to process images anyway. But it means that even if OpenWebMath were used to create a state-of-the-art mathematical language model, that model would have a fundamental capability gap compared to multimodal models (like GPT-4V) trained on text and figures together. The model would be unable to answer questions that require interpreting a geometric diagram, reading a graph, or understanding a visual proof.

Furthermore, for the text-only models that are the current target of OpenWebMath, the absence of images means the training text may contain "dangling references" to visual content that is not present—phrases like "as shown in Figure 3," "the red curve indicates," "the shaded region represents"—that are confusing or misleading without the accompanying figure. The current pipeline does not remove or annotate these references, so the model learns to process text that refers to non-existent visual content, which could produce peculiar behaviors when the model generates similar references at inference time.

What evidence exists in the paper. Appendix A explicitly lists this as a limitation and cites OpenAI (2023) for the importance of figures. The frequency of image dependence in the dataset is partially quantified by Table 6: "Math found in images" accounts for 6.96% of documents containing math. This is the second most common encoding format after MathJax delimiters, which means a non-trivial fraction of the mathematical content in OpenWebMath was originally presented as images. The paper extracts LaTeX from these images where possible (via alt text, URL decoding, etc.) but the underlying dominance of image-based math on the web—nearly 7% of mathematical documents—suggests that visual mathematics is a substantial component of the web's mathematical content that is lost.

Mitigation status. The paper does not attempt to address this limitation in the current release. It suggests that "future work should focus on... extracting images inline with math text" (Appendix A). This is a non-trivial engineering challenge: it would require not just downloading images (which Common Crawl may not reliably store) but also developing methods to associate the correct images with the surrounding text, handling figures that contain multiple sub-figures, and potentially using OCR or multimodal models to extract structured mathematical information from figures. The paper does not propose a specific approach.

The Dataset May Contain Problem-Answer Pairs from Public Benchmarks, Creating Contamination Risks for Evaluation

The assumption or constraint. OpenWebMath is sourced from the open web, which includes forum posts, educational websites, and reference pages where users discuss specific mathematical problems—including problems that appear in popular benchmarks like MATH, GSM8k, and MMLU. The paper performs no explicit contamination detection or removal. The deduplication step (Section 3.6) uses SimHash near-deduplication at the document level, which would remove identical copies of the same webpage but would NOT detect a MATH problem that appears verbatim within a longer Stack Exchange post, or a GSM8k problem that is paraphrased in a tutorial.

This is a general problem for web-sourced datasets, but it is particularly acute for mathematical datasets because:

1. Mathematical problems have precise, unique formulations—even small variations in numbers or wording constitute a "different" problem for most deduplication methods.
2. The MATH and GSM8k benchmarks are widely discussed online, with solutions posted on forums, blogs, and tutorial sites.
3. The paper's own domain analysis (Figure 2, Section 4) shows that 40% of OpenWebMath documents are forum posts and 21% are educational content—exactly the kinds of sources where benchmark problems and their solutions are likely to appear.

The consequence. A model trained on OpenWebMath may have seen MATH or GSM8k test problems (or close variants) during training. If this model is then evaluated on those benchmarks, its performance will reflect a combination of genuine mathematical reasoning ability and memorized problem-solution pairs. This makes it impossible to determine whether OpenWebMath-trained models are genuinely better at mathematical reasoning or simply better at recalling answers they've already seen.

The contamination risk is asymmetric across the baselines in the paper's experiments. The Pile (Gao et al., 2020) also contains web data and may contain benchmark contamination at some level. ProofPile (Azerbayev et al., 2023) contains Stack Exchange and arXiv data which likely include MATH and GSM8k problems and solutions. But without explicit contamination analysis for any of these datasets, the reader cannot assess whether the performance improvements attributed to OpenWebMath in Tables 1–2 are partially or wholly driven by differential contamination rates.

This is especially concerning for the paper's MATH Algebra-Easy accuracy results (Table 2), where OpenWebMath achieves 5.62% pass@1 versus 2.81% for ProofPile and 2.81% for The Pile. If OpenWebMath happens to contain more MATH Algebra-Easy problems in its training data than the comparison datasets—perhaps because Stack Exchange and tutorial sites that discuss MATH problems were more thoroughly captured by OpenWebMath's broad web crawl than by ProofPile's curated domain list—then the accuracy improvement would partly reflect memorization rather than generalization.

What evidence exists in the paper. None. The paper does not mention contamination, does not perform any contamination analysis, and does not discuss the issue as a limitation. There is no decontamination step in the pipeline (e.g., removing documents that contain benchmark problem text, holding out benchmarks that overlap with training data). The evaluation benchmarks (GSM8k, MATH, LILA-multiarith) are standard and well-known; their potential presence in web data is a known concern in the community, but the paper does not address it.

Mitigation status. None. The paper neither detects contamination nor discusses the risk. For practitioners using OpenWebMath, this means they must either (a) perform their own contamination analysis before evaluating on standard benchmarks, (b) use benchmarks that were released after the Common Crawl snapshots used (before May 2023) and are unlikely to be in the training data, or (c) accept that benchmark performance on MATH, GSM8k, and similar widely-discussed benchmarks may be inflated by memorization. None of these mitigations are discussed in the paper.

The Manual Inspection Step Introduces Subjective Judgments That Are Neither Documented nor Reproducible

The assumption or constraint. The final stage of the OpenWebMath pipeline involves "manual inspection" of top domains, longest documents, and the most frequent sources to create domain blacklists (Section 3.7). The paper describes this step as follows:

"Finally, we manually inspect the top domains by document count, the top domains by character count, and the longest documents in the dataset to ensure that the documents are high quality. We remove domains that are not high quality or clearly not mathematical by adding domains to a blacklist and adding domain filters such as removing user profile pages, abstract-hosting websites as in Lewkowycz et al. (2022), and removing search result pages."

This describes what was done but provides no operational criteria for the manual judgments involved. What constitutes "not high quality"? What makes a domain "clearly not mathematical"? How were user profile pages identified and distinguished from legitimate content pages? How many domains were blacklisted? Were the same criteria applied consistently across all inspected documents, or did the criteria evolve during the inspection process?

The consequence. The manual inspection step is not reproducible. A different team inspecting the same top domains might make different judgments about quality and mathematical relevance, producing a different blacklist and thus a different final dataset. Even the original authors, if they reran the pipeline on new Common Crawl data, might make different subjective judgments on different days. This means that OpenWebMath as released is a specific, one-time artifact that cannot be exactly reproduced from the documented pipeline alone—the human judgments are part of the pipeline but are not part of the documentation.

This matters for several reasons. First, it undermines the paper's goal of providing a reproducible dataset that can be audited and studied—a researcher wanting to understand the composition of the dataset cannot replicate the filtering decisions that produced it. Second, it introduces unknown biases: the authors' subjective notion of "high quality" may filter out content that is mathematically relevant but fails some unstated quality criterion (e.g., poorly formatted but content-rich forum posts, mathematical content on domains that also host non-mathematical content, mathematical documents from non-traditional or amateur sources). Third, it creates a maintenance problem: if Common Crawl continues to evolve, there is no documented procedure for applying equivalent manual judgments to new data.

The blacklisting of "abstract-hosting websites," following Lewkowycz et al. (2022), is a particularly consequential judgment. Abstract pages for mathematical papers often contain the full theorem statements, definitions, and proof sketches—they may lack the full paper text but can still contain substantial mathematical content. Deciding to exclude them is a judgment about what kind of mathematical content is valuable for training, and different researchers might reasonably disagree.

What evidence exists in the paper. Section 3.7 and Tables 3–4 provide the only documentation. Table 3 (top domains by document count) and Table 4 (top domains by character count) show the domains that survived the manual inspection, but do not list the domains that were blacklisted, the number of documents or characters removed, or the specific reasons for removal per domain. The paper states that filters were added for "user profile pages, abstract-hosting websites... and search result pages" but does not specify how many domains or documents were affected. The reader cannot assess whether a small number of obviously problematic domains were removed or whether a large, qualitatively significant fraction of the dataset was subjectively filtered.

Mitigation status. The paper does not address this limitation. The open-source code release presumably includes the domain blacklist, which would allow a researcher to see exactly which domains were removed—but this only tells you what was removed, not why, and does not provide criteria for making similar judgments on new data. Future work could develop automated quality criteria that approximate the authors' manual judgments (e.g., classifier-based detection of user profile pages, abstract pages, search result pages), but the paper does not suggest or pursue this.

7. Implications and Future Directions

How This Work Changes the Landscape

OpenWebMath does not introduce a new algorithm, architecture, or training method—and it would be a category error to evaluate it as though it did. Its contribution is infrastructure: it provides the first open, large-scale, LaTeX-preserving mathematical web corpus at a scale (14.7B tokens) comparable to the proprietary dataset that enabled Minerva's landmark reasoning results. The landscape shift is therefore not a conceptual breakthrough but an access shift: research directions that were previously only possible inside organizations with proprietary web crawls and post-JavaScript DOM rendering are now available to the broader research community.

Within the genre of dataset papers, OpenWebMath makes a methodological contribution that distinguishes it from prior releases like C4, The Pile, or RefinedWeb. The paper articulates a specific diagnostic framework—what might be called the extraction-filtering feedback loop—that explains why standard web data pipelines systematically exclude domain-specific content with unusual formatting. The core insight from Appendix C, that "if a text extractor fails to extract the main text... then both the classification and perplexity filters can filter out such examples," and that "this can lead to subtle biases in the dataset, where specific poorly-extracted websites are excluded entirely even though they do contain high quality content," is not merely an observation about this particular dataset. It is a reusable methodological lens that changes how practitioners should think about building domain-specific web corpora. The prescription—validate extraction quality across a diverse sample of target content before applying quality filters, because quality filters will mask extraction failures—inverts the standard workflow and identifies a failure mode that likely affects many existing datasets whose creators never checked for it.

This is an incremental refinement to dataset construction methodology, not a paradigm shift, but it is the kind of refinement that prevents systematic errors. Its significance is that it converts a silent failure mode (extraction bias masked by filtering) into a named, diagnosable problem that future dataset papers can explicitly address or rule out.

The paper also resolves a tension that existed in the open-source community between two facts: (1) Minerva showed that large-scale mathematical web pretraining dramatically improves quantitative reasoning, and (2) no open dataset existed that could replicate this finding. The tension was not a scientific contradiction but a practical impasse—researchers knew that mathematical web data was valuable but couldn't access any. OpenWebMath resolves this by providing the dataset, but more importantly, by documenting why previous open efforts failed to produce such a dataset. The paper's identification of the JavaScript execution gap (Appendix B)—the structural disadvantage that Common Crawl's pre-rendered HTML imposes on mathematical content extraction compared to proprietary crawls that can execute JavaScript—explains why a straightforward pipeline modeled on Minerva's description would recover only ~2% of available mathematical content. This diagnostic is arguably as valuable as the dataset itself, because it tells future dataset builders in other JavaScript-dependent domains (interactive code documentation, dynamic data journalism, client-side rendered scientific content) exactly what problem they need to solve.

The research directions that become more attractive after this release are those that require open, auditable training data: memorization vs. generalization studies (you can now train models on known data and test for contamination), reinforcement learning for mathematical reasoning (you can generate on-policy rollouts without distribution shift because you have the training distribution), and the development of new reasoning benchmarks (you can calibrate against a known training corpus). The research directions that become less critical are efforts to extract mathematical content from Common Crawl using simpler pipelines—the paper has demonstrated that those approaches miss the majority of available content, and the open-source release of OpenWebMath's extraction code provides a reusable foundation that makes reinventing this machinery unnecessary.

Follow-Up Research This Work Enables

Contamination analysis of OpenWebMath against standard mathematical reasoning benchmarks. The paper performs no decontamination against MATH, GSM8k, MMLU-STEM, or any other benchmark (Section 6, Limitations). Given that 40% of OpenWebMath documents are forum posts (Figure 2) where users discuss and solve specific problems, it is highly likely that benchmark problems appear in the training data. A concrete study would: (1) perform n-gram overlap detection between OpenWebMath and each benchmark's problem statements, (2) train duplicate models on contaminated vs. decontaminated versions of the dataset, and (3) measure the performance gap to estimate what fraction of OpenWebMath's benefit comes from memorization vs. generalization. This is newly tractable because the dataset is open—prior mathematical corpora were proprietary, making contamination analysis impossible for external researchers. A strong result would quantify the contamination rate and show whether the 20× efficiency advantage over The Pile (Tables 1–2) survives decontamination. A negative result—finding that most of the accuracy gain disappears after decontamination—would fundamentally change how the community interprets OpenWebMath's value, shifting it from "teaches reasoning" to "provides benchmark coverage."

Scaling laws for domain-specific pretraining data quality vs. quantity. The paper demonstrates that 14.7B tokens of mathematical data outperform 300B tokens of general data (Tables 1–2), but this is a single data point. A systematic study would train models at multiple scales (e.g., 100M, 400M, 1.4B, 7B parameters) on multiple dataset sizes (1B, 3B, 7B, 14.7B, 30B tokens) from both OpenWebMath and a general-domain baseline, then fit scaling laws that predict downstream mathematical reasoning performance as a function of model size, data quantity, and data domain. This is now possible because OpenWebMath provides enough tokens (~14.7B) to meaningfully vary data quantity at small-to-medium model scales, and because the dataset is fixed (unlike proprietary corpora where data composition is unknown). The key question: does the 20× efficiency multiplier hold across model scales, or does general-domain data "catch up" at larger model sizes? If the multiplier shrinks with scale, the case for domain-specific pretraining weakens for large models; if it grows or remains constant, the optimal pretraining mix for reasoning-capable models should be far more domain-heavy than current practice.

Ablation of the OpenWebMath filtering pipeline to identify which components drive downstream performance. The paper explicitly acknowledges "we were only able to run our pipeline once. Therefore, many of our choices are without empirical justification and we provide no ablation study" (Appendix A). A direct follow-up would train small models (e.g., 200M–400M parameters) on variants of OpenWebMath produced by toggling individual pipeline components: (a) with vs. without the MathScore filter, (b) with vs. without perplexity filtering, (c) with vs. without SimHash deduplication, (d) varying the MathScore thresholds, (e) varying the perplexity cutoff, (f) using Resiliparse vs. jusText vs. WET extraction, and (g) including vs. excluding each LaTeX encoding format (MathML conversion, CodeCogs extraction, WordPress plugin handling). The outcome would be a ranked list of which components matter most for downstream mathematical perplexity and accuracy. This would transform OpenWebMath from a one-off artifact into a reusable, evidence-based pipeline template that practitioners can adapt to other domains (extracting chemical notation, legal documents, medical text) with confidence about which design choices are load-bearing and which are incidental.

Fine-tuning existing pretrained models on OpenWebMath to test transfer learning for mathematical reasoning. Minerva's approach was to fine-tune a pretrained model (PaLM) on mathematical data, not to train from scratch. The paper's experiments train 1.4B models from scratch on 14.7B tokens, which tests whether OpenWebMath can teach mathematical language to a randomly initialized model but does not test whether it can inject mathematical reasoning capabilities into a model that already understands general language. A straightforward experiment: take Pythia 1.4B (pretrained on 300B tokens of The Pile) and fine-tune it on 14.7B tokens of OpenWebMath vs. 14.7B tokens of The Pile (control) vs. 14.7B tokens of ProofPile (existing mathematical baseline), then evaluate on MATH, GSM8k accuracy, MMLU-STEM, and OCWCourses. If OpenWebMath fine-tuning substantially outperforms the baselines on reasoning accuracy—not just perplexity—this would validate that the dataset provides the kind of reasoning transfer Minerva demonstrated, and would give the open-source community a concrete recipe for building mathematically capable models from existing open pretrained checkpoints. If fine-tuning doesn't help, that's equally informative: it would suggest that mathematical reasoning requires exposure during pretraining rather than post-hoc injection, which would change how the community uses domain-specific data.

Extending the pipeline to extract mathematical figures and diagrams alongside text. The paper notes that OpenWebMath "only contains the text from math on the web, not associated figures" and that figures "can be important for solving mathematical problems" (Appendix A). A natural extension would: (1) identify Common Crawl WARC records that contain image payloads for the mathematical pages in OpenWebMath, (2) download and associate those images with the surrounding text using positional information from the DOM, (3) train a multimodal model (text + image) on the resulting dataset, and (4) evaluate on multimodal mathematical reasoning benchmarks (geometry problems, graph interpretation, visual proofs). This is technically challenging—Common Crawl may not reliably store images, and associating figures with their textual referents requires solving a grounding problem—but the LaTeX extraction pipeline provides the scaffolding: the DOM processing stage (Section 3.4) already identifies figure-related elements, and the extracted text contains references like "as shown in Figure 3" that could anchor image-text alignment. A successful extension would produce the first open multimodal mathematical reasoning dataset, enabling research on whether visual mathematical intuition can be learned from web data.

Replication of the pipeline on the most recent Common Crawl snapshot to measure temporal drift in mathematical web content. The paper processes Common Crawl data from before May 2023. A direct replication using the open-source pipeline on a 2024 or 2025 Common Crawl snapshot would answer: (1) Has the distribution of mathematical encoding formats shifted? (Are more sites using MathJax 3+ with different delimiter conventions? Has MathML adoption changed? Are there new equation rendering services?) (2) Does the MathScore classifier trained on pre-2023 data generalize to current web text, or does it need retraining due to vocabulary drift? (3) Do models trained on more recent mathematical web data perform differently on benchmarks that were released after the original OpenWebMath cutoff (ensuring no contamination)? This would test the pipeline's robustness to temporal distribution shift and provide a concrete estimate of how often domain-specific web datasets need to be rebuilt to remain current. A negative finding—e.g., that the MathScore classifier's accuracy degrades substantially on 2024 data—would motivate research into continual or self-updating filtering pipelines.

Practical Applications and Downstream Use Cases

Open-source mathematical reasoning models via fine-tuning of existing checkpoints. The most immediate practical use of OpenWebMath is as a fine-tuning dataset for improving the mathematical reasoning of open-source language models like LLaMA, Mistral, or Pythia. A practitioner with access to a pretrained 7B or 13B model can fine-tune on OpenWebMath (14.7B tokens, one epoch) for a modest compute cost and plausibly see improvements on mathematical reasoning tasks analogous to Minerva's gains—something that was previously impossible without proprietary mathematical data. The paper's Mixture result (Tables 1–2) provides a concrete recipe: a 50/50 mix of OpenWebMath and ProofPile outperforms either alone, suggesting practitioners should combine web-sourced and curated mathematical data rather than choosing one. The 20× efficiency advantage over general-domain data (Tables 1–2) implies that even a small budget of continued pretraining on OpenWebMath could yield noticeable improvements on mathematical benchmarks, making this accessible to academic labs and individual researchers who cannot afford large-scale pretraining.

Difficulty estimation and problem routing for mathematical tutoring systems. OpenWebMath's domain composition (40% forum posts, 21% educational content; Figure 2) makes it a rich source of real-world mathematical questions and solutions at varying difficulty levels. An educational technology company could: (1) extract question-answer pairs from the forum and educational documents in OpenWebMath, (2) use the MathScore classifier (which captures the linguistic context of mathematics without needing LaTeX) as a lightweight filter for identifying new mathematical content in user queries, and (3) build a retrieval-augmented tutoring system that matches student questions to similar problems and step-by-step solutions from the dataset. The diversity of sources (Stack Exchange, physicsforums.com, mathoverflow.net, socratic.org, brilliant.org; Table 3) means the system would cover a wide range of mathematical topics and difficulty levels, from middle-school algebra to research-level mathematics.

Pretraining data mixing for LLM providers wanting to improve reasoning without scaling model size. The paper's central empirical finding—that 14.7B tokens of mathematical data outperforms 300B tokens of general data on mathematical metrics—has direct implications for commercial LLM providers making decisions about pretraining data composition. If the goal is to improve mathematical reasoning in a model of fixed size, adding a high-quality mathematical corpus to the pretraining mix is more token-efficient than simply scaling up general-domain data. The specific guidance from the paper: mathematical web data (OpenWebMath) and curated mathematical data (ProofPile) are complementary, so providers should include both sources rather than choosing one. The finding that "there is very little overlap between the two datasets" (Section 4) means the combined corpus is genuinely larger in information content, not just in token count. This has cost implications: for a given mathematical reasoning target, the total pretraining token budget can be smaller if it includes a substantial fraction of high-quality mathematical data, potentially saving millions of dollars in compute for large-scale training runs.

Calibration corpus for developing new mathematical reasoning benchmarks. When creating a new benchmark to measure mathematical reasoning in LLMs, a fundamental challenge is ensuring that benchmark problems are not already present in the models' training data—otherwise, benchmark performance measures memorization rather than reasoning. With proprietary models trained on proprietary data, this is impossible to verify. But OpenWebMath provides a public, fixed mathematical corpus that can serve as a calibration target: benchmark creators can check their proposed problems against OpenWebMath (and other open corpora like ProofPile) to ensure they don't appear, and can release benchmarks with the explicit guarantee that "problems were verified absent from OpenWebMath and ProofPile." Models trained on these open corpora can then be evaluated on the new benchmark, and performance can be attributed to generalization rather than contamination. This doesn't fully solve the contamination problem (models may have seen the problems in proprietary data), but it creates a contamination-controlled evaluation setting for the open-source research community that was previously unavailable. Benchmark creators can also use OpenWebMath's domain distribution (Figure 2: 50% mathematics, 17% physics, 12% computer science) to design benchmarks that specifically probe generalization across mathematical subfields.