daVinci-LLM:Towards the Science of Pretraining

🎯 Pitch

A 3B-parameter model trained with a principled ‘Data Darwinism’ pipeline matches the performance of a leading 7B model, proving that deep data processing can substitute for raw parameter scale. Over 200 controlled experiments reveal that ‘how you refine data’ is as critical as ‘how much data you use,’ with different domains like math and code saturating at distinct points and requiring tailored strategies. This massive open-science effort unlocks the black box of LLM pretraining, showing that capability ceilings are set by data quality, not just model size.

1. Executive Summary

This paper systematically investigates the foundational pretraining phase to establish it as a rigorous scientific discipline, documenting the design decisions behind daVinci-LLM, a 3B-parameter model trained from random initialization on 8T tokens. Using 200+ controlled ablations against the MATH, MMLU, and HumanEval benchmarks, the work operationalizes its core Data Darwinism framework (an L0–L9 taxonomy progressing from rule-based filtering through generative refinement to cognitive synthesis) and a two-stage adaptive curriculum (shifting from general foundation pretraining on web-scale corpora to reasoning-intensive enhancement with structured QA data). The resulting model achieves an overall score of 51.72, matching the 7B OLMo-3 baseline with less than half the parameters—a ~2.3× parameter-efficiency gain—while establishing that data processing depth can substitute for naive volume scaling only when capability-specific saturation patterns guide adaptive mixture adjustments across training stages.

2. Context and Motivation

The Core Problem: Pretraining Is a Black Box Despite Being the Capability Ceiling

The paper addresses a fundamental structural gap in LLM research: the foundational pretraining phase—which determines a model's capability ceiling—remains critically under-explored as a scientific discipline. This might seem counterintuitive given the explosion of LLM papers in recent years, but the authors identify a specific paradox: almost all accessible research focuses on post-training techniques (instruction tuning, RLHF, prompting strategies, test-time compute), while the pretraining process itself—where models acquire their fundamental knowledge, reasoning patterns, and linguistic competence from scratch—is treated as an opaque, heuristic-driven craft rather than a systematically investigated science.

The paper crystallizes this problem in Section 1:

"research attention has concentrated disproportionately on accessible post-training techniques, while the foundational pretraining phase, which determines a model's capability ceiling, remains critically under-explored."

This distinction between "capability ceiling" and post-training refinement is crucial to their argument. Post-training techniques (fine-tuning, alignment, prompting) can reshape how a model expresses its knowledge, but they struggle to fundamentally overcome the capability foundations established during pretraining. The paper cites evidence [26, 27, 28] that pretraining advantages are amplified—not compensated for—in subsequent phases. If a model fails to internalize multi-step mathematical reasoning during pretraining, no amount of chain-of-thought prompting will fully bridge that gap. Conversely, a model that has acquired deep reasoning patterns during pretraining can leverage them across diverse downstream tasks.

This means that understanding pretraining dynamics isn't just an academic exercise—it's the primary lever for improving model capabilities. Yet it remains the least understood phase of the LLM lifecycle.

A Structural Paradox: Resources and Freedom Are Seldom Co-Located

Why is pretraining under-explored? The authors identify a structural constraint they describe as a "fundamental paradox" (Section 1). The two ingredients needed for systematic pretraining research—large-scale computational resources and the freedom to publish comprehensive findings—are almost never found together:

Commercial labs (OpenAI, Anthropic, Google DeepMind) possess enormous computational resources for billion-parameter training runs. However, they operate under competitive pressures that favor rapid deployment over systematic exploration. Their technical reports (GPT-4 [2], Claude [3], Gemini [5]) describe capabilities at a high level but withhold critical details: data compositions, mixture ratios, training dynamics, failed experiments, and the decision-making logic behind architectural and data choices. The authors characterize this as "API-level access" to pretraining knowledge—useful for downstream applications but useless for understanding why certain design choices work.
Open-weight releases (LLaMA [6], Qwen [7, 8, 9, 10], DeepSeek [11]) provide model checkpoints, which is a significant step toward reproducibility. However, the technical reports accompanying these releases typically disclose what data was used at a high level but not how it was processed, why specific mixture ratios were chosen, or what alternatives were tried and abandoned. Table 1 in the paper starkly illustrates this: for LLaMA 3 and Qwen 3, data composition receives only partial disclosure (◎), while processing methodology, mixture rationale, decision transparency, and negative results are completely undisclosed (✗). As the paper notes, these releases "withhold critical pretraining details—data compositions, mixture ratios, and training dynamics remain largely undisclosed."
Academic efforts (OLMo [24, 25], YuLan [66]) achieve genuine transparency and research freedom—OLMo-3, for instance, releases training code, logs, intermediate checkpoints, and data composition. However, they face "severe scale limitations that make large-scale systematic exploration structurally infeasible" (Section 1). Even well-funded academic projects cannot afford to run 200+ controlled ablation experiments at the billion-parameter scale, each consuming hundreds of billions of tokens. The paper explicitly notes that OLMo faces "persistent challenges in sustaining both computational resources and key research personnel." The result is that academic pretraining research, while transparent, cannot conduct the extensive controlled experimentation needed to isolate causal mechanisms.

Figure 2 in the paper visualizes this stratification. The y-axis represents "research depth"—from surface-level artifacts (API access) to fundamental scientific questions about capability emergence and knowledge organization. The x-axis shows temporal progression from 2022 to 2026. Commercial labs occupy the resource-rich but depth-poor region. Open-weight releases provide model artifacts but limited scientific insight. Academic efforts achieve transparency but remain at limited scale. The top tier—combining both resources and freedom to publish comprehensive findings—remains "largely unexplored." daVinci-LLM positions itself at this unexplored intersection, conducting 200+ controlled ablations with full disclosure of both successes and failures.

Where Existing Pretraining Research Falls Short

Beyond the structural constraints on who can do pretraining research, the paper identifies specific methodological gaps in how pretraining research is currently conducted:

1. No systematic framework for reasoning about data processing depth. While the field broadly acknowledges that "data quality matters," there is no shared taxonomy for how data has been processed—whether it has merely been filtered by rules, assessed by lightweight classifiers, or actively transformed by generative models. This makes it impossible to compare data sources on a common quality scale or to reason about whether further processing would yield meaningful gains. The Data Darwinism framework (L0–L9) that the paper adopts (Section 2.1) was proposed precisely to address this gap, but prior to daVinci-LLM, no large-scale pretraining effort had systematically annotated its entire corpus by processing depth or conducted controlled ablations to measure the marginal benefit of advancing from one level to the next.

2. Static data mixtures treat all training phases as equivalent. Conventional pretraining recipes apply uniform data mixtures over predetermined token budgets—for example, 70% web text, 15% code, 10% science, 5% other, held constant throughout training. This implicitly assumes all capability dimensions develop at similar rates. The paper challenges this assumption directly, hypothesizing that different capabilities exhibit systematically different saturation patterns, and that adaptive mixture adjustment—reallocating compute from saturated to actively-learning dimensions—can substantially improve training efficiency. Prior to this work, there was no systematic empirical investigation of when and how to adjust data composition during training based on measured capability trajectories.

3. The tension between capability intensification and preservation is unexplored. As pretraining incorporates increasingly specialized data (structured QA, synthetic reasoning traces), a fundamental tradeoff emerges: aggressive concentration of reasoning-dense data may improve targeted capabilities but risks "catastrophic forgetting" of general knowledge acquired earlier. The field lacks principled understanding of what concentration ratios are safe at different training stages, how to balance multiple reasoning domains (code vs. science) within intensified mixtures, and whether compositional balance can prevent performance collapse while maximizing targeted enhancement.

4. Evaluation protocols are treated as interchangeable when they probe different capabilities. Base models are commonly evaluated under two distinct protocols: perplexity-based (PPL) evaluation, which measures whether the model assigns higher likelihood to correct answers among presented options, and generative evaluation, which requires the model to actively produce answers, often through chain-of-thought reasoning. These protocols are frequently used interchangeably, but the paper argues they probe fundamentally different aspects of capability—latent knowledge access versus operational knowledge deployment—and can produce ranking reversals between models (Section 4.4, Figure 12). Understanding which protocol matches the intended deployment scenario is essential but rarely discussed in pretraining literature.

5. The complete decision-making process is invisible. Even when papers report final training recipes, the path by which those recipes were arrived at—the hypotheses tested, the configurations that failed, the boundary conditions discovered—remains undocumented. This means the community cannot learn from negative results, cannot identify which design choices matter most versus which are incidental, and cannot accumulate knowledge across pretraining efforts. Each new training run starts largely from scratch, guided by intuition and folklore rather than accumulating empirical evidence.

How This Paper Positions Itself

The paper's positioning is distinctive in several ways:

It treats openness itself as scientific methodology. Rather than viewing transparency as a nice-to-have property, the paper argues (Section 1) that comprehensive disclosure—of data processing pipelines, training trajectories, ablation results, negative findings, and decision-making rationale—is essential for pretraining to function as a cumulative science. Table 1 systematically compares daVinci-LLM against existing models across three transparency dimensions (Model Artifacts, Data Openness, Scientific Process) and twelve specific sub-dimensions. daVinci-LLM is the only model that achieves full openness (✓) across all dimensions, including the most demanding ones: processing methodology (L0-9 annotation), pretraining ablations, mixture rationale, decision transparency, and negative results.

It transforms design choices into empirically testable research questions. Rather than presenting a single optimized recipe as a fait accompli, the paper structures its exploration around verifiable questions: Does data processing depth systematically improve reasoning? Do different capabilities saturate at different rates? How should data composition evolve as training progresses? What evaluation protocol actually measures what we care about? This question-driven approach (Section 4) means the paper's value extends beyond the specific model it produces—it provides a framework for how to investigate pretraining that other researchers can adopt.

It provides both a model and a methodology. The paper's contributions span three dimensions (Section 1): (1) complete research materials (model weights, checkpoints, datasets, training logs), (2) question-driven pretraining science (empirical understanding of data quality, training dynamics, and mixture strategies validated through 200+ ablations), and (3) transferable methodological foundations (the Data Darwinism framework as reusable research infrastructure). The authors explicitly frame their work as enabling "the community to build upon documented boundary conditions and form accumulative scientific knowledge in pretraining."

It addresses the pretraining/post-training imbalance. By demonstrating that principled pretraining can produce a 3B model matching 7B baselines—a ~2.3× parameter efficiency gain without relying on post-training enhancements—the paper provides concrete evidence for its central claim: investing in pretraining science is a high-leverage activity that the field has systematically underinvested in relative to post-training research.

It challenges the assumption that scale is the primary pretraining lever. While scaling laws (Hoffmann et al., 2022) have focused attention on model size and data volume as the dominant pretraining variables, this paper argues for data processing depth and adaptive mixture design as equally important dimensions. The finding that L5 cognitive completion (synthesizing reasoning structures) can substitute for multi-fold data volume increases (Section 4.1) suggests that the scaling laws picture is incomplete without accounting for quality-driven interventions.

3. Technical Approach

3.1 Reader Orientation

This is an empirical systems paper that systematically investigates the pretraining phase of large language model development, with the core idea that pretraining design decisions—data processing depth, mixture composition, and training dynamics—can be treated as scientifically testable hypotheses rather than heuristic conventions. The paper builds a complete end-to-end pretraining pipeline for a 3B-parameter model, but its primary contribution is not the model itself; rather, it is the framework and empirical methodology for investigating pretraining as a science: using the Data Darwinism taxonomy (L0–L9) to systematically vary data processing depth, tracking capability-specific saturation patterns to time adaptive curriculum interventions, and conducting 200+ controlled ablations to isolate causal effects of individual design choices.

3.2 Big-Picture Architecture (Diagram in Words)

The system has eight major components, each described at the level of its responsibility and the information it transforms:

Data Pool (Section 2.2): A collection of raw and pre-processed corpora spanning General (4.28T tokens), Code (~598B tokens), Science (~1.94T tokens), and QA (~734B tokens) domains. Each source is annotated with its Darwin Level (L0–L9) to make processing depth explicit. The total pool is ~7.58T tokens, of which 8T are consumed across training stages.
Data Darwinism Framework (Section 2.1): A principled ten-level taxonomy (L0–L9) that categorizes all data processing operations by their operation type (selection → transformation → synthesis) and the agent driving them (hand-crafted rules → lightweight models → frontier LLMs). This is the measurement instrument—it enables systematic comparison of processing depth across heterogeneous data sources.
Processing Pipelines (Sections 2.2.2–2.2.4): Concrete implementations of specific Darwin Levels applied to individual data sources. For example, L4 Generative Refinement using Qwen3-235B-A22B-Instruct to clean scientific papers, or L5 Cognitive Completion using GPT-OSS-120B to add pedagogical bridging to academic texts. Each pipeline has a specific prompt design and quality constraint (e.g., "no external knowledge may be introduced" at L4; "strict grounding in source text" at L5).
Training Scheduler (Section 3.3): A two-stage adaptive curriculum spanning 8T tokens. Stage 1 (6T tokens) establishes broad foundations with web-scale corpora; Stage 2 (2T tokens) shifts to reasoning-intensive enhancement with structured QA. Each stage contains two substages with different mixture compositions, learning rate schedules, and batch sizes. The stage boundaries are informed by capability-specific saturation tracking.
Model Architecture (Section 3.1): A 3B-parameter decoder-only transformer based on the Qwen2 design: 36 layers, hidden size 2048, 16 attention heads with 2 KV heads (GQA), SwiGLU MLP with intermediate size 11008, RoPE positional encoding (base θ=10000), RMSNorm, vocabulary size 151936 from the Qwen2 tokenizer, sequence length 4096.
Evaluation Protocol (Section 3.2): A suite of 19 benchmarks spanning General (9 tasks), Code (3 tasks), and Science (7 tasks). Two evaluation modes are used: PPL-based (measuring likelihood of correct answers) and generative (requiring active answer production). Checkpoints are evaluated every 5k steps to track capability emergence.
Ablation Infrastructure (Section 4): The experimental machinery for conducting 200+ controlled ablations. This is not a single component but a methodology: training isolated configurations that vary a single dimension (e.g., L2 vs. L3 filtering, L4 refinement vs. baseline, 10% vs. 70% QA concentration) while holding data volume and training compute constant, to isolate causal effects.
Release Artifacts: Model weights, intermediate checkpoints (every 5k steps), processed datasets, training logs, ablation results, prompts, and processing code—all publicly released to enable reproduction and extension.

Information flow: Raw corpora → processing pipelines (applying Darwin Level operations with specific prompts/models) → annotated data pool (with Level labels) → training scheduler (selecting subsets and composing mixtures per stage) → model training (8T tokens, adaptive curriculum) → checkpoint evaluation (every 5k steps, 19 benchmarks, two protocols) → ablation analysis (isolating causal effects of each design dimension). The difficulty estimation, search, and revision components from the reference paper are not present in this paper; instead, the core adaptive mechanism is capability-specific saturation tracking that informs mixture adjustments at stage boundaries.

3.3 Roadmap for the Deep Dive

First, the Data Darwinism framework (L0–L9) — the measurement instrument that makes systematic data processing investigation possible. I explain what each level represents, the underlying evolutionary logic, and how the framework is used to annotate every data source.
Second, the complete data pool across four domains (General, Code, Science, QA) — what sources are used, how they are processed, and their corresponding Darwin Levels. This establishes the raw material from which training mixtures are composed.
Third, the processing pipelines — concrete implementations of L3 model-based filtering, L4 generative refinement, and L5 cognitive completion with specific prompts, models, and quality constraints. This is where the Darwin Levels become operational.
Fourth, the model architecture and training methodology — the two-stage adaptive curriculum, the four substages, the learning rate schedules, batch size scaling, and the specific mixture compositions at each stage.
Fifth, the evaluation protocol — the 19 benchmarks, the two evaluation modes (PPL vs. generative), and why the choice of protocol matters for interpreting results.
Sixth, the ablation methodology — how experiments are designed to isolate causal effects, what is held constant, and how the 200+ ablations map onto the four investigative themes.

This order mirrors the paper's structure and builds from foundational concepts (data taxonomy) through concrete instantiations (pipelines, mixtures) to the experimental framework that generates the findings.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical systems paper that investigates pretraining as a scientific discipline by systematically varying data processing depth, training mixture composition, and curriculum design while measuring capability emergence through 200+ controlled ablations. The core idea is that pretraining design decisions—currently treated as heuristic conventions—can be decomposed into falsifiable research questions and answered through rigorous experimentation with full transparency about both successes and failures.

The Data Darwinism Framework (L0–L9): A Taxonomy of Processing Depth

The paper adopts the Data Darwinism framework (Section 2.1), a principled ten-level taxonomy (L0 through L9) that organizes all data processing operations into a hierarchy from basic acquisition to world-scale synthesis. This framework was proposed in prior work [29] and is used here as a measurement instrument: by annotating every data source with its Darwin Level, the paper makes processing depth explicit and comparable across heterogeneous corpora, enabling systematic investigation of whether advancing from one level to the next yields measurable capability improvements.

Underlying evolutionary logic. The framework is organized around two parallel progressions that co-evolve across the levels:

Operation type: Processing begins with selection and preservation of existing content (L0–L3), moves toward active rewriting and enrichment (L4–L5), and ultimately reaches synthesis of entirely new content from scratch (L6–L9). The key philosophical distinction is between operations that only filter (remove bad content, preserving what remains) versus operations that transform (modify existing content) versus operations that create (generate new content not present in the source).
Agent driving operations: The agents shift from hand-crafted deterministic rules (L0–L2), to lightweight classification models (L3), to frontier large language models capable of reasoning and generation (L4–L9). This progression reflects increasing reliance on learned semantic understanding rather than surface-level pattern matching.

A critical design principle is that levels are not mutually exclusive one-time passes: any operation at a given level can be applied multiple times, with different models, prompts, or parameters, each targeting different aspects of quality. Furthermore, the ordering of operations need not strictly follow the level hierarchy—a dataset may undergo L4 Generative Refinement before being subjected to L3 model-based filtering if such ordering better suits the data characteristics. The levels describe what kind of operation is performed, not when it is performed relative to other operations.

L0: Data Acquisition. Raw data is gathered from diverse sources: web crawls (Common Crawl), PDF repositories (scientific papers), code platforms (GitHub), and curated databases. The collected data exists in highly variable formats (HTML, PDF, binary files) and typically contains significant noise and duplication. The primary challenges are achieving broad coverage, maintaining data provenance (tracking where each document originated), and managing large-scale storage infrastructure. No quality filtering or transformation occurs at this level—it is purely collection.

L1: Format Normalization. Heterogeneous raw data is converted into unified, training-ready text representations. For document-based sources, key operations include OCR processing of scanned PDFs (converting image-based text to machine-readable characters) and HTML parsing to extract clean content from web markup. The critical design constraint is that no content is filtered at this stage—the goal is uniform processability while preserving structural fidelity across sources. For example, when processing an HTML page, the navigation sidebar and footer are converted to text alongside the main article; they will be removed later at higher levels once semantic understanding can distinguish signal from noise.

L2: Rule-based Filtering. This is the first stage of quality control, where deterministic pattern-based rules are applied to remove objectively identifiable problematic content. The key word is "objectively"—the rules must be based on surface-level patterns that do not require semantic understanding. Operations include: near-duplicate detection via MinHash LSH (Locality-Sensitive Hashing, which efficiently identifies documents with high Jaccard similarity without pairwise comparison), removal of excessively short or malformed text, language identification to filter non-target languages, and detection of garbled text from encoding errors. This stage requires no learned models, runs efficiently on CPU infrastructure, and achieves substantial volume reduction while remaining fully interpretable—every removed document can be traced to a specific rule violation.

L3: Lightweight Model Filtering. Unlike L2, which operates on surface patterns, this stage introduces semantic-level quality assessment using pretrained lightweight classifiers. The shift from "objective" to "semantic" is crucial: L2 can remove documents based on character-level patterns (e.g., too many garbled characters), but it cannot assess whether a document has educational value, belongs to a specific domain (math vs. literature), or represents a particular document type (research paper vs. blog post). L3 classifiers perform tasks such as educational value scoring (assigning a quality score to each document), domain identification (classifying into categories like math, code, science), and document type classification, enabling more nuanced filtering decisions. Importantly, this remains a pure selection stage: documents are retained or discarded based on predicted quality, but their content is never modified. The models used are "lightweight" relative to frontier LLMs—typically fastText classifiers or small transformer models—making them computationally feasible at web scale.

L4: Generative Refinement. This stage marks a qualitative shift from selection (keep or discard) to active, model-driven transformation of existing content. Medium-to-large generative models are deployed to purify content by removing structural noise while strictly adhering to the original content. The types of noise targeted include: navigation elements (table of contents entries, page headers/footers), reference lists, OCR artifacts (garbled characters, fragmented text), and formatting defects (broken equations, inconsistent whitespace). The critical constraint at this level is that the model must act as a faithful refiner: no external knowledge may be introduced, and the output must remain semantically equivalent to the input. This constraint is what distinguishes L4 from higher levels—the model can reorganize, repair, and clarify, but it cannot add new facts, derive new conclusions, or fill in missing reasoning steps. The paper provides specific prompts for L4 refinement in Appendix C, such as the Darwin-Science L4 processing prompt which instructs the model to identify and remove reference lists, page numbers, and OCR artifacts while preserving all academic content.

L5: Cognitive Completion. At this stage, frontier LLMs enrich data by making implicit reasoning explicit. Research and technical documents are typically written for expert audiences, characterized by compressed logical steps, assumed background knowledge, and implicit derivations that create a learnability gap for language models—the model sees the compressed final form but does not observe the intermediate reasoning that produced it. L5 bridges this gap through three types of augmentation:

Reasoning Reconstruction: Expanding implicit logical leaps into explicit step-by-step derivations. When a paper states "it follows that..." or "clearly..." or "trivially..." without showing the intermediate steps, L5 generates those missing steps.
Terminological Explication: Contextualizing domain-specific terminology within the narrative flow rather than assuming prior mastery. When a term like "manifold" or "eigenvalue" appears, L5 adds brief pedagogical explanations that let the reader understand the term in context without consulting external references.
Pedagogical Bridging: Grounding abstract concepts in concrete analogies and established knowledge. An abstract mathematical structure might be explained through a physical analogy; a complex biological pathway might be connected to a more familiar everyday process.

The paper provides the full L5 Darwin-Science prompt in Appendix C, which is extensive (spanning multiple pages in the appendix). The prompt instructs the model with both "The Unbreakable Rules" (scientific correctness, structural integrity, contextual limitation) and "The Core Mandate" (deconstruct and narrate the 'why', transition from jargon to insight, invent vivid analogies, create contextual bridges). The strictest rule concerns termination: "your output must terminate exactly where the provided chunk terminates. If the chunk ends abruptly... your output must be cut off at that exact same point." This prevents hallucination of content beyond the source document's boundaries.

L6–L9: Higher-Order Synthesis. The upper levels address increasingly ambitious forms of data generation that go beyond enriching existing content. Contextual Completion (L6) expands documents by integrating external references and background knowledge to create self-contained artifacts—for example, a research paper extract might be augmented with the full content of its citations. Environment Synthesis (L7) constructs executable environments where data objects can be validated through actual execution—code with test suites that verify correctness, or scientific experiments with simulated results. Ecosystem Synthesis (L8) builds dynamic multi-agent systems where diverse intelligent entities interact and generate emergent data through sustained collaboration. World Synthesis (L9) represents the theoretical apex, aspiring to construct comprehensive simulated worlds as a source of essentially unlimited synthetic training data. The paper does not implement L6–L9; these levels are described for completeness of the taxonomy but the empirical investigation focuses on L0–L5.

The Data Pool: Composition, Scale, and Darwin Level Annotation

The pretraining corpus is organized into four categories—General, Code, Science, and QA—with each source annotated by its Darwin Level and further processed where meaningful quality gains can be achieved (Section 2.2, Table 2, Figure 3). The total data pool is approximately 7.58T tokens, of which 8T are consumed across training stages (some sources are used in multiple stages, so the pool is larger than the total consumed tokens).

General Domain (4.28T tokens, all L3). The general corpus is drawn entirely from Nemotron-CC-v1, a Common Crawl-derived dataset built from 99 snapshots spanning multiple years. The processing pipeline places it at L3: text extraction from HTML, English language filtering, and global deduplication, followed by an ensemble of classifiers that combine educational value and informativeness signals. Each document is scored and grouped into five quality tiers based on downstream task performance, enabling precise control over the quality–diversity trade-off across training stages. This tiered structure is important: rather than a binary keep/discard decision, the quality scoring allows the training scheduler to sample from higher-quality tiers more heavily during later stages while maintaining broad coverage from lower tiers during early foundation building. The total contribution is approximately 4.28T tokens, making it the dominant component of Stage 1 training (68.2% in Stage 1-1, 55.42% in Stage 1-2).

Code Domain (598B tokens, L2–L5). The code corpus spans four sources with processing depths from L2 to L5:

Self-Crawled GitHub (187B tokens, L3): Repositories are crawled directly from public GitHub with a minimum 10-star threshold as a rule-based quality gate (L2). The collected source files are then processed through OpenCoder's filtering pipeline, which applies lightweight model-based assessment to remove low-quality or non-informative code files—for example, configuration files dominated by hard-coded data, auto-generated boilerplate, or files with minimal logic. This brings the dataset to L3.
Nemotron-Pretraining-Code-v1-non-synthetic (220B tokens, L3): Real-world source code from GitHub, filtered for permissive licenses and deduplicated (both exact and fuzzy to handle cross-repository redundancy), then passed through the same OpenCoder pipeline. The license filtering is notable: it ensures the training data is legally permissible for model training, a consideration that raw web crawls often ignore.
Nemotron-Pretraining-Code-v1-synthetic-code (171B tokens, L5): LLM-generated question-answer pairs grounded in short code snippets across 11 programming languages. The model is prompted to both formulate and solve coding questions, with outputs filtered post-hoc through language-specific heuristics such as Python AST (Abstract Syntax Tree) parsing to verify syntactic validity. This is L5 because it involves synthesis of new reasoning content (question-answer pairs) from existing code, but the synthesis is grounded in and verified against executable code.
TxT360-Stack-Exchange (20B tokens, L2): Technical community discourse from 364 Stack Exchange sub-communities. Raw data is extracted from archived XML dumps, with posts and comments parsed to reconstruct the full threaded discussion hierarchy, preserving the collaborative reasoning process (questions, multiple answers, comments, accepted solutions). Only rule-based cleaning and format normalization are applied, placing this at L2.

Science Domain (1.94T tokens, L3–L5). The science corpus is the most complex in terms of processing depth, spanning multiple sources with extensive L4 and L5 processing:

MegaMath family (from L3 to L5): Three subsets from the MegaMath dataset, curated from Common Crawl using a two-stage coarse-to-fine web extraction pipeline:
- MegaMath-Web (231B tokens, L3): Mathematical content extracted via Resiliparse (fast initial pass) followed by trafilatura (high-fidelity re-extraction on math-optimized HTML), with fastText-based math filtering and MinHash LSH deduplication. A fastText classifier trained on LLM-annotated math-relevance labels provides the final quality filter.
- MegaMath-Web-Pro (13B tokens, L4): A high-quality subset selected by applying the FineMath classifier with a dynamic threshold (score ≥4 for older Common Crawl snapshots, ≥3 for recent ones), followed by LLM-driven refinement using Llama-3.3-70B-Instruct to remove noise and reorganize content into logically structured, information-dense form while preserving original length. The dynamic threshold acknowledges that older snapshots have more accumulated noise and need stricter filtering.
- MegaMath-Synth-Code (5B tokens, L5): LLM-generated natural language–code interleaved pairs targeting mathematical reasoning, with syntax and runtime verification via AST filtering and execution checks.
MegaMath Refined (176B tokens, L4): The paper applies its own L4 Generative Refinement to the full MegaMath-Web using Qwen3-235B-A22B-Instruct, with a prompt that instructs the model to "extract key facts and concepts, remove noisy or irrelevant content, and reorganize the material into a logically structured, information-dense form while preserving the original length" (see Appendix C for the full prompt, titled "Rewriting Prompt for Constructing MegaMath-Web-Pro"). The refined version reaches L4.
Nemotron-CC-Math-v1 family (from L4 to L5): Three subsets from a high-quality mathematical corpus built from 98 Common Crawl snapshots spanning 2014–2024:
- Nemotron-CC-Math-v1-3 (81B tokens, L4): Documents with FineMath classifier scores of 3, with Phi-4-based cleanup pass that removes boilerplate and normalizes mathematical representations into unified LaTeX format. The cleanup pass itself constitutes an L4 Generative Refinement operation.
- Nemotron-CC-Math-v1-4+ (52B tokens, L4): The highest-quality tier, retaining documents with FineMath scores of 4–5. Same Phi-4 cleanup pass as above.
- Nemotron-CC-Math-v1-4+-MIND (74B tokens, L5): The 4+ subset processed through the MIND framework, which converts mathematical documents into structured multi-turn dialogues using diverse conversational templates (Teacher-Student, Problem-Solving, Debate). These dialogues reconstruct implicit reasoning steps and lower the cognitive barrier for model internalization while strictly preserving original content without introducing external knowledge—this constraint is what keeps the operation at L5 rather than L6.
Nemotron-CC-Math-v1 Refined (115B tokens total, L4): The paper applies a second round of L4 refinement to the Nemotron-CC-Math-v1-3 and v1-4+ subsets using Qwen3-235B-A22B-Instruct. The key insight here is that Darwin Levels can be applied iteratively: the original Phi-4 cleanup was already L4, but a stronger model (Qwen3-235B) can produce additional quality improvements through refinement. This iterative application is explicitly noted in Section 2.1: "any operation at a given level can be applied multiple times, with different models, prompts, or parameters." The refined versions contribute 68B (v1-3 Refined) and 47B (v1-4+ Refined) tokens.
Darwin-Science (drawn from L4 and L5 processing of books and papers): This is the paper's own constructed scientific corpus, built from raw PDFs of scientific books and academic papers sourced from publicly accessible online repositories including PubMed and arXiv:
- Common L0–L3 pipeline: Raw PDFs are converted to machine-readable text using olmOCR-7B-0225-preview, a vision-language model optimized for document text extraction (this handles scanned PDFs where text is embedded in images). The resulting text undergoes deduplication via MinHash LSH, rule-based filtering (discarding files below 8KB, documents with excessive garbled characters from OCR errors, and non-English content), and annotation using EAI-Distill-0.5B, a lightweight classifier that performs educational value scoring and field-of-discipline classification across nine major domains. Documents with zero educational value are filtered out. All documents are then classified into book and paper categories using metadata where available and Qwen2.5-7B-Instruct for ambiguous cases.
- Darwin-Science-Book (251B tokens, L4): Books processed through L4 Generative Refinement using GPT-OSS-120B, which removes structural noise (table of contents, reference lists, headers/footers, OCR artifacts) while repairing formatting defects (fragmented text, damaged formulas) without altering content. The prompt (Appendix C, "Darwin-Science L4 Processing Prompt") is extensive and covers document structural deletion, academic content deletion, invalid and redundant content deletion, OCR error correction, and content protection rules.
- Darwin-Science-Paper (215B tokens, L4): Same L4 pipeline applied to academic papers using GPT-OSS-120B.
- Darwin-Science-Paper-GPT (290B tokens, L5): L5 Cognitive Completion applied to L4-processed papers using GPT-OSS-120B. The augmentation targets reasoning reconstruction, terminological explication, and pedagogical bridging. The prompt (Appendix C, "Darwin-Science L5 Processing Prompt") is the most elaborate in the paper, combining "The Unbreakable Rules" (scientific correctness, structural integrity, contextual limitation, strict termination) with "The Core Mandate" (deconstruct and narrate the 'why', transition from jargon to insight, invent vivid analogies, create contextual bridges, think like a learner, prioritize narrative flow). The termination rule is particularly strict: "You MUST terminate your output at the EXACT same point the provided chunk terminates. Do not write a single character past the end of the original chunk."
- Darwin-Science-Paper-Qwen (440B tokens, L5): The same L5 Cognitive Completion applied using Qwen3-235B-A22B-Instruct instead of GPT-OSS-120B. The authors hypothesize that "a stronger model is better equipped to reconstruct implicit reasoning and produce richer pedagogical enrichment, potentially yielding greater learnability gains" (Section 2.2.3). This creates a natural experiment: the same L5 operation performed by two different models on the same source data, enabling comparison of model strength effects on downstream training outcomes.

QA Domain (734B tokens, all L5). Question-answer pairs spanning three domains (Code, General, Science), all at Darwin Level L5 because they involve synthesis of reasoning structures. The sources include:

Nemotron-Pretraining-SFT-v1 (171B tokens): Short-form SFT-style data in code, math, and general domains. The code subset (21B) uses the Genetic-Instruct framework, an evolutionary algorithm with crossover and mutation operations that iteratively generates diverse coding problems. The math subset (138B) uses OpenMathInstruct-2, where a strong teacher model generates multiple solutions for seed questions, with high-quality solutions selected by answer correctness. The general subset (12B) consists of MMLU-style QA pairs synthesized from curated documents.
Llama-Nemotron-Post-Training-Dataset (15.4B tokens): Code, science, and math subsets. The code subset (5B) is used directly. The science and math subsets are processed through rejection sampling using Qwen3-32B in non-thinking mode: the model generates responses to prompts, and only responses whose final answers are verifiably correct against ground-truth answers are retained. This is an important quality filter—it ensures the QA pairs used for training have been validated for correctness.
MegaMath-QA-RS and MegaScience-RS (10B tokens): Additional rejection-sampled QA pairs from MegaMath and MegaScience, filtered through the same Qwen3-32B correctness verification pipeline.
Darwin-Science-Book QA (46B tokens): QA pairs generated directly from Darwin-Science Book using Qwen3-235B-A22B-Instruct with domain-specific prompts. The authors design separate prompts for each scientific discipline (biology, chemistry, computer science, engineering, humanities/social sciences, math, medicine, physics, STEM-others) to account for distinct knowledge structures and expository styles. The model is prompted to identify key knowledge points from source passages and formulate QA pairs that are strictly grounded in the original text, with the constraint that every answer must be directly verifiable against its source. The model also supplies intermediate reasoning steps to bridge questions to answers, making implicit derivations explicit. The full set of domain-specific prompts is provided in Appendix C, spanning multiple pages.
Nemotron-CC-v1 Synthetic QA (492B tokens): The largest single QA source, generated from high-quality Common Crawl documents. An LLM is prompted to generate QA pairs in multiple forms (yes/no, open-ended, multiple-choice) at different cognitive levels, probing factual information in source text. The grounding in diverse web text makes this a broad-coverage complement to domain-specific QA sources.

Token allocation across stages. Table 2 shows how this pool is allocated: Stage 1-1 (4T tokens) uses primarily General (68.2%), Code (9.53%), and Science (22.27%) with no QA. Stage 1-2 (2T tokens) reduces General to 55.42% and increases Code and Science. Stage 2-1 (1T tokens) shifts dramatically to 30% QA, 30% Code, 30% Science, 10% General. Stage 2-2 (1T tokens) intensifies QA to 70% with reduced Code (2.61%) and Science (8.85%) but sustained General (18.84%). The progressive shift from raw text to structured QA is the central curriculum design choice.

Data Processing Pipelines: Operationalizing Darwin Levels

The Data Darwinism framework provides the taxonomy; the processing pipelines provide the concrete implementations. Each pipeline is characterized by a specific model, prompt, quality constraint, and target data source.

L3 Model-Based Filtering (Code Domain). The paper's L3 filtering for code uses GPT-OSS-120B quality scoring following SeedCoder's assessment framework (Section 4.1.1). The framework identifies specific categories of low-quality code artifacts: configuration files with extensive hard-coded data (e.g., JSON files with thousands of static key-value pairs), data files dominated by constants (e.g., lookup tables with no logic), code with minimal logic (e.g., one-line wrapper functions), and auto-generated content (e.g., machine-generated boilerplate). The model scores each file, and files below a quality threshold are discarded. This is L3 rather than L2 because it requires semantic understanding of what constitutes "low-quality code"—a rule-based system cannot distinguish a legitimate data file from boilerplate, but a language model can assess whether code contains meaningful logic.

The comparison in Section 4.1.1 (Table 7) confirms this provides modest but consistent improvements over L2 rule-based filtering: HumanEval improves by +0.27 points, EvalPlus by -0.19, MBPP by +3.40, with an overall improvement of +0.27. The differential pattern—where foundational programming tasks (MBPP) benefit more than advanced algorithmic challenges (HumanEval)—is interpreted as evidence that "basic programming competence is more sensitive to training example clarity, as models acquiring fundamental patterns benefit from removing noisy artifacts."

L4 Generative Refinement (Math and Science Domains). The paper applies L4 refinement to multiple math and science sources using two strong models:

Qwen3-235B-A22B-Instruct for MegaMath and Nemotron-CC-Math datasets. The prompt (Appendix C, "Rewriting Prompt for Constructing MegaMath-Web-Pro") instructs the model to "extract key facts, concrete details, important numbers, and core concepts" while removing "irrelevant or noisy information" and reorganizing content "into a logically structured, information-dense, and concise version that is easy to learn from." A critical instruction: "Strive to maintain the original length as much as possible (avoid excessive shortening)." This constraint prevents the model from simply summarizing content—it must preserve the information density of the original while improving structure and removing noise.
GPT-OSS-120B for Darwin-Science books and papers. The Darwin-Science L4 prompt (Appendix C) is much more extensive, with detailed deletion rules organized into categories: document structural deletion (table of contents, reference lists, front/back matter, publication metadata, page headers/footers/page numbers), academic content deletion (pure indexing appendices, image files and placeholders), invalid and redundant content deletion (OCR artifacts, garbage content, duplicate content, navigation markers, URLs), and OCR error correction (text fragmentation, whitespace, character errors, punctuation, truncated words, academic formatting). The prompt also includes "Content Protection Rules" that explicitly preserve: technical content (formulas, equations, proofs, symbols), in-text references and citations, table structures, code blocks, educational content, explanatory content, and chunk boundary content. The model is instructed: "If the entire chunk should be deleted, leave the output tags completely empty."

The key design principle for L4 is the semantic equivalence constraint: the output must be a cleaner, better-structured version of the input, but it must not introduce new facts, derive conclusions, or fill in missing reasoning. The paper explicitly states (Section 2.1): "no external knowledge may be introduced, and the output must remain semantically equivalent to the input."

The empirical results in Section 4.1.2 (Table 8) show that L4 refinement of math data yields substantial but uneven gains: MATH improves by +7.00 points, while GSM8K improves by only +1.37. The interpretation is that "structural purification, specifically the transition from messy exposition to pedagogical logic, disproportionately benefits complex, multi-step reasoning. While basic word problems in GSM8K follow relatively simple and linear solution paths, the abstract problems in MATH are highly sensitive to the clarity of the underlying logical flow."

L5 Cognitive Completion (Science Domain). The L5 Darwin-Science prompt (Appendix C) is the most elaborate processing specification in the paper, combining strict rules with open-ended pedagogical transformation. The prompt frames the model as "a master science communicator and pedagogical expert" with a mission to "transform dense, expert-level text into vibrant, crystal-clear educational material." It is governed by two sets of principles:

The Unbreakable Rules (what must never be violated):

Scientific and factual correctness: all data, formulas, definitions, theories, and logical arguments must be preserved without altering meaning or context.
Structural integrity: all section headers, figure/table labels, equation numbers must be kept exactly as they appear.
Contextual limitation: the model works strictly within the provided text chunk, with no invention of definitions or reference to external goals.
Strict termination: the output must terminate at the exact point the input chunk terminates, even if mid-sentence. This is described as "the single most critical rule for preventing hallucination and ensuring continuity."

The Core Mandate (what must be actively done):

Deconstruct and narrate the 'why': expand logical leaps, explain the expert's internal monologue ("What's our goal here?" "Why use technique X for this specific job?"), detail intermediate steps.
Transition from jargon to insight: pause to explain crucial terms and variables with simpler language, analogies, and concrete examples.
Invent vivid analogies and concrete examples beyond the text.
Create contextual bridges: connect current ideas to broader fields of knowledge.
Think like a learner: proactively identify points of confusion and answer potential questions.
Prioritize narrative flow: write as "a continuous, guided tour through the material" rather than disconnected facts.

The prompt explicitly instructs the model to avoid "the overuse of overly-structured, point-by-point expressions" and to "let the main text flow logically and tell a story, adopting the persona of an extremely patient and encouraging teacher."

Model Architecture

The model architecture (Section 3.1, Table 4) follows the Qwen2 design family with a 3B-parameter configuration. The architecture is not novel—it uses standard transformer components—but the specific configuration choices reflect design principles for this scale regime:

Core configuration: 36 layers, hidden size 2048, 16 attention heads, vocabulary size 151936 tokens from the Qwen2 tokenizer, maximum sequence length 4096 tokens, precision bfloat16.

Grouped-Query Attention (GQA): 2 key-value heads shared across 16 query heads. This means the model computes attention queries for 16 heads but only 2 sets of keys and values, which are broadcast across groups of 8 query heads each. The motivation is memory bandwidth reduction: the KV cache during inference is proportional to the number of KV heads times sequence length, so GQA with 2 KV heads reduces the cache by 8× compared to full multi-head attention (16 KV heads) while preserving most of the attention expressiveness. The head dimension is 128, making the total attention dimension 16 × 128 = 2048, matching the hidden size.

SwiGLU activation in MLP layers: The intermediate size is 11008, giving an expansion ratio of 11008/2048 ≈ 5.37×. SwiGLU is a gated variant of the GLU (Gated Linear Unit) family that uses the Swish activation function for the gate. The standard formulation computes (xW₁ ⊙ Swish(xW₂))W₃ where the gate Swish(xW₂) controls information flow from the linear projection xW₁. The expansion ratio represents how much the hidden dimension is expanded in the MLP before being projected back down—higher ratios provide more representational capacity per layer at the cost of more parameters and compute.

RMSNorm for pre-normalization: Applied before both attention and MLP sublayers with epsilon = 10⁻⁶ for numerical stability. RMSNorm is a simplification of LayerNorm that removes the mean-centering step and only scales by the root-mean-square of activations, making it computationally cheaper while maintaining training stability.

RoPE positional encoding: Rotary Position Embeddings with base frequency θ = 10000. RoPE encodes position information by rotating query and key vectors in the attention computation by an angle proportional to their position. The base frequency controls the wavelength of the rotations—θ = 10000 means the lowest-frequency component has a wavelength of approximately 10000 tokens, which determines the maximum sequence length the model can effectively distinguish. The 4096 sequence length is well within this range.

Design philosophy: The "prioritizing depth (36 layers) with moderate hidden dimensions (2048)" strategy is noted as "a strategy that has proven effective for balancing parameter count, training throughput, and downstream task performance in the 3B scale regime" (Section 3.1). Deeper, narrower models generally have more sequential processing capacity (more layers of transformation) at a given parameter count compared to shallower, wider models, which benefits tasks requiring multi-step reasoning.

Training Methodology: The Two-Stage Adaptive Curriculum

The training methodology (Section 3.3) is the paper's central design contribution, implementing a progressive curriculum across 8T tokens organized into two primary stages, each with two substages. The key insight driving the stage boundaries is differential capability saturation: different capability dimensions (general knowledge, code reasoning, science reasoning) mature at different rates during training, requiring adaptive interventions to sustain growth.

Stage 1: General Foundation Pretraining (6T tokens). This stage establishes broad foundational capabilities from random initialization through diverse web-scale corpora.

Stage 1-1 (Foundation Building, 4T tokens): The first 4 trillion tokens use a progressive global batch size (GBS) scaling strategy: starting at 1,024 sequences per step for 70k steps, increasing to 2,048 for 40k steps, and reaching 4,096 for the remainder. This progressive scaling is motivated by training stability: smaller batch sizes early in training provide more frequent gradient updates and can help navigate the initial loss landscape, while larger batch sizes later improve throughput once the model has stabilized. The learning rate is held constant at 3 × 10⁻⁴ after a 2,000-step linear warmup. The AdamW optimizer uses β₁ = 0.9 and β₂ = 0.95 with weight decay 0.1. The data mixture is dominated by general web text: Common Crawl 68.2%, Code 9.53%, Science 22.27%, and no QA data.
Stage 1-2 (Reasoning Enhancement, 2T tokens): At the 4T-token boundary, the system transitions based on observed saturation patterns (Section 4.2.1, Figure 8): general knowledge benchmarks have largely plateaued within the first 1T tokens, while code and science reasoning continue to grow. The mixture is adjusted to reduce Common Crawl to 55.42% and increase Code to 11.66% (+2.13%) and Science to 32.92% (+10.65%). The batch size is fixed at 4,096, and the learning rate transitions from constant to cosine decay, decreasing from 3 × 10⁻⁴ to 3 × 10⁻⁵ over the 2T tokens. Cosine decay schedules reduce the learning rate following a cosine function from the initial to final value, meaning the rate decreases slowly at first and more rapidly near the end, allowing the model to first explore aggressively and then settle into a stable minimum.

The training dynamics (Figure 5) demonstrate stability throughout: training loss (Figure 5a) decreases smoothly from approximately 3.0 to approximately 2.1 over 300k steps, and the gradient norm (Figure 5b) remains stable between 0 and 7 throughout, with "no significant gradient spikes or loss divergences" enabling continuous training without manual intervention.

Stage 2: Reasoning Capability Enhancement (2T tokens). This stage shifts from broad linguistic modeling to structured logic acquisition, motivated by the observation (Section 4.2.1) that continued domain proportion adjustments in raw text encounter diminishing returns—once standard pretraining corpora approach saturation, merely reallocating proportions cannot sustain growth. The solution is to introduce data with explicit reasoning structures (QA pairs) that provide inherent problem-solving scaffolds.

Stage 2-1: Balanced Foundation Building (1T tokens): A balanced mixture of 30% structured QA, 30% code, 30% L4/L5-processed science, and 10% high-quality web text. This balance is designed "to prevent overfitting or domain collapse when introduced to high-intensity reasoning data" (Section 3.3.2). The learning rate uses a 2,000-step warmup to 3 × 10⁻⁵ followed by a constant schedule. The batch size remains at 4,096 with sequence length 4,096.
Stage 2-2: QA-Intensive Enhancement (1T tokens): The final trillion tokens intensify QA concentration to 70%, with code reduced to 2.61%, science to 8.85%, and general sustained at 18.84%. The learning rate remains constant at 3 × 10⁻⁵ (after empirical comparison in Appendix B shows cosine decay provides modest but consistent gains of +0.34 overall score in Stage 2-2). The mixture shift is motivated by the observation (Section 4.3.2, Figure 10) that Stage 2-1's balanced training establishes a "necessary foundation that enables the model to effectively internalize high-intensity supervision without capability loss."

Stage dependency of QA tolerance. A critical finding documented in Section 4.3.2 (Figure 11b) is that the model's tolerance for high QA concentration is stage-dependent. In Stage 2-1, increasing QA beyond 30% causes code performance to collapse (despite science continuing to improve), with general knowledge degrading sharply at 100% QA. The paper hypothesizes this is due to compositional imbalance: the Stage 2-1 QA pool is approximately 80% science-weighted, so high total QA concentrations create insufficient code sample diversity, potentially triggering "premature over-fitting or representation collapse." In contrast, Stage 2-2—building on the balanced foundation—can safely intensify to 70% QA with monotonic improvements across all domains (Table 9 shows QA-70% achieving the highest scores at every training checkpoint).

Implementation details (Appendix B). Two practical training decisions are examined:

Learning rate decay scheduling: Comparing constant LR against cosine decay (3 × 10⁻⁵ to 3 × 10⁻⁶) for Stage 2-2 with 70% QA shows cosine decay yields consistent improvements: general +0.43, code +0.99, science -0.05, overall +0.34. The benefits align with a "capability building then refinement" pattern.
QA masking policy: Comparing masking question tokens (only answer tokens contribute to loss, following SFT convention) against treating QA as continuous text (all tokens contribute) shows masking achieves only marginal gains (+0.37 overall) with high-quality data. The interpretation is that "in contrast to SFT—where masking is essential to prevent shortcut learning—pretraining's broader knowledge acquisition objective means that models benefit from learning both question understanding and answer generation when questions are sufficiently diverse."

Evaluation Protocol

The evaluation protocol (Section 3.2) spans 19 benchmarks organized into three capability domains, with two complementary evaluation strategies chosen based on task characteristics.

Benchmark domains (detailed in Appendix A):

General (9 tasks): MMLU (57 subjects, multiple-choice, 5-shot PPL), MMLU-Pro (10-option enhanced variant, 5-shot generative with CoT), AGIEval (standardized exams, 0-shot generative), HellaSwag (commonsense NLI, 0-shot generative), TriviaQA (reading comprehension, 5-shot generative), RACE (reading comprehension from exams, 0-shot generative), WinoGrande (commonsense reasoning, 0-shot generative), OpenBookQA (science QA with provided facts, 5-shot PPL), PIQA (physical commonsense, 0-shot PPL).
Code (3 tasks): HumanEval (164 Python problems, 0-shot generative), EvalPlus (stricter test suites for HumanEval, 0-shot generative), MBPP (974 basic Python tasks, 3-shot generative).
Science (7 tasks): GSM8K (grade school math, 8-shot generative with CoT), GSM-Plus (robustness-tested math, 5-shot generative), MATH (competition mathematics, 4-shot generative), GPQA-Main (graduate-level science QA, 5-shot PPL), SuperGPQA (extended graduate science, 5-shot generative), MMLU-STEM (STEM subset of MMLU, 5-shot PPL), MMLU-Pro-STEM (STEM subset of MMLU-Pro, 5-shot generative).

Two evaluation modes. The choice between PPL-based and generative evaluation is not arbitrary—it reflects which aspect of model capability is being measured (Section 4.4, Figure 12):

PPL-based evaluation is applied to multiple-choice tasks where the model scores each candidate answer directly by computing its likelihood under the model. The model does not need to produce free-form text; it only needs to assign higher probability to the correct option among the presented choices. This probes latent knowledge access: does the model know the correct answer, even if it cannot articulate it? PPL-based evaluation is used for: PIQA (0-shot), MMLU (5-shot), OpenBookQA (5-shot), GPQA-Main (5-shot), MMLU-STEM (5-shot).
Generative-based evaluation requires the model to actively produce a free-form response, often involving Chain-of-Thought (CoT) reasoning. The model must not only know the answer but also operationalize that knowledge through explicit answer production, following reasoning chains that may span multiple steps. This probes operational knowledge deployment. Generative evaluation is used for: MATH (4-shot), MMLU-Pro (5-shot), SuperGPQA (5-shot), MMLU-Pro-STEM (5-shot), GSM8K (8-shot), HumanEval (0-shot), EvalPlus (0-shot), MBPP (3-shot), AGIEval (0-shot), HellaSwag (0-shot), TriviaQA (5-shot), RACE (0-shot), WinoGrande (0-shot), GSM-Plus (5-shot).

The fundamental difference between these protocols—and why it matters—is documented in Section 4.4: models with heavier QA exposure during pretraining (like daVinci-LLM with 1T tokens of QA data in Stage 2) tend to gain disproportionately under generative evaluation because "QA-style pretraining does not merely inject factual content; it also trains the model in the behavioral format of mapping questions to explicit answers." Figure 12 shows a ranking reversal: OLMO-2-7B outperforms Qwen-2.5-3B under PPL evaluation (65.93 vs. 65.73), but the ranking reverses under generative evaluation (57.41 vs. 60.31), a 3.10% swing.

Evaluation infrastructure and tracking. All evaluations use the lm-eval-harness framework (EleutherAI). Checkpoints are evaluated every 5,000 steps throughout training to track capability emergence (Figures 4 and 6 show these trajectories). Since evaluated models are base checkpoints (not instruction-tuned), inference is performed under greedy decoding (temperature 0) to ensure deterministic, reproducible results.

Baselines. Six open base models provide comparison points at similar parameter scales: OLMo-3 7B (primary reference, 7B parameters, academic transparency), OLMo-2 7B (previous generation), Qwen-3.5-4B, Qwen-3-4B, Qwen-2.5-3B (closest architecture-matched baseline since daVinci-LLM uses Qwen2 architecture), LLaMa-3.2-3B, and Yulan-Mini-2.4B. The baselines span both scale-matched (3-4B parameter) and capability-matched (7B models with stronger absolute performance) comparisons.

Ablation Methodology

The ablation methodology (Section 4) is the experimental engine that generates the paper's scientific findings. It is characterized by a systematic, question-driven approach that isolates individual design dimensions while controlling for confounds.

Experimental design principles. The paper conducts 200+ controlled ablations, each varying a single dimension of the pretraining process while holding all other factors constant:

Data volume is held constant within comparison sets. When comparing L2 vs. L3 filtering or baseline vs. L4 refinement, both configurations train on the same total token count. This isolates the effect of processing depth from the effect of simply having more data.
Training compute is held constant within comparison sets. All configurations within an ablation are trained for the same number of tokens with the same model architecture, batch size, and sequence length. This isolates data quality effects from compute scaling effects.
Evaluation is at fixed checkpoints. Results are reported at consistent training intervals (e.g., after 500B tokens for the code filtering ablation, after 1T tokens for Stage 2 mixture ablations), enabling direct comparison at matched training budgets.
Domain-specific metrics are tracked separately. Rather than reporting only an aggregate score, all ablations report performance on General, Code, and Science domains independently. This enables detection of capability-specific effects—for example, L5 CodeQA improves code benchmarks but shows "limited cross-domain transfer to unrelated areas like general knowledge or science" (Section 4.1.3).

Four investigative themes. The ablations are organized around four research questions:

Data Processing Depth (Section 4.1): "Does data processing depth systematically improve reasoning capabilities?" Tested through controlled comparisons at each Darwin Level: L2 vs. L3 for code filtering (Table 7), baseline vs. L4 for math refinement (Table 8), and L5 synthetic QA for different domains (Figure 7). Each comparison trains from the same starting checkpoint for the same number of tokens.
Training Dynamics and Adaptation (Section 4.2): "Do different capability dimensions saturate at different rates, and can we use this to guide adaptive training?" Tested through systematic convergence tracking across Stage 1's 6T-token trajectory (Figure 8) and comparison of continued domain adjustment (Stage 1-3) against QA introduction (Stage 2) in Figure 9.
Intensification and Preservation (Section 4.3): "How to manage the intensification of reasoning data to maximize gains without compromising overall capability breadth?" Tested through systematic sweeps of code-science proportions (Figure 11a) and QA concentration ratios (Figure 11b, Table 9) at different training stages.
Evaluation Validity (Section 4.4): "Which evaluation protocols reliably measure base model pretraining progress?" Tested by comparing model rankings under PPL-based vs. generative evaluation on MMLU across three representative models (Figure 12).

Stage 1-3 as a counterfactual experiment. An important ablation not explicitly named as such is the Stage 1-3 experiment (Section 4.2.2, Figure 9). To test whether continued domain proportion adjustments could match the benefits of introducing QA data, the paper trains a Stage 1-3 configuration that further increases code and science proportions (CC 51.42%, Code 13.64%, Math 13.58%, Science 21.36%) without adding any QA. This serves as a control experiment: it isolates whether the gains in Stage 2 come from QA data specifically or merely from further reallocation of raw text proportions. The result (Figure 9) shows Stage 2 substantially outperforming Stage 1-3, establishing that structured QA—not just more reasoning-domain text—is the causal factor.

Rejection sampling as quality filter. For QA data sources that use existing post-training datasets (Llama-Nemotron, MegaMath-QA, MegaScience), the paper applies rejection sampling using Qwen3-32B in non-thinking mode. The procedure: the model generates a response to each prompt, the response is checked against ground-truth answers for correctness, and only correct responses are retained for training. This creates a quality-guaranteed QA corpus: every training example has been validated against a known answer. The filter rate is not reported, but the final token counts (e.g., 10B for MegaMath-QA-RS, 1B for MegaScience-RS) represent only verified-correct pairs.

Cross-domain QA extraction prompts. The Darwin-Science-Book QA generation uses domain-specific prompts (Appendix C, extending over multiple pages) for nine scientific domains: biology, chemistry, computer science, engineering, humanities/social sciences, math, medicine, physics, and STEM-others. Each prompt is customized to the knowledge structures of its domain. For example, the biology prompt instructs the model to create questions about "mechanisms and processes," "structure-function relationships," "phenomena and causes," "diseases," and "evolution/ecology," while the math prompt focuses on "theorem/proposition/corollary + proof" pairs and "formulas" with a strict rule: "If you see only solution steps without a question, DO NOT create a question." The medicine prompt includes a "No Data-Dependent Conclusions" rule: "Do NOT extract questions whose answers rely heavily on complex data tables, diagnostic images, or clinical charts that cannot be adequately described in text." All prompts share common constraints: zero references to source documents, complete independence of each QA pair, answer-first principle (only extract if complete answer exists), and the $ \boxed{answer} $ notation for answers requiring exact matching.

This domain-specific prompt engineering is a non-trivial design choice: it reflects the recognition that knowledge in different scientific fields is structured differently (biological mechanisms are not the same as mathematical proofs), and QA extraction must respect these structural differences to produce valid training data.

4. Key Insights and Innovations

Innovation 1: Data Darwinism as a Scientific Instrument, Not Just a Taxonomy

The paper's most distinctive intellectual contribution is not the L0–L9 taxonomy itself—which originated in prior work [29]—but rather its operationalization as a measurement instrument for causal investigation. The field has long acknowledged that "data quality matters," but this platitude has been scientifically sterile because there was no shared vocabulary for what kind of quality intervention was applied and therefore no way to measure whether advancing from one quality tier to the next produced reliable gains. The paper transforms Data Darwinism from a descriptive labeling scheme into an experimental variable: by annotating every data source with its Darwin Level and then conducting controlled ablations that vary processing depth while holding volume constant, the authors can measure the marginal return on processing investment in a way that generalizes across heterogeneous data sources.

This is fundamentally different from how prior pretraining efforts have discussed data quality. Commercial technical reports (GPT-4, Claude, Gemini) describe filtering at a high level without quantifying its impact. Open-weight releases (LLaMA, Qwen) specify data sources but rarely characterize their processing depth in comparable terms. Academic efforts like OLMo document their filtering pipeline but don't systematically vary processing depth as an independent variable. The Data Darwinism framework makes it possible to ask—and answer—questions like "Does advancing from L2 rule-based filtering to L3 model-based filtering produce gains that justify the computational cost?" and "Does L4 refinement help more for math reasoning than for basic programming?" These questions were unaskable before because there was no common yardstick for "amount of processing."

The evidence that this works as a measurement instrument comes from the systematic patterns observed across levels. L3 filtering (Table 7) produces consistent but modest improvements (+0.27 overall). L4 refinement (Table 8) delivers substantial but uneven gains (+7.00 on MATH, +1.37 on GSM8K). L5 synthesis (Figure 7) shows strong source-target alignment; its benefits are domain-specific rather than universal. This graded pattern—with each level producing qualitatively different improvement characteristics—validates that the taxonomy captures real distinctions in how data processing affects downstream capabilities, not arbitrary categorization. A taxonomy that failed to differentiate would show similar gains across all levels; the fact that it doesn't confirms its utility as a scientific instrument.

The framework's iterative nature—the explicit recognition that operations at any level can be applied multiple times with different models—is also conceptually important. The paper demonstrates this by applying L4 refinement twice to Nemotron-CC-Math: first with Phi-4 during original construction, then again with Qwen3-235B during daVinci's own processing. This treats processing depth not as a binary flag but as a continuous dimension that can be deepened through stronger models or repeated application, opening up a new axis for optimization that prior work had no vocabulary to describe.

Innovation 2: Capability-Specific Saturation Tracking as a Curriculum Design Principle

The dominant pretraining paradigm—applying a fixed data mixture for a predetermined token budget—rests on an implicit assumption that all capability dimensions develop at roughly similar rates during training. The paper presents convincing evidence that this assumption is false, and more importantly, that the differential saturation pattern itself provides the signal for when and how to adjust the training curriculum. This transforms curriculum design from a heuristic art ("add more code data at some point") into an empirically-grounded methodology ("monitor domain-specific convergence rates and reallocate compute from saturated to actively-learning dimensions when improvement plateaus are detected").

The evidence for differential saturation comes from systematic convergence tracking across Stage 1's 6T-token trajectory (Section 4.2.1, Figure 8). General knowledge benchmarks plateau within the first 1T tokens—well before the halfway point of Stage 1—while code and science reasoning sustain growth through 4T tokens and beyond. Critically, the paper demonstrates that benchmarks within each domain exhibit consistent convergence behavior: all general knowledge tasks saturate early, while all reasoning tasks maintain growth trajectories. This rules out the possibility that the pattern is an artifact of specific benchmarks and establishes that it reflects domain-level characteristics—a finding with direct implications for how practitioners should think about their training budgets.

The adaptive intervention—reducing web text proportion and increasing code/science at the 4T-token boundary (Stage 1-2)—validates the approach by producing renewed acceleration in reasoning domains while maintaining general knowledge stability. But the deeper insight comes from what happens when this strategy is pushed further. The Stage 1-3 experiment (Section 4.2.2, Figure 9) shows that continued domain proportion adjustments encounter diminishing returns: once standard pretraining corpora collectively approach saturation, merely reallocating proportions among textual formats cannot sustain growth. This negative result is as important as the positive one because it delineates the boundary condition for proportion-based curriculum adjustment and motivates the qualitative shift to structured QA data in Stage 2.

This framework—track capability-specific convergence, adjust when domains diverge, recognize when format shifts become necessary—constitutes a principled methodology for curriculum design that did not previously exist in the pretraining literature. Prior work on curriculum learning for LLMs has largely focused on simple heuristics (train on easier data first, gradually increase difficulty) or on token-level ordering within batches. This paper's approach is different: it treats the entire pretraining process as an adaptive system where intervention timing is guided by empirical measurement of learning dynamics, not predetermined schedules. The finding that Stage 2's QA introduction substantially outperforms Stage 1-3's continued proportion adjustment—+20.34 on Code, +15.75 on Science (Figure 9)—provides quantitative evidence that the choice of when to change data format is consequential, not just the choice of what data to use.

Innovation 3: Compositional Balance as the Mechanism for Capability Preservation Under Intensification

The tension between targeted capability improvement and general capability preservation is a fundamental challenge in any form of specialized training—the "catastrophic forgetting" problem familiar from continual learning and fine-tuning. In the pretraining context, aggressive concentration of reasoning-intensive data (structured QA, synthetic reasoning traces) creates a risk that the model will over-specialize, losing the broad linguistic competence and general knowledge that were the primary focus of earlier training stages. The paper provides what may be the first systematic empirical characterization of how compositional balance within intensified mixtures prevents this collapse, and more importantly, demonstrates that the safe concentration ratio is stage-dependent.

The finding that code and science must maintain internal balance within an intensified mixture (Section 4.3.1, Figure 11a) is particularly revealing. When QA is fixed at 30%, the balanced configuration (C-30-S-30) achieves superior overall performance compared to the code-heavy C-40-S-20, despite allocating more total tokens to reasoning domains (60% vs. 60%—the same total reasoning proportion). The failure mode of the unbalanced configuration is not that code improves at the expense of everything else—it's that overall capability degrades because single-domain over-specialization creates representational distortions that harm even the intensified domain. This is a more subtle finding than simple catastrophic forgetting: it suggests that the model's internal representations become qualitatively worse when training data is concentrated too narrowly, not just that untrained capabilities fade.

The stage-dependence of safe QA concentration (Section 4.3.2) is perhaps the paper's most practically actionable insight for curriculum design. In Stage 2-1, increasing QA beyond 30% causes code performance to collapse, while Stage 2-2—building on the balanced foundation established in Stage 2-1—can safely intensify to 70% QA with monotonic improvements across all domains (Table 9). The paper's hypothesis—that Stage 2-1's QA pool is compositionally imbalanced (~80% science) and that insufficient code sample diversity at high concentrations triggers representation collapse—explains why the tolerance is stage-dependent: Stage 2-1 creates the diverse representational base that makes later high-concentration training safe.

This insight reframes how practitioners should think about data mixture design. The conventional wisdom—"use diverse data to prevent overfitting"—is correct but underspecified. The paper provides the missing specificity: diversity within intensified domains matters as much as diversity across domains, and the safe concentration ratio depends on the representational foundation already established. A stage-1 model trained on 6T tokens of diverse web text can tolerate much higher QA concentration than the same model would have tolerated if QA had been introduced earlier. This has direct implications for anyone designing multi-stage pretraining curricula: the ordering of data introduction matters, and early-stage balance enables later-stage intensification.

Innovation 4: Evaluation Protocol as a Diagnostic for Pretraining Data Composition

The paper's analysis of PPL-based versus generative evaluation (Section 4.4, Figure 12) makes a conceptual contribution that extends beyond the specific model or benchmarks studied. The finding that these evaluation protocols are not interchangeable—that they probe fundamentally different aspects of model capability and can produce ranking reversals between models—is not entirely novel (the discrepancy between likelihood-based and generation-based evaluation has been noted in prior work). What is novel is the causal attribution of the discrepancy to pretraining data composition, specifically QA exposure.

The argument is that QA-style pretraining does not merely inject factual content; it also trains models in the behavioral format of mapping questions to explicit answers. Models with heavier QA exposure therefore gain disproportionately under generative evaluation, which requires exactly this skill of operationalizing knowledge through answer production. Under PPL evaluation—which only requires the model to assign higher likelihood to correct options among presented choices—this behavioral advantage is invisible. The ranking reversal between OLMO-2-7B and Qwen-2.5-3B (Figure 12), with Qwen gaining +3.10% relative ground when switching from PPL to generative evaluation, is presented as evidence for this mechanism.

The implication is that evaluation protocol choice is not a neutral technical decision—it embeds assumptions about what kind of capability is being measured, and these assumptions interact with pretraining data composition in predictable ways. For the field, this means that comparing base models evaluated under different protocols is scientifically hazardous, and that the common practice of treating PPL and generative scores as interchangeable proxies for "model quality" can lead to systematically wrong conclusions. For practitioners, it means that evaluation should be aligned with the intended deployment scenario: chatbots and QA systems should be evaluated generatively, while ranking and scoring applications should use PPL-based metrics.

This insight also serves as a diagnostic tool. If two models rank similarly under PPL evaluation but diverge substantially under generative evaluation, that discrepancy signals a meaningful difference in their pretraining data composition—specifically, differential exposure to structured QA or instruction-formatted data. The paper demonstrates this with the OLMO vs. Qwen comparison, but the principle generalizes: evaluation protocol discrepancies can serve as an investigative lens into training data characteristics that may not be disclosed in technical reports. This transforms evaluation from a purely summative exercise (how good is this model?) into a potentially informative one (what does this model's evaluation profile tell us about how it was trained?).

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper does not use a single monolithic benchmark for its primary evaluation. Instead, it uses a suite of 19 benchmarks organized into three capability domains: General (9 tasks including MMLU, MMLU-Pro, AGIEval, HellaSwag, TriviaQA, RACE, WinoGrande, OpenBookQA, PIQA), Code (3 tasks: HumanEval, EvalPlus, MBPP), and Science (7 tasks: GSM8K, GSM-Plus, MATH, GPQA, SuperGPQA, MMLU-STEM, MMLU-Pro-STEM). Detailed descriptions of each benchmark are provided in Appendix A. The paper does not report training/validation/test splits for these benchmarks—they are standard public benchmarks used in their canonical evaluation configurations.
Base model(s). The model being developed and analyzed is daVinci-LLM-3B, a 3.09B-parameter decoder-only transformer trained from random initialization on 8T tokens using the Qwen2 architecture (Section 3.1, Table 4). The architecture choice follows Qwen2's design philosophy: 36 layers, hidden size 2048, 16 attention heads with 2 KV heads (GQA), SwiGLU activation with intermediate size 11008, RoPE positional encoding with base θ=10000, RMSNorm, and the Qwen2 tokenizer with vocabulary size 151,936. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 3.1), though this claim is asserted rather than systematically validated.
Metrics. The primary metric is average accuracy across all 19 benchmarks, reported as an "Overall Average" at the bottom of Table 6. Domain-specific averages (Avg General, Avg Code, Avg Science) are also reported for all main results and ablations. For PPL-based evaluation tasks (PIQA, MMLU, OpenBookQA, GPQA-Main, MMLU-STEM), accuracy is measured by whether the model assigns highest likelihood to the correct answer among candidates. For generative evaluation tasks (all others), accuracy is measured by whether the model's greedy-decoded response matches the ground-truth answer according to the lm-eval-harness framework. MATH uses the grading function from Hendrycks et al. (2021) as noted in the Qwen2 technical report [7]; other benchmarks use their standard evaluation scripts.
Baselines. The paper compares against six open base models: OLMo-3 7B [25] (primary reference, 7B parameters, representative of academic transparency efforts), OLMo-2 7B [24] (previous generation), Qwen-3.5-4B [10], Qwen-3-4B [9], Qwen-2.5-3B [8] (closest architecture-matched baseline since daVinci-LLM uses the Qwen2 architecture), LLaMa-3.2-3B [6] (parameter-matched from Meta), and Yulan-Mini-2.4B [66] (academic effort at similar scale). These span both parameter-matched (3-4B) and capability-matched (7B) comparisons. All baselines are evaluated using the same lm-eval-harness framework with the same prompt configurations as daVinci-LLM.
Generation budget / compute accounting. The paper does not use a formal token-based or FLOPs-based compute budget for comparing methods because the primary experiments compare data processing strategies at fixed training token counts. For ablations, all configurations are trained on the same number of tokens (e.g., 500B tokens for the code filtering ablation in Section 4.1.1, 1T tokens for Stage 2 mixture ablations). This makes the comparison fair in terms of training compute: same architecture, same tokens, same batch size, differing only in the data processing variable being tested. The computational cost of producing the processed data (e.g., running L5 Cognitive Completion on scientific papers) is not amortized into the training budget and is not accounted for in any comparison—this is an acknowledged limitation.
Cross-validation / statistical protocol. The paper does not employ cross-validation for the main results. The 200+ ablations are individual training runs, each producing a single set of evaluation scores at fixed checkpoints. There is no reporting of confidence intervals, standard deviations, or statistical significance tests for any result in the paper. For the final model (Table 6), each benchmark is evaluated once on the final checkpoint. The evaluation framework (lm-eval-harness) may use internal bootstrapping for some benchmarks, but this is not reported. The absence of any statistical protocol is a genuine weakness: with 19 benchmarks and 200+ ablations, some number of "best" configurations will emerge through random variation alone, and there is no mechanism to distinguish signal from noise.

Main Quantitative Results

Final Model Performance (Table 6)

The headline result is that daVinci-LLM-3B achieves an Overall Average of 51.72 across all 19 benchmarks, matching OLMo-3 7B's 51.65 despite having less than half the parameters (3B vs. 7B). Table 6 presents the full per-benchmark breakdown:

General domain (Avg 52.96): daVinci-LLM-3B trails all Qwen models (Qwen-2.5-3B: 55.16, Qwen-3-4B: 58.60, Qwen-3.5-4B: 59.13) and OLMo-3 7B (55.13), outperforms LLaMa-3.2-3B (51.08) and Yulan-Mini-2.4B (46.74). The largest gaps relative to parameter-matched Qwen-2.5-3B appear on MMLU (62.53 vs. 65.73, −3.20) and TriviaQA (49.90 vs. 51.20, −1.30), while daVinci leads on MMLU-Pro (43.50 vs. 39.00, +4.50), HellaSwag (71.17 vs. 73.60, −2.43), and several other tasks within tight margins.
Code domain (Avg 55.99): daVinci-LLM-3B substantially outperforms all baselines except Qwen-3-4B and Qwen-3.5-4B. The key comparisons: versus OLMo-3 7B (55.99 vs. 54.42, +1.57), versus Qwen-2.5-3B (55.99 vs. 56.13, −0.14, essentially tied), versus LLaMa-3.2-3B (55.99 vs. 32.40, +23.59—a massive gap). On HumanEval specifically, daVinci achieves 61.64, exceeding OLMo-3 7B's 59.05 and approaching Qwen-2.5-3B's 60.17. On MBPP, daVinci (49.00) trails Qwen-2.5-3B (55.00) by 6 points.
Science domain (Avg 48.30): daVinci-LLM-3B exceeds OLMo-3 7B (45.98, +2.32) and substantially outperforms all parameter-matched baselines: Qwen-2.5-3B (44.65, +3.65), LLaMa-3.2-3B (22.45, +25.85). The most striking single-benchmark result is MATH: daVinci achieves 62.80, exceeding OLMo-3 7B's 39.60 by 23.20 points and Qwen-2.5-3B's 37.20 by 25.60 points. This MATH score is the single strongest piece of evidence for the paper's central claim that principled pretraining data processing can produce disproportionate reasoning gains. However, daVinci trails OLMo-3 7B on GSM8K (72.86 vs. 76.80, −3.94) and GPQA-Main (32.37 vs. 37.05, −4.68), and trails Qwen-2.5-3B on MMLU-STEM (53.41 vs. 61.91, −8.50).

Interpretation of Table 6. The overall average masks substantial domain-specific variation. daVinci-LLM-3B's strength is concentrated in reasoning-intensive benchmarks (MATH, code generation, MMLU-Pro), while it underperforms on general knowledge tasks relative to parameter-matched baselines. This pattern is consistent with the paper's training methodology: Stage 2's heavy QA emphasis and L4/L5 science processing directly target reasoning capabilities, while the reduced Common Crawl proportion in later stages may limit general knowledge acquisition. The model's ability to match a 7B model's overall score while trailing on general knowledge and dominating on reasoning suggests a specific capability profile rather than uniform improvement.

Progressive Training Results (Figures 4, 6, 8, 10)

The paper tracks performance at 5,000-step intervals throughout training, producing capability emergence curves that reveal differential saturation patterns.

Stage 1 trajectory (Figure 4, Section 3.3.1). Figure 4 displays the progressive training results across Stage 1-1 (4T tokens) and Stage 1-2 (2T tokens). The vertical dashed line marks the stage boundary. The key quantitative observations:

General knowledge (red curve) rises rapidly from approximately 10 at 0 tokens to approximately 50 by 1T tokens, then largely plateaus with minor fluctuations through 6T tokens. The paper reports this as "+34.73" for Stage 1-1 and "+4.90" for Stage 1-2.
Code (green curve) shows more gradual, sustained growth from approximately 5 at 0 tokens to approximately 35 by 6T tokens, with no clear plateau.
Science (yellow curve) similarly shows sustained growth from near 0 to approximately 30 by 6T tokens, with noticeable acceleration during Stage 1-2 following the reasoning-heavy mixture adjustment.
The overall average (blue curve) reaches 39.58 at the Stage 1 endpoint.

Figure 8 (Section 4.2.1) provides a more granular view with individual benchmark curves (light lines) and domain averages (dark lines) across three substages (1-1, 1-2, 1-3). The key quantitative markers: General average reaches 49.8 by the end of Stage 1-2, Code average reaches 34.6, and Science average reaches 29.4. The critical pattern is that individual benchmarks within each domain cluster together—all general knowledge tasks saturate early and similarly, while all reasoning tasks sustain growth—confirming domain-level rather than task-level saturation.

Stage 2 trajectory (Figure 6, Section 3.3.2). Figure 6 shows the transition from Stage 1 baseline (39.58 overall) through Stage 2-1 and Stage 2-2. The stage boundary is marked by a vertical dashed line at the point where Stage 1 ends and Stage 2 begins. The key jumps:

Stage 2-1 (1T tokens): Overall average increases from 39.58 to 48.60 (+8.97 within 1T tokens). This efficiency—nearly 9 points in 1T tokens versus approximately 30 points accumulated over the preceding 6T tokens—is attributed to the introduction of structured QA data and L4/L5-processed science content.
Stage 2-2 (1T tokens): Overall average increases from 48.60 to 51.02 (+2.43 within 1T tokens), with the model reaching 51.72 at peak (the final result reported in Table 6).
Code and Science show the largest relative gains in Stage 2-1, while General remains largely stable (consistent with its early saturation pattern).

Figure 10 (Section 4.2.2) provides the individual benchmark breakdown for Stage 2, with vertical dashed lines marking substage boundaries. General average is essentially flat throughout Stage 2 (approximately 52.1), while Code rises from approximately 30 to 52.3 and Science rises from approximately 20 to 45.4 by the end of Stage 2-1, accelerating again in Stage 2-2.

Stage 1-3 as counterfactual (Figure 9, Section 4.2.2). This is the critical controlled experiment that isolates the value of QA introduction. Starting from the Stage 1-2 checkpoint:

Stage 1-3 (continued domain proportion adjustment without QA): General −0.15, Code +2.62, Science −0.24 over the 1T-token continuation.
Stage 2 (30% QA introduction): General +2.15, Code +20.34, Science +15.75 over the same token budget.

The differential is stark: Stage 2 outperforms Stage 1-3 by approximately 17.72 points on Code and 15.99 points on Science within the same compute budget, confirming that structured QA—not merely more reasoning-domain text—drives the capability gains.

Data Processing Depth Ablations (Section 4.1)

L3 Model-Based Filtering vs. L2 Rule-Based Filtering (Table 7, Section 4.1.1). Training for 500B tokens on code data with GPT-OSS-120B quality scoring (L3) versus rule-based deduplication and pattern matching (L2):

HumanEval: L3 54.70 vs. L2 54.43 (+0.27)
EvalPlus: L3 48.39 vs. L2 48.58 (−0.19)
MBPP: L3 45.80 vs. L2 42.40 (+3.40)
Avg General: +1.16
Avg Code: +0.07
Avg Science: +0.14
Overall: 46.93 vs. 46.66 (+0.27)

The gain is concentrated on MBPP, with HumanEval nearly unchanged and EvalPlus slightly negative. The overall +0.27 improvement is modest, validating that advancing from L2 to L3 provides measurable but incremental benefits.

L4 Generative Refinement vs. Baseline Math Data (Table 8, Section 4.1.2). Training for 500B tokens comparing Qwen3-235B-refined math data against the original MegaMath and Nemotron-CC-Math baseline:

GSM8K: L4 65.43 vs. Baseline 64.06 (+1.37)
GSM-Plus: L4 42.38 vs. Baseline 40.58 (+1.80)
MATH: L4 45.00 vs. Baseline 38.00 (+7.00)
Avg General: L4 51.40 vs. Baseline 51.69 (−0.29)
Avg Code: L4 50.15 vs. Baseline 49.80 (+0.35)
Avg Science: L4 42.25 vs. Baseline 40.52 (+1.73)
Overall: 47.83 vs. 47.27 (+0.56)

The +7.00 gain on MATH is the standout result, validated across the remaining metrics showing modest improvements. The −0.29 on Avg General suggests minor trade-offs outside the target domain.

L5 Synthetic QA (Figure 7, Section 4.1.3). Comparing domain-specific L5 QA data sources in Stage 2-2:

CC-QA (general web QA): Overall rises from approximately 48 to 53 (+5.06), with Science driving most of the gain.
CodeQA (code-specific QA): Overall rises from approximately 47 to 51.3 (+4.29), with Code driving most of the gain.

The pattern shows source-target alignment: code-domain QA improves code benchmarks but provides limited cross-domain transfer; web-text QA improves science reasoning but has less impact on code. The exact per-domain breakdowns are visualized in Figure 7 but specific numerical values for individual domains are not reported in the text.

Domain Balance and QA Concentration Ablations (Section 4.3)

Code-Science internal balance (Figure 11a, Section 4.3.1). Training for 500B tokens from Stage 1 checkpoint with QA fixed at 30% and varying code-science proportions:

C-30-S-30 (balanced): EvalPlus 55.50, HumanEval 60.52, GSM-8K 64.06, MMLU 53.00 (estimated from radar plot), GPQA-Main 30.80, MATH 38.00
C-40-S-20 (code-heavy): Performance on EvalPlus is higher, but GSM-8K, MMLU, GPQA, and MATH all decline relative to C-30-S-30, with the overall average reported as lower.
Lower-concentration configurations (C-10-S-10, C-10-S-30, C-30-S-10) all substantially underperform the higher-concentration ones, validating the need for aggressive reasoning-data allocation.

The radar plot shows that C-30-S-30 achieves the most balanced polygon shape with the largest area, while C-40-S-20 creates a distorted shape emphasizing code benchmarks at the expense of science and general knowledge.

QA concentration sweep in Stage 2-1 (Figure 11b, Section 4.3.2). Training with QA ratios from 10% to 100%:

General: Stable at ~51 from 10% to 30%, then declines (50.9 at 10%, 51.4 at 30%, 48.1 at 100%)
Code: Rises from 29.9 at 10% to 31.2 at 30%, then collapses to 26.9 at 100%—a non-monotonic pattern
Science: Increases monotonically from 35.8 at 10% to 48.2 at 100%, but the paper selects 30% QA for Stage 2-1
Overall: Peaks at 51.8 at 70% QA, but this masks the code collapse, so the conservative 30% is chosen

The specific numbers from Figure 11b: 10% QA (50.9, 29.9, 35.8, 42.0), 30% QA (51.4, 31.2, 39.1, 43.6), 50% QA (51.7, 27.9, 41.8, 44.0), 70% QA (51.8, 26.1, 44.1, 45.2), 100% QA (48.1, 26.9, 48.2, 44.8), where the four numbers represent General, Code, Science, Overall respectively.

QA concentration in Stage 2-2 (Table 9, Section 4.3.2). Continuing from the Stage 2-1 checkpoint, training with 30%, 50%, and 70% QA and measuring at 84B, 252B, and 419B tokens:

QA-30%: Overall 46.80 → 46.61 → 46.94 (essentially flat across token budgets)
QA-50%: Overall 47.70 → 48.06 → 48.43 (consistent improvement with more tokens)
QA-70%: Overall 48.17 → 49.13 → 49.84 (highest at every checkpoint, strongest growth)

At 419B tokens: Code (QA-70%: 52.40 vs. QA-50%: 49.89 vs. QA-30%: 47.16), Science (QA-70%: 45.77 vs. QA-50%: 43.42 vs. QA-30%: 40.59), General (QA-70%: 52.16 vs. QA-50%: 51.83 vs. QA-30%: 51.80). Critically, in Stage 2-2, Code does not collapse at high QA concentrations—unlike in Stage 2-1—validating the hypothesis that Stage 2-1's balanced foundation enables safe later intensification.

Evaluation Protocol Comparison (Figure 12, Section 4.4)

Comparing PPL-based vs. generative (CoT) evaluation on MMLU for three models:

LLaMa-3.2-3B: PPL 54.91, Generative 49.01 (gap: −5.90)
Qwen-2.5-3B: PPL 65.73, Generative 60.31 (gap: −5.42)
OLMO-2-7B: PPL 65.93, Generative 57.41 (gap: −8.52)

Under PPL evaluation: OLMO-2-7B (65.93) > Qwen-2.5-3B (65.73). Under generative evaluation: Qwen-2.5-3B (60.31) > OLMO-2-7B (57.41). This represents a ranking reversal with a 3.10% swing in relative performance. The gap relative to Qwen-2.5-3B (Figure 12b) shows OLMO-2-7B going from +0.20% (PPL) to −2.90% (generative), while LLaMa-3.2-3B goes from −10.82% (PPL) to −11.30% (generative), a smaller swing.

Training Implementation Decisions (Appendix B, Table 10, Figure 13)

Learning rate decay (Table 10). Comparing constant LR vs. cosine decay (3e-5 to 3e-6) for Stage 2-2 with 70% QA:

Constant: General 52.06, Code 52.32, Science 45.44, Overall 49.66
Cosine decay: General 52.49, Code 53.31, Science 45.39, Overall 50.00
Delta: +0.43, +0.99, −0.05, +0.34

Cosine decay provides consistent but modest gains (+0.34 overall), with most benefit in Code.

QA masking policy (Figure 13). Comparing masking question tokens against treating QA as continuous text in Stage 2-2:

At 84B tokens: Masked achieves approximately 51.0 General, 49.0 Code, 41.5 Science; Unmasked achieves nearly identical values.
At 503B tokens: Masked achieves approximately 52.5 General, 52.5 Code, 44.0 Science; Unmasked is marginally lower across all three.
The paper reports that at 30k steps, masking achieves 49.14 overall vs. 48.77 without masking (+0.37), with per-domain gains of +0.38 (General), +0.35 (Code), +0.35 (Science).

The marginal gains mean that "question quality and diversity matter more than masking strategy" (Appendix B).

Ablation Studies and Robustness Checks

Stage 1-3 as counterfactual for QA introduction (Figure 9): The most critical ablation. Continued domain proportion adjustment without QA yields negligible improvements (+2.62 Code, −0.15 General, −0.24 Science), while QA introduction yields +20.34 Code, +2.15 General, +15.75 Science over the same token budget. This confirms that the gains attributed to QA are not achievable through further proportion optimization, establishing a causal role for data format.
Code internal balance under QA intensification (Figure 11a): The balanced C-30-S-30 configuration achieves superior overall performance to the code-heavy C-40-S-20 despite allocating the same total proportion to reasoning domains (60%). This demonstrates that internal balance within intensified mixtures matters independently of total allocation—over-specialization harms overall capability even when the intensified domain would be expected to benefit.
Stage-dependent QA tolerance (Figure 11b vs. Table 9): In Stage 2-1, code performance collapses beyond 30% QA (dropping from 31.2 at 30% to 26.1 at 70%); in Stage 2-2, code performance monotonically improves with QA concentration (rising from 47.16 at 30% to 52.40 at 70% at 419B tokens). This is a non-trivial finding—it means the same QA proportion has qualitatively different effects depending on when in training it is introduced, validating the stage-dependent curriculum design.
L5 synthesis exhibits domain-specific steering (Figure 7): CodeQA improves Code benchmarks but shows limited cross-domain transfer to Science; CC-QA improves Science but impacts Code less. This confirms that L5 synthesis is a precision tool for targeted capability development rather than a universal booster.
Learning rate schedule (Table 10): Cosine decay provides +0.34 overall improvement over constant LR in Stage 2-2. The benefits are concentrated in Code (+0.99) with Science essentially unchanged (−0.05), suggesting the decay schedule's benefits are domain-specific rather than uniform.
QA masking (Figure 13): Masking question tokens provides only marginal gains (+0.37 overall) when QA data is already diverse and high-quality, with training dynamics remaining stable under both strategies. This suggests that masking—essential in SFT to prevent shortcut learning—is less critical in pretraining where broader knowledge acquisition objectives dominate.
L3 vs. L2 filtering for code (Table 7): The +0.27 overall gain is modest, but the differential pattern—MBPP (+3.40) benefiting more than HumanEval (+0.27)—is informative: basic programming competence is more sensitive to example clarity than advanced algorithmic reasoning, suggesting filtering benefits are task-complexity-dependent.
L4 refinement differential impact (Table 8): The +7.00 on MATH vs. +1.37 on GSM8K confirms that content purification benefits complex multi-step reasoning more than simpler problem-solving, consistent with the hypothesis that MATH's abstract problems are more sensitive to logical flow clarity than GSM8K's linear word problems.
Negative result: ReST^EM revision model degrades performance (Appendix K, Figure 16): An attempt to further optimize the revision model using ReST^EM (on-policy RL-based training) causes sequential revisions to substantially hurt performance. At 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio. This demonstrates sensitivity of revision training to data generation methodology and the non-triviality of combining on-policy optimization with pretraining data processing.

Critical Assessment

The experiments in this paper collectively provide substantial empirical support for its framework-level claims, but several important caveats and potential overclaims require examination.

Claim 1: Processing depth systematically enhances capabilities, establishing it as a critical dimension alongside volume scaling.

This claim is supported within the specific context tested—mathematical and scientific reasoning data processed through L4 generative refinement and L5 cognitive completion with frontier LLMs—but the evidence is narrow. Only three transitions are tested: L2→L3 for code filtering (Table 7, +0.27 overall), baseline→L4 for math data (Table 8, +0.56 overall, +7.00 on MATH), and L5 QA synthesis for code and general domains (Figure 7, +4–5 overall per domain). The gains are real but the magnitudes vary dramatically: the MATH gain is enormous and domain-specific; the code filtering gain is marginal. "Systematically enhances capabilities" overstates the uniformity of the effect. Moreover, all L4 and L5 processing uses frontier models (GPT-OSS-120B, Qwen3-235B-A22B-Instruct) that themselves cost enormous compute to run—the cost of processing in these experiments likely exceeds the cost of training the 3B model on the resulting data, yet this is never accounted for. The paper demonstrates that better-processed data improves training outcomes, but does not establish that this is an efficient tradeoff relative to simply collecting more data of the original quality.

What would strengthen this claim: A cost-benefit analysis that amortizes processing compute into the training budget. A comparison showing that 500B tokens of L4-refined data outperforms 1T tokens of baseline data—demonstrating the "substitute for volume scaling" claim explicitly rather than implicitly. Tests on non-reasoning domains (general web text, dialogue, creative writing) to establish whether processing depth benefits are reasoning-specific.

Claim 2: Different domains exhibit distinct saturation dynamics, necessitating adaptive strategies from proportion adjustments to format shifts.

This is the strongest-evidenced claim in the paper. The saturation tracking in Figures 4, 6, 8, and 10 consistently shows general knowledge plateauing by 1T tokens while code and science sustain growth through 6T+ tokens. The Stage 1-3 vs. Stage 2 comparison (Figure 9) cleanly demonstrates that proportion adjustment alone cannot sustain growth and that format shifts become necessary. The evidence is internally consistent across multiple figures and stages. The main limitation is that this pattern is demonstrated for a specific model architecture (Qwen2-3B), a specific data mixture, and a specific benchmark suite—whether the same saturation hierarchy (general knowledge saturating first, reasoning later) holds across architectures, scales, and domains is an open question that the paper does not address.

Claim 3: Compositional balance enables targeted intensification while preventing performance collapse.

Figure 11a and Table 9 collectively support this claim with quantitative specificity. The finding that code collapses at high QA in Stage 2-1 but not Stage 2-2 is a robust stage-dependence effect. However, the mechanism proposed—"compositional imbalance of the QA corpus"—is a post-hoc explanation, not a tested hypothesis. The paper hypothesizes that the Stage 2-1 QA pool is ~80% science, causing code collapse at high concentrations due to insufficient code sample diversity. But no experiment varies the internal composition of the QA pool to test this mechanism directly (e.g., training with a code-balanced QA pool at 70% concentration in Stage 2-1). Without this control, the attribution to compositional imbalance is plausible but unverified.

What would strengthen this claim: An ablation in Stage 2-1 using a QA pool with balanced code/science proportions to test whether the code collapse is indeed caused by QA composition rather than by some other property of Stage 2-1 training (e.g., the learning rate, the model's developmental stage). A direct comparison of equal-proportion vs. science-heavy QA in Stage 2-1 at high concentrations.

Claim 4: Evaluation protocol choices shape our understanding of pretraining progress.

Figure 12 demonstrates the PPL vs. generative discrepancy convincingly for MMLU, with a 3.10% ranking reversal between OLMO-2-7B and Qwen-2.5-3B. The mechanism proposed—QA exposure trains models in answer-production behavior—is plausible and consistent with the observed pattern (Qwen, which likely has more QA exposure in its training, gains under generative evaluation). However, the paper only tests this on one benchmark (MMLU) with three models, none of which are daVinci-LLM itself. The paper does not report PPL vs. generative discrepancies for daVinci-LLM-3B's own evaluation, nor does it systematically vary QA exposure in a controlled setting to demonstrate the causal link. The claim that "QA-style pretraining... trains the model in the behavioral format of mapping questions to explicit answers" is asserted based on correlational evidence.

What would strengthen this claim: A controlled experiment training two models with identical data except one has QA exposure and one does not, then comparing PPL vs. generative discrepancies. Reporting PPL vs. generative discrepancies across all 19 benchmarks for daVinci-LLM-3B itself. Testing whether the discrepancy is proportional to the amount of QA data in training (dose-response relationship).

Genuine weaknesses in the experimental design:

Single model architecture, single scale. All experiments use the Qwen2-3B architecture at exactly 3B parameters. There is no evidence that the findings—differential saturation patterns, optimal QA ratios, L4/L5 benefit magnitudes—generalize to other architectures (e.g., LLaMA, OLMo) or scales (e.g., 1B, 7B, 13B). The paper claims daVinci-LLM is "representative," but provides no evidence.
No statistical rigor. With 200+ ablations, 19 benchmarks, and no confidence intervals, standard deviations, or significance tests reported, the likelihood that some "best" configurations emerge through random variation is non-trivial. The paper's core claims rely on comparing point estimates that could differ by ±1-2 points due to training stochasticity alone, yet this uncertainty is never quantified.
The 7B model comparison is not FLOPs-matched. daVinci-LLM-3B matching OLMo-3 7B is impressive, but the paper does not compare against what a 3B model could achieve with the same total compute as the 7B model's training. The OLMo-3 7B was presumably trained on more total tokens (the paper does not report this), and daVinci-LLM-3B's 8T tokens represent a specific training budget. A proper efficiency comparison would control for total FLOPs, not just parameter count.
Data processing cost is externalized. The L4 and L5 processing pipelines use massive frontier models (GPT-OSS-120B, Qwen3-235B-A22B) to process terabytes of text. The compute for this processing is not amortized into any comparison. A realistic deployment accounting would show that the processing cost may dominate the training cost for a 3B model.
Benchmark decontamination is not discussed. The paper uses public benchmarks (MATH, MMLU, HumanEval) that may appear in the training data of the frontier models used for L4/L5 processing (GPT-OSS-120B, Qwen3-235B). If these models have memorized benchmark content, the L5 Cognitive Completion could inadvertently inject benchmark answers into training data, inflating scores. The paper mentions "benchmark decontamination" in the description of Nemotron-CC-Math-v1 but does not describe applying it to Darwin-Science or QA sources.
Only one random seed. All training runs appear to use a single random initialization. Training the same architecture with different seeds could produce performance variation of 1-3 points on individual benchmarks; without seed replication, it is impossible to distinguish true signal from initialization noise.
The "overall average" metric masks domain tradeoffs. Averaging across 19 heterogeneous benchmarks treats a 1-point gain on PIQA as equivalent to a 1-point gain on MATH, even though these represent very different levels of difficulty and practical significance. The choice of equal weighting is never justified. A domain-weighted or difficulty-weighted metric might produce different rankings.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted for in Reported Gains

The assumption or constraint. The paper's core adaptive mechanism—adjusting data mixtures and curriculum stage boundaries based on capability-specific saturation patterns—requires tracking benchmark performance at 5,000-step intervals across 19 benchmarks throughout the entire 8T-token training trajectory. This evaluation infrastructure is integral to the methodology: without it, the differential saturation patterns that motivate Stage 1-2's mixture adjustment (Section 4.2.1, Figure 8) and the Stage 1-3 vs. Stage 2 comparison that validates QA introduction (Section 4.2.2, Figure 9) would be invisible. The paper treats this evaluation cost as external to the training budget, but in practice, evaluating 19 benchmarks at 5k-step intervals over 8T tokens means running thousands of evaluation passes—each requiring forward passes on held-out data—which adds non-trivial computational overhead relative to the training itself, particularly for a 3B model where evaluation costs are proportionally larger than they would be for larger models.

The consequence. The reported efficiency gains—most prominently, the claim that daVinci-LLM-3B matches a 7B model with less than half the parameters—are computed after the adaptive curriculum has been designed using full-trajectory evaluation data. A practitioner attempting to replicate this methodology would need to either (a) run the same expensive evaluation infrastructure, adding substantial overhead to the total compute budget, or (b) design a cheaper proxy for capability saturation that may not capture the same patterns. The paper does not quantify this evaluation cost, does not amortize it into any efficiency comparison, and does not demonstrate that the key curriculum decisions (the 4T-token boundary for Stage 1-2, the transition to QA at Stage 2, the 30%→70% QA intensification) could have been identified with sparser evaluation or cheaper proxies. This makes the headline efficiency numbers an upper bound—the true cost of the methodology is higher than reported, potentially substantially so for smaller models where evaluation overhead is proportionally larger.

What evidence exists in the paper. The paper explicitly acknowledges a related cost concern in a different context. In Section 4.1, after presenting the L4 Generative Refinement results (Table 8), it notes that the processing cost of running frontier models for data refinement is not amortized into the training budget. However, the evaluation cost is never mentioned as a limitation. The paper reports checkpoint evaluations at 5k-step intervals in Figures 4, 6, 8, and 10, demonstrating that this dense evaluation was actually performed, but it never quantifies the FLOPs or wall-clock time consumed by these evaluations relative to training. The lm-eval-harness framework used for evaluation (Section 3.2) typically runs multiple forward passes per benchmark (for few-shot examples and candidate scoring), and with 19 benchmarks, this constitutes a meaningful fraction of the forward passes consumed by training itself—but this fraction is never estimated.

Mitigation status. Not addressed. The paper does not discuss evaluation cost, does not propose cheaper proxies for capability saturation tracking, and does not report what fraction of total compute was spent on evaluation versus training. A natural mitigation would be to demonstrate that the key curriculum decisions are robust to sparser evaluation (e.g., checking benchmarks every 50k steps instead of 5k, or using a subset of representative benchmarks rather than all 19). This is left entirely to future work.

All Results Are on a Single Model Architecture at a Single Scale

The assumption or constraint. Every experiment in the paper—the 200+ ablations, the saturation tracking, the L4/L5 processing benefit measurements, the QA concentration sweeps, the Stage 1-3 vs. Stage 2 comparison—is conducted on a single model: a 3B-parameter Qwen2-based architecture trained on 8T tokens. The paper states in Section 3.1 that the authors "believe this model is representative of the capabilities of many contemporary LLMs," but this is an assertion, not a finding. The architecture uses specific design choices (Grouped-Query Attention with 2 KV heads, SwiGLU activation, RoPE with base θ=10000, RMSNorm, 36 layers with hidden size 2048) that may interact with the data processing and curriculum design choices in unknown ways.

The consequence. The paper's central empirical claims—that processing depth systematically enhances capabilities, that different domains exhibit distinct saturation dynamics, that compositional balance enables targeted intensification without collapse—are all conditional on this specific model configuration. Several failure modes are possible:

A different architecture might exhibit different saturation timescales for different domains, changing the optimal stage boundaries. A wider, shallower model (e.g., 24 layers with hidden size 2560) might plateau differently than daVinci-LLM's deep-narrow design.
The interaction between data processing depth and model scale is unknown. L4 Generative Refinement and L5 Cognitive Completion use frontier models (GPT-OSS-120B, Qwen3-235B-A22B) to enrich data. At larger model scales (7B, 13B, 70B), the marginal benefit of this enrichment might shrink because larger models are better at extracting patterns from raw data—or it might grow because larger models can better leverage enriched reasoning structures. The paper provides no evidence either way.
The Qwen2-specific tokenizer (vocabulary size 151,936) may benefit differentially from certain data sources compared to tokenizers used in other model families (LLaMA, OLMo), affecting the measured domain-specific improvements.
The "representativeness" assumption is particularly questionable for the QA concentration findings (Section 4.3.2). The non-monotonic code performance in Stage 2-1 (peaking at 30% QA, collapsing at higher concentrations; Figure 11b) and the stage-dependent tolerance could be artifacts of this specific architecture's training dynamics rather than general principles.

What evidence exists in the paper. None. The paper does not include any experiments with alternative architectures, different model scales, or different tokenizers. The comparison baselines (Table 6) include models with different architectures—LLaMA-3.2-3B, OLMo-3 7B, Qwen-2.5-3B—but these are evaluated as final products, not used to test whether the methodology (adaptive curriculum driven by saturation tracking, L4/L5 processing) transfers. No ablation varies the architecture while holding the data processing constant to measure interaction effects.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, does not discuss architecture-dependence of any findings, and does not suggest multi-architecture validation as future work. Given that the paper's stated goal is to advance "the science of pretraining" with "transferable methodological foundations" (Section 1), the single-architecture evaluation is a significant gap in establishing transferability.

Data Processing Compute Costs Are Externalized from All Efficiency Comparisons

The assumption or constraint. The L4 Generative Refinement and L5 Cognitive Completion pipelines use frontier-scale models—GPT-OSS-120B and Qwen3-235B-A22B-Instruct—to process terabytes of training data. The paper's Darwin-Science L5 processing (Section 2.2.3), for instance, applies an elaborate prompt (Appendix C, spanning multiple pages) to scientific papers, generating pedagogically enriched text that is "more detailed and thus longer than the original" by design. The compute cost of running these massive models on this scale of data—processing 290B tokens of Darwin-Science-Paper-GPT, 440B tokens of Darwin-Science-Paper-Qwen, 492B tokens of Nemotron-CC-v1 Synthetic QA, and so forth—likely exceeds the compute cost of training the 3B model itself on the resulting data. Yet none of this processing cost is amortized into any efficiency comparison. The paper treats processed data as an input whose production cost is external to the system.

The consequence. The paper's central economic argument—that data processing depth can substitute for naive volume scaling—is evaluated in a setting where processing cost is invisible. A practitioner deciding whether to apply L5 Cognitive Completion to their pretraining corpus faces a concrete resource allocation question: given a fixed total compute budget, should they spend it on (a) processing existing data to higher quality using frontier models and training on less total data, or (b) training on more data of lower processing depth? The paper's ablation in Section 4.1 answers the narrow question "does L4-refined data produce better models than unrefined data at the same training token count?" but cannot answer the broader economic question because the cost of producing the refined data is not accounted for. If L4 refinement costs 10× more compute per token than the training itself, then a practitioner with a fixed budget might get better results from training on 10× more unrefined tokens—a comparison the paper never makes.

This is particularly acute for the L5 Cognitive Completion results (Section 4.1.3, Figure 7). The +5.06 overall gain from CC-QA and +4.29 from CodeQA in Stage 2-2 are achieved by adding synthetically generated QA data. But generating 492B tokens of CC-QA using frontier models is enormously expensive. The paper never reports what fraction of total system compute (processing + training) was consumed by data generation, making it impossible to assess whether these gains are cost-effective.

What evidence exists in the paper. The paper acknowledges this implicitly in Section 4.1's takeaway box but never quantifies it:

"These findings establish a systematic pathway from filtering to synthesis for data-centric capability improvement."

The word "systematic pathway" describes the method but not the cost. The ablation results in Tables 7-8 and Figure 7 all hold training tokens constant while varying data processing, meaning the comparison is fair for measuring the effect of processing but not for measuring its efficiency relative to other uses of the same total compute budget. The paper never reports FLOPs for data processing, never compares "L4-refined data at N tokens" against "unrefined data at 2N tokens," and never discusses the break-even point where processing cost exceeds training benefit.

Mitigation status. Not addressed. The paper does not report processing compute costs, does not include processing cost in any budget calculation, does not compare equal-compute configurations across processing levels, and does not discuss this as a limitation. This is particularly notable because the paper's framing—"processing depth can substitute for multi-fold data volume increases" (Section 4.1 takeaway)—directly invokes an economic substitution argument without providing the economic data needed to evaluate it.

Benchmark Contamination Risk from Frontier Models Used in Data Processing Is Not Assessed

The assumption or constraint. The L4 and L5 processing pipelines rely on frontier LLMs—GPT-OSS-120B and Qwen3-235B-A22B-Instruct—to transform training data. These models were themselves trained on large-scale web corpora that almost certainly include content from the benchmarks used for evaluation (MATH, MMLU, HumanEval, GSM8K, etc.). When these models perform L5 Cognitive Completion on scientific papers, they are instructed to "invent vivid analogies and concrete examples" and "deconstruct and narrate the 'why'" (Appendix C, Darwin-Science L5 Processing Prompt). If a frontier model has memorized benchmark problems or solutions from its own training, it could inadvertently inject benchmark content—rephrased as "analogies" or "pedagogical explanations"—into the pretraining data for daVinci-LLM-3B. The paper assumes that its processing pipelines are contamination-free but never verifies this.

The consequence. If benchmark contamination occurred during L4/L5 processing, daVinci-LLM-3B's evaluation scores would be inflated relative to its true generalization capability. This would affect every result in the paper, but would be most problematic for the claims that depend on large-magnitude improvements on specific benchmarks:

The +7.00 MATH gain from L4 refinement (Table 8) and the final MATH score of 62.80—a 23-point lead over OLMo-3 7B—are the paper's strongest single-benchmark results. If the L4/L5 processing inadvertently introduced MATH-like content into training data, these gains would partially reflect memorization rather than genuine reasoning improvement.
The +15.75 Science gain from QA introduction in Stage 2 (Figure 9) and the sustained growth on science benchmarks throughout Stage 2 (Figure 10) could be partially driven by benchmark-relevant content in the 1T tokens of L5-processed science data and structured QA.

The risk is not hypothetical: the frontier models used for processing (GPT-OSS-120B, Qwen3-235B-A22B) are known to have strong performance on these benchmarks, which implies exposure during their own training. The L5 prompt explicitly encourages generating "concrete examples" and "vivid analogies"—exactly the kind of generation where a model might draw on memorized benchmark content.

What evidence exists in the paper. The paper mentions "benchmark decontamination" exactly once, in the description of Nemotron-CC-Math-v1 (Section 2.2.3): the dataset's pipeline "uses the FineMath classifier to score each document... followed by fuzzy deduplication and benchmark decontamination." But this decontamination is performed at the source dataset level during original construction—it is not applied by the daVinci-LLM team to their own processed data. There is no description of any decontamination procedure applied to Darwin-Science L4/L5 outputs, QA synthesis from Darwin-Science-Book, or MegaMath Refined. The paper provides no n-gram overlap analysis, no embedding similarity search, and no discussion of contamination risk as a limitation.

Mitigation status. Not addressed. The paper does not mention contamination risk from processing models, does not describe any decontamination procedure for processed data, and does not report any analysis (e.g., n-gram overlap, benchmark substring matching) that would detect contamination. This is a significant omission given that the processing pipelines are explicitly designed to expand and enrich content in ways that could mask contamination (rephrased benchmark problems embedded in "pedagogical explanations" would not be caught by simple exact-match decontamination).

No Statistical Rigor for Distinguishing Signal from Noise Across 200+ Ablations

The assumption or constraint. The paper reports results from 200+ controlled ablations, each producing evaluation scores across up to 19 benchmarks, with many ablations varying hyperparameters continuously (QA concentration from 10% to 100% in Figure 11b, code-science proportions across five configurations in Figure 11a, QA ratios across three levels and three training checkpoints in Table 9). None of these results include confidence intervals, standard deviations, standard errors, or any form of statistical significance testing. The paper does not report whether training runs were replicated with different random seeds, and all reported numbers are point estimates from single training runs. This means that observed differences between configurations—the +0.56 overall gain from L4 refinement (Table 8), the +0.27 gain from L3 filtering (Table 7), the +0.34 gain from cosine decay (Table 10)—cannot be distinguished from sampling noise.

The consequence. With 200+ ablations and no statistical rigor, two problems arise:

False positives from multiple comparisons: Testing many configurations and selecting the best-performing one without multiplicity correction will identify some configurations as "best" purely through random variation. The paper's methodology—training many configurations and then selecting the one with highest performance—is essentially a grid search without statistical guards. The compute-optimal policy selection in Section 4.3 (choosing 30% QA for Stage 2-1, 70% for Stage 2-2) is based on the highest point estimate among tested configurations, but if the true performance difference between 30% and 50% QA is small relative to training noise, the "optimal" choice may reflect the particular random seed rather than a genuine superiority of that configuration.
Unreliable magnitude estimates for small effects: The paper draws conclusions from small-magnitude differences: the +0.27 overall gain from L3 filtering (Table 7) is presented as evidence that "L3 model-based filtering provides consistent but modest improvements." But with a single training run and no estimate of variance, a +0.27 difference on an aggregate score derived from 19 benchmarks could easily be within the noise floor. The same applies to the +0.34 gain from cosine decay (Table 10) and the +0.37 gain from QA masking (Appendix B). These effects may be real, but the paper provides no evidence that they are distinguishable from zero.

This is not an abstract concern: training large language models is inherently stochastic due to random initialization, data shuffling, and dropout. Prior work has shown that identical training runs with different random seeds can produce evaluation score differences of 1-3 points on individual benchmarks. The paper's "overall average" metric aggregates 19 benchmarks, which reduces variance but does not eliminate it, and the per-benchmark results that are used to diagnose domain-specific effects (e.g., the MBPP +3.40 gain from L3 filtering in Table 7) have no associated uncertainty.

What evidence exists in the paper. The absence of statistical reporting is itself the evidence. No table or figure in the paper reports any measure of uncertainty. The training dynamics figures (Figures 4, 5, 6, 8, 10) plot single trajectories without error bands. The radar plot (Figure 11a) shows point estimates without variability. Table 9 reports scores to two decimal places without any indication of whether a 0.1-point difference is meaningful. The paper does not mention random seeds, does not describe any replication, and does not discuss statistical methodology.

Mitigation status. Not addressed. The paper does not acknowledge the absence of statistical rigor as a limitation, does not report confidence intervals for any result, and does not discuss the multiple-comparisons problem inherent in selecting best configurations from 200+ ablations. This is particularly concerning given the paper's framing as "the science of pretraining" (Section 1)—scientific conclusions require distinguishing signal from noise, and the current methodology cannot reliably do this for small-to-moderate effect sizes.

The "Overall Average" Metric Masks Domain Tradeoffs and Assumes Equal Weighting Without Justification

The assumption or constraint. The paper's primary aggregate metric is the arithmetic mean across all 19 benchmarks, reported as "Overall Average" in Table 6 and throughout the ablation results. This metric treats a 1-point improvement on PIQA (a physical commonsense benchmark with high baseline performance, where daVinci-LLM-3B scores 77.26 and the ceiling is near 80-85) as equivalent to a 1-point improvement on MATH (a competition mathematics benchmark, where daVinci-LLM-3B scores 62.80). The benchmarks span fundamentally different difficulty levels, score ranges, and practical significance, yet they receive equal weight in the aggregate. The paper never justifies this equal-weighting scheme and never reports alternative weightings (e.g., domain-weighted, difficulty-weighted, capability-cluster-weighted).

The consequence. The "Overall Average" can be dominated by improvements on benchmarks where gains are easiest to achieve or where the baseline is high, while masking degradations on benchmarks that are more important for specific use cases. Concretely:

daVinci-LLM-3B trails Qwen-2.5-3B by 3.20 points on MMLU (62.53 vs. 65.73) and by 8.50 points on MMLU-STEM (53.41 vs. 61.91), but these gaps are diluted in the overall average by daVinci-LLM's large leads on MATH (+25.60) and HumanEval (+1.47). A practitioner who cares primarily about broad general knowledge might prefer Qwen-2.5-3B, but the overall average obscures this.
The Stage 2-1 QA concentration sweep (Figure 11b) shows that 70% QA achieves the highest overall average (45.2 vs. 43.6 for 30% QA) but suffers a code collapse (26.1 vs. 31.2). The paper correctly selects 30% QA for Stage 2-1 despite the lower overall average, but this decision required looking past the headline metric. In other contexts where domain-specific breakdowns are less carefully examined, the overall average could drive suboptimal decisions.
The domain-averaging within General, Code, and Science clusters (Avg General, Avg Code, Avg Science) uses the same equal-weighting assumption. Within Science, GSM8K (where daVinci-LLM-3B scores 72.86) receives the same weight as SuperGPQA (where it scores 19.56), even though these represent very different capability levels and practical relevance.

What evidence exists in the paper. The paper consistently reports domain-specific averages alongside the overall average (Table 6, Figures 4, 6, 8, 10), which partially mitigates the problem by allowing readers to examine domain-specific patterns. However, even within domains, the equal-weighting assumption persists, and the paper never discusses whether equal weighting is appropriate. The evaluation protocol analysis in Section 4.4 demonstrates awareness that metric choices matter—the PPL vs. generative discrepancy shows how evaluation design shapes conclusions—but this awareness is not extended to the choice of aggregation function.

Mitigation status. Partially. The paper reports per-benchmark and per-domain breakdowns (Table 6, radar plot in Figure 11a, per-domain averages in Figures 11b and Table 9), which provides the data needed for readers to apply their own weighting schemes. The paper's decision to select 30% QA for Stage 2-1 despite 70% QA having a higher overall average (Section 4.3.2) demonstrates awareness that domain-specific patterns can override aggregate metrics. However, the paper never formalizes this awareness—it never proposes a principled weighting scheme, never discusses the tradeoffs embedded in equal weighting, and never reports sensitivity of conclusions to alternative aggregation methods. A simple robustness check (e.g., reporting whether the ranking of daVinci-LLM-3B vs. OLMo-3 7B changes under difficulty-weighted or domain-weighted averaging) would substantially strengthen confidence that the headline comparisons are not artifacts of the aggregation function.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reframes pretraining from an intuition-guided craft into a domain where design decisions can be treated as falsifiable research questions and answered through systematic, controlled experimentation. The magnitude of this contribution is best understood not as a paradigm shift—pretraining still involves training large transformers on web-scale corpora using standard optimization—but as a methodological reframing with significant practical leverage. The key conceptual move is transforming "data quality matters" from a platitude into an operationalized measurement framework (Data Darwinism L0–L9) and demonstrating that processing depth can be treated as an experimental variable whose marginal returns can be measured, compared, and optimized.

The most landscape-changing specific finding is the demonstration that capability-specific saturation patterns provide the signal for curriculum design. Before this work, the dominant pretraining paradigm applied static data mixtures over predetermined token budgets, implicitly assuming that all capability dimensions develop at similar timescales. The paper's convergence tracking across Stage 1's 6T-token trajectory (Section 4.2.1, Figure 8)—showing general knowledge plateauing at 1T tokens while code and science reasoning sustain growth through 4T+ tokens—provides the first large-scale empirical evidence that this assumption is false, and more importantly, that the saturation pattern itself can guide when to adjust mixtures and when to shift data formats. This transforms curriculum design from a heuristic to a measurement-driven process.

The paper also resolves a latent tension in prior work about the role of data processing. On one side, the scaling laws literature (Hoffmann et al., 2022) has focused attention on model size and data volume as the dominant pretraining variables, implicitly treating data as a homogeneous commodity. On the other side, practitioners have long suspected that "data quality" matters enormously but lacked a shared vocabulary for characterizing quality interventions or a methodology for measuring their impact. The Data Darwinism framework bridges this gap by providing exactly that shared vocabulary, and the 200+ controlled ablations demonstrate that processing depth is not merely a matter of subjective preference—it produces measurable, systematic, and sometimes large-magnitude capability improvements (+7.00 on MATH from L4 refinement in Table 8; +15–20 on reasoning domains from structured QA introduction in Figure 9). This establishes data processing depth as a first-class optimization dimension alongside model scale and data volume, a conceptual shift with direct implications for how pretraining compute budgets should be allocated.

The identification of stage-dependent tolerance for data intensification (Section 4.3.2) is another finding that changes how practitioners should think about multi-stage training. The observation that the same QA concentration (e.g., 70%) causes code collapse in Stage 2-1 but enables monotonic improvement in Stage 2-2 (Table 9) demonstrates that the ordering of data introduction matters as much as the proportion. This finding challenges the common practice of specifying a single data mixture for the entire training run and suggests that mixture design should be treated as a dynamic, stage-aware optimization problem. The practical implication is that early-stage balance enables late-stage intensification—a principle that practitioners can apply immediately to their own training pipelines without needing the full ablation infrastructure that produced the finding.

The paper also redirects research attention in several important ways:

Away from post-training as the primary lever for capability improvement. The paper's explicit claim that "post-training techniques... struggle to fundamentally overcome the capability foundations established during pretraining" (Section 1), backed by citations [26, 27, 28] and demonstrated by daVinci-LLM-3B's large reasoning gains without any post-training, argues that the field's disproportionate focus on accessible post-training techniques may be misallocated relative to the leverage available from pretraining science. This does not make post-training research less valuable, but it suggests that pretraining improvements compound—a model that acquires better reasoning patterns during pretraining will benefit more from post-training refinement than one that acquires weaker foundations. This reframes the pretraining/post-training relationship as complementary rather than substitutive, with pretraining determining the ceiling that post-training can approach but not exceed.

Away from scale as the only pretraining lever. The finding that a 3B model can match a 7B model (Table 6) and that L5 cognitive completion can substitute for multi-fold data volume increases (Section 4.1 takeaway) challenges the implicit assumption in the scaling laws literature that parameters and tokens are the dominant variables. The paper does not claim that scale is unimportant—8T tokens is still a substantial training budget—but it demonstrates that data processing quality and adaptive curriculum design are powerful complementary variables that can produce efficiency gains comparable to significant scale increases. This suggests that the scaling laws picture is incomplete without accounting for quality-driven interventions, opening a new axis for scaling laws research that incorporates processing depth as a variable.

Toward openness as scientific methodology. The paper's most distinctive meta-contribution is treating transparency not as a nice-to-have property but as essential infrastructure for cumulative scientific progress in pretraining. Table 1 systematically documents the transparency gap across existing models, and the paper's release of complete training trajectories, data specifications, ablation results, and negative findings provides a template for what comprehensive pretraining disclosure looks like. The argument—that without such disclosure, each pretraining effort starts largely from scratch, guided by intuition and folklore rather than accumulating empirical knowledge—is a direct challenge to the commercial and open-weight paradigms that dominate the field. Whether this template is adopted by other groups remains to be seen, but the paper makes the strongest case to date that the current transparency norms are scientifically inadequate for building cumulative understanding of pretraining dynamics.

Follow-Up Research This Work Enables

1. Scaling the Data Darwinism framework across model sizes and architectures. The paper's central finding—that processing depth systematically improves downstream capabilities—is established for a single 3B Qwen2 architecture. The most important follow-up is to test whether the marginal benefit of advancing from one Darwin Level to the next is scale-dependent. A concrete experiment: replicate the L4 refinement ablation (Table 8) at 1B, 3B, 7B, and 13B parameter scales using the same architecture family, same training tokens, and same evaluation benchmarks. The critical question is whether the +7.00 MATH gain from L4 refinement at 3B shrinks at larger scales (because bigger models extract patterns from noisy data more effectively), stays constant, or grows (because bigger models can better leverage enriched reasoning structures). If the benefit shrinks, processing depth is primarily a small-model strategy; if it persists or grows, it becomes increasingly important as models scale. A negative result—showing that L4/L5 benefits diminish to near zero at 13B+ scale—would be as scientifically valuable as a positive one because it would establish a boundary condition on when data processing investments are worthwhile. The paper's release of complete processing pipelines and prompts (Appendix C) makes this replication feasible without re-engineering the processing infrastructure.

2. Cheap difficulty estimation and dynamic curriculum adjustment without full-trajectory evaluation. The paper's adaptive curriculum (Stage 1-2's mixture adjustment at 4T tokens, Stage 2's QA introduction, Stage 2-2's QA intensification) depends on tracking benchmark performance at 5,000-step intervals across 19 benchmarks—an expensive evaluation infrastructure that is not amortized into any efficiency comparison. The natural research direction this enables is developing cheap proxies for capability saturation that can guide curriculum decisions without full benchmark evaluation. A concrete proposal: train a small probe model (e.g., a linear classifier on intermediate layer activations) to predict downstream benchmark performance from the model's internal representations at a given training step. If such a probe could accurately estimate whether a capability dimension is approaching saturation—using only a small held-out validation set rather than 19 full benchmark evaluations—the entire adaptive curriculum methodology becomes deployable at low overhead. The daVinci-LLM paper provides the perfect substrate for this research: the released intermediate checkpoints at 5k-step intervals, combined with the benchmark evaluation logs, provide labeled training data (checkpoint representations → benchmark scores) for training such probes. A strong follow-up would demonstrate that probe-predicted saturation points match the actual saturation patterns observed in Figure 8, and that a probe-guided curriculum achieves comparable final performance to the full-evaluation-guided curriculum at a small fraction of the evaluation cost.

3. Testing whether internal QA composition, not just total QA concentration, drives the Stage 2-1 code collapse. The paper's most intriguing negative result—code performance collapses in Stage 2-1 when QA exceeds 30% but recovers and improves monotonically in Stage 2-2 at 70% QA (Figure 11b vs. Table 9)—is attributed post-hoc to compositional imbalance in the QA pool (hypothesized to be ~80% science). But this mechanism is never directly tested. A clean follow-up experiment: replicate Stage 2-1 training with QA concentration fixed at 70% but systematically vary the internal code-to-science ratio of the QA pool—for instance, 50/50 code/science QA, 80/20, and 20/80. If the code collapse disappears when the QA pool is code-balanced (50/50), the compositional imbalance hypothesis is confirmed and the practical prescription becomes clear: when intensifying QA, balance the internal domain composition of the QA pool, not just the total ratio. If the code collapse persists even with balanced QA, the mechanism is something else—perhaps a fundamental property of the model's developmental stage that makes it unable to process high concentrations of structured supervision before certain representational foundations are established. Either outcome advances understanding of the stage-dependence phenomenon beyond the paper's current post-hoc explanation. The paper's release of all QA data sources with domain annotations makes this experiment trivial to set up.

4. Measuring contamination from frontier models used in data processing. The paper's L4 and L5 processing pipelines use frontier LLMs (GPT-OSS-120B, Qwen3-235B-A22B) to transform training data, but the paper never assesses whether these models inadvertently inject benchmark content into processed text. This is a tractable measurement problem. A specific experiment: take the L5-processed Darwin-Science-Paper-GPT and Darwin-Science-Paper-Qwen datasets, and run n-gram overlap analysis (e.g., 13-gram matching) against the MATH, MMLU, GSM8K, and HumanEval benchmark questions and solutions. Also run embedding similarity search: embed all processed text chunks and benchmark items using a sentence transformer, and flag any processed chunks whose cosine similarity to any benchmark item exceeds a threshold. If contamination is detected at non-trivial levels, train a version of daVinci-LLM with contaminated data removed and measure the performance difference on contaminated benchmarks. This would quantify how much of the paper's reported gains—particularly the +23.20 MATH lead over OLMo-3 7B (Table 6)—are attributable to genuine reasoning improvement versus memorization of benchmark-relevant content. A negative result (no detectable contamination) would strengthen confidence in the paper's claims; a positive result would establish an important methodological requirement for any future work using frontier models for data processing. The experiment is feasible because the paper releases all processed datasets.

5. Decoupling the effect of data format (QA structure) from data content (reasoning supervision) in Stage 2 gains. Stage 2's QA introduction produces dramatic improvements over continued domain proportion adjustment (Figure 9: +20.34 Code, +15.75 Science), but the QA data differs from raw text along two dimensions simultaneously: it provides explicit reasoning supervision (correct answers with intermediate steps), and it uses a structured format (question-answer pairs). A valuable follow-up would attempt to decouple these. A concrete experiment: create a "raw text with reasoning" control by taking the same scientific content used to generate QA pairs and formatting it as continuous prose that explicitly articulates reasoning steps (e.g., "To solve this problem, we first... then... therefore the answer is..."), matching the information content of QA data but without the structural format of explicit question-answer pairs. Train a Stage 2 configuration with this control data at the same proportion as the QA data in the paper, and compare against both the Stage 2 baseline and Stage 1-3. If the control achieves most of the QA gain, the benefit is primarily from reasoning supervision content, and structured QA is just one convenient way to deliver it. If the control substantially underperforms QA, the structural format itself (the behavioral training of mapping questions to explicit answers) is causally important—consistent with the paper's evaluation protocol analysis in Section 4.4. This would refine our understanding of why QA data works and guide future data synthesis efforts toward the dimension that actually matters.

6. Replication of differential saturation patterns across model architectures and domains. The paper's finding that general knowledge saturates early while reasoning capabilities sustain growth throughout training (Section 4.2.1, Figure 8) is the empirical foundation for its adaptive curriculum methodology. But this finding is demonstrated for one architecture (Qwen2-3B), one data mixture, and one benchmark suite. A multi-group replication effort—testing whether the same saturation hierarchy appears when training LLaMA, OLMo, or Mamba architectures on different data mixtures—would establish whether this is a universal property of language model training dynamics or an architecture-specific or data-specific artifact. The key measurement: track separate benchmark clusters (general knowledge, code, math, science) at regular intervals during pretraining and test whether the ordering of saturation times (general first, reasoning later) is consistent across architectures and data compositions. If the pattern is universal, it becomes a reliable design principle for any pretraining curriculum. If it varies—for example, if some architectures saturate math before general knowledge—then curriculum design must be architecture-aware, which would be an important negative result qualifying the paper's claims of generalizability. The paper's release of training logs and evaluation methodology makes the measurement protocol replicable; the main barrier is the computational cost of multiple full-training-run replications, which would require multi-institutional collaboration of the kind the paper's open-release philosophy is designed to enable.

Practical Applications and Downstream Use Cases

1. Cost-efficient pretraining for organizations with fixed compute budgets. The paper's core practical insight—that principled data processing investments can achieve parameter efficiency gains of approximately 2.3× (3B matching 7B)—directly translates to reduced training costs. For an organization deciding between training a 7B model for N dollars or a 3B model with invest-in-data-processing for approximately N/2 dollars, the paper provides evidence that the latter can match or exceed the former's performance, particularly on reasoning-intensive tasks. The specific recipe: allocate a portion of the pretraining budget to L4 Generative Refinement and L5 Cognitive Completion of reasoning-domain data using strong but not necessarily frontier models (the paper shows gains from Qwen3-235B which, while large, is publicly available), and implement a two-stage curriculum with QA introduction guided by convergence tracking. The practical barrier is the upfront processing cost—the paper does not report processing FLOPs—but for organizations that already have access to strong models for data processing (increasingly common as open-weight frontier models proliferate), the marginal cost of processing may be low relative to the training cost savings. The key metric a practitioner would compute: (cost of processing X tokens to L4/L5 + cost of training on those tokens) versus (cost of training on 2.3× more unprocessed tokens for a larger model). The paper provides the performance data but not the cost data needed to make this calculation; a practical deployment would require measuring both.

2. Targeted capability improvement for domain-specific deployments without full retraining. The paper's demonstration that L5 synthetic QA exhibits strong source-target alignment (Figure 7, Section 4.1.3)—code-domain QA primarily improves code benchmarks, science-domain QA primarily improves science reasoning—suggests a strategy for organizations deploying models in specific domains. If a model needs stronger medical reasoning, generate L5 QA from medical textbooks using domain-specific prompts (the paper provides the medicine QA extraction prompt in Appendix C) and continue training with a Stage 2-like mixture that concentrates this domain-specific QA. The finding that Stage 2-2 can safely intensify QA to 70% without capability collapse (Table 9) provides an existence proof that aggressive domain-specific intensification is viable, but with the important caveat that this requires a balanced foundation stage first (Stage 2-1 at 30% QA). The practical workflow: (1) process domain-specific documents through L4 Generative Refinement to clean and structure them; (2) apply L5 Cognitive Completion to generate pedagogically enriched text and domain-specific QA pairs; (3) train a balanced mixture (30% QA, complemented by diverse general and code data) for ~1T tokens to establish a stable representational foundation; (4) intensify to 70% domain-specific QA for another ~1T tokens. The paper's MATH score of 62.80 (exceeding 7B baselines by 23+ points) suggests that this recipe can produce disproportionate gains in targeted reasoning domains, though the caveat about benchmark contamination risk from processing models applies.

3. Diagnostic evaluation protocols for pretraining data composition. The paper's analysis of PPL-based versus generative evaluation discrepancies (Section 4.4, Figure 12) provides a practical diagnostic tool for organizations evaluating third-party models or their own training runs. The key observation: models with heavier QA exposure during pretraining gain disproportionately under generative evaluation because QA-style training teaches the behavioral format of mapping questions to explicit answers. A practitioner who observes a model performing similarly to a baseline under PPL evaluation but substantially better under generative evaluation can infer—without access to training data details—that the model likely received more structured QA or instruction-formatted data during pretraining. Conversely, a model that excels under PPL but underperforms under generative evaluation may have strong latent knowledge but weak answer-production skills, suggesting that continued training with QA-formatted data could unlock latent capability. The practical application: when selecting among candidate base models for a downstream task requiring open-ended generation (chat, QA, code synthesis), evaluate under generative protocols rather than PPL, and use the PPL-to-generative gap as a diagnostic for whether additional QA-formatted training might help. The paper demonstrates this diagnostic on OLMO-2-7B vs. Qwen-2.5-3B, but the principle generalizes to any model comparison where training data composition is partially unknown.

4. Multi-stage curriculum design as a default pretraining paradigm. The paper's two-stage curriculum—broad foundation on diverse web text (Stage 1), followed by reasoning intensification with structured QA (Stage 2), with adaptive mixture adjustments at empirically-determined saturation boundaries—provides a concrete template that practitioners can adopt with minimal modification. The specific design principles: (1) monitor capability-specific convergence during Stage 1 to identify when general knowledge plateaus and reasoning still grows; (2) at that boundary, reduce web text proportion and increase reasoning-domain concentrations; (3) when further proportion adjustments yield diminishing returns (Stage 1-3's failure in Figure 9), transition to structured QA data; (4) start with a balanced QA mixture (30% QA, complemented by diverse code and science) to establish a stable foundation; (5) intensify QA concentration only after this foundation is established. The paper's release of complete mixture compositions at each substage (Table 2), training hyperparameters (Table 5), and negative results (Stage 1-3's failure, Stage 2-1's code collapse at high QA) makes this template actionable: a practitioner can adopt the specific numbers as starting points and adapt based on their own convergence tracking. The main adaptation needed is for different model scales—the saturation timescales and optimal QA ratios may shift at 7B or 13B parameters, but the paper's methodology for finding these boundaries (monitor, adjust when domains diverge, shift formats when proportion adjustment saturates) is scale-agnostic.