Data Science and Technology Towards AGI Part I: Tiered Data Management

ArXiv: 2602.09003

🎯 Pitch

Low-quality data doesn't just waste compute—it actively sabotages later training stages by saturating performance and blocking further gains. This paper shows that simply stratifying data into quality tiers and feeding L1→L2→L3 across training stages yields a 1.7× faster learning rate in final tuning compared to mixing everything together, while the highest-quality tier alone boosts math performance over 7 points.

---

1. Executive Summary

This paper proposes a tiered data management framework — an L0–L4 quality hierarchy spanning raw archived resources (L0) to organized, verifiable knowledge (L4) — and systematically validates it across the LLM training lifecycle using MiniCPM-1.2B models trained on math, web, code, and Chinese web corpora. The framework introduces two complementary mechanisms: quality-stratified data tiering (progressively refining corpora from heuristic-filtered L1 to model-selected L2 to synthetic/edited L3) and tiered training schedules (introducing higher-quality data in later training stages rather than uniformly mixing all tiers), demonstrating that L3 data consistently outperforms L1 by 1.70–7.06 average percentage points across domains, with Math-L3 additionally driving cross-domain gains in language understanding and code generation. A tiered training strategy that allocates L1 → L2 → L3 data across consecutive stages outperforms mixed training by 1.49 percentage points overall (31.66 vs. 30.17) and sustains later-stage learning improvement at ~1.7× the growth rate of the mixed baseline, establishing that quality-stratified data scheduling prevents the performance saturation caused by low-quality interference.

2. Context and Motivation

The Core Problem: LLM Data Management Has Scaled Quantity Without Quality

The fundamental issue this paper tackles is a structural mismatch in how the field manages training data for large language models. Over the past decade, the dominant paradigm has been straightforward: collect more data, filter it with basic heuristics, and train larger models on larger corpora. This approach — what the paper calls "Data-Driven Learning" (Section 1, Figure 1) — has delivered remarkable progress through unidirectional scaling of data volume, but it is now colliding with three hard constraints that make continued scaling unsustainable.

The paper articulates these constraints explicitly in the introduction (Section 1):

Constraint 1: High-quality public data is becoming scarce. The paper cites Villalobos et al. (2022) — the widely-discussed "will we run out of data?" analysis — and argues that:

"Future model development cannot rely solely on expanding data scale. Instead, data science and technology must shift from pursuing scale toward more careful data management and utilization."

This is not merely a hypothetical concern. The paper's own data analysis (Table 1) reveals the asymmetry: ultra-large-scale corpora like Common Crawl contain vast quantities of raw text, but the yield after quality filtering drops precipitously. For example, UltraData-Math-L1 (filtered math data) is 170B tokens, but UltraData-Math-L2 (model-selected high-quality math) shrinks to 33B tokens — roughly an 80% reduction — and UltraData-Math-L3 (synthetic and refined math) recovers to 88B tokens only through substantial generative augmentation. Simply scaling the L1 pool further would add predominantly low-quality tokens with diminishing returns, a dynamic the field has already encountered with web-scale filtering projects.

Constraint 2: Different training stages have fundamentally different data requirements. The paper observes that LLM training is not a monolithic process — it spans pre-training (knowledge acquisition), mid-training (domain specialization), and alignment (behavioral fine-tuning), each with distinct objectives. A corpus optimized for one stage can actively harm another. For example, pre-training benefits from scale and diversity even at moderate quality, while mid-training and annealing stages are sharply sensitive to data quality — the paper notes that "concentrating high-quality data during specific phases, such as annealing or mid-training, can significantly enhance model performance" (Section 2). Yet existing data management frameworks (Section 2.1.1) treat these stages as independent pipelines with separate management standards, creating fragmentation rather than a unified quality hierarchy.

Constraint 3: Data management costs must be balanced against marginal training benefit. Different processing operations — heuristic filtering, model-based selection, LLM-based editing and synthesis — have orders-of-magnitude different computational costs. The paper argues this explicitly in Section 1:

"In the early stage of data management, lightweight and low-cost methods (such as heuristic filtering) should be adopted, while in deeper management stage, more fine-grained and higher-cost approaches (such as LLM-based labeling) should be used."

Without a principled framework for when to invest in expensive processing, practitioners either under-invest (leaving quality gains on the table) or over-invest (spending compute on processing data that yields minimal incremental benefit). The paper draws attention to the strategic dimension: "strategically deploying valuable data at critical training moments — such as mid-training phases or annealing stages — can maximize data effectiveness while keeping overall costs manageable."

The Structural Gap: Processing Methods Exist, But a Unified Quality Framework Does Not

The paper provides an extensive survey of existing data management techniques in Section 2.1.2, organized by processing methodology — data parsing, filtering, selection, editing, and synthesis. This survey reveals a critical pattern: the field has developed sophisticated point solutions for individual processing tasks, but these solutions operate in isolation without a shared quality vocabulary or integrated evaluation framework.

Consider what exists (Section 2.1.2 in detail):

• Data parsing has evolved from heuristic HTML extraction (Trafilatura) to LLM-based semantic parsing for math (OpenWebMath's LaTeX extraction) and documents (MinerU, olmOCR). Each domain developed bespoke parsers without quality-level standardization.
• Data filtering progressed from regex-based rules (C4) to fuzzy deduplication (RefinedWeb's MinHash-LSH) to semantic deduplication (SemDeDup's embedding-based approach). These methods produce cleaner data but provide no mechanism for ranking the remaining content by quality tier.
• Data selection introduced model-driven classifiers — DCLM, FineWeb-Edu, Ultra-FineWeb — that score samples for educational value or domain relevance. Yet these classifiers output continuous or discrete scores without an agreed-upon mapping to quality levels that span pre-training, mid-training, and alignment stages.
• Data editing and synthesis can produce textbook-quality content (Phi series, Nemotron-CC), but there is no framework for relating this synthetic data to the original filtered/selected corpora — should refined data be treated as a separate dataset, mixed uniformly, or concentrated in specific training phases?

The paper's central diagnosis is that these methods are "severely fragmented" (Section 2.1.2 closing paragraph):

"Management strategies tend to be monolithic, failing to implement differentiated processing based on data value, intended use, or training stage. Moreover, data processing pipelines are severely fragmented—collection, cleaning, selection, and validation are often conducted independently without unified quality metrics or closed-loop feedback mechanisms."

This fragmentation manifests concretely: a practitioner who builds an L2-quality math corpus using a classifier (like FineMath) and a colleague who builds an L3-quality math corpus through synthesis (like Nemotron-CC-Math) have no shared framework for comparing their outputs, deciding how to combine them, or scheduling them across training stages. The pipeline outputs are technically incompatible at the metadata and quality-assessment level.

Why Prior Stage-Oriented and Method-Oriented Approaches Fall Short

The paper identifies two dominant existing frameworks — stage-oriented and method-oriented — and explains their specific inadequacies (Section 2.1).

Stage-oriented management (Section 2.1.1) organizes data processing by training phase: pre-training data management, mid-training data management, post-training data management. This framework emerged organically as practitioners developed specialized pipelines for each phase. For pre-training, the emphasis is on scale, diversity, and basic cleanliness (e.g., C4, RefinedWeb, FineWeb). For mid-training, the emphasis shifts to domain relevance and reasoning density (e.g., OpenWebMath for math, The Stack v2 for code). For post-training, the focus narrows further to instruction quality, preference alignment, and verifiable correctness (e.g., UltraFeedback, verifiable reward signals in RL).

The limitation is that this framework couples data quality to training stage rather than treating quality as an independent axis. A high-quality pre-training corpus and a high-quality mid-training corpus may both deserve the label "high-quality," but the stage-oriented framework provides no mechanism for recognizing this shared property. More critically, the same underlying dataset might serve multiple stages at different quality tiers — a raw math corpus could be L1 for pre-training, L2 for mid-training after classifier filtering, and L3 for RL after synthesis — but the stage-oriented view treats these as separate datasets rather than progressive refinements of a single resource. The paper's proposed tiering (L0-L4) directly addresses this by making quality an explicit, stage-independent attribute that can be tracked across the entire data lifecycle.

Method-oriented management (Section 2.1.2) organizes data processing by the complexity and intelligence of the operation — parsing, filtering, selection, editing, synthesis. This taxonomy is useful for understanding the tools available, but it provides no guidance on when to apply which method or how the outputs relate to each other. The paper argues:

"Despite methodological diversity, existing practices remain hampered by the lack of systematic and hierarchical management. The absence of unified tiered standards makes it difficult to identify and prioritize high-value data effectively."

This is more than a classification problem — it is a resource allocation problem. Given a fixed compute budget for data processing, should a practitioner invest in model-based selection (L1→L2 transition) or in synthetic augmentation (L2→L3 transition)? Without a quality tiering framework that quantifies the marginal benefit of each transition, these decisions are made ad hoc.

The Paper's Position: From Data-Driven Scaling to Data-Model Co-Evolution

The paper positions itself not as proposing yet another data processing method, but as establishing the architectural framework within which existing and future methods can be systematically organized, compared, and deployed. This is captured in the central thesis (Section 1, Figure 1):

"We argue that AI development should transition toward 'Data-Model Co-Evolution,' wherein models improve data management practices while high-quality data further refines model performance, creating a positive feedback cycle."

The term "Data-Model Co-Evolution" is deliberately chosen to contrast with "Data-Driven Learning." In the data-driven paradigm, the relationship is unidirectional: more data → better models. The model is a passive consumer of externally provided data. In the co-evolution paradigm, the relationship is bidirectional: models actively participate in data management (through quality scoring, content editing, synthesis, and selection), and the resulting high-quality data, in turn, amplifies model capabilities. This creates a feedback loop where better models produce better data management, which produces better training data, which produces better models.

The L0-L4 tiered framework (Section 2.2, Figure 2) operationalizes this vision by defining five quality levels with explicit properties, management strategies, and training roles:

• L0 (Raw Data): Archive-grade, unprocessed resources. Maintained for traceability and reprocessing potential. Not used directly in training.
• L1 (Filtered Data): Heuristic-cleaned, deduplicated, format-standardized corpora. Low processing cost, high scalability. Primary resource for large-scale pre-training.
• L2 (Selected Data): Model-driven quality selection using classifiers and quality scorers. Higher information density than L1, with domain-specific curation. Suitable for decay stages and mid-training.
• L3 (Refined Data): Edited, synthesized, or human-refined content with explicit educational intent. Maximum learnability, textbook-quality standards. Core resource for mid-training, SFT, and RL stages.
• L4 (Organized Data): Structured, verifiable knowledge in graph or database formats. Trustworthy factual substrate for retrieval-augmented generation.

The framework is explicitly designed to balance three competing forces (Section 1): data quality (which increases with tier level), acquisition cost (which also increases with tier level), and marginal training benefit (which depends on the training stage and the model's current capability). This is what distinguishes the framework from a simple "more processing is better" philosophy — it provides the vocabulary and structure for making strategic allocation decisions.

A crucial insight embedded in the framework is that the L0-L4 taxonomy is not just about static data properties — it is about the operational processes that produce each tier. Section 2.2 details the specific operators (parsers, filters, classifiers, editors, synthesizers, orchestrators) that transition data between levels. This means the framework supports data lineage tracking: for any sample in an L3 corpus, you can trace its provenance back through L2 (which classifier selected it) to L1 (which heuristic pipeline cleaned it) to L0 (which Common Crawl snapshot it originated from). This traceability — which the paper explicitly notes is lacking in current practice — enables systematic debugging of data quality issues, ablation studies on processing decisions, and incremental improvement of the pipeline itself.

The Experimental Gap That Motivates the Validation Strategy

The paper's experimental design (Section 3) is motivated by a specific evidentiary gap: while many individual processing methods have been validated in isolation (FineWeb vs. C4, FineWeb-Edu vs. FineWeb, etc.), there is essentially no prior work that:

1. Constructs L1, L2, and L3 corpora for the same underlying data source across multiple domains,
2. Evaluates the quality gradient under a consistent training protocol (same model architecture, same training budget, same evaluation benchmarks), and
3. Tests whether tiered training schedules (sequentially introducing higher-quality data) outperform uniform mixing.

Without this kind of controlled comparison, the field cannot distinguish between two competing hypotheses: (a) higher-quality data is always beneficial regardless of how it is scheduled, or (b) the sequencing of data quality matters independently of the total quality-weighted volume. The paper's tiered-vs-mixed training experiment (Section 3.4) directly tests this, and the result — tiered training achieves 1.7× the late-stage learning rate of mixed training — provides evidence for hypothesis (b). This finding has direct practical implications: it means that even with a fixed set of available data across quality tiers, how you sequence them during training can change final model performance.

The cross-domain math experiment (Section 3.3) is similarly motivated by a gap in the literature. Prior work has shown that high-quality math data improves math reasoning (unsurprising), but the claim that math data quality drives cross-domain gains in language understanding and code generation — with Math-L3 improving English benchmarks by 3.45 pp over Math-L1 and code benchmarks by 3.80 pp — is a stronger statement about the transfer properties of reasoning-intensive training data. This connects to emerging evidence that mathematical reasoning capability serves as a foundation for more general logical competence, but validates it through the lens of data quality tiering rather than just data domain.

Summary of the Motivation

The paper's motivation can be distilled into a single claim: the field's current approach to data management — treating data quality as a binary (clean vs. noisy) and mixing all available data uniformly — is leaving substantial model performance on the table, and a systematic, quality-tiered framework with stage-aware scheduling can recover these gains. The theoretical contribution is the L0-L4 taxonomy and the data-model co-evolution perspective. The empirical contribution is the demonstration that quality stratification matters independently of quality volume, and that tiered scheduling outperforms uniform mixing even when both strategies use identical data in identical proportions.

3. Technical Approach

3.1 Reader Orientation

This paper constructs an L0–L4 tiered data management framework — not a single model or algorithm, but a systematic quality hierarchy and operational pipeline that transforms raw, noisy web-scale corpora (L0) into progressively higher-quality training datasets (L1 through L4) and then schedules those datasets across different stages of LLM training. The core problem it solves is the current industry practice of treating all available training data as a single undifferentiated pool — the framework provides both a quality vocabulary (what makes L2 different from L3?) and a resource allocation logic (when should you pay the cost of L3 refinement rather than settling for L1 or L2?), demonstrating that quality-stratified scheduling recovers statistically significant performance gains that uniform mixing leaves on the table.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major operational layers, each producing data of increasing quality from the layer below it, and a training scheduler that determines how data from different layers feeds into the LLM training lifecycle:

1. L0 (Raw Data Archive): Ingest heterogeneous sources (Common Crawl WARC/WET files, arXiv LaTeX, GitHub repositories) through domain-specific parsers. This layer preserves original content for traceability — no data is modified or discarded.
2. L1 (Filtered Data Pipeline): Apply heuristic cleaning (URL filtering, language identification, regex rules), format standardization (e.g., LaTeX repair for math, boilerplate removal for web), and global deduplication (MinHash) to produce a clean, uniform-text corpus ready for large-scale pre-training.
3. L2 (Selected Data Pipeline): Use model-driven quality classifiers (fastText or LLM-based scorers trained on synthetic annotations) to score and retain only high-value samples from the L1 pool. This layer produces domain-specific corpora with substantially higher information density — e.g., educational-value-filtered web pages, reasoning-dense math problems, or permissively-licensed code with documentation.
4. L3 (Refined Data Pipeline): Apply LLM-based editing (removing residual noise, fixing formatting, repairing broken reasoning chains) and multi-format synthetic generation (difficulty-stratified Q&A, multi-turn dialogues, knowledge-driven textbooks, style-diverse rewrites) to produce textbook-quality corpora suitable for mid-training, supervised fine-tuning, and reinforcement learning.
5. L4 (Organized Data Layer): Transform unstructured high-quality text into structured, verifiable knowledge assets (knowledge graphs, databases) through orchestration and fact-verification against trusted sources.

The training scheduler (Section 3.4) then allocates data from these tiers across training stages. In the tiered strategy: L1 data dominates early pre-training (broad linguistic foundation); L2 data is introduced in mid-training (high-information-density knowledge); L3 data concentrates in late training / annealing (logical reasoning and capability refinement). This contrasts with the baseline "mix training" strategy, which uniformly blends L1, L2, and L3 data at all stages.

Information flows vertically upward through the tiers (L0 → L1 → L2 → L3, with each tier progressively refining the output of the previous one) and then horizontally into the training scheduler, which allocates data by tier to specific training phases. The framework also supports feedback: LLMs used for L2 classification and L3 synthesis/editing are themselves trained on earlier iterations of the tiered data, creating the "data-model co-evolution" loop that the paper advocates.

3.3 Roadmap for the Deep Dive

• First, the L0-L4 tier definitions in full operational detail — what specific processing operators are applied at each layer, what the inputs and outputs are, and what the data properties become (Section 2.2.1–2.2.5).
• Second, the three verification protocols — efficient verification, pre-train verification, and decay verification — including their exact hyperparameter configurations, because understanding the experimental results requires knowing what "verification" means and what it costs (Section 3.1).
• Third, the model architecture and training setup, including why MiniCPM-1.2B with μP was chosen and how the tokenizer and Megatron-LM training framework are configured.
• Fourth, the domain-specific data construction pipelines for math (the UltraData-Math L0→L3 pipeline, the most detailed case study) and web data (the Ultra-FineWeb L1→L3 pipeline), since these are the concrete implementations that validate the framework.
• Fifth, the tiered-vs-mixed training experimental design (Section 3.4), which is the paper's key causal test of whether quality sequencing matters independently of quality volume.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a framework and empirical validation paper whose core idea is that data quality can be stratified into an explicit five-level hierarchy (L0–L4), that this hierarchy captures meaningful and measurable differences in downstream model performance, and that scheduling training data by tier (low-quality first, high-quality later) is strictly better than uniform mixing, because it prevents low-quality data from interfering with late-stage capability refinement.

---

The L0-L4 Tier Definitions (Section 2.2)

The framework defines five levels through which data ascends, with each transition requiring additional computational investment and producing progressively higher-quality, higher-cost corpora. The levels are designed to be domain-agnostic — the same quality concepts apply whether the underlying data is web text, mathematical content, code, or academic literature — while the specific operators at each level are domain-adapted.

L0: Raw Data

L0 data is defined by its archival property: it is stored in its native format or undergoes only minimal structural conversion, with no filtering, deduplication, or quality improvement. The defining characteristic is that L0 preserves data provenance and enables reprocessing — if a downstream pipeline improves (e.g., a better parser or classifier becomes available), the raw material can be re-ingested without loss.

The input sources include Common Crawl (the largest open-source web crawl, with snapshots spanning 15+ years and 30+ billion pages as of January 2026, in three formats: WARC for complete HTTP responses, WAT for metadata/link graphs, WET for pre-extracted plaintext), academic repositories (arXiv's LaTeX source and PDF versions), code hosting platforms (GitHub's full commit histories), and Q&A repositories (Stack Overflow). For domain-specific math data, sources like MathOverflow (a Stack Exchange community for research-level mathematics) provide L0 material.

The processing at L0 consists of format parsing only — no content modification. For HTML, tools such as MinerU-HTML, Resiliparse, and Trafilatura extract text while preserving document structure. For PDF documents, tools such as MinerU, olmOCR, and Nougat perform OCR and layout analysis. For mathematical content, domain-specific enhanced HTML parsers (like the UltraData-Math-Parser released with this paper) preserve LaTeX formulas and mathematical notation that general-purpose HTML parsers often corrupt.

The L0 level does not directly participate in training. Its purpose is to serve as the "source of truth" for data lineage — enabling traceability from a training sample in an L3 corpus back to the original Common Crawl snapshot, the specific WARC file, and the raw HTML that produced it.

L1: Filtered Data

L1 is the first training-eligible tier and is defined by standardized text formatting, basic readability, and the removal of obvious noise. The processing is entirely heuristic — no learned models are involved — making L1 the cheapest quality tier per token.

The paper's canonical L1 construction example is FineWeb (Penedo et al., 2024b), whose pipeline the paper reviews in detail as a reference architecture:

1. Text extraction: FineWeb extracts text from Common Crawl WARC files using Trafilatura (a heuristic HTML-to-text extractor that identifies main content regions by analyzing DOM structure, text density, and link-to-text ratios). The paper notes an important empirical finding: extracting from WARC files with Trafilatura significantly outperforms using the pre-extracted WET files, because the WET extraction discards structural information that Trafilatura uses to identify boilerplate.

2. Base filtering: URL blacklists (inherited from RefinedWeb, which maintained blocklists of spam, pornographic, and machine-generated domains) and fastText language identification (a lightweight linear classifier trained on n-gram features that achieves near-perfect language detection at minimal computational cost) reduce the corpus to approximately 36 trillion tokens.

3. Heuristic quality filtering: The paper describes custom filters developed by comparing statistical metric distributions between high-quality and low-quality datasets. Three specific filters are operationalized:

   • Line-ending punctuation ratio: ≤ 0.12. This filters documents where lines rarely end with punctuation, indicating either fragmented text (no proper sentences) or code without documentation.
   • Duplicated line character ratio: ≥ 0.1. This filters documents where a high fraction of characters belong to lines that are duplicated elsewhere in the same document, indicating template-generated content or repetitive boilerplate.
   • Short line ratio: ≥ 0.67. This filters documents where most lines are very short, indicating navigation menus, lists of links, or other non-prose content.

   These three filters collectively remove approximately 22% of tokens and improve aggregate benchmark scores by about 1%.

4. Deduplication: FineWeb uses MinHash-based deduplication with 5-grams, 112 hash functions, and 14 buckets, targeting 75% similarity. Notably, it applies deduplication per-snapshot rather than globally — the paper reports that global MinHash across all Common Crawl snapshots increases the sampling rate of low-quality data in older snapshots (because older content has more near-duplicates across snapshots, so deduplication preferentially retains higher-quality newer content), leading to performance degradation compared to per-snapshot deduplication.

The final FineWeb dataset contains 15 trillion tokens and outperforms existing public datasets on multiple benchmarks.

For domain-specific L1 data, the paper introduces UltraData-Math-L1, which implements mathematical content cleaning through two categories of operators:

• Format repair mappers (record-preserving): These remove invisible characters and control codes, consolidate excessive consecutive line breaks, and strip residual interface noise (navigation bars, pagination buttons, advertisement fragments that survived initial parsing). These operations transform noisy extractions into clean readable text without discarding any records.
• Content filters (record-discarding): These apply heuristic rules to remove records that fail basic usability standards — specifically, short articles lacking proper punctuation (indicating extraction failures or non-text content) and documents with abnormal text lengths (truncated extractions or concatenated multi-page artifacts).

The key design principle at L1 is low cost + high scalability: all operations are embarrassingly parallelizable (each document is processed independently), require no GPU inference, and apply deterministic rules that produce reproducible outputs. This enables processing petabyte-scale corpora at manageable cost, establishing a standardized foundation for the more expensive model-driven operations at higher tiers.

L2: Selected Data

L2 marks the first transition to model-driven processing and is defined by higher information density, thematic relevance, and domain-appropriate content selection. The core mechanism is a quality classifier — typically a lightweight model like fastText trained on LLM-generated annotations — that scores each L1 sample and retains only those above a quality threshold.

The paper's canonical L2 construction example for web data is Ultra-FineWeb (Wang et al., 2025b), whose methodology the paper describes in detail:

1. Seed data selection: The central hypothesis is that "high-quality seed data benefits LLM training and fosters stronger classifiers to identify similar beneficial data." Ultra-FineWeb addresses the prohibitive cost of traditional validation (training many models to convergence to evaluate data subsets) by introducing an efficient validation strategy based on a weight-decay scheduler and a two-stage annealing phase. This approach uses a base model pre-trained for 1.1T tokens (stable stage) followed by a 0.1T-token decay stage, where candidate data subsets are mixed at 30% into the default distribution and model performance after the decay phase is measured. The paper reports this significantly reduces GPU hours required, enabling rapid assessment of different data subsets.

2. Classifier training: Using this efficient validation, researchers select high-quality samples from a candidate pool to serve as seed data for training a fastText classifier. fastText is a linear classifier over bag-of-n-grams representations — it represents each document as the average of its word n-gram embeddings and applies a softmax layer. This is computationally orders of magnitude cheaper than LLM-based inference: a fastText classifier can score billions of tokens on CPU, whereas LLM-based scoring would require GPU inference at roughly $N_{\text{layers}} \times 2 \times \text{FLOPs per token}$ cost.

3. Inference: The trained fastText classifier scores every document in the L1 pool (FineWeb). Documents above a quality threshold are retained as L2 data.

The paper reports that the resulting Ultra-FineWeb corpus significantly outperforms the underlying L1 FineWeb in downstream evaluations, validating the model-driven selection hypothesis.

For math data, the paper's L2 construction example is UltraData-Math-L2, which uses the UltraData-Math Classifier — a domain-specific model trained to identify mathematical content with high reasoning density and educational value. The classifier operates on the L1 math corpus (UltraData-Math-L1, 170B tokens) and retains only high-scoring samples, producing a 33B-token L2 corpus. This ~80% reduction in volume (170B → 33B) with improved downstream performance is the core value proposition of L2: trading quantity for quality, measured by the downstream accuracy gain per training token.

The paper also documents FineMath (Allal et al., 2025) as an alternative L2 math pipeline, providing a more detailed view of the classification process:

1. First-pass classification: A classifier trained on Llama-3.1-70B-Instruct scores extracted math pages on a 3-point scale to identify high-quality mathematical domains.
2. Re-extraction: The OpenWebMath pipeline (which preserves LaTeX formatting and removes boilerplate) is used to re-extract all pages from the identified domains, yielding 7.1B pages and 6.5T tokens.
3. Second-pass classification: A second classifier scores on a 5-point scale, specifically filtering for pages containing reasoning and educational content from middle school to early college levels.
4. Deduplication and decontamination: Single-band MinHash LSH (10 hashes), fastText language classification (retaining English only), and benchmark decontamination (removing samples that overlap with evaluation sets).
5. Output: Two versions — FineMath-4+ (10B tokens, retaining only scores 4-5) and FineMath-3+ (34B tokens, retaining scores 3-5). FineMath-4+ achieves a 2× performance increase on GSM8K and a 6× increase on MATH compared to OpenWebMath and InfiMM-WebMath.

For code data, the L2 example is Stack-Edu (Allal et al., 2025), which applies educational-value filtering to the L1 Stack-v2 code corpus.

The key design principle at L2 is model-guided value distillation: using models (either lightweight classifiers or LLMs for annotation) to identify which L1 samples are worth the training budget. The cost increase over L1 is substantial — training and running a classifier requires GPU computation — but the paper argues this is justified by the downstream performance gains and the fact that L2 data can be strategically allocated to training phases (mid-training, decay) where quality sensitivity is highest.

L3: Refined Data

L3 is the highest-quality training tier and is defined by structured content with clear reasoning chains, explicit educational intent, and maximum learnability — essentially "textbook-quality" corpora. The processing transitions from selective (keeping the best existing samples) at L2 to generative (creating or substantially editing content) at L3.

The paper describes two distinct L3 production strategies: editing refinement (for web data) and synthetic generation (for math and code data).

Editing Refinement (Ultra-FineWeb-L3): The paper's L3 construction for general web data draws on the Nemotron-CC approach (Su et al., 2025). The insight is that even L2-quality web pages — which have been selected by classifiers for high educational value — still contain significant noise from parser limitations: residual boilerplate text (sidebars, headers, footers), navigation elements, OCR errors, broken code indentation, and grammatical inconsistencies. The L2 classifier can identify high-value content but cannot fix these defects.

The L3 pipeline treats data processing as semantic distillation: an LLM reads each L2 document and reconstructs the underlying content in its purest form. Concretely:

• The LLM removes non-content elements (sidebars, headers, footers, advertisements).
• The LLM corrects OCR errors, fixes broken code indentation, and resolves grammatical inconsistencies.
• The LLM improves text coherence and readability while strictly preserving the original semantics.
• Documents that fail to meet information density thresholds after editing are discarded.

This "filtering + editing" paradigm is a key architectural choice: rather than simply discarding noisy documents (which loses potentially valuable content), or keeping them as-is (which degrades training quality), the pipeline attempts to recover the signal from the noise. The paper argues this overcomes "the brittleness of heuristic filtering, achieving secondary purification of data quality."

For the English web domain, Ultra-FineWeb-en-L3 is a 200B-token corpus produced by this editing pipeline applied to Ultra-FineWeb-en (the L2 corpus). The paper notes that the L3 data comprises five types of synthesized content to ensure diversity: diverse QA, distill (extractive summarization of knowledge), extract knowledge (structured extraction of facts and relationships), knowledge list (enumerative presentations of domain knowledge), and wiki style (encyclopedic article formatting).

Synthetic Generation (UltraData-Math-L3): For mathematical data, editing alone is insufficient because web-mined math content has fundamental limitations: it is biased toward certain problem types, lacks systematic coverage of difficulty levels, and rarely includes step-by-step reasoning. The L3 math pipeline therefore uses generative synthesis to create novel content that goes beyond what exists in the L1/L2 corpora.

The UltraData-Math-L3 construction pipeline (Section 2.2.4) proceeds through these stages:

1. Seed cleaning and standardization: L2 math documents are cleaned and standardized into a unified LaTeX format, removing formatting noise so the synthesis model focuses purely on mathematical logic.

2. Multi-model ensemble synthesis: Rather than relying on a single LLM (which would produce stylistically homogeneous output), the pipeline employs a multi-model ensemble to transform seeds into five diverse instructional formats:

   • Difficulty-stratified Q&A pairs: Problems spanning a curriculum-aligned progression from primary school to undergraduate levels, providing clear supervision signals (problem → step-by-step solution → final answer).
   • Multi-turn teacher-student dialogues: Seven persona pairs (e.g., "patient tutor and struggling student," "Socratic questioner and advanced learner") simulate interactive teaching scenarios that introduce structural complexity and context-maintenance requirements.
   • Multi-style rewrites: Each mathematical core logic is rewritten in multiple presentation styles — Wikipedia article style, blog post style, academic paper style, textbook style — to decouple the mathematical reasoning from surface-level linguistic patterns, preventing the model from overfitting to narrow writing conventions.
   • Knowledge-driven textbook modules: Theorems, axioms, and definitions are extracted from L2 documents and used to generate pedagogical explanations and multi-level practice problems, creating structured learning materials.
   • Persona-integrated synthesis: Professional educational materials are simulated by generating content that a specific persona (e.g., a university professor, a textbook author) would produce.

3. Quality filtering: All synthetic outputs undergo rigorous filtering for LaTeX syntax errors (e.g., unmatched braces, invalid commands) and logical incompleteness (e.g., problems without solutions, solutions that don't address the stated problem). This ensures the final corpus maintains high information density.

The output is UltraData-Math-L3, an 88B-token corpus. Notably, while L2 dramatically reduced volume (170B → 33B, an 80% reduction), L3 expands it again (33B → 88B) through synthesis, producing a corpus that is both higher quality than L2 and larger in scale. This illustrates the paper's argument that quality and scale are not necessarily in tension — L3 can achieve both through generative augmentation.

For code data, the L3 construction (Code Textbook) uses "textbook-style rewriting, including code explanations and programming exercises" applied to the L2 code corpus. The paper provides fewer details on this pipeline compared to the math and web pipelines.

The key design principle at L3 is value creation through generation: rather than merely filtering existing content, L3 actively produces new training material optimized for learnability. This is the most expensive tier per token (LLM inference for editing or synthesis costs substantially more than classifier inference), and the paper explicitly positions it for critical training moments where quality sensitivity is highest.

L4: Organized Data

L4 is the knowledge-verification tier and is defined by structured, trustworthy, and verifiable data — typically in knowledge graph or database formats rather than continuous text. While L1-L3 focus on the linguistic and semantic quality of training corpora, L4 emphasizes factual reliability and structured queryability.

The paper describes two core operations at L4:

• Data orchestration: Scattered information from diverse sources is unified under coherent thematic frameworks and interconnected knowledge structures. This is not simply merging datasets — it requires identifying equivalent entities across sources, resolving contradictions, and establishing relationships (e.g., "this theorem from source A depends on this definition from source B").
• Fact verification: Entries are cross-referenced with trusted sources to eliminate factual inconsistencies present in raw web corpora. This is particularly important for knowledge-intensive tasks where hallucination is costly (medical, legal, scientific domains).

Representative L4 datasets include Wikidata (a highly structured, multilingual knowledge base supporting precise entity-relation querying via SPARQL) and UltraData-arXiv (a reorganization of scholarly literature where mathematical formulas, citations, and experimental results are standardized into searchable and interconnected formats).

L4 data serves downstream applications like retrieval-augmented generation (RAG) — by providing a trusted factual substrate that models can query during inference, L4 acts as a defense against hallucination and ensures factual precision for expert-level reasoning. The paper positions L4 as fundamentally different from L1-L3: while those tiers are designed for training (improving model parameters), L4 is designed for inference-time augmentation (providing factual grounding during generation).

---

Verification Protocols (Section 3.1)

The paper uses three distinct verification protocols to evaluate data quality at different scales and training phases. Understanding these protocols is essential for interpreting the experimental results, because each protocol trades off evaluation fidelity against computational cost.

Efficient Verification

This is the lightweight, high-throughput protocol used for rapid data quality comparison across tiers and domains (Section 3.2). The core idea is to measure data quality through a short annealing phase on a pre-trained base model — rather than training from scratch, which would be prohibitively expensive for the number of comparisons needed.

The protocol proceeds as follows:

1. Base model: A MiniCPM-1.2B model is pre-trained from scratch on 1.1T tokens using the MiniCPM-3-4B corpus (the default training distribution for that model family). The training uses a Warmup-Stable-Decay (WSD) scheduler: a 1T-token stable stage (constant learning rate) followed by a 0.1T-token decay stage (learning rate decay to near zero).

2. Annealing phase: Starting from this pre-trained checkpoint, the model is further trained on 10B tokens (referred to as "10B" for simplicity; the exact calculation is SeqLen × GBS × TrainStep = 4096 × 512 × 5000 = 10.5B tokens). The data mixture for annealing is 30% verification data (the candidate dataset being evaluated — e.g., Math-L1, Math-L2, or Math-L3) and 70% of the default distribution (the same distribution used during pre-training). This 30/70 split is chosen to provide enough signal from the verification data to measure quality differences while preventing catastrophic forgetting of the pre-training distribution.

3. Training parameters: Sequence length of 4096 tokens, weight decay of 0.1, gradient clipping threshold of 1.0, global batch size of 512 (implemented with micro-batch size of 16). The optimization uses an exponential decay schedule with a 500-step warm-up, scaling the learning rate from a peak of $1 \times 10^{-3}$ to a minimum of $5 \times 10^{-5}$.

4. Stabilization: All experiments use Maximal Update Parameterization (μP) (Yang et al., 2022) to improve training stability. μP is a specific parameterization of neural network initialization and learning rates that ensures the optimal hyperparameters transfer across model sizes — meaning the learning rate schedule tuned for MiniCPM-1.2B would transfer to larger or smaller architectures without re-tuning.

The output of efficient verification is a set of benchmark scores after the 10B-token annealing phase. Because the annealing phase is short and uses a pre-trained base, this protocol can evaluate many candidate datasets at manageable cost, but the results may have higher variance than longer training runs. The paper explicitly acknowledges this trade-off: "Although efficient verification enables higher iteration efficiency with significantly lower resource requirements, the limited training budget may introduce higher variance in results."

Pre-train Verification

This is the medium-scale protocol used for evaluating data quality when training from scratch — used specifically in the tiered-vs-mixed training experiment (Section 3.4). Rather than annealing on a pre-trained base, this protocol trains a model from random initialization on 120B tokens (computed as SeqLen × GBS × TrainStep = 4096 × 2048 × 15000 = 125.8B tokens, simplified to 120B).

The training parameters are:

• Sequence length: 4096 tokens.
• Global batch size: 2048 (micro-batch size of 16). This is 4× larger than the efficient verification global batch size, providing more stable gradient estimates.
• Learning rate schedule: Cosine decay with a 1000-step warm-up. The learning rate starts at $1 \times 10^{-5}$, peaks at $1 \times 10^{-2}$, and decays to $5 \times 10^{-4}$. The cosine schedule is chosen over exponential because it provides a smoother decay curve that has been empirically found to work well for mid-scale training.
• Weight decay: 0.1.
• Gradient clipping: 1.0.
• μP is used for training stability.

The 120B-token budget is a deliberate compromise: it is large enough to observe meaningful training dynamics and final performance differentiation, but small enough to run multiple experiments (comparing different data tiers and training strategies) within practical compute constraints. The paper notes that this budget yields models that are far from convergence but provide reliable relative comparisons.

Decay Verification

This is the full-scale validation protocol used for the UltraData-Math case study (Section 3.3) and represents the most reliable evaluation because it closely mimics the final pre-training phase of production models. The core idea is to start from a near-converged model and apply a full-scale decay phase with the candidate data.

The protocol proceeds as follows:

1. Base model: A MiniCPM-1.2B model is pre-trained on 1.3T tokens from the MiniCPM-4 corpus, having completed both warmup and stable stages. This is a substantially larger pre-training budget than the efficient verification base model (1.3T vs. 1.1T tokens), providing a stronger baseline.

2. Decay phase: The model is further trained on approximately 100B tokens (computed as SeqLen × GBS × TrainStep = 4096 × 1280 × 20000 = 104.9B tokens). The data mixture is 30% new verification data and 70% of the default distribution (same ratio as efficient verification, but at 10× the scale).

3. Training parameters: Sequence length of 4096, global batch size of 1280 (micro-batch size of 10), exponential decay schedule that scales the learning rate from the stable stage's $7.5 \times 10^{-4}$ down to a minimum of $3.75 \times 10^{-5}$. Other hyperparameters (weight decay 0.1, gradient clipping 1.0, μP) remain consistent.

The paper positions decay verification as the gold standard: "Decay verification addresses this [variance issue] by employing a full-scale decay phase, providing a more robust and definitive evaluation that closely reflects the final pre-training performance."

---

Model Architecture and Training Configuration

All experiments use the MiniCPM-1.2B model architecture with the MiniCPM3-4B tokenizer (Table 3). The architecture specifications are:

• Non-embedding parameters: 1,247,442,432 (~1.25B).
• Vocabulary size: 73,448 tokens. This is notable because it uses the MiniCPM3-4B tokenizer vocabulary rather than training a new tokenizer, ensuring compatibility with the broader MiniCPM ecosystem.
• Hidden dimension ($d_m$): 1,536. This is the dimensionality of the transformer's residual stream.
• Feedforward bottleneck dimension ($d_{ff}$): 3,840. This is the hidden dimension of the feedforward network (FFN) within each transformer layer, typically 2.5× the hidden dimension in standard configurations — here it's exactly 2.5×.
• Attention head dimension ($d_h$): 64. With 24 query heads ($n_{head}$), the total attention dimension is $24 \times 64 = 1536$, matching the hidden dimension.
• Number of query heads ($n_{head}$): 24.
• Number of key/value heads ($n_{kv}$): 8. This indicates grouped-query attention (GQA) with a 3:1 query-to-key-value ratio (24 query heads sharing 8 key-value heads), which reduces memory bandwidth during inference while preserving most of the quality of multi-head attention.
• Number of layers ($n_{Layer}$): 52.

All models are trained using the Megatron-LM library (Shoeybi et al., 2019), which provides distributed training infrastructure (tensor parallelism, pipeline parallelism, data parallelism) for transformer models at scale.

The choice of MiniCPM-1.2B as the evaluation architecture is strategic: it is small enough that multiple training runs (pre-train verification at 120B tokens, decay verification at 100B tokens) are feasible within a research compute budget, but large enough that training dynamics and scaling trends are representative of larger models. The paper does not claim the results would transfer unchanged to billion-parameter models, but the consistent tier-quality ordering across domains and protocols suggests the quality stratification is architecture-robust.

---

Domain-Specific Pipeline: UltraData-Math (L0 → L3)

The mathematics domain receives the most detailed treatment as a case study, and the pipeline illustrates the complete L0→L3 progression with concrete operators. The paper released this pipeline as open source under the UltraData-Math project.

From L0 to L1: UltraData-Math-L1 (170B tokens)

1. Input: Raw mathematical web pages collected from sources like Common Crawl and MathOverflow. These are in HTML format with embedded LaTeX, images, and web formatting.

2. Parsing (L0 → L1 boundary): The UltraData-Math-Parser — an enhanced HTML parser specifically designed for mathematical content — transforms raw HTML into clean text with LaTeX formulas preserved. This is the key domain adaptation: general-purpose HTML parsers often corrupt LaTeX by misinterpreting backslashes, braces, and dollar signs, but the UltraData-Math-Parser is trained or configured to recognize mathematical notation boundaries.

3. L1 processing: The suite of format repair mappers and content filters described in the L1 definition is applied:

   • Format repair mappers: remove invisible characters and control codes (ASCII 0-31 except newline/tab, Unicode zero-width characters), consolidate excessive consecutive line breaks (merging sequences of 3+ newlines into 2), strip residual interface noise (identified by pattern matching against known navigation bar text, pagination button text, and advertisement phrases).
   • Content filters: remove records shorter than a minimum character threshold (indicating failed extraction) and records with abnormal text lengths (both too short — truncated — and too long — indicating concatenated multi-page artifacts, based on statistical outlier detection in the length distribution).

The output is UltraData-Math-L1, a 170B-token corpus of filtered mathematical text suitable for pre-training.

From L1 to L2: UltraData-Math-L2 (33B tokens)

1. Input: UltraData-Math-L1 (170B tokens).
2. Classifier training: The UltraData-Math Classifier is trained (details not fully specified in the paper — the paper references the released code and models) to distinguish high-quality mathematical content from lower-quality content. Based on the FineMath parallel, this likely involves training on LLM-annotated scores for a subset of L1 samples.
3. Selection: The classifier scores every document in L1. Documents above a quality threshold are retained; others are discarded. The threshold is set to achieve the desired quality-volume trade-off, producing 33B tokens — approximately 19% of the L1 volume.

The output is UltraData-Math-L2, a 33B-token corpus of high-quality, model-selected mathematical content suitable for mid-training and decay stages.

From L2 to L3: UltraData-Math-L3 (88B tokens)

1. Input: UltraData-Math-L2 (33B tokens), plus the L2 seed documents as raw material for synthesis.

2. Seed standardization: L2 documents are cleaned and standardized into unified LaTeX format. This involves:

   • Normalizing LaTeX command variants (e.g., \frac vs. \dfrac, \binom vs. \choose).
   • Ensuring consistent math environment delimiters ($...$ for inline, $$...$$ or \[...\] for display).
   • Fixing common LaTeX errors (unmatched braces, missing \right for \left, incorrect use of \text vs. \mathrm).

3. Multi-model ensemble synthesis: The UltraData-Math-Generator (released as open source) uses multiple LLMs to generate content in five instructional formats, as described in the L3 definition above. The multi-model approach is critical: by using different models (or different prompting strategies on the same model), the pipeline produces diverse outputs that prevent the training model from overfitting to a single generator's stylistic biases or error patterns.

4. Quality filtering: All synthetic outputs are validated:

   • LaTeX syntax checking: detecting and discarding outputs with invalid LaTeX that would cause rendering failures.
   • Logical completeness checking: verifying that problems have solutions, solutions address the stated problems, and multi-step reasoning chains are not truncated.
   • Format compliance: ensuring each output matches its intended format (e.g., Q&A pairs have clearly separated questions and answers; dialogues have alternating speaker turns).

The output is UltraData-Math-L3, an 88B-token corpus of synthetic and refined mathematical content suitable for mid-training, supervised fine-tuning, and reinforcement learning. Note that L3 is larger than L2 (88B vs. 33B) — the synthesis process has expanded the corpus beyond the original selection volume, creating new content rather than merely refining existing content.

---

Domain-Specific Pipeline: Ultra-FineWeb (L1 → L3)

For web data, the paper provides separate pipelines for English and Chinese, following the same tier structure.

English Web: FineWeb (L1) → Ultra-FineWeb-en (L2) → Ultra-FineWeb-en-L3 (L3)

• L1: FineWeb (Penedo et al., 2024c), the 15T-token English web corpus produced by the heuristic pipeline described in the L1 section.
• L2: Ultra-FineWeb-en (Wang et al., 2025b), a 1,800B-token corpus produced by training and applying a fastText classifier on L1 data, as described in the L2 section. The classifier is the Ultra-FineWeb-en-Classifier, released as open source.
• L3: Ultra-FineWeb-en-L3, a 200B-token corpus produced by LLM-based editing refinement of L2 data, following the Nemotron-CC distillation approach. The L3 data includes five synthesis types: diverse QA, distill, extract knowledge, knowledge list, and wiki style.

Chinese Web: Chinese FineWeb (L1) → Ultra-FineWeb-zh (L2) → Ultra-FineWeb-zh-L3 (L3)

• L1: Chinese FineWeb (Yu et al., 2025, specifically from Chinese FineWeb-edu-v2), a Chinese web corpus produced by heuristic filtering.
• L2: Ultra-FineWeb-zh (Wang et al., 2025b), a 120B-token corpus produced by a Chinese-specific fastText classifier (the Ultra-FineWeb-zh-Classifier, released as open source).
• L3: Ultra-FineWeb-zh-L3, a 200B-token corpus produced by LLM-based editing of L2 data, with the same synthesis types as the English version.

The scale difference between English (1,800B L2) and Chinese (120B L2) reflects the underlying availability of web data in each language. The fact that both L3 corpora are 200B tokens suggests the editing+synthesis pipeline may have a target output scale independent of input scale — the LLM-based process can generate L3 content up to a practical compute budget rather than being limited by the L2 volume.

---

Tiered Training Experimental Design (Section 3.4)

This experiment is the paper's causal test of whether the tiered framework matters beyond simply having higher-quality data in aggregate. The question is: given identical data in identical proportions, does the sequence of data quality during training affect final model performance?

Domain distribution. Both strategies use the same domain distribution: 50% Web-en, 25% Web-zh, 8% Math, 17% Code. This distribution is held constant to ensure any performance differences come from the scheduling strategy, not from different domain exposures.

Mix training (baseline). All 120B tokens are pooled into a single unified training corpus with an equal 1:1:1 ratio of L1, L2, and L3 data. That is, the model sees L1, L2, and L3 samples interleaved randomly throughout the entire 120B-token training run. This represents the current dominant practice: "data of varying quality are often indiscriminately mixed during training" (Section 2).

Tiered training (proposed). The same 120B tokens are partitioned into three consecutive 40B-token stages:

• Stage 1 (0–40B tokens): Only L1 data is used. This corresponds to the early pre-training phase where the model builds broad linguistic representations and basic knowledge — the paper argues that L1 data, with its high diversity and scale, is optimal for this stage.
• Stage 2 (40–80B tokens): Only L2 data is used. This introduces higher information density and domain-specific curation as the model transitions to mid-training — the model-selected L2 data provides more efficient knowledge acquisition than L1 at this stage.
• Stage 3 (80–120B tokens): Only L3 data is used. This concentrates the highest-quality, textbook-level, reasoning-dense data in the final stage where the model is refining its capabilities — the paper hypothesizes that L3 data is most valuable when the model has already acquired foundational knowledge and can benefit from structured, pedagogical content.

Both strategies use the identical total data volume (120B tokens) and identical data quality composition (1:1:1 L1:L2:L3 ratio by token count). The only difference is temporal allocation: mixed interleaves all tiers throughout training, while tiered sequences them L1 → L2 → L3.

The paper evaluates both strategies using the pre-train verification protocol (training from scratch on 120B tokens, as described above), and compares final benchmark scores and intermediate training curves (Figure 3). The results are discussed in the Experimental Analysis section of the full paper summary, but the experimental design is the core technical contribution of Section 3.4: it establishes the causal identification strategy for testing whether quality sequencing matters.

Why this design answers the question. If tiered training outperforms mixed training, it cannot be because tiered training uses higher-quality data (both strategies use identical data). It also cannot be because tiered training uses more data (identical total volume). Nor can it be because tiered training uses a different domain distribution (identical distribution in both strategies). The only explanatory variable is the temporal sequencing of data quality. This is a clean causal design that isolates the mechanism the paper is proposing: that low-quality data in late training stages interferes with capability refinement, and that concentrating high-quality data at the end of training prevents this interference.

---

Benchmarks and Evaluation (Section 3.1)

The paper uses OpenCompass (Contributors, 2023) as the evaluation framework, a standardized benchmark suite for Chinese and English LLMs. The evaluation covers four domains with specific datasets and settings:

General English (10 benchmarks):

• MMLU (5-shot, PPL-based): Massive Multitask Language Understanding — 57 subjects across STEM, humanities, and social sciences. PPL-based evaluation means the model is scored by its perplexity on correct answer choices rather than by generation.
• ARC-C (0-shot, PPL): AI2 Reasoning Challenge (Challenge set) — grade-school science questions requiring reasoning. The "Challenge" subset filters for questions that simple statistical methods fail on.
• ARC-E (0-shot, PPL): AI2 Reasoning Challenge (Easy set).
• BBH (3-shot, Gen): BigBench Hard — 23 challenging tasks from the BIG-Bench benchmark, using few-shot generation rather than PPL. This evaluates reasoning capabilities that are difficult to measure with multiple-choice perplexity.
• CSQA (8-shot, PPL): CommonSenseQA — commonsense reasoning questions.
• HellaSwag (0-shot, PPL): Commonsense natural language inference — choosing the most plausible continuation of a scenario.
• OpenbookQA (0-shot, PPL): Open-book question answering requiring elementary science knowledge and reasoning.
• PIQA (0-shot, PPL): Physical commonsense reasoning — questions about everyday physical interactions.
• SIQA (0-shot, PPL): Social commonsense reasoning about social situations.
• Winogrande (0-shot, Loglikelihood): Winograd schema challenge — pronoun resolution requiring commonsense reasoning. Evaluated by log-likelihood of the correct option rather than perplexity.

General Chinese (2 benchmarks):

• C-Eval (5-shot, PPL): Chinese multi-level multi-discipline evaluation — analogous to MMLU for Chinese.
• CMMLU (5-shot, PPL): Chinese Massive Multitask Language Understanding.

Math Reasoning (2 benchmarks):

• MATH500 (4-shot, Gen): Subset of the MATH competition dataset, with 500 problems. Evaluated by generation (the model produces a solution, and the final answer is extracted and compared to the ground truth).
• GSM8K (4-shot, Gen): Grade-school math word problems. Evaluated by generation.

Code Reasoning (2 benchmarks):

• MBPP (3-shot, Gen): Mostly Basic Programming Problems — Python programming tasks. Evaluated by generation and execution of the produced code against test cases.
• HumanEval (0-shot, Gen): Hand-written Python programming problems from OpenAI. Evaluated by generation and execution against test cases.

The distinction between PPL-based and generation-based evaluation matters because PPL-based evaluation measures the model's ability to recognize correct answers in a multiple-choice setting (a weaker capability signal), while generation-based evaluation measures the model's ability to produce correct answers from scratch (a stronger capability signal that requires reasoning and generation quality). The math and code benchmarks use generation because these domains require producing novel solutions; the general knowledge benchmarks use PPL because efficient multiple-choice evaluation is well-established and correlates with more expensive generation-based evaluation for these tasks.

The average score metric used throughout the paper is the unweighted mean across all benchmarks in each domain (e.g., "English Avg." is the mean of the 10 English benchmark scores, "All Avg." is the mean of all 16 benchmarks). This treats each benchmark as equally important.

4. Key Insights and Innovations

Innovation 1: Data Quality as an Independent Axis, Not a Training Stage Attribute

The paper's most fundamental conceptual move is the decoupling of data quality from training stage. Before this work, the dominant organizational frameworks for LLM data management were either stage-oriented (what data do you need for pre-training vs. mid-training vs. post-training?) or method-oriented (what processing operations do you apply — filtering, selection, editing, synthesis?). Both frameworks treat quality as an emergent consequence of the processing pipeline or the intended use case, not as a first-class property with its own explicit taxonomy.

The paper's L0–L4 tiering framework changes this by defining quality as an intrinsic, measurable, and monotonic attribute of the data itself, independent of which training stage it feeds. An L2 math corpus is L2-quality regardless of whether you use it for pre-training, mid-training, or reinforcement learning. An L3 web corpus is L3-quality irrespective of domain. This seems subtle but has profound implications: it means data quality can be tracked, compared, and scheduled across training stages rather than being reinvented within each stage's pipeline. A practitioner who upgrades their L1→L2 pipeline for math data doesn't need to re-evaluate whether the output is "good for pre-training" or "good for mid-training" — the tier label captures the quality level, and the scheduling decision (when to deploy it) becomes a separate, explicit choice.

This is a fundamental reframing, not an incremental improvement. Prior work implicitly treated data quality as a binary — clean vs. noisy (C4, RefinedWeb, FineWeb) — or attached quality judgments to specific stages (pre-training data must be large-scale and diverse; mid-training data must be domain-specific; post-training data must be instruction-formatted). The tiered framework unifies these perspectives by making quality a continuous spectrum with explicit transition operators (heuristic filtering at L1, model-based selection at L2, editing/synthesis at L3, verification at L4), each with defined inputs, outputs, and cost characteristics. The evidence that this reframing captures real structure comes from Table 5: across four domains (English web, Chinese web, math, code), the performance hierarchy L3 > L2 > L1 holds without exception under efficient verification, confirming that the tiers correspond to genuine quality distinctions that are measurable and monotonic.

Innovation 2: Quality Sequencing as an Independent Lever from Quality Volume

The paper's tiered-vs-mixed training experiment (Section 3.4, Table 7) establishes a finding that is, to my knowledge, novel in the LLM training literature: given identical data in identical proportions, the temporal sequence of data quality during training causally affects final model performance. Both strategies use 120B tokens at a 1:1:1 L1:L2:L3 ratio with identical domain distributions (50% Web-en, 25% Web-zh, 8% Math, 17% Code). The only difference is that mixed training interleaves all tiers uniformly throughout training, while tiered training sequences them L1 → L2 → L3 in three 40B-token stages.

The result — tiered training achieves 31.66 average score vs. 30.17 for mixed training (+1.49 pp), with late-stage learning improvement at ~1.7× the growth rate of the mixed baseline (Figure 3) — is significant not because it is a large absolute gain, but because it isolates a mechanism that has been largely overlooked: low-quality data in late training stages actively interferes with capability refinement. In the mixed strategy, the model continues seeing L1-quality samples throughout the entire 120B-token run, including the final stages where it is trying to consolidate reasoning capabilities. The tiered strategy eliminates this interference by concentrating L1 data in early training (where it provides useful linguistic diversity and broad knowledge) and reserving L3 data exclusively for the final stage (where its textbook-quality, reasoning-dense content can be absorbed without dilution).

This finding has direct implications for how practitioners should think about data curation. The dominant assumption in the field — implicit in the widespread practice of mixing all available data uniformly — is that the total quality-weighted volume of training data determines model performance, and that the ordering of samples is irrelevant beyond basic learning rate scheduling. The tiered training result challenges this: it suggests that data quality scheduling is an independent lever that can be optimized alongside data volume, model architecture, and learning rate schedules. The paper does not claim to have found the optimal schedule — the 1:1:1 ratio and 40B-token stages are heuristic choices — but the demonstration that any schedule matters opens a new dimension for optimization that the field has not systematically explored.

This is an empirical discovery with theoretical implications, not a theoretical advance per se. The paper does not propose a formal model of why quality sequencing matters (e.g., through the lens of curriculum learning theory or loss landscape analysis). But the clean experimental design — holding total data volume, quality composition, domain distribution, model architecture, and hyperparameters constant while varying only temporal allocation — makes the causal claim unusually robust for an applied ML paper, and the finding that quality scheduling can produce a 1.7× improvement in late-stage learning rate is practically significant even at the modest scale tested.

Innovation 3: High-Quality Math Data as a Driver of General Reasoning Capability

The paper's case study on UltraData-Math (Section 3.3, Table 6) demonstrates a cross-domain transfer effect that goes substantially beyond what prior work has shown. Training on Math-L3 (refined, synthetic mathematical data) not only improves math reasoning — which is expected — but produces statistically significant gains across English language understanding (+3.45 pp over Math-L1, from 51.32 to 54.77 average), Chinese language understanding (+1.89 pp, from 51.59 to 53.48), and code generation (+3.80 pp, from 37.30 to 41.10).

The magnitude and breadth of these gains challenge the conventional view that domain-specific data primarily benefits its own domain. The paper reports that Math-L3 improves ARC-E by +5.29 pp, ARC-C by +1.69 pp, BBH by +5.87 pp, and OpenbookQA by +15.40 pp over Math-L1 — these are reasoning-intensive English benchmarks that have no mathematical content. The gains on code generation (MBPP +4.56 pp, HumanEval +3.05 pp over Math-L1) further suggest that the benefit is not merely linguistic but reflects improved structured reasoning and abstraction capability that transfers across modalities and task formats.

This finding contextualizes a growing body of evidence that mathematical reasoning ability serves as a foundation for more general logical competence, but adds a specific and actionable dimension: the transfer effect is mediated by data quality, not just data domain. Math-L1 (filtered web math, 170B tokens) produces weaker transfer than Math-L3 (synthetic textbook-quality math, 88B tokens), even though both are mathematical data. The quality tier matters more than the raw volume — Math-L3 is actually smaller than Math-L1 (88B vs. 170B tokens) but produces dramatically better cross-domain transfer. This suggests that the mechanism is not "exposure to math tokens improves reasoning" but rather "exposure to high-quality, structured, pedagogical math content builds reasoning circuits that generalize." It is the reasoning structure in the data, not the math domain per se, that drives transfer.

This is an empirical discovery with significant practical implications for data mixture design. If high-quality math data produces general reasoning improvements at lower token cost than equivalent volumes of general web data, practitioners should allocate a disproportionate fraction of their high-quality data budget to math — not just for math benchmarks, but for overall model capability. The paper does not explore whether similar transfer effects exist from other reasoning-intensive domains (code, formal logic, scientific reasoning), but the math finding provides a template for such investigations and establishes that cross-domain transfer from high-quality reasoning data is a real and substantial effect, not a speculative hypothesis.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates across four domains with domain-specific datasets: English web (FineWeb for L1, Ultra-FineWeb-en for L2, Ultra-FineWeb-en-L3 for L3); Chinese web (Chinese FineWeb for L1, Ultra-FineWeb-zh for L2, Ultra-FineWeb-zh-L3 for L3); math (UltraData-Math-L1/L2/L3); and code (Stack-v2 for L1, Stack-Edu for L2, a textbook-style rewritten corpus for L3). Each tier within each domain is constructed from the same underlying data source, enabling controlled quality-gradient comparisons. Dataset scales are reported in Table 1 and Table 4: e.g., Math-L1 is 170B tokens, Math-L2 is 33B tokens, Math-L3 is 88B tokens; Web-en L2 is 1,800B tokens while Web-en L3 is 200B tokens.

• Base model(s). All experiments use the MiniCPM-1.2B architecture (1.25B non-embedding parameters, 52 layers, hidden dimension 1,536, 24 query heads with 8 key-value heads for grouped-query attention, vocabulary size 73,448 tokens from the MiniCPM3-4B tokenizer) trained via the Megatron-LM library. The paper argues this model is representative of current small-scale LLM training dynamics while being computationally tractable for the many training runs needed (multiple 120B-token pre-training runs, 100B-token decay runs, and 10B-token efficient verification runs across four domains). The choice of a 1.2B model rather than a larger one is a deliberate trade-off: it enables the controlled multi-condition experimental design (L1 vs. L2 vs. L3 per domain, tiered vs. mixed training, cross-domain transfer analysis) that would be prohibitive at 7B+ scale.

• Metrics. The primary metric is benchmark accuracy (% correct) evaluated through OpenCompass across 16 benchmarks spanning four domains. General English (10 benchmarks): MMLU (5-shot, PPL), ARC-C (0-shot, PPL), ARC-E (0-shot, PPL), BBH (3-shot, Gen), CSQA (8-shot, PPL), HellaSwag (0-shot, PPL), OpenbookQA (0-shot, PPL), PIQA (0-shot, PPL), SIQA (0-shot, PPL), Winogrande (0-shot, Loglikelihood). General Chinese (2 benchmarks): C-Eval (5-shot, PPL), CMMLU (5-shot, PPL). Math Reasoning (2 benchmarks): MATH500 (4-shot, Gen), GSM8K (4-shot, Gen). Code Reasoning (2 benchmarks): MBPP (3-shot, Gen), HumanEval (0-shot, Gen). Domain averages are unweighted means across the benchmarks in each domain; "All Avg." is the unweighted mean across all 16 benchmarks. The distinction between PPL-based evaluation (measuring recognition of correct multiple-choice answers) and generation-based evaluation (measuring production of correct open-ended solutions) matters: math and code benchmarks use generation because they require producing novel solutions, while general knowledge benchmarks use PPL for computational efficiency.

• Baselines. The paper does not compare against external models or prior datasets as baselines. Instead, it uses within-framework baselines: the L1 tier serves as the baseline for evaluating L2 and L3 quality gains within each domain (Table 5, Table 6), and the mix training strategy (uniform 1:1:1 blending of L1, L2, and L3 data across all 120B training tokens) serves as the baseline for evaluating the tiered training strategy (Table 7). This design choice means the paper demonstrates relative improvements from its own framework elements rather than claiming superiority over specific prior datasets — the evaluation asks "does L3 outperform L2 outperform L1?" and "does tiered scheduling outperform mixed scheduling?" rather than "does UltraData-Math-L3 outperform OpenWebMath?"

• Generation budget / compute accounting. Compute is measured in training tokens (not FLOPs or GPU-hours). Three verification protocols operate at different token scales: Efficient verification uses 10B tokens (30% verification data + 70% default distribution) applied as an annealing phase on a 1.1T-token pre-trained base model. Pre-train verification uses 120B tokens trained from scratch (random initialization). Decay verification uses ~100B tokens applied as a full-scale decay phase on a 1.3T-token pre-trained base model. Each protocol represents a different point on the cost-fidelity trade-off curve: efficient verification is cheapest but highest-variance; decay verification is most expensive but most representative of final pre-training performance; pre-train verification sits in between. All protocols use the MiniCPM-1.2B architecture with μP for stability, weight decay 0.1, gradient clipping 1.0, and sequence length 4096. Specific batch sizes and learning rate schedules differ by protocol (detailed in Section 3.1). The paper does not report total GPU-hours or FLOPs consumed.

• Cross-validation / statistical protocol. The paper does not report cross-validation, statistical significance testing, confidence intervals, or multiple-run error bars for any experimental result. All benchmark scores in Tables 5-7 and Figure 3 are reported as point estimates from single training runs. The efficient verification protocol is explicitly acknowledged as having "higher variance in results" due to the limited training budget, but no quantification of this variance (e.g., standard deviation across random seeds or data shuffles) is provided. This is a significant methodological limitation: without variance estimates, it is impossible to determine whether the reported differences between tiers (e.g., L3 vs. L2 average scores differing by 0.60-6.02 pp depending on domain) are statistically reliable or within the noise range of the training process. The pre-train verification and decay verification protocols, with their larger budgets, likely have lower variance than efficient verification, but this is not empirically established.

Data Quality Gradient Across Tiers (Section 3.2)

The paper's foundational empirical claim is that the L0–L4 tiering framework produces meaningful, monotonic quality differentiation — that L3 data is genuinely higher-quality than L2, which is genuinely higher-quality than L1, and that these differences are detectable through downstream model performance even under lightweight evaluation protocols.

Efficient verification results (Table 5). Using the efficient verification protocol (10B-token annealing on a 1.1T pre-trained base, with 30% verification data), the paper evaluates L1, L2, and L3 data across four domains. The results show a strict L3 > L2 > L1 ordering in all cases:

• English web: Average scores are 52.26 pp (L1), 53.36 pp (L2), 53.96 pp (L3). The L1→L2 gain is +1.10 pp; L2→L3 is +0.60 pp; total L1→L3 gain is +1.70 pp. The improvements are distributed across most individual benchmarks but are modest in magnitude — on MMLU, L3 actually regresses slightly from L2 (47.25 vs. 46.73 for L1, suggesting noise in the estimate) before recovering at L3 (47.25). The largest single-benchmark gain is OpenbookQA, jumping from 56.00 (L1) to 66.20 (L2) to 72.40 (L3), a +16.40 pp improvement — this is notable because OpenbookQA evaluates elementary science reasoning, suggesting L2/L3 web data may be particularly enriched for educational science content.

• Chinese web: Average scores are 49.44 pp (L1), 50.66 pp (L2), 51.48 pp (L3), for a total L1→L3 gain of +2.04 pp. Both C-Eval and CMMLU show consistent improvement, with C-Eval gaining +1.71 pp (49.51 → 51.22) and CMMLU gaining +2.37 pp (49.37 → 51.74). The gains are more uniformly distributed than in English, suggesting the Chinese L2/L3 tiers may provide more balanced quality improvement across knowledge domains.

• Math: Average scores are 23.78 pp (L1), 24.82 pp (L2), 30.84 pp (L3), for a total L1→L3 gain of +7.06 pp — the largest absolute improvement across all four domains. The pattern reveals a non-linear quality gradient: L1→L2 yields only +1.04 pp (23.78 → 24.82), while L2→L3 yields +6.02 pp (24.82 → 30.84). This suggests that the transition from model-selected math (L2) to synthetic/refined math (L3) is disproportionately beneficial compared to the transition from filtered math (L1) to selected math (L2). GSM8K shows the strongest improvement: +8.72 pp (32.75 → 41.47), while MATH500 improves by +5.40 pp (14.80 → 20.20). The larger gain on GSM8K (which tests grade-school math word problems) compared to MATH500 (which tests competition-level mathematics) may reflect that L3 synthetic data is particularly effective at building foundational problem-solving patterns, or that competition-level math requires capabilities beyond what 10B tokens of annealing can impart.

• Code: Average scores are 34.49 pp (L1), 35.29 pp (L2), 36.28 pp (L3), for a total L1→L3 gain of +1.79 pp — the second-smallest improvement among the domains. MBPP improves from 43.97 (L1) to 44.36 (L2) to 45.73 (L3), a modest +1.76 pp gain. HumanEval improves from 25.00 (L1) to 26.22 (L2) to 26.83 (L3), a +1.83 pp gain. The code domain shows the most linear quality gradient of the four, with roughly equal improvements at each tier transition, suggesting that the code L2 (Stack-Edu) and L3 (textbook rewrites) provide incremental rather than transformative quality improvements under this evaluation budget.

Interpretation and caveats. The consistent L3 > L2 > L1 ordering across all four domains is evidence that the tiered framework captures genuine quality distinctions. However, several aspects of these results warrant caution:

First, the absolute performance levels are very low — even L3 English web data produces only 53.96 pp average across 10 benchmarks, and L3 Math produces 30.84 pp average. These are models trained on only 10B tokens of annealing on top of a 1.1T pre-trained base; the absolute scores primarily reflect the limited verification budget rather than the ceiling of what the data can achieve. The meaningful signal is the relative ordering, not the absolute values.

Second, the quality gradient is not uniform across domains. Math shows a strongly non-linear pattern (L1→L2: +1.04 pp; L2→L3: +6.02 pp) while code shows a nearly linear pattern (L1→L2: +0.80 pp; L2→L3: +0.99 pp). This suggests that the L2→L3 transition — which involves LLM-based editing and synthesis — provides substantially more value for math (where synthetic generation can create novel, reasoning-dense problems) than for code (where textbook-style rewriting provides only modest improvements over already-filtered code repositories). The paper does not discuss this domain-dependent non-linearity, but it has practical implications: it suggests the cost-benefit ratio of L3 processing is much more favorable for math than for code.

Third, the efficient verification protocol uses a 30/70 mixture (30% verification data, 70% default distribution). This means the model sees only 3B tokens of the candidate dataset during annealing — a tiny fraction of the data available (e.g., Math-L3 is 88B tokens, so only ~3.4% is actually used). The evaluation therefore measures the marginal benefit of adding a small amount of tiered data to a pre-existing training distribution, not the quality of the tiered data in isolation or at scale. This is a reasonable protocol for comparing data quality efficiently, but it means the reported quality gaps may not reflect what would happen if models were trained exclusively or primarily on each tier.

Math Data Case Study at Scale (Section 3.3)

To test whether the quality advantages observed under efficient verification persist at larger training scales, the paper conducts a dedicated case study on UltraData-Math using the decay verification protocol (100B tokens of training on a 1.3T-token pre-trained base, 30% math verification data). This represents a 10× increase in verification budget over the efficient verification math experiments and a more realistic assessment of how tiered math data affects near-convergence model performance.

Math domain results (Table 6). Under decay verification, the L3 > L2 > L1 ordering is preserved and the absolute gaps are more informative:

• MATH500: L1 achieves 27.78 pp, L2 achieves 29.20 pp (+1.42 pp), L3 achieves 37.02 pp (+7.82 pp over L2, +9.24 pp over L1). The L2→L3 jump remains disproportionately large, confirming the efficient verification finding that synthetic/refined math data is substantially more valuable than model-selected math data.
• GSM8K: L1 achieves 54.66 pp, L2 achieves 52.92 pp (−1.74 pp), L3 achieves 61.79 pp (+8.87 pp over L2, +7.13 pp over L1). Notably, L2 underperforms L1 on GSM8K — this is the only case in all reported experiments where a higher tier produces lower performance than a lower tier. The paper does not comment on this anomaly, but it suggests that the L2 selection classifier may filter out GSM8K-relevant content (perhaps simple arithmetic word problems that score low on "reasoning density" but are valuable for grade-school math capability), while L3 synthesis recovers this content through difficulty-stratified Q&A generation.
• Math average: L1: 41.22 pp, L2: 41.06 pp (essentially identical to L1 at −0.16 pp), L3: 49.41 pp (+8.35 pp over L2, +8.19 pp over L1). The math average improvement from L1 to L3 is +8.19 pp, comparable to the +7.06 pp observed under efficient verification — suggesting that the relative benefit of higher-tier math data is stable across different evaluation scales.

Cross-domain transfer results (Table 6). The most striking finding in this experiment is that Math-L3 produces substantial improvements on non-math benchmarks compared to Math-L1 and Math-L2:

• English average: Math-L3 achieves 54.77 pp, compared to 51.32 pp (Math-L1) and 51.69 pp (Math-L2). The gain over Math-L1 is +3.45 pp across 10 English benchmarks — larger than the +1.70 pp gain achieved by upgrading English web data itself from L1 to L3 (Table 5, Web-En). This means that improving math data quality produces more English benchmark improvement than improving English data quality, at least under these evaluation protocols and data scales.
• Reasoning-intensive English benchmarks: The gains are concentrated on reasoning tasks rather than knowledge tasks. ARC-E improves by +5.29 pp (54.50 → 59.79), ARC-C by +1.69 pp (37.29 → 38.98), BBH by +5.87 pp (37.75 → 43.62), and OpenbookQA by +15.40 pp (41.60 → 57.00). In contrast, knowledge-heavy benchmarks like MMLU show modest improvement (+1.10 pp: 50.57 → 51.67), and commonsense reasoning benchmarks like HellaSwag and PIQA show minimal change. This pattern — large gains on structured reasoning tasks, small gains on factual knowledge tasks — supports the interpretation that high-quality math data improves general reasoning circuitry rather than providing domain-specific knowledge.
• Chinese average: Math-L3 achieves 53.48 pp, compared to 51.59 pp (Math-L1) and 50.84 pp (Math-L2), a gain of +1.89 pp over L1 and +2.64 pp over L2. The improvement is observed on both CMMLU (+1.59 pp) and C-Eval (+2.19 pp). This cross-lingual transfer — English math data improving Chinese language understanding — is particularly notable because it suggests the reasoning improvements are language-agnostic, operating at the level of logical operations rather than surface-form linguistic patterns.
• Code average: Math-L3 achieves 41.10 pp, compared to 37.30 pp (Math-L1) and 38.41 pp (Math-L2), gains of +3.80 pp and +2.69 pp respectively. MBPP improves by +4.56 pp (44.71 → 49.27), HumanEval by +3.05 pp (29.88 → 32.93). The code improvement is consistent with the reasoning-transfer hypothesis: mathematical reasoning and code generation share underlying requirements for structured problem decomposition, step-by-step logical progression, and attention to formal constraints.
• Overall average: Math-L3 achieves 58.27 pp compared to 48.39 pp (Math-L1) and 48.59 pp (Math-L2), a gain of +9.88 pp over L1 and +9.68 pp over L2. Math-L2 provides essentially no improvement over Math-L1 in overall average (48.59 vs. 48.39, a negligible +0.20 pp), while Math-L3 provides a dramatic jump. This reinforces the domain-dependent non-linearity observed earlier: for math data, the L1→L2 transition (model-based selection) is low-value or even neutral, while the L2→L3 transition (synthetic generation) is high-value.

Interpretation. The decay verification results strengthen the efficient verification findings in several ways. First, they confirm that the L3 quality advantage is not an artifact of limited training budgets — it persists and in some cases amplifies at 10× the verification scale. Second, the cross-domain transfer effects demonstrate that the benefits of high-quality reasoning data extend far beyond the training domain, with math data serving as a surprisingly effective driver of general logical competence. Third, the Math-L2 anomaly (underperforming Math-L1 on GSM8K and showing no overall improvement) raises questions about what the L2 classifier is selecting for — if it prioritizes advanced mathematical reasoning at the expense of foundational problem-solving, the data may be optimized for MATH500 at the expense of GSM8K, an unintended consequence of the classifier's training objective.

However, the decay verification shares the efficient verification's limitation of evaluating only a 30% mixture of verification data. The model still receives 70% of its tokens from the default distribution, meaning the measured effects represent the marginal benefit of enriching an existing training distribution with tiered math data, not the quality of tiered math data as a standalone pre-training corpus. The cross-domain transfer effects, while impressive in relative terms, are achieved by replacing 30% of the decay-phase tokens with math data — it is possible that a different 30% allocation (e.g., structured logic puzzles, formal reasoning datasets) would produce even stronger transfer, but the paper does not explore alternative domain allocations.

Tiered vs. Mixed Training (Section 3.4)

This experiment tests the paper's central operational claim: that scheduling data by quality tier during training (L1 → L2 → L3 sequentially) outperforms uniformly mixing all tiers, holding total data volume, quality composition, and domain distribution constant.

Experimental setup. Both strategies use the MiniCPM-1.2B model trained from scratch via pre-train verification (120B tokens, cosine decay schedule). The domain distribution is fixed at 50% Web-en, 25% Web-zh, 8% Math, 17% Code. The total L1:L2:L3 ratio is 1:1:1 in both conditions, meaning each tier contributes 40B tokens to the overall 120B budget. In the mix training condition, all 120B tokens are pooled and sampled uniformly throughout training. In the tiered training condition, the 120B tokens are partitioned into three consecutive 40B-token stages: Stage 1 uses only L1 data, Stage 2 uses only L2 data, Stage 3 uses only L3 data.

Final benchmark results (Table 7). Tiered training achieves an overall average of 31.66 pp compared to 30.17 pp for mix training, an improvement of +1.49 pp. The gains are distributed across all four domains:

• English: Tiered achieves 42.29 pp vs. mix 41.41 pp (+0.88 pp). Individual benchmark improvements include MMLU +0.89 pp (28.26 → 29.15), ARC-C +4.75 pp (26.78 → 31.53), ARC-E +1.77 pp (48.32 → 50.09), BBH +2.17 pp (26.20 → 28.37), and OpenbookQA +3.00 pp (26.00 → 29.00). Small decreases are observed on HellaSwag (−1.68 pp), PIQA (−0.82 pp), SIQA (−0.56 pp), and Winogrande (−0.87 pp) — tasks that the paper describes as "focusing more on the breadth rather than depth of common sense reasoning and language understanding."
• Chinese: Tiered achieves 27.54 pp vs. mix 24.72 pp (+2.82 pp). C-Eval improves by +4.40 pp (23.97 → 28.37), CMMLU by +1.24 pp (25.47 → 26.71).
• Math: Tiered achieves 4.60 pp vs. mix 2.36 pp (+2.24 pp). MATH500 improves by +2.60 pp (1.60 → 4.20), GSM8K by +1.89 pp (3.11 → 5.00). The absolute performance is extremely low for both strategies — this is expected for a 1.2B model trained on only 120B tokens with 8% math data (9.6B tokens of math total), but it means the relative improvement, while proportionally large (+95% for math), operates in a very low-absolute-performance regime.
• Code: Tiered achieves 9.70 pp vs. mix 7.25 pp (+2.45 pp). MBPP improves by +4.28 pp (12.06 → 16.34), HumanEval by +0.61 pp (2.44 → 3.05).

Training dynamics (Figure 3). The training curves reveal the mechanism behind the final performance gap. In the early training stage (approximately 0–40B tokens), both strategies exhibit similar growth, with performance improving from roughly 24.7 pp to approximately 28.3 pp. At this stage, tiered training is using only L1 data, while mix training is using a uniform blend of L1/L2/L3 — the similar performance suggests that L2 and L3 data provide limited benefit in early training when the model is still acquiring basic linguistic representations.

In the later training stages (40–120B tokens), the curves diverge sharply. Tiered training, which introduces L2 data at 40B tokens and L3 data at 80B tokens, shows sustained performance improvement from approximately 28.35 pp to 31.66 pp, an increase of +3.31 pp. Mix training, continuing to sample uniformly from all tiers, improves from approximately 28.26 pp to 30.17 pp, an increase of only +1.91 pp. The late-stage learning rate of tiered training is approximately 1.7× that of mix training (3.31 / 1.91 ≈ 1.73).

Interpretation. The paper interprets this result as evidence that low-quality data in late training stages interferes with capability refinement: in the mix condition, the model continues seeing L1-quality samples throughout the entire training run, including the final stages where it should be consolidating complex reasoning capabilities learned from L2 and L3 data. The tiered condition eliminates this interference by concentrating L1 data in early training (where it provides useful linguistic diversity) and reserving L3 data for the final stage (where it can be absorbed without dilution).

This interpretation is consistent with the pattern of benchmark improvements: the largest gains from tiered training are on reasoning-intensive tasks (ARC-C +4.75 pp, BBH +2.17 pp, OpenbookQA +3.00 pp, MATH500 +2.60 pp) and knowledge-intensive tasks (MMLU +0.89 pp, C-Eval +4.40 pp, CMMLU +1.24 pp), while commonsense reasoning tasks that benefit from broad exposure to diverse situations (HellaSwag, PIQA, SIQA) show small decreases. The decreases are consistent with the hypothesis that tiered training's exclusion of L1 data in later stages reduces exposure to the diverse commonsense scenarios that L1 data contains — a trade-off that the framework accepts in exchange for larger gains on reasoning and knowledge tasks.

Caveats. Several features of this experiment limit the strength of the conclusions. First, the absolute performance is low across all benchmarks — the overall average of 31.66 pp for tiered training and 30.17 pp for mix training reflects models trained on only 120B tokens, far from convergence. Whether the tiered advantage persists, amplifies, or diminishes at production scales (trillions of tokens) is not established. The sustained improvement trend in the tiered training curve (Figure 3) suggests it may continue, but the curve has not plateaued, so extrapolation is uncertain.

Second, the experiment tests only one specific schedule (three equal 40B-token stages, 1:1:1 ratio). The paper does not explore whether a different allocation (e.g., 20B L1 → 40B L2 → 60B L3, or L1 → L3 without L2, or a gradual transition rather than discrete stage boundaries) would perform better or worse. The 1:1:1 equal-stage design is a reasonable proof-of-concept but does not establish that this particular schedule is optimal or even close to optimal.

Third, the domain distribution (50% Web-en, 25% Web-zh, 8% Math, 17% Code) is held constant and not ablated. If tiered training's benefits derive primarily from concentrating L3 math data in the final stage (where its cross-domain reasoning benefits manifest), the results may be sensitive to the fraction of math data in the mixture. The paper does not report per-domain training curves, making it impossible to determine which domain transitions are driving the late-stage divergence.

Ablation Studies and Robustness Checks

The paper does not include a dedicated ablation studies section. The following are implicit ablations and robustness checks that can be extracted from the experimental design and results:

Domain variation as implicit ablation of the tier framework: The four-domain evaluation (English web, Chinese web, math, code) can be viewed as an ablation over data modality. The consistent L3 > L2 > L1 ordering across all four domains (Table 5) demonstrates that the tiered framework generalizes across text types. However, the varying magnitude of the quality gradient — strongest for math (+7.06 pp L1→L3), weakest for English web (+1.70 pp L1→L3) — reveals that the value of higher-tier processing is domain-dependent, an interaction the paper does not systematically explore.

Verification protocol as implicit robustness check on scale: The math data results are evaluated under both efficient verification (10B tokens, Table 5) and decay verification (100B tokens, Table 6). The L3 advantage persists at both scales (+7.06 pp and +8.19 pp respectively), suggesting the quality stratification is robust to evaluation budget. However, the Math-L2 anomaly (underperforming Math-L1 on GSM8K and math average under decay verification but not efficient verification) indicates that some tier effects are scale-sensitive — L2's weakness relative to L1 only appears at larger training budgets.

Math-L2 regression on GSM8K: Under decay verification (Table 6), Math-L2 scores 52.92 pp on GSM8K compared to Math-L1's 54.66 pp, a decrease of −1.74 pp. This is the only case in all experiments where a higher tier performs worse than a lower tier on any benchmark. The paper does not ablate the L2 classifier's training data, threshold, or selection criteria to diagnose this regression. Possible explanations include: the L2 classifier over-prioritizes advanced mathematical content at the expense of simple arithmetic (which GSM8K primarily tests); the L2 selection process reduces data diversity in ways that disproportionately affect grade-school math; or the result is simply noise within the variance of the decay verification protocol. Without variance estimates or classifier ablations, the cause cannot be determined.

Tier proportion as implicit ablation in the tiered training experiment: The tiered training experiment holds the total L1:L2:L3 ratio constant at 1:1:1 while varying only the schedule. This implicitly ablates the hypothesis that the schedule matters independently of the proportion — the result confirms that schedule does matter, but does not explore whether a different proportion (e.g., 1:2:3 or 3:2:1) would produce larger or smaller schedule effects.

Missing ablations. Several ablations that would strengthen the paper's claims are absent:

• No classifier quality ablation: The L1→L2 transition quality depends on the L2 classifier. The paper does not compare different classifiers (e.g., fastText vs. lightweight transformer vs. LLM-based scoring), different classification thresholds, or different seed data selection strategies for classifier training. This makes it impossible to determine whether the observed L1→L2 gains are near the ceiling of what model-based selection can achieve or could be substantially improved.
• No synthesis quality ablation: The L2→L3 transition for math uses a multi-model ensemble generating five instructional formats. The paper does not ablate the number of models, the number of formats, or the synthesis approach (e.g., comparing single-model synthesis to multi-model ensemble, or comparing editing-based L3 to generation-based L3 for math). The large L2→L3 gap for math makes this ablation particularly important — understanding which synthesis components drive the gain would guide practical adoption.
• No tiered schedule ablation: The tiered training experiment tests only one schedule (40B per stage, 1:1:1 ratio). Ablating the stage boundaries (when to switch tiers), the stage ratios (how many tokens per tier), and the transition strategy (discrete switches vs. gradual interpolation between tiers) would characterize the sensitivity of the tiered advantage to schedule parameters.
• No model scale ablation: All experiments use MiniCPM-1.2B. The paper does not test whether the tiered data advantages and the tiered training benefits scale with model size. It is possible that larger models, with their greater capacity, benefit more from high-tier data (because they can absorb more complex patterns) or less (because they can extract more value from lower-tier data through emergent capabilities).

Critical Assessment

The paper makes three central claims that the experiments aim to support: (1) the L0–L4 tiered framework produces meaningful, monotonic quality differentiation (L3 > L2 > L1), (2) tiered training schedules (L1 → L2 → L3 sequentially) outperform uniform mixing of all tiers, and (3) high-quality math data produces cross-domain improvements in general reasoning, language understanding, and code generation. The experiments provide varying degrees of support for each claim, with specific conditions and limitations that warrant careful examination.

Claim 1: L3 > L2 > L1 quality ordering is meaningful and monotonic. The evidence from Table 5 (efficient verification across four domains) and Table 6 (decay verification for math) is consistent with this claim: the L3 > L2 > L1 ordering holds in 7 of the 8 domain-protocol combinations (the exception being Math-L2 underperforming Math-L1 on GSM8K and math average in decay verification). However, what "meaningful" means requires qualification. The quality gap between L1 and L2 is small for several domains: +1.10 pp for English web, +1.22 pp for Chinese web, +1.04 pp for math, +0.80 pp for code under efficient verification. These are modest improvements that, given the absence of variance estimates, could plausibly be within the noise range of the evaluation protocol. The L2→L3 gap is larger in all cases, but the magnitude varies dramatically: +6.02 pp for math vs. +0.60 pp for English web vs. +0.82 pp for Chinese web vs. +0.99 pp for code. This domain-dependent non-linearity — which the paper does not discuss — suggests that the tier framework does not produce a uniform "quality increment" per tier transition. Practitioners adopting the framework should expect that the L1→L2→L3 progression provides sharply diminishing returns for web data but strongly accelerating returns for math data, a pattern driven by the nature of the L3 processing (synthetic generation is transformative for math, editing refinement is incremental for web text).

The experiments also do not establish that L2 data is cost-effective relative to L1, or that L3 data is cost-effective relative to L2. The paper acknowledges the cost hierarchy (L3 > L2 > L1) but provides no cost accounting — no FLOP counts, GPU-hours, or dollar estimates for the processing pipelines. Without cost data, a practitioner cannot determine whether the +1.70 pp gain from upgrading English web data from L1 to L3 justifies the additional computational investment, or whether it would be more efficient to simply train on more L1 tokens.

Claim 2: Tiered training outperforms mixed training. The evidence from Table 7 and Figure 3 supports a narrower version of this claim than the paper's framing suggests. The experiment demonstrates that, for a 1.2B model trained on 120B tokens with a 1:1:1 L1:L2:L3 ratio and a specific domain distribution, the L1 → L2 → L3 schedule outperforms uniform mixing by +1.49 pp overall. This is a proof-of-concept that scheduling matters, but it does not establish the general superiority of tiered training, nor does it provide guidance on how to optimize the schedule.

Specific limitations of this experiment:

• Single model scale (1.2B). The paper does not demonstrate that tiered training scales to larger models where pre-training dynamics may differ (e.g., larger models may learn faster from high-quality data in early stages, reducing the benefit of L1 pre-training, or may be more robust to low-quality interference in late stages).
• Single training budget (120B tokens). At this scale, the model is far from convergence on all benchmarks (overall average ~31 pp). The tiered advantage is observed as a difference in learning rate in the later stages (Figure 3) — whether this translates to a difference in asymptotic performance at convergence is not tested.
• Fixed schedule and ratio. The 1:1:1 ratio with 40B-token stages is one arbitrary point in a large schedule space. The paper does not establish that this schedule is near-optimal, or characterize the sensitivity of the tiered advantage to schedule parameters. It is possible that other schedules (e.g., more L3 tokens, earlier L2 introduction, gradual transitions) would outperform the tested configuration.
• Unclear mechanism. The paper attributes the tiered advantage to "low-quality data interference" in late training, but provides no direct evidence for this mechanism — no analysis of gradient conflict between tiers, no measurement of forgetting when switching tiers, no comparison of loss landscapes between mixed and tiered models. The training curve evidence (Figure 3) is consistent with the interference hypothesis but also consistent with alternative explanations (e.g., L3 data simply provides more learnable signal per token at late training stages, independent of interference from L1 data).

Claim 3: High-quality math data drives cross-domain transfer. The evidence from Table 6 strongly supports this claim for the specific configuration tested: Math-L3 (88B tokens of synthetic/refined content) produces substantial improvements on English benchmarks (+3.45 pp over Math-L1), Chinese benchmarks (+1.89 pp), and code benchmarks (+3.80 pp) when used as 30% of the data in a 100B-token decay phase. The pattern of improvements — concentrated on reasoning-intensive tasks (ARC-C, BBH, OpenbookQA) rather than factual knowledge tasks — is internally consistent with the paper's interpretation that structured reasoning capability transfers across domains.

However, several aspects of this result warrant qualification:

• The transfer is from replacing 30% of tokens, not from training on math alone. The model receives 70% of its decay-phase tokens from the default distribution. The cross-domain improvements are therefore the marginal effect of enriching an already-diverse training distribution with high-quality math, not the effect of math data in isolation. It is possible that the observed transfer would not manifest in a model trained primarily on math data (where the narrow domain distribution might cause catastrophic forgetting of non-math capabilities).
• The transfer mechanism is not isolated. The paper claims the transfer is driven by improved "general logical consistency and problem-solving capabilities," but the experiment does not distinguish between several plausible mechanisms: (a) math data directly trains reasoning circuits that apply to non-math tasks, (b) math data improves the model's ability to follow complex multi-step instructions, which benefits all generation-based evaluations, (c) math data improves the model's attention to detail and precision, reducing careless errors across domains, or (d) math data serves as a regularizer that prevents overfitting to surface-level patterns in web data. Distinguishing these mechanisms would require targeted experiments (e.g., testing whether synthetic data without mathematical content but with similar structural complexity produces similar transfer).
• The Math-L2 baseline is effectively flat. Math-L2 produces essentially no improvement over Math-L1 on non-math benchmarks (overall average: 48.59 vs. 48.39, +0.20 pp). This means the cross-domain transfer is entirely driven by the L2→L3 transition — i.e., by synthetic generation specifically, not by mathematical data quality generally. This is an important boundary condition: simply having "better math data" (L2) is insufficient for cross-domain transfer; you need L3-level synthetic data with its explicit pedagogical structure, multi-format diversity, and difficulty-stratified curriculum. The paper does not explicitly flag this condition, but it follows directly from the data.

What is genuinely established and what is not. The paper convincingly establishes that:

1. Within its tiered framework, L3 data consistently outperforms L1 data across four domains, with math showing the largest quality gradient and code/web showing more modest gradients (Tables 5, 6).
2. For a 1.2B model at 120B tokens, scheduling data as L1 → L2 → L3 outperforms uniform mixing, with the advantage emerging primarily in later training stages (Table 7, Figure 3).
3. High-quality synthetic math data (L3) produces cross-domain improvements on reasoning benchmarks when used in a 30% mixture during decay-phase training (Table 6).

The paper does not establish:

1. That the tiered framework provides cost-effective quality improvements — no cost accounting is provided for any tier transition.
2. That the specific tier definitions (what counts as L1 vs. L2 vs. L3) are optimal or even close to optimal — no alternative tiering schemes are tested.
3. That tiered training scales to larger models, larger budgets, or different domain distributions — all experiments use a single model scale, a single training budget, and a single domain mixture.
4. That the cross-domain transfer from math data is unique to math or would not be matched by other reasoning-intensive domains — no comparison to logic, formal reasoning, or structured science data is provided.
5. The statistical reliability of the reported differences — no confidence intervals, error bars, or multiple-run variance estimates are reported for any experiment.

Missing experiments that would strengthen the paper:

• Cost-normalized comparisons: For each domain, compare models trained on L1, L2, and L3 data under a fixed compute budget that accounts for both data processing cost and training cost. This would answer the question practitioners actually face: "given my total budget, should I invest in higher-tier data or in more training on lower-tier data?"
• Model scale ablation: Repeat the tiered-vs-mixed experiment at 0.5B, 1.2B, and 3B+ scales to determine whether the tiered training advantage scales with model size.
• Schedule optimization: Sweep stage boundaries, stage ratios, and transition strategies to characterize the sensitivity of the tiered advantage and identify near-optimal schedules.
• Classifier comparison for L2: Compare fastText vs. lightweight transformer vs. LLM-based classifiers for the L2 selection step, measuring both downstream performance and processing cost.
• Direct interference measurement: In the mixed training condition, track per-tier loss and gradient statistics in late training to directly test whether L1 data produces interfering gradients relative to L3 data, providing mechanistic evidence for the paper's interference interpretation.
• Alternative domain transfer: Replace math L3 data with logic, structured science, or formal reasoning L3 data in the decay verification protocol to determine whether the cross-domain transfer is specific to math or general to structured reasoning.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Amortized Into the Framework's Efficiency

The assumption. The entire tiered data management framework rests on the ability to distinguish between quality tiers — L1 from L2 from L3 — and to route data through increasingly expensive processing pipelines accordingly. The paper provides detailed descriptions of the operators that perform these transitions: heuristic filters for L1, model-based classifiers for L2, LLM-based editors and synthesizers for L3. However, the paper provides no cost accounting for any of these operations. There are no measurements of GPU-hours, FLOPs, wall-clock time, or dollar cost for:

• Training the L2 quality classifiers (fastText models trained on LLM-annotated seed data)
• Running L2 classifier inference over hundreds of billions of tokens
• Running LLM-based editing refinement at L3 (for web data)
• Running multi-model ensemble synthesis at L3 (for math data, involving five instructional formats with multiple LLMs)
• The quality filtering and validation stages at each tier

This absence is acknowledged implicitly through the paper's consistent reference to a cost hierarchy — "data quality increases with each tier, albeit with a corresponding rise in acquisition and computational costs" (Section 2.2) — but this qualitative statement is never operationalized with numbers.

The consequence. Without cost accounting, the paper's central value proposition — that tiered data management provides a principled framework for balancing "data quality, acquisition cost, and marginal training benefit" (Section 1, Section 2.2) — remains unvalidated for the cost dimension. A practitioner reading this paper cannot answer the most basic resource-allocation question: given a fixed compute budget, should I invest in upgrading my data from L1 to L2, from L2 to L3, or simply in training on more L1 tokens?

This gap is particularly acute because the paper's own results show that the quality gradient is highly domain-dependent. For English web data, L1 → L3 yields only a +1.70 pp improvement under efficient verification (Table 5). For math data, the same transition yields +7.06 pp. If the cost of producing L3 English web data (LLM-based editing of 200B tokens) is comparable to the cost of producing L3 math data (multi-model ensemble synthesis of 88B tokens), then the return on investment is roughly 4× higher for math. But if L3 web editing is substantially cheaper than L3 math synthesis (because editing existing text is less expensive than generating novel problems), the cost-effectiveness ordering might reverse. The paper provides no data to resolve this.

The tiered-vs-mixed training experiment (Section 3.4) suffers from a related cost-blindness: the tiered strategy produces better models, but the processing cost of separating data into L1, L2, and L3 tiers is not accounted for. If producing tiered data costs 2× more than simply filtering everything to L1 quality, a fair comparison would give the mixed-training baseline 2× more tokens to compensate. The paper acknowledges this implicitly when noting that "in the early stage of data management, lightweight and low-cost methods (such as heuristic filtering) should be adopted, while in deeper management stage, more fine-grained and higher-cost approaches (such as LLM-based labeling) should be used" (Section 1), but never integrates cost into the experimental comparisons.

Evidence in the paper. There is none. No figure, table, or section reports processing costs. The paper's efficient verification and decay verification protocols measure training cost (in tokens) but not data preparation cost. The open-source release of datasets and tools (Table 1) enables external cost measurement, but the paper itself does not provide it.

Mitigation status. The paper does not attempt to address this limitation. It is framed as future work in the conclusion's call for "more rigorous methods for scientific data value assessment, enabling models to quantitatively estimate the marginal utility of data across tiers and training stages" (Section 4), but this is a forward-looking aspiration rather than a concrete mitigation. The "Ultra-FineWeb" methodology (Section 2.2.3) includes an efficient validation strategy for classifier training that reduces GPU-hours relative to naive approaches, but this only addresses a sub-component of the overall pipeline cost and provides only qualitative comparison relative to an unspecified baseline, not absolute cost measurements.

---

The Framework Is Validated on a Single Model Scale and Training Budget

The assumption. All experiments in the paper use the MiniCPM-1.2B architecture trained on token budgets ranging from 10B (efficient verification) to 120B (pre-train verification) to ~100B (decay verification, applied on top of a 1.3T pre-trained base). The paper implicitly assumes that the observed quality stratifications and tiered-training advantages will transfer to larger model scales and larger training budgets. The authors state that MiniCPM-1.2B is "representative of current small-scale LLM training dynamics" but provide no empirical evidence that these dynamics are scale-invariant.

The consequence. This limitation creates uncertainty about the framework's applicability in two distinct regimes:

Regime 1: Larger models. Modern production LLMs are typically in the 7B–70B+ parameter range, 6–60× larger than the tested architecture. Larger models have greater capacity and may exhibit different data-quality sensitivity. Several plausible failure modes exist:

• Larger models may benefit more from L1 data scale and diversity (because their greater capacity allows them to extract useful signal from noisy corpora that smaller models cannot), reducing the relative advantage of higher tiers.
• Conversely, larger models may saturate more slowly on high-quality data (because they have more capacity to absorb complex patterns), increasing the relative advantage of L3 data.
• Larger models may be more robust to low-quality interference in late training (because their overparameterization provides implicit regularization), weakening the tiered-training advantage.
• Larger models may require different optimal tier ratios — the 1:1:1 L1:L2:L3 split tested at 1.2B scale may not be appropriate at 7B+ scale.

The paper provides no evidence to distinguish these scenarios. The μP parameterization (used for training stability) is designed to make optimal hyperparameters transfer across model scales, but this only addresses training dynamics, not data-quality sensitivity.

Regime 2: Larger training budgets. The tiered-vs-mixed training experiment (Section 3.4) operates at 120B tokens, which is far from convergence — the overall average accuracy for both strategies is ~30–32 pp, well below what the architecture could achieve with continued training. The training curves (Figure 3) show that both strategies are still improving at the 120B cutoff, with no sign of plateau. The paper interprets the tiered advantage as evidence that quality scheduling matters, but the experiment cannot distinguish between two interpretations: (a) tiered training achieves a permanently higher learning rate that will produce a larger gap at convergence, or (b) tiered training converges faster initially but both strategies asymptote to the same performance, with the tiered advantage being a transient effect of accelerated early-stage learning.

Distinguishing these interpretations matters enormously for a practitioner: if (a), tiered training is a fundamental improvement that should be adopted for all large-scale training runs. If (b), tiered training is a useful acceleration technique for compute-limited settings but provides no asymptotic benefit for well-resourced training runs.

Evidence in the paper. The paper provides no model-scale ablation and no training-budget sweep for the tiered-training experiment. The training curves (Figure 3) show 5 evaluation checkpoints at approximately 20B-token intervals, but none are near convergence. The paper reports that tiered training achieves "a sustained and stable trend of learning capability improvement" (Section 3.4), but this observation is limited to the studied budget range.

Mitigation status. The paper does not address this limitation beyond acknowledging the single-model-scale choice as a deliberate trade-off "to balance validation comprehensiveness with computational efficiency" (Section 3.1). The μP parameterization provides some transferability for hyperparameters but not for data-quality sensitivity. The conclusion calls for extending the framework to "broader modalities and application domains" (Section 4) but does not specifically prioritize scale-validation studies.

---

The Tiered Training Advantage Is Small in Absolute Terms, and Domain-Dependent Regressions Are Unexplained

The assumption. The paper presents the tiered-training advantage (Section 3.4) as evidence that quality-stratified scheduling is strictly superior to uniform mixing. The experimental design cleanly isolates scheduling as the causal variable, and the reported +1.49 pp overall improvement (31.66 vs. 30.17) is directionally consistent with the paper's thesis. However, the paper treats this as an unqualified positive result without examining the distribution of effects across benchmarks or the statistical reliability of the measured differences.

The consequence. A closer examination of the per-benchmark results (Table 7) reveals that the tiered advantage is not uniform — several benchmarks show performance decreases under tiered training:

These four benchmarks represent commonsense reasoning and language understanding tasks that "focus more on the breadth rather than depth of common sense reasoning and language understanding" (Section 3.4). The paper acknowledges these decreases and attributes them to the reduced L1 data exposure in later training stages — the model sees less diverse commonsense scenarios when L1 data is excluded from the final 80B tokens.

This creates a fundamental trade-off that the paper does not resolve: tiered training improves reasoning and knowledge tasks at the expense of broad commonsense coverage. A practitioner who deploys tiered training must accept degraded performance on some capability dimensions in exchange for gains on others. The paper's reporting of an overall average improvement masks this trade-off — the average gain is driven by larger improvements on reasoning tasks (ARC-C +4.75, BBH +2.17, OpenbookQA +3.00, MATH500 +2.60, MBPP +4.28) that outweigh the decreases on commonsense tasks. But for an application where commonsense reasoning is the primary requirement (e.g., a general-purpose chatbot, a social media content moderator), tiered training would produce a worse model.

Furthermore, the paper provides no variance estimates — no confidence intervals, standard errors, or multiple-run replications for any experiment. Without these, it is impossible to determine whether the reported differences are statistically reliable or within the noise range of the training process. The +1.49 pp overall improvement for tiered training, distributed across 16 benchmarks, corresponds to an average per-benchmark improvement of ~0.09 pp. Individual benchmark differences range from −1.68 to +4.75 pp. In the absence of variance estimates, a skeptical reader could reasonably question whether the positive results reflect true effects or are partially attributable to training stochasticity — especially since the paper's efficient verification protocol is explicitly noted to "have higher variance in results" due to limited training budget, and the pre-train verification protocol at 120B tokens shares structural similarities (limited budget relative to convergence).

Evidence in the paper. The per-benchmark regressions are clearly visible in Table 7 (HellaSwag, PIQA, SIQA, Winogrande). The paper acknowledges these decreases with a single sentence attribution to breadth-vs-depth trade-offs but provides no analysis of whether the decreases are statistically significant, whether they would persist or diminish with more training, or whether alternative schedules (e.g., including a small fraction of L1 data in later stages) could mitigate them. The absence of variance estimates is a methodological gap across all experiments (Tables 5–7, Figure 3).

Mitigation status. The paper does not attempt to address this trade-off. It does not explore hybrid schedules that might preserve commonsense performance (e.g., L1 → L1+L2 → L2+L3 with gradually changing mixtures rather than discrete tier switches). It does not characterize whether the commonsense regressions are a necessary consequence of tiered training or an artifact of the specific 1:1:1 schedule and 40B stage boundaries. It does not provide the variance estimates that would allow practitioners to assess the reliability of the reported differences. The conclusion calls for "dynamic data–model feedback mechanisms, where model signals continuously inform data selection, refinement, and allocation during training" (Section 4), which could in principle enable adaptive scheduling that avoids these regressions, but this is a forward-looking direction rather than a mitigation of the current limitation.

---

The Cross-Domain Transfer Claim Lacks Mechanistic Evidence and Alternative-Explanation Controls

The assumption. Section 3.3 presents the finding that Math-L3 data produces improvements on English (+3.45 pp), Chinese (+1.89 pp), and Code (+3.80 pp) benchmarks as evidence that "high-quality mathematical data is a fundamental driver of enhancing a model's general logical consistency and problem-solving capabilities across diverse languages and tasks" (Section 3.3). The paper interprets this as a causal transfer effect: training on structured mathematical reasoning content directly improves the model's ability to reason across domains.

The consequence. This interpretation, while plausible and consistent with the observed benchmark improvements, is underdetermined by the experimental design. The decay verification protocol mixes 30% Math-L3 data with 70% default distribution data — meaning the model receives both the math data and a diverse general corpus during the decay phase. The observed cross-domain improvements could arise from several distinct mechanisms, and the experiment cannot distinguish them:

Mechanism A (Direct reasoning transfer): Math-L3 data trains reasoning circuits that generalize to non-math tasks. This is the paper's preferred interpretation — the structured, step-by-step logical patterns learned from math problems transfer directly to logical reasoning in language understanding and code generation.

Mechanism B (Interaction effect with general data): Math-L3 data does not produce reasoning improvements on its own, but rather enhances the model's ability to learn from the general data in the 70% default distribution. The math data might, for example, improve the model's attention to logical structure, which then enables more efficient extraction of reasoning patterns from the general corpus. In this case, the cross-domain gains would not manifest in a model trained on math data alone — they depend on the interaction between math data and diverse general data.

Mechanism C (Reduced exposure to noisy data): Replacing 30% of the decay-phase tokens with Math-L3 data necessarily reduces exposure to whatever was in the default distribution. If the default distribution contains noisy or low-quality samples that interfere with reasoning capability, the improvement might come from simply removing those samples rather than adding math data. The paper cannot rule this out because there is no control condition that replaces 30% of tokens with a non-math high-quality dataset (e.g., L3 web data, L3 code data) to measure domain-specific transfer.

Mechanism D (Benchmark contamination or format similarity): If the Math-L3 synthetic data uses problem formats, reasoning templates, or linguistic patterns that resemble the evaluation benchmarks (even without content overlap), the model might learn surface-level features that improve benchmark scores without improving general reasoning capability. The paper does not perform decontamination analysis on the synthetic L3 data against the evaluation benchmarks.

The paper's confident causal attribution ("fundamental driver") is not justified without experiments that distinguish these mechanisms. The fact that Math-L2 — which is also mathematical data, but of lower quality — produces essentially no cross-domain transfer (overall average: 48.59 for L2 vs. 48.39 for L1, a negligible +0.20 pp) narrows the candidates (the transfer is specific to L3-quality math, ruling out generic "exposure to math tokens"), but does not isolate a mechanism.

Evidence in the paper. The cross-domain improvements are clearly documented in Table 6. The paper reports benchmark scores but provides no mechanistic analysis — no probing of internal representations, no comparison of attention patterns between math and non-math tasks, no measurement of whether the model's reasoning chains on English benchmarks become more "math-like" after Math-L3 training.

Mitigation status. The paper does not address this limitation. It does not include a control condition using non-math high-quality data, does not perform decontamination analysis, and does not analyze model internals to provide mechanistic evidence. The conclusion's call for "more rigorous methods for scientific data value assessment" (Section 4) could support future mechanistic studies, but the current paper presents its cross-domain transfer interpretation as an established finding rather than a hypothesis requiring further validation.

---

The Framework Provides No Guidance on Tier Thresholds, Ratios, or Schedule Optimization

The assumption. The L0–L4 framework defines five quality tiers with distinct properties and processing operators (Section 2.2). The experiments then test specific instantiations of these tiers — e.g., UltraData-Math-L1/L2/L3 — and a specific training schedule — three equal 40B-token stages with a 1:1:1 L1:L2:L3 ratio (Section 3.4). The paper implicitly assumes that these specific operationalizations are representative of what the framework can achieve, and that the observed improvements generalize to the framework's conceptual value.

The consequence. This limitation manifests in three distinct gaps that make the framework difficult to operationalize without extensive additional experimentation:

Gap 1: No guidance on tier boundary placement. The paper defines L1 as heuristic-filtered data, L2 as model-selected data, and L3 as edited/synthetic data. But within each tier, there is a continuous spectrum of processing intensity. For L2: how stringent should the classifier threshold be? The paper reports that UltraData-Math-L2 retains 33B tokens out of 170B (19% retention), while FineMath offers two versions: FineMath-4+ (10B tokens, retaining only top-scoring samples) and FineMath-3+ (34B tokens, broader retention). Both are "L2" but represent very different quality-volume trade-offs. The framework provides no principled basis for choosing a threshold — it is left as an implementation detail.

For L3: how much synthesis is enough? UltraData-Math-L3 is 88B tokens — larger than both L1 (170B) and L2 (33B) for the same domain. But the paper never justifies this scale. Could a 44B-token L3 corpus achieve most of the benefit at half the generation cost? Could a 176B-token L3 corpus provide even larger gains? The framework provides no guidance.

Gap 2: No guidance on inter-tier ratios in training mixtures. The tiered training experiment uses a 1:1:1 L1:L2:L3 ratio with 40B tokens per stage. This is one arbitrary point in a three-dimensional space (tokens per tier). The paper provides no evidence that this ratio is near-optimal, or any characterization of how sensitive the tiered advantage is to the ratio. A practitioner with limited L3 data (e.g., 10B tokens of synthetic math rather than 88B) cannot determine from this paper how to allocate it — should it all be concentrated in the final stage? Spread across multiple stages? Mixed with L2 data?

Gap 3: No guidance on stage boundaries and transition strategies. The tiered training experiment uses hard discrete switches between tiers: 0–40B tokens of L1, 40–80B of L2, 80–120B of L3. The paper provides no evidence that discrete switches are superior to gradual interpolation (e.g., linearly decreasing L1 fraction and increasing L2 fraction across a transition window), or that the 40B-token stage boundaries are appropriate for the model scale and training budget.

These gaps mean that the paper demonstrates the existence of a tiered-training advantage (for one specific configuration) without providing the engineering heuristics needed to realize that advantage in practice. A practitioner who adopts the L0–L4 framework would need to independently re-discover threshold choices, ratio allocations, and schedule parameters through their own experimentation — at which point the framework provides conceptual vocabulary but limited operational guidance.

Evidence in the paper. The lack of threshold, ratio, and schedule ablations is visible in the experimental design. Section 3.2 defines specific dataset choices for each tier (Table 4) but provides no justification for why these particular datasets and retention ratios were selected. Section 3.4 describes the tiered training experiment but provides no sensitivity analysis on stage boundaries or ratios. The paper does not acknowledge these gaps as limitations.

Mitigation status. The paper does not address these gaps. The conclusion calls for future work on "dynamic data–model feedback mechanisms, where model signals continuously inform data selection, refinement, and allocation during training" (Section 4), which could in principle automate the optimization of tier thresholds, ratios, and schedules. But this is a forward-looking aspiration — the current paper provides a static framework with manually chosen parameters and no optimization methodology.

---

The Training Verification Protocols Use Fixed 30/70 Mixtures, Limiting the Generality of the Quality Measurements

The assumption. All three verification protocols — efficient verification (Section 3.1), pre-train verification (Section 3.4), and decay verification (Section 3.3) — use a data mixture of 30% verification data and 70% default distribution when evaluating tiered data. This means the model never sees the tiered data in isolation or at high concentration. The quality measurements (Tables 5–7) therefore reflect the marginal benefit of adding a small amount of tiered data to an existing diverse training distribution, not the quality of the tiered data as a standalone training corpus or at high proportions.

The paper explicitly describes this protocol: for efficient verification, "a data mixture composed of 30% verification data and 70% of the default distribution" (Section 3.1); for decay verification, "a data mixture composed of 30% new verification data and 70% of the default distribution" (Section 3.1). The justification — to provide enough signal while preventing catastrophic forgetting — is reasonable, but it imposes a structural limitation on what the experiments can measure.

The consequence. This protocol design creates uncertainty about the framework's applicability in two practical scenarios:

Scenario 1: Training primarily on tiered data. A practitioner building a domain-specific model (e.g., a math-specialized LLM) might want to train primarily on L3 math data rather than using it as a 30% additive to a general distribution. The paper's experiments cannot predict how such a model would perform — the 70% default distribution in the verification protocols contributes substantial general knowledge and linguistic diversity that would be absent in a primarily-L3 training run. The cross-domain transfer effects observed from Math-L3 (Table 6) might not manifest if the model lacked the general English and Chinese exposure provided by the default distribution.

Scenario 2: Optimizing the mixture ratio. The 30/70 split is fixed across all experiments. If the optimal mixture ratio for L3 data is higher (e.g., 50%) or lower (e.g., 10%), the paper provides no evidence to guide that choice. For tiered training (Section 3.4), the experiment fixes the data composition at 1:1:1 L1:L2:L3 but the schedule varies — a different design choice that focuses on scheduling rather than mixture ratios. The interaction between scheduling and mixture ratio is unexplored.

Furthermore, the 30/70 protocol means the model receives only a fraction of the available tiered data during evaluation. For the math case study (Table 6), the decay verification uses 100B tokens at 30% math verification data — meaning only 30B tokens of the 88B-token Math-L3 corpus are actually seen. The evaluation therefore measures the quality of the first 30B tokens sampled from the tiered corpus, not the marginal value of the remaining 58B tokens. If the corpus has internal quality variation (e.g., the highest-quality synthetic samples are generated first and distributed throughout), this is not captured.

Evidence in the paper. The 30/70 mixture is explicitly stated in the experimental protocol descriptions (Section 3.1). The paper does not ablate this ratio in any experiment, nor does it discuss the implications of the protocol design for the generality of the findings.

Mitigation status. The paper does not address this limitation. The consistent 30/70 ratio across all verification protocols suggests it was chosen as a practical default rather than based on optimization or sensitivity analysis. The conclusion does not identify mixture-ratio optimization as future work. A practitioner who adopts the framework would need to independently determine appropriate mixture ratios for their specific training setup — the paper provides no transferable guidance.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new model architecture, training algorithm, or data processing technique. Instead, it proposes something more fundamental: a shared vocabulary and organizing framework for data quality management across the LLM training lifecycle. The L0–L4 tiered framework changes how the field can talk about data quality, which in turn changes what kinds of experiments are possible and what kinds of comparisons are meaningful.

From binary quality to stratified quality. The most immediate conceptual shift is the rejection of clean-vs-noisy as the only quality axis. Before this work, the dominant data management narrative was: start with raw web data (noisy), apply deduplication and heuristic filtering (clean), optionally apply model-based selection (cleaner), and mix everything together for training. Quality was treated as a binary property — data was either acceptable for training or not — and pipelines were designed to maximize the volume of "acceptable" data while minimizing cost. The L0–L4 framework replaces this binary view with a five-level explicit quality hierarchy, where each level has defined properties (L1: standardized, L2: selected, L3: refined, L4: verified), defined processing operators (heuristic filters → classifier selection → LLM editing/synthesis → knowledge orchestration), and an explicit cost gradient (increasing with each tier). This is not merely a more granular classification system — it is a reframing of data quality from a threshold-based filtering problem to a tiered resource allocation problem.

This reframing has immediate consequences for how the field designs and reports data pipelines. Under the binary-clean paradigm, papers report datasets by volume and broad processing steps (e.g., "filtered from Common Crawl using heuristic rules, 15T tokens"). Under a tiered paradigm, datasets can be reported with quality levels — "UltraData-Math-L2 (33B tokens, model-selected)" — that enable direct comparison across research groups and processing methods. The paper's Table 2, which maps dozens of existing open-source datasets and tools onto the L0–L4 framework, demonstrates that this taxonomy is retrospective — it can classify prior work — and prospective — it can guide future dataset construction. This is a standardization contribution, not an algorithmic one, but in a field where data quality claims are notoriously difficult to compare, standardization can be as impactful as innovation.

From method-centric to quality-centric data science. The paper's survey of existing data management frameworks (Section 2.1) reveals a field organized around two orthogonal axes: training stage (pre-training data, mid-training data, post-training data) and processing methodology (parsing, filtering, selection, editing, synthesis). Both axes have produced sophisticated techniques but no shared quality vocabulary. The paper's tiered framework introduces quality as an independent axis that cuts across both: L2 data is L2-quality regardless of whether it was produced for pre-training or mid-training, and regardless of whether the L2 classification used fastText or an LLM-based scorer.

This decoupling has practical consequences. It means that a research group developing a new classifier for math content can evaluate its output by measuring how much L1→L2 quality improvement it achieves, using a standardized evaluation protocol (the paper's efficient verification, for example) rather than needing to demonstrate end-to-end model training benefits. It means that practitioners can make cost-quality trade-off decisions at each tier transition independently — "should I invest in L1→L2 for my English web data?" becomes a separable question from "should I invest in L2→L3 for my math data?" — rather than needing to optimize the entire pipeline jointly. This modularity is essential for systematic data engineering because it transforms data quality from an emergent property of complex pipelines into a measurable, comparable, and optimizable quantity at each processing stage.

Reconciling contradictory findings on data quality. The paper's own experimental results provide a unifying explanation for why different studies have reached conflicting conclusions about the value of data processing. Model-based selection (L1→L2) provides modest gains for web data (+1.10 pp for English, +1.22 pp for Chinese) but essentially no gain for math (+1.04 pp under efficient verification, actually negative on GSM8K under decay verification). LLM-based editing and synthesis (L2→L3) provides modest gains for web data (+0.60 to +0.82 pp) but dramatic gains for math (+6.02 pp under efficient verification, +8.35 pp under decay verification). A study that only evaluated classifier-based selection on math data would conclude that model-driven curation is low-value; a study that only evaluated synthetic generation on math data would conclude it is transformative. Both conclusions would be correct within their scope but misleading when generalized. The tiered framework provides the vocabulary to express these domain-dependent effects precisely: the value of each tier transition depends on the domain, and the framework makes this dependence explicit rather than leaving it implicit in conflicting paper-level findings.

Where research attention should shift. The experimental results have clear implications for research prioritization:

Direction that becomes more attractive: Synthetic data generation for reasoning domains. The L2→L3 transition for math produces the largest quality gain observed in the paper (+6.02 pp under efficient verification, +8.35 pp under decay verification), and the cross-domain transfer effects from Math-L3 to English, Chinese, and Code benchmarks suggest that synthetic reasoning data has benefits far beyond its domain. This makes synthetic data generation — particularly multi-format, difficulty-stratified, curriculum-aligned synthesis — a high-return investment for reasoning-intensive domains (math, formal logic, structured science, competitive programming). The paper's detailed description of the UltraData-Math-L3 synthesis pipeline (five instructional formats, multi-model ensemble, quality filtering) provides a concrete recipe that other research groups can replicate and extend.

Direction that becomes less attractive: Incremental improvements to L1 filtering. The L1→L2 quality gain for web data is modest (+1.10 to +1.22 pp), and the paper's own FineWeb discussion notes that custom heuristic filters (line-ending punctuation ratio, duplicated line character ratio, short line ratio) removed 22% of tokens for a ~1% benchmark improvement. This suggests diminishing returns to further heuristic filtering improvements — the easy gains have been captured, and additional rule engineering is unlikely to produce transformative quality improvements. Research attention should shift toward L2 and L3 processing for web data (better classifiers, more sophisticated editing/synthesis) rather than incremental L1 refinements.

Direction that becomes newly tractable: Data quality scheduling as an optimization problem. The tiered-vs-mixed training experiment (Section 3.4) demonstrates that data quality scheduling matters — the temporal sequence of data tiers during training causally affects final model performance even when total data volume and composition are held constant. This opens a new optimization dimension that the field has not systematically explored: given a corpus with known quality tiers, what is the optimal schedule for introducing each tier during training? The paper tests only one schedule (three equal 40B-token stages, 1:1:1 ratio) and finds a 1.73× late-stage learning rate improvement over uniform mixing, suggesting substantial room for optimization. This connects naturally to the curriculum learning literature but with a novel quality-tiered framing: rather than ordering examples by difficulty (the traditional curriculum learning approach), the optimization variable is the quality tier mixture ratio as a function of training progress.

A note on the magnitude of the contribution. This paper is best understood as a conceptual framework with empirical validation, not as an empirical breakthrough. The performance improvements it demonstrates — +1.70 to +7.06 pp for L1→L3 quality upgrades across domains, +1.49 pp for tiered vs. mixed training — are meaningful but not transformative at the scales tested. The paper's primary value is in providing the architecture for thinking about data quality systematically: the tier definitions, the processing operators at each tier, the verification protocols for measuring quality, and the demonstration that tiered scheduling matters. In this sense, it plays a role analogous to the Chinchilla scaling laws (Hoffmann et al., 2022) for pretraining compute allocation, or the LLaMA recipe (Touvron et al., 2023) for open-weight model training — it provides a reference framework that enables the community to build, compare, and optimize collectively rather than in fragmented isolation.

---

Follow-Up Research This Work Enables

Cost-normalized tier comparisons to establish return on investment. The most urgent gap this paper leaves is the absence of cost accounting. The L0–L4 framework is explicitly designed to balance "data quality, acquisition cost, and marginal training benefit," but the paper measures only quality and training benefit, never cost. A natural follow-up would quantify the full processing cost for each tier transition in each domain: GPU-hours (or FLOPs) for classifier training and inference at L2, for LLM-based editing at L3 (web data), and for multi-model ensemble synthesis at L3 (math data), plus the training cost to reach equivalent downstream performance. The experimental design would train models from scratch on different tier-quality corpora under a fixed total FLOP budget (processing FLOPs + training FLOPs), testing whether investing in higher-tier data or more lower-tier training tokens yields better downstream performance. This would transform the paper's qualitative cost hierarchy into quantitative cost-effectiveness ratios — e.g., "for math data, L3 costs X FLOPs per token to produce and yields Y benchmark improvement per training FLOP, compared to L1 which costs X' and yields Y'" — that practitioners can use for budgeting decisions.

Model scale replication of the tiered training advantage. The paper's tiered-vs-mixed training result (Section 3.4, Table 7) is demonstrated only at 1.2B parameters and 120B training tokens. Whether the tiered advantage scales with model size is unknown. A natural follow-up would replicate the experiment at multiple model scales — e.g., 0.5B, 1.2B, 3B, 7B — using the same data tiers, same domain distribution, and same 1:1:1 L1:L2:L3 ratio with proportional stage boundaries. The key measurement would be the tiered-vs-mixed performance gap as a function of model size. If the gap shrinks with scale (because larger models extract more value from L1 data, reducing the relative benefit of tiering), tiered training is primarily valuable for small-to-medium models under compute constraints. If the gap grows with scale (because larger models benefit disproportionately from high-quality data in late training), tiered training becomes increasingly important for frontier-scale training runs. A negative result at scale — e.g., the tiered advantage disappearing at 7B — would establish an important boundary condition on the framework's applicability.

Schedule optimization through dynamic mixture ratios. The paper's tiered training experiment uses hard discrete switches between tiers at fixed boundaries (0–40B L1, 40–80B L2, 80–120B L3). The training curves (Figure 3) suggest this is suboptimal: tiered and mixed training perform similarly in early stages, diverging only after approximately 40B tokens. A follow-up could treat the tier mixture ratio as a continuous function of training progress — r(t) = (r_L1(t), r_L2(t), r_L3(t)) where r_i(t) is the fraction of tokens from tier i at training step t — and optimize this function using Bayesian optimization or learned scheduling policies. The measurement of interest would be the optimal schedule's performance vs. the paper's fixed schedule, and the shape of the optimal schedule (e.g., does it transition gradually between tiers, or does it maintain sharp boundaries but at different token counts?). This would also test whether including a small fraction of L1 data in late training (e.g., 5–10%) can mitigate the commonsense reasoning regressions observed in Table 7 (HellaSwag −1.68 pp, PIQA −0.82 pp) while preserving the reasoning gains.

Mechanistic investigation of cross-domain transfer from math data. The paper's finding that Math-L3 improves English, Chinese, and Code benchmarks (Table 6) is a correlation, not a demonstrated mechanism. A follow-up using mechanistic interpretability techniques could test the paper's claim that the transfer is driven by improved "general logical consistency and problem-solving capabilities." Specific experiments: (1) Compare attention patterns on reasoning benchmarks between models trained with Math-L3 vs. Math-L1 — do Math-L3 models deploy more structured, step-by-step attention that resembles mathematical reasoning patterns? (2) Probe internal representations for logical operators (AND, OR, NOT, implication) before and after Math-L3 training — does the model develop more disentangled representations of logical structure? (3) Test whether the transfer is specific to math or general to structured reasoning by comparing Math-L3 transfer to transfer from equivalent-quality synthetic data in other reasoning domains (formal logic, structured science, competitive programming). A negative result — finding no mechanistic evidence for reasoning transfer despite benchmark improvements — would suggest the improvements come from indirect effects (improved format following, reduced noise exposure) rather than genuine reasoning capability enhancement, which would substantially change how the field interprets the value of domain-specific high-quality data.

Tier-aware data mixture optimization for production training. All experiments in the paper use a fixed domain distribution (50% Web-en, 25% Web-zh, 8% Math, 17% Code for tiered training) and a fixed quality composition (1:1:1 L1:L2:L3). A natural extension is to treat the entire data mixture — domain distribution × quality tier distribution × temporal schedule — as a joint optimization problem for a given training budget. This would involve: (1) constructing tiered corpora across all domains of interest, (2) defining a parameterized mixture function with domain weights, tier weights, and schedule parameters, (3) optimizing these parameters using efficient validation (the paper's efficient verification protocol provides a template) to maximize downstream performance under a total FLOP budget that includes both data processing cost and training cost. The result would be a "compute-optimal data mixture" that specifies, for a given model scale and training budget, how many tokens of each domain at each quality tier should be consumed at each training stage. This is the data-management analog of the Chinchilla scaling laws — moving from "what mixture works empirically?" to "what mixture is optimal for a given budget?" — and the L0–L4 framework provides the quality vocabulary needed to make this optimization well-defined.

Domain boundary characterization for the L2→L3 transition. The paper reveals a striking domain asymmetry: the L2→L3 transition is high-value for math (+6.02 to +8.35 pp) but low-value for English web (+0.60 pp), Chinese web (+0.82 pp), and code (+0.99 pp). This is a single datapoint per domain, not a characterized boundary. A systematic follow-up would apply L3 processing (LLM-based editing and multi-format synthesis) to a spectrum of domain types — from highly structured (formal logic, mathematical proofs, legal documents) through semi-structured (scientific papers, technical documentation, encyclopedia articles) to unstructured (general web text, social media, creative writing) — and measure the L2→L3 quality gain for each. The hypothesis would be that the value of L3 processing is proportional to the degree of latent structure in the domain: synthetic generation is transformative when it can exploit formal constraints (math proofs, code execution, logical entailment) to create novel, verifiable content, but merely incremental when it can only perform stylistic rewriting (general web text). Establishing this boundary would enable practitioners to triage domains for L3 investment without needing to run full-scale experiments for each domain.

---

Practical Applications and Downstream Use Cases

Open-source dataset construction with explicit quality tiers. The paper's Table 1 lists 10 open-source datasets and 6 tools released under the UltraData project, including UltraData-Math-L1/L2/L3 (170B, 33B, 88B tokens), Ultra-FineWeb-en L2/L3 (1,800B, 200B tokens), and Ultra-FineWeb-zh L2/L3 (120B, 200B tokens), along with parsers, generators, and classifiers. These resources are immediately usable by any research group training LLMs. A team building an open-source model could adopt the tiered data directly — using UltraData-Math-L3 for their mid-training and annealing phases, Ultra-FineWeb-en-L3 for high-quality English web data, and the released classifiers to produce additional L2 data from their own L1 corpora. The tier labels provide quality guarantees that simple dataset names cannot: "UltraData-Math-L3" communicates that the data has undergone heuristic filtering, model-based selection, and multi-format synthetic generation with quality validation, which is far more informative than "our math corpus" or "filtered web math." This standardization reduces the information asymmetry that currently plagues dataset releases and enables more principled data mixture design.

Cost-efficient data processing for mid-training and annealing phases. The paper's key operational insight — that high-tier data is most valuable when concentrated in specific training phases rather than spread uniformly — has direct implications for teams with limited data processing budgets. A team that can afford to produce only 50B tokens of L3 math data should concentrate that data in the final 10–20% of their training run (the decay or annealing phase) rather than mixing it uniformly. This is not speculative — the tiered training experiment (Section 3.4) demonstrates that concentrating L3 data in the final 40B tokens of a 120B-token run produces better results than spreading the same L3 volume across the entire run, and the 1.73× late-stage learning rate improvement (Figure 3) quantifies the benefit. Similarly, the paper's efficient verification protocol (30% candidate data, 70% default distribution, 10B-token annealing evaluation) provides a lightweight template for data quality testing: teams can evaluate candidate datasets at ~1% of the cost of full training runs, enabling rapid iteration on data processing pipelines before committing to large-scale processing.

Domain-specific data investment strategy informed by the quality gradient. The paper's finding that the L1→L3 quality gain varies dramatically by domain — +7.06 pp for math vs. +1.70 pp for English web vs. +1.79 pp for code (Table 5) — provides an evidence-based heuristic for allocating data processing budgets across domains. An organization with a fixed compute budget for data preparation should prioritize L3 processing for math (where the return on investment is highest), followed by L2 processing for code and web data (where the L1→L2 gain is modest but non-zero), and should be skeptical about the value of L3 processing for general web data (where the additional +0.60 pp gain over L2 may not justify the cost). This domain-prioritization heuristic is directly grounded in the paper's numbers, not in hand-waving, and can be refined as the community produces additional domain-specific quality gradient measurements.

Self-improving data pipelines through the data-model co-evolution loop. The paper's L2 and L3 pipelines both depend on LLMs — for classifier annotation at L2 and for editing/synthesis at L3 — but the paper uses fixed external models for these operations. The "data-model co-evolution" thesis (Section 1, Figure 1) suggests a practical deployment pattern: train a model on tiered data (L1 → L2 → L3), then use that model to process the next generation of tiered data (improved classifiers for L2, improved synthesis for L3), then train a better model on the improved data, and repeat. The open-source release of the UltraData processing tools (parsers, generators, classifiers) makes this loop operational: a team can take UltraData-Math-Parser (for L0→L1), the UltraData-Math Classifier (for L1→L2), and UltraData-Math-Generator (for L2→L3), substitute their own progressively improving models for the LLM components, and measure whether each co-evolution iteration produces measurable quality improvements using the paper's efficient verification protocol. This transforms data processing from a one-time pre-training step into an ongoing capability-building process that scales with model capability.

---

When to Prefer This Method

The paper does not position the L0–L4 tiered framework against a named alternative methodology. It is proposed as a general organizing framework for data management rather than as a competing method to a specific existing approach (e.g., "use tiered data management instead of FineWeb's filtering pipeline"). The paper argues that current practice is fragmented and ad hoc — data of varying quality is "often indiscriminately mixed during training" (Section 2) — and that the tiered framework provides structure where none currently exists. The experimental validation compares within-framework configurations (L1 vs. L2 vs. L3, tiered vs. mixed training) rather than against external alternatives. As such, a formal "prefer X when Y" trade-off matrix would impose a comparison the paper does not make. The practical decision the framework enables is whether and how much to invest in each tier transition for each domain, which depends on the cost-effectiveness ratios that the paper does not yet provide (see the first follow-up direction above).