Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training

🎯 Pitch

An automated clustering and iterative search method finds optimal data mixtures directly from unlabeled web crawls, enabling a 1B model trained on just 400B tokens to beat Llama-3.2-1B by 2.0% across reasoning tasks. Targeting a single domain like Social Sciences yields a 5% boost over random data sampling, demonstrating that semantic clustering alone can surface high-value training subsets without curated labels.

1. Executive Summary

This paper introduces CLustering-based Iterative Data Mixture Bootstrapping (Nemotron-CLIMB), an automated framework that discovers, evaluates, and refines data mixtures for language model pre-training without requiring manually curated domain labels. Using PaLM-style Transformer models (62M–1B parameters) and the MATH-derived reasoning benchmarks, CLIMB operates through two complementary mechanisms: data preprocessing via embedding-based clustering (mapping documents to a semantic space, then grouping and merging them into ~20 super-clusters that serve as the atomic units for mixing) and iterative bootstrapping with a weak predictor (sampling mixture configurations, training lightweight proxy models to score them, and fitting a LightGBM regressor to guide subsequent sampling toward higher-quality subspaces, alternating over three iterations). When continuously trained on 400B tokens with the CLIMB-optimized mixture, the authors' 1B model surpasses Llama-3.2-1B by 2.0% on average across 12 reasoning benchmarks, while optimizing specifically for Social Sciences yields a 5% improvement over random sampling—establishing that iterative, predictor-guided mixture search over semantically clustered data enables domain adaptation without explicit labels, but only when the clustering granularity and iterative compute allocation are balanced to avoid both under-exploration from too few iterations and diluted signal from too many.

2. Context and Motivation

The Core Problem: We Don't Know What Data to Feed Language Models

The central problem this paper tackles is deceptively straightforward: given a massive, unlabeled corpus of web-scraped text, how do you decide what mixture of which subsets to feed a language model during pre-training to maximize performance on downstream tasks? This question matters enormously because data composition—not just data quantity—has emerged as one of the most consequential design choices in language model development, yet the dominant data sources used in practice (Common Crawl and its derivatives) arrive without any inherent organization that tells you which portions are good for learning reasoning, which are good for learning facts, and which are simply noise.

The practical stakes are substantial. Pre-training runs for modern LLMs consume thousands of GPU-days and trillions of tokens. A suboptimal data mixture means you are spending enormous computational resources teaching your model from the wrong examples. Worse, once the pre-training is done, the model's capabilities are largely baked in—mid-training or fine-tuning can nudge performance, but they cannot fully compensate for a poorly constructed pre-training corpus. The paper's Figure 1 illustrates exactly this dynamic: two models trained on the same number of tokens but different data mixtures diverge substantially in performance, with the CLIMB-optimized mixture yielding steady gains throughout training while competing datasets plateau earlier. This gap compounds over training tokens, meaning the cost of a bad mixture choice is not fixed—it grows with your training budget.

Beyond the immediate efficiency argument, there is a deeper domain adaptation problem that motivates this work. Organizations building LLMs increasingly want models that excel not just at general-purpose chat but at specific valuable capabilities: medical reasoning, legal analysis, code generation, mathematical problem-solving. The naive approach—throw more domain-specific data at the model—often backfires because over-saturating on one domain can degrade general capabilities without guaranteeing the desired specialization. The paper frames this tension explicitly in Section 1:

"Optimizing a model for coding tasks requires not just programming-related content but also complementary knowledge from mathematics, reasoning, and security."

What is needed is a principled way to discover which combinations of which data subsets produce the best blend of general competence and domain expertise. Absent such a method, practitioners are left with expensive trial-and-error—training many large models on different mixtures and hoping for the best.

The problem of data mixture optimization can be decomposed into two sub-problems, each of which the paper identifies as unsolved:

Gap 1: Web-scale datasets lack domain structure. The raw material for modern LLM pre-training—Common Crawl snapshots, FineWeb, C4—is a vast, undifferentiated soup of text spanning every conceivable topic, quality level, and format. While curated datasets like The Pile (Gao et al., 2020) provide explicit domain annotations (e.g., "PubMed Central," "ArXiv," "GitHub"), producing such annotations requires enormous manual effort and domain expertise. The paper states this directly (Section 1):

"Large-scale datasets such as Common Crawl offer unmatched diversity and scale but lack explicit domain labels, making it difficult to extract domain-relevant content. Filtering data often relies on general-purpose heuristics like perplexity or educational value, which do not necessarily capture the most informative or high-quality content for specific domains."

This means that most data mixture optimization methods—which assume you know which documents belong to which domain—cannot be applied to the largest, most diverse corpora that dominate real-world pre-training. You are forced to either (a) use manually curated datasets with domain labels, limiting scale and diversity, or (b) operate on the raw web corpus with crude heuristics that don't capture semantic content. Neither option is satisfying.

Gap 2: Even with domain labels, finding the optimal mixture is non-trivial. Suppose you do have a dataset divided into domains (e.g., The Pile's 22 components). Determining the right sampling weights across those domains is itself a challenging optimization problem. The relationship between domain proportions and downstream performance is complex and non-linear: doubling the math data might help on GSM8K but hurt on HellaSwag; adding code data might transfer positively to reasoning tasks through improved logical capabilities; the optimal mixture for a 1B model might differ from that for a 7B model due to capacity differences. Traditional approaches—manually defined heuristics, uniform sampling, or intuition-based upweighting of "high-quality" domains—capture none of this complexity. The paper explicitly notes (Section 1):

"the complex, nonlinear relationship between dataset composition and model performance... Even with curated datasets like The Pile with domain annotations, selecting an optimal data mixture is non-trivial."

Where Prior Approaches Fall Short

To understand precisely what CLIMB contributes, we need to examine the landscape of existing data selection and mixing methods and identify their specific limitations relative to the two gaps above.

Manual and Heuristic Mixture Design

The earliest and still most common approach is to design data mixtures by hand, typically by a team of researchers making educated guesses about what proportions of what data sources will produce good results. The Pile was constructed this way: its creators chose 22 data sources and assigned them weights based on intuition about diversity and quality. GLaM (Du et al., 2022) used a similar manual process. SlimPajama-DC (Shen et al., 2023) systematically evaluated the impact of different predefined configurations, but the configurations themselves were human-designed.

The limitation is obvious: manual design does not scale. As the number of potential data sources grows and as domain-specific optimization objectives multiply, the space of possible mixtures explodes beyond what human intuition can navigate. A practitioner wanting to optimize for five different domain-specific benchmarks cannot manually tune 20+ domain weights—the combinatorial space is simply too large. Moreover, human design relies on assumptions about what data is "good for" what capability, assumptions that are frequently wrong (the paper's analysis in Appendix D.2 shows that cluster similarity to a downstream task is not a sufficient predictor of that cluster's contribution to performance—some highly similar clusters provide limited benefit, while apparently dissimilar clusters become crucial).

Learning-Based Mixture Optimization (DoReMi, DoGE, RegMix)

A more recent class of methods attempts to learn the optimal domain weights automatically. These methods represent the state of the art against which CLIMB compares, and understanding their mechanics is essential to appreciating what CLIMB changes.

DoReMi (Domain Reweighting with Minimax Optimization; Xie et al., 2023) operates through a two-stage process. In the first stage, a small proxy model is trained with Group Distributionally Robust Optimization (Group DRO), which dynamically upweights domains where the model's loss is high and downweights domains where it is low. The intuition is that harder domains deserve more training emphasis. The domain weights learned by the proxy model are then frozen and used to resample the training data for the much larger target model. The key assumption here—and the key limitation—is that the reference model's training dynamics on a domain reliably signal that domain's importance for downstream performance. This assumption can break: a domain could have high loss simply because it contains noisy or irrelevant content, not because it is a valuable challenge that would improve downstream capabilities. DoReMi learns to emphasize where the model currently struggles during pre-training, which is a proxy for—but not equivalent to—what will improve downstream task performance.

DoGE (Domain Reweighting with Generalization Estimation; Fan et al., 2023) similarly optimizes domain weights using proxy models but differs in using a generalization-based objective rather than Group DRO.

RegMix (Data Mixture as Regression; Liu et al., 2024) takes a fundamentally different approach that is the most direct precursor to CLIMB. Rather than deriving weights from training dynamics, RegMix treats data mixture optimization as a regression problem: (1) randomly sample N mixture configurations from the simplex of domain weights, (2) train a proxy model on each configuration and measure its downstream performance, (3) fit a regression model (also LightGBM) on the (mixture, performance) pairs, and (4) use the regressor to predict and select the optimal mixture. This is essentially a single iteration of the approach that CLIMB extends iteratively. RegMix's key advantage is that it directly models the relationship between mixture weights and task performance, bypassing the proxy-via-training-loss approach of DoReMi. However, as formulated, RegMix is a "one-shot" method: it samples configurations uniformly from the entire space, trains on all of them, fits one predictor, and selects one final mixture. This has two related weaknesses:

Uniform sampling is inefficient. Most randomly sampled configurations from a high-dimensional simplex will be near-uniform mixtures that achieve mediocre performance. The predictor model spends most of its capacity learning to distinguish various flavors of mediocrity rather than accurately predicting what makes the top 5% of mixtures special. With a fixed budget of, say, 100 proxy model trainings, uniform sampling might test only 1–2 configurations in the truly promising region of the simplex.
One predictor must do everything. The predictor is fit once on whatever data happened to be sampled. If the initial sampling missed high-performing regions (which is likely with uniform sampling), the predictor cannot extrapolate to them accurately. The method has no mechanism to iteratively refine its understanding or focus sampling where the predictor is uncertain about high-potential regions.

The paper explicitly frames RegMix as the limiting case of CLIMB with one iteration (Section 2.2):

"Existing methods [referring to RegMix] can be seen as running the above coordinate descent process for only a single iteration, which is a special case of our more general framework."

Domain-Specific Data Selection Methods

Another thread of related work focuses on selecting specific data points (rather than mixing whole domains) for domain adaptation. DSIR (Xie et al., 2023) uses importance resampling with n-gram features to match a target distribution. CRISP (Grangier et al., 2025) clusters a generalist dataset and samples clusters proportional to their frequency in a specialist dataset. Training-dynamics-based methods like S2L (Yang et al., 2024) select data based on loss trajectories during fine-tuning. LESS (Xia et al., 2024) selects instruction-tuning examples with high gradient similarity to a target task.

These methods are effective for their intended use cases—fine-tuning or domain adaptation on labeled target data—but they share a fundamental constraint that limits applicability to large-scale pre-training: they require either explicit domain labels, a target-domain reference dataset, or computationally expensive per-datum gradient calculations. For pre-training from scratch on trillions of tokens from unlabeled web corpora, you have none of these luxuries. You are trying to build a general-purpose model, not adapt to a single known target distribution, and you are operating at a scale where per-datum analysis is infeasible.

Quality-Filtering Approaches

A parallel line of work focuses on filtering raw web data for quality without domain awareness. FineWeb-Edu (Penedo et al., 2024) uses an educational-value classifier to retain only "educational" web pages. DCLM (Li et al., 2024) systematically benchmarks filtering strategies. Nemotron-CC (Su et al., 2024), which CLIMB actually uses as a source dataset, applies sophisticated quality annotation to Common Crawl.

The limitation of pure quality filtering is that quality is not the same as relevance. A perfectly written Wikipedia article about 18th-century harpsichord construction may score highly on every quality metric but contribute nothing to a model's code generation capabilities. Conversely, a rough-but-functional GitHub README with code examples might score poorly on "educational value" or "polished writing" but be exactly what the model needs. Quality filtering alone cannot make these distinctions—it removes noise uniformly, not intelligently.

A Gap That Spans the Landscape

Synthesizing these lines of work reveals the specific gap that CLIMB addresses: no existing method can automatically discover semantically meaningful data groups from an unlabeled web corpus and then iteratively optimize their mixture weights for arbitrary downstream objectives.

Manual methods can't scale.
DoReMi/RegMix/DoGE require pre-existing domain labels.
Domain-specific selection methods require a target distribution or are computationally prohibitive at pre-training scale.
Quality filtering doesn't capture domain relevance.

How This Paper Positions Itself

CLIMB's position in this landscape is defined by two architectural choices that directly address the two gaps identified above:

For Gap 1 (no domain structure in web data): CLIMB introduces a preprocessing pipeline that creates domain structure from raw text via embedding-based clustering. Instead of requiring pre-annotated domains, CLIMB embeds every document in a semantic space using a pre-trained encoder (stella_en_400M_v5), clusters these embeddings with k-means (1000 initial clusters), prunes low-quality clusters using fastText quality classifiers, and merges the remaining clusters into ~20 super-clusters based on centroid distance. The result is a set of semantically coherent data groups—the paper's Table 4 shows these clusters naturally correspond to interpretable topics like "Biology, Genetics, Astronomy, Climate Science" (Cluster 8), "Python, Code" (Cluster 20), and "History, Culture, Economy, Energy, Market, Policy" (Cluster 19)—without any human labeling. These clusters become the "domains" over which mixture weights are optimized.

This is not simply clustering for its own sake. It directly enables the application of mixture optimization techniques (like DoReMi or RegMix) to unlabeled web corpora by creating the domain divisions those techniques require. The paper explicitly contrasts this with parallel work like WebOrganizer (Wettig et al., 2025), which uses classifiers to annotate data with topic labels, positioning CLIMB's clustering approach as "more straightforward, and readily scalable" (Section 7).

For Gap 2 (finding optimal mixtures efficiently): CLIMB extends RegMix's one-shot regression approach into an iterative bootstrapping framework. The key insight—and the paper's most significant algorithmic contribution—is that you can allocate a fixed computational budget of proxy model trainings much more effectively by running multiple iterations of (sample → train → fit predictor → sample smarter) than by doing it all in one shot. Early iterations use cheap, uniform-ish sampling to get a rough map of the performance landscape. Later iterations use the predictor to focus sampling on the promising regions of the simplex, allowing the predictor to become progressively more accurate exactly where accuracy matters most—near the optimum. The paper's Figure 3 visualizes this process using t-SNE: the sampled configurations concentrate and converge over iterations, shifting from broad exploration of the space to targeted exploitation of high-performing regions.

This is essentially applying the explore-then-exploit paradigm from Bayesian optimization and multi-armed bandits to data mixture search. The "weak predictor" (LightGBM) serves as a surrogate model for the expensive true objective (train a proxy model and evaluate), and the iterative sampling procedure balances gathering information about uncertain regions (exploration) with testing configurations predicted to be good (exploitation). The paper's formulation in Equations 3–4 makes this explicit: at each iteration $k$ , the predictor $f^k$ scores all unsampled configurations, the top $N$ become candidates, $M$ are randomly sampled from among them (the randomness providing exploration within the promising set), and the predictor is then refit on the expanded set $S^{k+1}$ .

The broader narrative. The paper situates itself within a larger trend in LLM research: the shift from "throw more data at the model" toward "be smarter about what data you use." This mirrors contemporaneous developments in data-constrained scaling laws (Muennighoff et al., 2023), where researchers discovered that when data is limited, data quality and composition matter enormously. CLIMB contributes the automated machinery to make "be smarter about data" practically achievable without the manual curation bottleneck that has historically made it expensive and non-scalable.

The paper also draws an implicit parallel to neural architecture search (NAS) and hyperparameter optimization: just as those fields moved from manual design to automated search, data mixture optimization is moving from manual heuristics to learned search procedures. The iterative predictor-guided approach CLIMB employs is directly inspired by techniques like Bayesian optimization with Gaussian process surrogates, adapted to the specifics of data mixture search where each "function evaluation" requires training an entire language model.

Why Now?

Three converging trends make the timing of this work significant:

Pre-training data is being exhausted. Several studies have suggested that high-quality web text may run out within this decade. When data is abundant, suboptimal mixture choices are masked by sheer volume—the model eventually sees enough of everything. When data is constrained, mixture quality becomes the dominant factor in achievable performance. CLIMB provides a framework for extracting maximum value from a finite data pool.
Domain-specialized models are increasingly prized. The market for LLMs is fragmenting into general-purpose assistants and domain-specialized systems for medicine, law, finance, and science. Building each of these currently requires substantial manual data curation effort. An automated mixture optimization method that can target arbitrary downstream objectives dramatically reduces the cost of producing domain experts.
Compute costs for pre-training continue to rise. Frontier model training runs now cost tens to hundreds of millions of dollars. The cost of a suboptimal data mixture—paid in wasted GPU hours—has never been higher. Methods that can shrink the search cost for good mixtures from multiple large training runs to a series of small proxy model trainings offer enormous practical savings. CLIMB's proxy models cost ~45 GPU hours each (Appendix C.4), compared to ~6,400 GPU hours for a full target model training run—a 142× cost reduction per evaluation.

3. Technical Approach

3.1 Reader Orientation

CLIMB is a search system that automatically discovers how to mix different topics from a massive, unlabeled text corpus to produce the best possible language model for a specific task or domain. It solves the problem of "what should my model read?" by replacing human guesswork with an iterative process: first grouping similar documents using embeddings so that semantically coherent units exist, then running a series of cheap small-model training experiments guided by a learned predictor that gets smarter each round, progressively zeroing in on the optimal recipe.

3.2 Big-Picture Architecture (Diagram in Words)

The CLIMB system has two major phases connected by a shared output:

Phase 1 — Data Preprocessing (once, upfront):

Embedding: A pre-trained encoder maps every document in the raw corpus to a dense vector in semantic space.
Clustering: k-means groups these vectors into many fine-grained clusters (1000).
Pruning and Merging: Low-quality clusters are removed using fastText quality classifiers, then remaining clusters are merged by centroid proximity into ~20 super-clusters. These super-clusters are now the atomic "domains" over which mixing happens.

Phase 2 — Iterative Mixture Bootstrapping (the search loop, per optimization target):

Configuration sampling: Mixture weight vectors (summing to 1, assigning a proportion to each super-cluster) are drawn from the simplex according to the current iteration's strategy.
Proxy model training: A small language model (62M–350M parameters) is trained from scratch or continued on a short run (40B tokens) using each sampled mixture, then evaluated on validation benchmarks.
Predictor fitting: A LightGBM regression model is trained on the collected (mixture weights → validation score) pairs, learning to predict how good any untested mixture would be.
Predictor-guided sampling: The fitted predictor scores all possible mixtures; the most promising are added to the pool for the next iteration, where proxy models will be trained on them.

The loop runs for 3 iterations (64, then 32, then 16 new proxy trainings), producing progressively better mixture candidates. The final predictor's top-ranked mixture becomes the recipe for training the full-size target model.

Information flows: raw text → embeddings → clusters → mixture weight vectors → proxy model scores → predictor → refined mixture vectors → ... → optimal mixture → target model training.

3.3 Roadmap for the Deep Dive

First, the clustering pipeline (Section 2.1/3.1): Because clusters are the fundamental units over which mixing operates, understanding what they are and how they are made is prerequisite to everything else. I will cover embedding, k-means, the fastText quality pruning, and the centroid-distance merging—explaining why 1000 initial clusters → 240 pruned → 21 enhanced is a deliberate design choice about granularity and computational cost.
Second, the bi-level optimization formulation (Section 2.2, Eq. 1): This formalizes what "optimal mixture" means mathematically and defines the expensive inner loop (training a model on a mixture) and the outer loop (searching over mixtures) that motivates the entire iterative approach.
Third, the predictor surrogate model (Section 2.2, Eq. 2): Because the true objective requires training a full model, the paper replaces it with a learned regressor. I will explain how LightGBM serves as the surrogate, what data it is trained on, and how its use converts an intractable optimization into a feasible one.
Fourth, the iterative coordinate descent algorithm (Section 2.2, Eqs. 3–4 and implementation): This is the core mechanism that distinguishes CLIMB from one-shot methods like RegMix. I will walk through each iteration's two subroutines (configuration sampling and predictor fitting), explaining the exploration-exploitation balance created by sampling from the top-N predicted configurations, and why three iterations with a 4:2:1 compute allocation is the sweet spot.
Fifth, the full implementation stack (Section 3.1): The paper makes specific choices about embedding model, clustering hyperparameters, quality thresholds, initialization distributions, predictor regularization, and compute allocation. I will enumerate these precisely and explain why each matters.
Sixth, the evaluation protocol and proxy/target model training (Sections 3 and 4): How the system is validated—the phase-1 pre-training, the WSD schedule, the 40B-token continuous pre-training protocol, and the benchmark suite—to make clear what "performance" means throughout.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core idea is that data mixture optimization for pre-training can be automated by combining semantic clustering (to create domain-like units from unlabeled text) with an iterative predictor-guided search (to efficiently explore the high-dimensional simplex of mixture weights using cheap proxy models), and that the iterative approach significantly outperforms both random sampling and one-shot regression-based methods under a fixed compute budget.

The Clustering Pipeline: Creating Domains from Raw Text

The clustering pipeline is the bridge that allows mixture optimization techniques—which fundamentally require discrete "domains" or "buckets" to assign weights to—to operate on raw, unlabeled web corpora. Without this step, the downstream iterative search would have nothing to search over. The pipeline has three sequential stages, each addressing a specific sub-problem.

Stage 1: Text Embedding. The raw input is a large dataset $\hat{\mathcal{D}} = \{D_1, D_2, \ldots, D_n\}$ containing $n$ documents. Each document $D_i$ is fed through a frozen embedding model $\mathcal{M}_e$ , producing a fixed-dimensional vector $E_i = \mathcal{M}_e(D_i)$ . The paper uses stella_en_400M_v5, a 400M-parameter encoder chosen because "it efficiently encodes large-scale text with excellent performance" (Section 3.1). The output is a set of embedding vectors $\mathcal{E} = \{E_1, E_2, \ldots, E_n\}$ .

Why embeddings rather than raw text features: The paper wants clusters that capture semantic similarity (documents about genetics should cluster with documents about biology, not with documents that happen to share the same vocabulary but discuss different topics). Embedding models trained on contrastive or retrieval objectives map semantically similar texts to nearby points in the vector space, which is precisely the property that k-means clustering exploits. Alternatives like n-gram overlap or bag-of-words TF-IDF would cluster documents by surface lexical similarity, failing to group "cellular mitosis mechanism" with "chromosome replication process" while potentially grouping it with "cell phone repair guide" based on shared words. The embedding space abstracts away surface form to capture underlying topic.

Stage 2: Embedding Clustering. The embedding vectors $\mathcal{E}$ are clustered using k-means from the FAISS library (Johnson et al., 2019; Douze et al., 2024), which is optimized for billion-scale similarity search on GPUs. The initial number of clusters $K_{\text{init}}$ is set to 1000. K-means iteratively assigns each embedding to the nearest of $K_{\text{init}}$ centroids and updates centroids to be the mean of their assigned points, minimizing the within-cluster sum of squared Euclidean distances.

Why 1000 initial clusters: The paper states this relatively large value is chosen "to ensure the clusters are as fine-grained as possible for subsequent processing" (Section 2.1). The strategy is deliberately to over-cluster: produce many small, tightly coherent groups that are likely single-topic, then aggregate later. This is preferable to under-clustering (e.g., 20 clusters initially), which would force k-means to lump together documents that are only weakly related, losing the semantic precision that makes the rest of the pipeline work. The extreme granularity means individual clusters might only contain 0.1% of the total data, but they will be thematically pure—which makes the subsequent merging step meaningful rather than arbitrary.

A subtle point about feasibility: Clustering 800B tokens' worth of embeddings (which is the corpus size after filtering) is computationally non-trivial. FAISS enables this by using approximate nearest-neighbor search and GPU acceleration, but the paper does not report the computational cost of this embedding and clustering step relative to the rest of the pipeline. This is a notable omission, since embedding 800B tokens—even with a relatively small 400M-parameter encoder—represents a substantial one-time cost that contributes to the method's total computational footprint.

Stage 3: Cluster Pruning and Merging. After k-means produces 1000 clusters, two operations transform them into the final set of super-clusters:

Cluster-level pruning. The paper trains several fastText models (Joulin et al., 2016) to score each document on four quality dimensions: overall quality, educational value, informational value, and advertisement score, each on a 1–5 scale. These fastText classifiers are trained on 1 million texts that were annotated by Nemotron-340B (Adler et al., 2024), a 340B-parameter language model, using a carefully designed prompt template (provided in Appendix D.11). The prompt instructs the annotator model to evaluate each text on well-defined rubrics—for example, advertisement score gets +1 for each level of freedom from promotional language, from "minimal promotional elements, not distracting" to "no detectable promotional content."

Once every document has quality scores, cluster-level aggregates are computed. Clusters with average quality below a threshold of 3.0 across the four dimensions are removed. This threshold is described as "relatively loose" (Section 3.1), and results in retaining 240 clusters (i.e., $K_{\text{pruned}} = 240$ ). The pruning removes clusters dominated by boilerplate, spam, or incoherent text—documents that would waste training compute regardless of topic. The choice of 3.0 as threshold: since the scale is 1–5, 3.0 represents "mostly meets the criterion," meaning clusters where documents are, on average, at least moderately high-quality across all dimensions survive. A tighter threshold would remove more data but risk losing topic coverage; a looser threshold would retain more noise.

Cluster merging. The 240 surviving clusters are merged based on the Euclidean distance between their centroids in the embedding space. Specifically, clusters whose centroids are within a distance threshold of 1.5 are grouped together into super-clusters. This produces $K_{\text{enhanced}} \approx 21$ super-clusters (the paper uses 21 in most experiments, though some ablation settings test 15 and 30; Section 5, Table 3 "Abl.clus").

Why merge at all: With 240 clusters, the search space for mixture weights is a 240-dimensional simplex, which is far too large for the iterative bootstrapping to explore effectively with only ~100 proxy model trainings. Merging reduces the dimensionality to ~20, making the search tractable. The merging criterion—centroid proximity—ensures that clusters being merged are semantically similar, so the resulting super-clusters remain coherent topics rather than arbitrary aggregations. This is the crucial design choice: the method creates "domains" not through human curation but through geometric proximity in embedding space, which naturally produces interpretable groupings. Table 4 validates this: the 21 clusters correspond to recognizable topics (Cluster 20 = "Python, Code"; Cluster 8 = "Biology, Genetics, Astronomy, Climate Science"; Cluster 19 = "History, Culture, Economy, Energy, Market, Policy").

What the entire pipeline produces: A filtered corpus $\mathcal{D}$ containing reduced data from the original $\hat{\mathcal{D}}$ (bad clusters are removed, not just downweighted), partitioned into $K_{\text{enhanced}}$ super-clusters $\mathcal{D} = \{\mathcal{D}_1, \mathcal{D}_2, \ldots, \mathcal{D}_k\}$ , where $k \in \{15, 21, 30\}$ depending on the setting. Each super-cluster has a known token count. This structured corpus is the input to the mixture weight search.

The Bi-Level Optimization Formulation

With structured data domains in hand, the paper formalizes the mixture weight search as a bi-level optimization problem (Equation 1). This formalization is important because it makes explicit why the problem is hard and what approximations the iterative bootstrapping method is making.

$\min_{\alpha \in \mathcal{A}} \ell_{\text{val}}(\alpha, \omega^*(\alpha)) \quad \text{s.t.} \quad \omega^*(\alpha) = \arg\min_{\omega} \ell_{\text{train}}(\alpha, \omega) \quad \text{s.t.} \quad \sum_{i=1}^{k} \alpha_i = 1, \quad \alpha_i \geq 0$

where $\alpha = (\alpha_1, \ldots, \alpha_k)$ is a vector of mixture weights, one per cluster, constrained to lie in the $k$ -dimensional simplex $\mathcal{A}$ (all weights non-negative, sum to 1); $\omega$ represents the parameters (weights) of a language model; $\ell_{\text{train}}(\alpha, \omega)$ is the training loss (standard next-token prediction cross-entropy) when the model is trained on data sampled according to mixture $\alpha$ ; $\omega^*(\alpha)$ is the resulting optimal model parameters for that mixture; and $\ell_{\text{val}}(\alpha, \omega^*(\alpha))$ is the validation loss (or negative accuracy) of that model on the target downstream benchmarks.

What this computes: The outer minimization finds the mixture weights $\alpha$ that produce the best downstream task performance, but evaluating any candidate $\alpha$ requires solving the inner minimization—training an entire model to convergence on that mixture. This is a bi-level problem because the outer objective depends on the result of the inner optimization.

Why this form: The bi-level structure reflects a fundamental reality of data mixture optimization: you cannot evaluate a mixture without training on it. There is no closed-form relationship between mixture proportions and downstream performance that could be computed directly—the interaction between data composition and model learning is too complex. The inner loop (training) is the only "oracle" that can tell you how good a mixture is, and it is extremely expensive. This motivates the entire surrogate-model approach: since calling the inner loop is costly, you want to call it as few times as possible, and you want each call to be maximally informative for estimating the outer objective.

Why the simplex constraint: Mixture weights must be non-negative (you cannot "anti-sample" data) and must sum to 1 (they represent proportions of a finite training budget). The simplex is a $(k-1)$ -dimensional manifold—with 21 clusters, the search space is a 20-dimensional surface embedded in 21-dimensional space. This is still very high-dimensional for black-box optimization with expensive evaluations, which is why the iterative predictor-guided approach is necessary.

The Predictor Surrogate Model

Since evaluating $\ell_{\text{val}}(\alpha, \omega^*(\alpha))$ for every candidate $\alpha$ is computationally prohibitive (it requires training an entire model), the paper introduces a learned predictor $f_\theta(\alpha)$ that approximates this function. This converts the intractable bi-level problem into a tractable surrogate optimization (Equation 2):

$\min_{\alpha \in \mathcal{A}} f(\alpha \mid \mathcal{S}) \quad \text{s.t.} \quad f = \arg\min_{\tilde{f} \in \tilde{\mathcal{F}}} \sum_{s \in \mathcal{S}} L(\tilde{f}(s), \ell(s, \omega^*))$

where $\mathcal{S}$ is a set of mixture configurations that have been evaluated (by training a proxy model and measuring validation performance); $\ell(s, \omega^*)$ is the true performance for configuration $s$ ; $\tilde{f}$ is a candidate predictor from the function class $\tilde{\mathcal{F}}$ (the set of all possible regression models); $L$ is the loss function used to train the predictor (typically mean squared error for regression); and $f(\alpha \mid \mathcal{S})$ is the predictor's estimated performance for any mixture $\alpha$ given the training data $\mathcal{S}$ .

What this computes: Given a set of evaluated (mixture, performance) pairs, we train a regression model $f$ that maps mixture vectors to predicted performance scores, then optimize over $\alpha$ using $f$ as a cheap stand-in for the true objective. The inner minimization (fitting $f$ ) is cheap; the outer minimization (searching over $\alpha$ using $f$ ) is also cheap because $f$ evaluation costs milliseconds rather than GPU-hours.

Why this works: The predictor can generalize—it learns the structure of how mixture proportions relate to performance, so it can predict scores for mixtures it has never seen. The key assumption is that this relationship is smooth enough to be learnable from a relatively small number of samples (hundreds, not millions). LightGBM is chosen as the function class $\tilde{\mathcal{F}}$ because it "fits mixture-performance pairs well with limited data" (Section 3.1), builds an ensemble of decision trees that can capture non-linear interactions between cluster weights, and has built-in regularization to prevent overfitting when $|\mathcal{S}|$ is small relative to the dimensionality of $\alpha$ . The paper uses L1 and L2 regularization (combined elastic net), early stopping after 20 rounds of no improvement on a held-out validation set, a maximum tree depth of 4, and a minimum of 5 samples per leaf (Section 3.1).

Why this form over alternatives: An alternative would be to use a Gaussian Process (as in standard Bayesian optimization), but GPs scale poorly with dimensionality and struggle with the 21-dimensional input space. LightGBM's tree-based approach handles high-dimensional inputs gracefully and naturally captures interactions between clusters (e.g., "Cluster 8 helps only when combined with Cluster 9, but not alone"), which are expected to be critical given the complex interplay between data domains. Another alternative would be to skip the predictor entirely and simply test many configurations with the cheapest possible proxy model, but this would waste evaluations on clearly bad mixtures—the predictor allows the system to avoid spending GPU-hours training on mixtures that are almost certainly suboptimal.

The critical constraint: The set $\mathcal{S}$ is bounded by a sampling budget $C$ (i.e., $|\mathcal{S}| \leq C$ ). $C$ is directly tied to the total training cost of proxy models. With a 350M proxy model, each evaluation costs ~45 GPU-hours (Appendix C.4), so $C = 112$ in the main experiment represents ~5,040 GPU-hours for the entire search. If the predictor is inaccurate—because $\mathcal{S}$ doesn't cover the high-performing region—the optimization will converge to a suboptimal mixture. This is the exact failure mode of one-shot methods like RegMix, which CLIMB's iterative approach is designed to prevent.

The Iterative Coordinate Descent Algorithm

The centerpiece of CLIMB is an iterative procedure that alternates between improving the sampling set $\mathcal{S}$ and improving the predictor $f$ , formalized as a coordinate descent on the bi-level problem from Equation 2. This is where CLIMB diverges from one-shot regression methods like RegMix and where the majority of the paper's technical novelty resides. The algorithm is specified in Equations 3 and 4, and I will first present the equations, then explain the implementation in concrete operational steps.

Iteration $k$ consists of two alternating subroutines:

$\tilde{\mathcal{P}}^k = \{f^k(s) \mid s \in \mathcal{A} \setminus \mathcal{S}^k\}, \quad \mathcal{S}^M \subset \text{TopN}(\tilde{\mathcal{P}}^k), \quad \mathcal{S}^{k+1} = \mathcal{S}^M \cup \mathcal{S}^k$

$\alpha^* = \arg\min_{\alpha \in \mathcal{A}} f(\alpha \mid \mathcal{S}^{k+1}), \quad \text{s.t.} \quad f^{k+1} = \arg\min_{f^k \in \tilde{\mathcal{F}}} \sum_{s \in \mathcal{S}^{k+1}} L(f(s), \ell(s, \omega^*))$

where $\mathcal{S}^k$ is the set of evaluated configurations at the start of iteration $k$ ; $f^k$ is the predictor trained on $\mathcal{S}^k$ ; $\tilde{\mathcal{P}}^k$ is the vector of predicted performance scores for all unevaluated configurations in the simplex $\mathcal{A} \setminus \mathcal{S}^k$ ; $\text{TopN}(\tilde{\mathcal{P}}^k)$ selects the $N$ highest-scoring configurations according to the current predictor; $\mathcal{S}^M$ is a randomly sampled subset of $M$ configurations from those top $N$ (where $M < N$ ); and $\mathcal{S}^{k+1}$ is the expanded evaluation set for the next iteration.

What the first equation (Configuration Sampling) computes: The current predictor $f^k$ is applied to every possible mixture configuration that has not yet been evaluated—effectively a "virtual screening" of the entire search space using the cheap surrogate model. The top $N$ configurations by predicted performance are identified as the most promising candidates. From these $N$ candidates, $M$ are randomly selected to actually be evaluated (by training proxy models) in the next iteration. These $M$ new evaluations are added to the existing set $\mathcal{S}^k$ to form $\mathcal{S}^{k+1}$ .

What the second equation (Predictor Fitting) computes: With the expanded evaluation set $\mathcal{S}^{k+1}$ , a new predictor $f^{k+1}$ is trained to replace $f^k$ . This predictor is then used to solve the surrogate optimization $\min_\alpha f(\alpha \mid \mathcal{S}^{k+1})$ , giving the current best estimate of the optimal mixture $\alpha^*$ . In practice, the final $\alpha^*$ is selected after the last iteration $K$ .

Why the top-N + random sampling structure: If the algorithm deterministically selected the top $M$ configurations and only evaluated those, it would quickly collapse to a narrow region of the simplex based on the predictor's (potentially inaccurate) early estimates—the classic exploitation trap. By selecting a larger pool of top $N$ candidates and then randomly sampling $M$ from among them, the algorithm maintains diversity in the evaluations. The randomness provides exploration within the promising region: two configurations with similar predicted scores might have very different true performances, and testing both helps the next predictor learn which subtle differences matter. The ratio $M/N$ controls the exploration-exploitation tradeoff; the paper does not explicitly state the $N$ value (it is implied to be larger than $M$ , the number sampled per iteration), but the decreasing $M$ across iterations (64 → 32 → 16) implicitly reduces exploration as the predictor becomes more accurate.

Implementation in Concrete Steps (Section 2.2 and Section 3.1):

The algorithm runs for $K = 3$ iterations with a total budget of $C = 64 + 32 + 16 = 112$ proxy model trainings (this is the 100% compute budget in Table 3). Here is exactly what happens:

Initialization (Iteration 1 setup): The initial set $\mathcal{S}^1$ is small—just a few configurations are sampled to bootstrap the first predictor. The sampling uses a Dirichlet distribution parameterized by each cluster's token count (so clusters with more tokens are initially sampled at roughly their natural frequency, providing a reasonable starting point). The Dirichlet distribution produces vectors on the simplex with concentration controlled by the token counts. The paper specifies this as Dirichlet-based initialization (Section 3.1) and ablates it against random initialization (Table 3, "Abl.init"), finding Dirichlet slightly better (60.41% vs. 60.21% average) because it starts the search closer to a reasonable operating point. The exact size of $\mathcal{S}^1$ is not given (it is small, used only to train the first predictor).

Iteration 1 — Broad Exploration (64 proxy trainings): The predictor $f^1$ is trained on $\mathcal{S}^1$ . It predicts scores for all unevaluated configurations, selecting the top $N$ (some large number, not explicitly stated). From these, $M_1 = 64$ configurations are randomly sampled and evaluated by training 350M proxy models on each mixture for 40B tokens and measuring validation performance on PIQA, ARC_E, and HellaSwag. The resulting 64 (mixture, score) pairs are added to form $\mathcal{S}^2$ . A new predictor $f^2$ is fitted on $\mathcal{S}^2$ .

Iteration 2 — Focused Exploration (32 proxy trainings): The improved predictor $f^2$ —now trained on approximately 64+initial data points—makes better predictions. The top $N$ predicted configurations are identified; the predictor should now be concentrating these in genuinely higher-performing regions of the simplex. From them, $M_2 = 32$ are randomly sampled and evaluated. These are added to form $\mathcal{S}^3$ , and predictor $f^3$ is fitted.

Iteration 3 — Exploitation (16 proxy trainings): The predictor $f^3$ , trained on approximately 64+32+initial ≈ 96+ data points, should be quite accurate in the promising regions. The top $N$ are identified and $M_3 = 16$ are randomly sampled and evaluated. These are added to form $\mathcal{S}^4$ , and the final predictor $f^4$ is fitted. The mixture $\alpha^*$ that maximizes $f^4(\alpha)$ over the simplex is selected as the optimal data mixture.

Why three iterations: The paper ablates this in Table 3 ("Abl.allo") under the "Effects of Compute Allocation" analysis. With the same total budget of 112 proxy trainings, allocating them as 6:1 (roughly 96:16, two iterations), 4:2:1 (64:32:16, three iterations), or 2:2:1:1 (28:28:28:28, four iterations) produces different results. The 4:2:1 allocation achieves the best average performance (60.41%), while 6:1 achieves 60.05% and 2:2:1:1 achieves 60.14%. The paper's interpretation: too few iterations (6:1) means the predictor only gets one major refinement, so its sampling in the second iteration is still relatively uninformed—it hasn't had a chance to correct its initial errors. Too many iterations (2:2:1:1) spreads the proxy training budget too thin per iteration—with only 28 evaluations per round, the predictor may not get enough new information in each iteration to significantly improve, and the 28 evaluations may not sufficiently explore the promising region identified by the previous predictor. Three iterations with decreasing budgets (64 → 32 → 16) balances the depth of refinement (multiple predictor updates) with the breadth of each search round (enough evaluations per iteration to meaningfully improve the predictor).

Why decreasing budgets (64, 32, 16): Early iterations need more samples because the predictor is inaccurate and the promising region is poorly characterized—you need to cast a wide net. Later iterations need fewer samples because the predictor has narrowed down the region of interest, and each new evaluation in that region provides proportionally more useful information for refining the predictor around the optimum. This is the same logic as simulated annealing or adaptive Bayesian optimization schedules: start with high exploration, transition to focused exploitation.

The predictor iteration as coordinate descent: The paper frames the algorithm as coordinate descent on (sampling set, predictor), but it is worth clarifying what "coordinate" means here. The two subroutines are:

Fix the predictor $f^k$ , update the sampling set $\mathcal{S}^{k+1}$ by selecting new points where $f^k$ predicts high performance. This is a "configuration sampling" step that expands the training data for the predictor.
Fix the sampling set $\mathcal{S}^{k+1}$ , update the predictor $f^{k+1}$ by refitting on the expanded data. This is the standard regression step.

The alternation ensures that the predictor is always trained on data that includes recent explorations of promising regions, and that the sampling always leverages the most up-to-date predictor. This is distinct from Bayesian optimization, where the acquisition function balances predicted mean and uncertainty—CLIMB's sampling strategy only uses predicted mean (top-N by score), with randomness providing the exploration. A potential limitation: because the predictor only sees evaluated points as training data, it may become overconfident in regions it has sampled heavily and underconfident in unexplored regions, potentially missing good mixtures that happen to be far from any evaluated point. The top-N + random sampling partially mitigates this by ensuring some diversity, but it is a heuristic, not a principled uncertainty quantification.

The Phase-1 Pre-Training and Continuous Pre-Training Protocol

The paper's experimental design involves a specific training protocol that is important to understand because it defines what "training on a mixture" means in practice and constrains how mixtures can affect performance.

Phase-1 (foundation) pre-training. Before any data mixture experiments begin, three sizes of standard Transformer decoder-only models—62M, 350M, and 1B parameters—are pre-trained from scratch on 10 trillion tokens of a general-purpose dataset combining DCLM (Li et al., 2024) and TxT360 (Tang et al., 2024). This training uses the warmup-stable-decay (WSD) learning rate schedule (Hu et al., 2024), which has three phases: a warmup phase where the learning rate increases from zero to a peak value (standard practice), a long stable phase where it stays at the peak value, and a decay phase where it linearly anneals to near zero. Models are saved at the end of the stable phase (before decay starts). The key property of WSD is that it "supports resuming at any time of the stable stage" (Section 3)—you can continue training from a WSD checkpoint without restarting the schedule, which is crucial for the experimental protocol where many different mixtures are tested via continued training.

Why 10T tokens of general pre-training before mixture experiments: This is arguably the most important experimental design choice in the paper, and it deserves careful scrutiny. The models are not trained from scratch on the CLIMB-optimized mixtures; they are continuously pre-trained on the mixture for 40B tokens starting from a checkpoint that has already seen 10T tokens of general data. This means CLIMB is optimizing mixtures for the final stage of pre-training (sometimes called "mid-training" or "annealing"), not for pre-training from the start. The paper acknowledges this implicitly: "we focus on the data mixing research in the decay stage" (Section 3, Model description).

Why this matters for interpreting the results: A mixture that works well for 40B tokens of continuous training might not work well for training from scratch. During continuous training, the model already has strong general capabilities from the 10T-token foundation; the mixture only needs to provide the marginal benefit of domain refinement. From scratch, the mixture would need to provide both the general foundation and the domain specialization, which might require different proportions. The paper's NEMOTRON-CLIMBMIX experiment (Section 6, Figure 1) partly addresses this by training from scratch, but the main comparisons in Tables 1 and 2 use continuous pre-training. A reader should understand that "CLIMB finds the optimal data mixture" means "CLIMB finds the optimal data mixture for the final 40B tokens of training, given a 10T-token general foundation."

Why WSD specifically: The WSD schedule enables an important practical convenience for the search process. With standard cosine or linear schedules, you cannot simply "continue" training from an intermediate checkpoint—you would need to restart the schedule, which wastes tokens on re-warming up or changes the effective learning rate trajectory. WSD's stable phase means that any checkpoint from within it has the same learning rate, so you can resume training with that same rate and then apply a fresh decay phase. This makes the proxy model evaluations in the search loop consistent: each 40B-token mixture evaluation starts from the same foundational model state with the same learning rate, isolating the effect of the mixture itself.

Training hyperparameters (Appendix C.4): The AdamW optimizer is used. During the stable stage, the learning rate is set to $5 \times 10^{-5}$ , and during the decay stage it anneals to $1 \times 10^{-5}$ . The batch size is 2M tokens. Training uses 256 NVIDIA H100 GPUs. A single lightweight proxy model training (40B tokens, 350M parameters) takes approximately 45 GPU hours; a full target model training (also 40B tokens, 1B parameters) takes approximately 6,400 GPU hours. This 142× cost ratio (45 vs. 6400) is the fundamental economic motivation for the proxy-based search—you can evaluate 142 mixture candidates using the proxy model for the same cost as one evaluation using the target model.

Mixture sampling during training: When training on a mixture defined by weights $\alpha = (\alpha_1, \ldots, \alpha_k)$ , each training batch of 2M tokens is constructed by sampling tokens from cluster $i$ with probability $\alpha_i$ . This is standard proportional sampling. The mixture weights control the expected composition of each batch, but due to the randomness of sampling, any individual batch may deviate from the exact proportions. Over 40B tokens (20,000 batches of 2M tokens), the law of large numbers ensures the realized proportions closely match the target $\alpha$ .

The Evaluation Protocol and Metrics

Target benchmarks (Section 3): The primary optimization target for the main experiments is general reasoning, operationalized through three validation sets: PIQA (physical commonsense reasoning), ARC_E (AI2 Reasoning Challenge, Easy set), and HellaSwag (commonsense natural language inference). The choice to optimize on only three tasks and then evaluate on a broader set is deliberate: it tests whether a mixture that improves the optimized tasks also transfers to held-out reasoning tasks, providing evidence for generalization rather than overfitting to the validation set. The paper states this explicitly: "Although the optimization objective is confined to the validation sets of PIQA, ARC_E, and HellaSwag, we observe that the resulting performance gains carry over to all the benchmark tasks" (Section 4.1).

Full evaluation suite: The downstream evaluation includes PIQA, ARC_C (Challenge set), ARC_E, HellaSwag, WinoGrande (pronoun resolution), SIQA (social commonsense), MMLU (massive multitask language understanding, 5-shot), OBQA (OpenBookQA), BoolQ (Boolean question answering), RACE (reading comprehension), LAMBADA (language modeling), and TruthfulQA (factuality). All except MMLU (5-shot) are evaluated in 0-shot settings using the LM-Evaluation Harness (Gao et al., 2024). The average across these 12 benchmarks is the primary metric reported in Table 2 and Figure 1.

Perplexity metrics: For language modeling quality, the paper also reports perplexity on WikiText and LAMBADA (Table 1). These capture the model's fundamental next-token prediction ability independent of downstream task formatting.

Optimization objective for search: During proxy model evaluation in the search loop, the predictor is trained to predict the average accuracy on the validation sets of PIQA, ARC_E, and HellaSwag. The paper does not report using perplexity in the optimization objective, only accuracy. This is a choice to optimize directly for downstream task performance rather than for a proxy metric like perplexity, which prior work has shown can be anti-correlated with task performance in some regimes (models can achieve low perplexity by memorizing frequent patterns while failing on reasoning tasks).

MMLU domain experiments: For the domain-specific optimization experiments (Section 5, Figure 5), MMLU's pre-defined subject categories are used to create three domain groupings: STEM (science, technology, engineering, math subjects), Humanities (history, philosophy, law, etc.), and Social Sciences (economics, sociology, psychology, etc.). The optimization objective for each domain is the average accuracy on that domain's MMLU subjects in the validation set. This tests CLIMB's ability to customize mixtures for different downstream specializations.

Full Implementation Stack: Exact Numbers and Hyperparameters

This subsection consolidates all specific hyperparameters and design choices into one reference location, since they are scattered across Sections 2, 3, and Appendix C.

Embedding and Clustering:

Embedding model: stella_en_400M_v5 (400M parameters)
Clustering algorithm: k-means (FAISS)
Initial clusters: $K_{\text{init}} = 1000$
Data annotated for quality fastText: 1 million texts
Annotator model: Nemotron-340B using prompt template in Appendix D.11
Quality dimensions: overall quality, educational value, informational value, advertisement score (each 1–5)
Pruning threshold: average quality score 3.0 (retains $K_{\text{pruned}} = 240$ clusters)
Merging distance threshold: Euclidean distance 1.5 between centroids
Final super-clusters: $K_{\text{enhanced}} = 21$ (default; ablations at 15 and 30)
Total tokens after clustering/filtering: approximately 800B

Iterative Bootstrapping:

Number of iterations: $K = 3$
Proxy trainings per iteration: 64, 32, 16 (total 112, the "100% compute" budget)
Ablation compute budgets: 150% (168 total) and 200% (224 total)
Ablation compute allocations: 6:1, 4:2:1 (default), 2:2:1:1
Initial sampling: Dirichlet distribution parameterized by cluster token counts
Predictor model: LightGBM regression
LightGBM hyperparameters: L1 and L2 regularization (elastic net), maximum depth 4, minimum 5 samples per leaf, early stopping after 20 rounds of no improvement on held-out validation set
Optimization targets for search: validation accuracy on PIQA + ARC_E + HellaSwag

Model Training:

Architecture: standard Transformer decoder-only
Model sizes: 62M, 350M, 1B parameters (proxy models use 62M or 350M; target models use 350M or 1B)
Phase-1 pre-training: 10T tokens on DCLM + TxT360 mix
Continuous pre-training for mixture evaluation: 40B tokens
Learning rate schedule: WSD (warmup → stable at $5 \times 10^{-5}$ → decay to $1 \times 10^{-5}$ )
Batch size: 2M tokens
Optimizer: AdamW
Hardware: 256 NVIDIA H100 GPUs
Proxy training cost: ~45 GPU-hours (350M model, 40B tokens)
Target training cost: ~6,400 GPU-hours (1B model, 40B tokens)
Final reported models trained on 400B tokens for comparison with SOTA (Table 2)

Source Data for Clustering:

Nemotron-CC (Su et al., 2024), highest-quality bucket
SmolLM-corpus (Ben Allal et al., 2024) for NEMOTRON-CLIMBLAB and NEMOTRON-CLIMBMIX (Section 6)

CLIMB vs. RegMix: What the Iterative Extension Changes

Because the paper explicitly positions RegMix as the "single iteration" special case (Section 2.2), it is worth concretely enumerating what the iterative extension adds and why it outperforms:

Progressive focusing of the training data for the predictor. In RegMix's one-shot approach, all mixture candidates are drawn uniformly from the simplex, meaning most are near-uniform mediocre mixtures. The predictor is trained primarily on mediocre data and must extrapolate to the high-performing tail. In CLIMB, iterations 2 and 3 add training data concentrated in the promising region, so the predictor learns to distinguish excellent from very-good mixtures, not just good from mediocre. This produces a more accurate predictor in the region that matters for final selection.
Adaptive allocation of limited proxy training budget. RegMix spends its entire budget on configurations that are mostly mediocre. CLIMB spends the majority (64/112) in broad exploration but reserves substantial budget (48/112) for focused refinement. If the high-performing region occupies 5% of the simplex, RegMix might test only 5–6 configurations in it (5% of 112), while CLIMB might test 30–40 by the final iteration, giving the predictor much richer data about what actually works.
Multiple rounds of predictor refinement. The predictor in iteration 1 will make errors—it might overestimate some mediocre configurations due to noise in the proxy evaluations or underestimate genuinely good configurations that happen to be in an under-sampled region. By iteration 3, these errors are corrected because the predictor has been refit on data that includes evaluations of both the overestimated and underestimated configurations. RegMix gets one shot; if the initial predictor is wrong about the optimal mixture, there is no mechanism to recover.
The exploration-exploitation schedule. CLIMB's decreasing budget per iteration (64 → 32 → 16) implements a deliberate schedule: start broad, get narrower. This is not simply "run RegMix three times" because each iteration's sampling is conditioned on the previous iteration's predictor. If you ran three independent RegMix-style searches and picked the best result, you would have three independent uniform samples with no information transfer between them—the search would not systematically improve.

The empirical validation of these mechanisms appears in the ablation results (Table 3): the 4:2:1 allocation (three iterations, decreasing breadth) outperforms both the "tall tree" (6:1, effectively two iterations with one being undersized) and the "fat tree" (2:2:1:1, four iterations with insufficient samples per iteration to refine the predictor). This confirms that the iterative structure itself—not just the total compute—matters for search efficiency.

4. Key Insights and Innovations

Innovation 1: Data Mixture Optimization as an Iterative Predictor-Guided Search — Not a One-Shot Regression

The paper's most significant intellectual contribution is recognizing that the problem of learning a data mixture is fundamentally not a one-shot regression problem — it is a sequential decision problem where each expensive evaluation should inform where to look next. The prior state of the art, RegMix (Liu et al., 2024), treated mixture discovery as: (1) uniformly sample N configurations from the simplex, (2) train proxy models on all of them, (3) fit a regressor, (4) pick the best predicted configuration. This is conceptually clean but strategically naive — it spends the entire evaluation budget before the learner has any idea where the promising configurations live, meaning most evaluations are wasted on near-uniform mediocre mixtures that teach the predictor about regions nobody cares about. The predictor's training data is dominated by points far from the optimum, and it must extrapolate to the high-performing tail from a sample that may contain only a handful of good configurations by chance.

CLIMB's conceptual move is to treat mixture search as a resource allocation problem over a sequence of decisions, where each round of proxy evaluations is conditioned on everything learned so far. The predictor is not a one-time oracle but a progressively improving surrogate model whose training data becomes more concentrated in high-value regions each iteration. This shifts the framing from "collect data, fit once, optimize once" to "collect a little, fit, sample smarter, collect more, refit, sample even smarter." The difference is not incremental — it changes the statistical properties of the predictor's training distribution from uniform over the simplex to importance-weighted toward regions that matter, which in turn changes the predictor's accuracy exactly where accuracy is needed for final mixture selection.

What makes this more than just "run RegMix three times" is the coupling between iterations. At iteration k+1, the new configurations are sampled from a set identified as promising by the predictor from iteration k. This creates an information flow — mistakes in the predictor at iteration 1 (overestimating some mediocre mixtures, underestimating some genuinely good ones) are corrected in iteration 2 because the newly evaluated configurations provide ground-truth data about the regions the predictor was uncertain about or wrong about. Running RegMix three times independently would produce three independent uniform samples with no information transfer; CLIMB's sequential conditioning means the effective sample size in the optimal region compounds across iterations.

The evidence for this claim is in the ablation results (Table 3, "Abl.allo"), where the 4:2:1 allocation (three coupled iterations) outperforms both the 6:1 allocation (two iterations, less coupling) and the 2:2:1:1 allocation (four iterations, too thinly sliced per iteration to meaningfully refine the predictor). The optimal structure is not the one with the most iterations, nor the one with the most evaluations per iteration — it is the one where the predictor has enough new information per round to correct its errors while also having enough rounds to progressively focus. This is the signature of a method where the dynamics of iterative refinement, not just the total compute budget, determine success.

Significance: This reframes data mixture optimization from a static curve-fitting problem to a dynamic sequential decision problem. It opens the door to more sophisticated frameworks — Bayesian optimization with proper uncertainty quantification, multi-fidelity optimization where early iterations use even cheaper proxies, or adaptive stopping rules that allocate budget dynamically based on predictor convergence diagnostics. The paper demonstrates the principle with a simple top-N sampling heuristic, but the conceptual framework is far more general than the specific implementation.

Innovation 2: Semantic Clustering as a Domain Discovery Mechanism — Removing the Manual Annotation Bottleneck

The dominant assumption in prior data mixture work — both manual methods (The Pile, GLaM) and learning-based methods (DoReMi, RegMix, DoGE) — is that the dataset arrives with pre-existing domain divisions. Whether those divisions come from human curation (PubMed vs. GitHub vs. ArXiv) or from source metadata (different Common Crawl snapshots, different websites), the optimization operates over a fixed, given partition of the data. This assumption is so deeply embedded that it is rarely stated explicitly — it is simply the water the field swims in.

CLIMB's second fundamental innovation is eliminating this assumption by treating domain discovery as an algorithmic preprocessing step rather than a human-dependent prerequisite. The embedding-clustering-pruning-merging pipeline converts raw, unlabeled web text into semantically coherent groups without any human domain knowledge. The clusters that emerge — documented in Table 4 — are not random. They capture genuine topic structure: Cluster 20 is "Python, Code," Cluster 8 is "Biology, Genetics, Astronomy, Climate Science," Cluster 19 is "History, Culture, Economy, Energy, Market, Policy." These groupings emerge from geometric proximity in the embedding space of a pre-trained encoder, validated by quality filtering to remove noise, and merged to a tractable dimensionality for search.

What makes this intellectually distinctive is not the use of clustering per se — semantic clustering of text has existed for decades — but the recognition that the quality of the domain partition critically determines the quality of the downstream mixture optimization, and that this partition can and should be algorithmically optimized rather than assumed as given. Prior work that clustered pre-training data (e.g., DSIR for n-gram-based importance sampling, or SemDeDup for deduplication) used clustering for filtering or selection — removing near-duplicates, selecting data similar to a target distribution. CLIMB uses clustering for structure creation: building the atomic units over which mixture weights will be learned. This is a different purpose with different desiderata — clusters should be semantically coherent (so mixing over them is meaningful), distinct (so they provide non-redundant signals), and interpretable enough that practitioners can understand what the final mixture represents.

The design choice of over-clustering followed by quality-pruning followed by centroid-distance merging reflects a deliberate tradeoff logic that the paper makes explicit. Over-clustering (1000 initial clusters) ensures that each atomic unit is thematically pure — documents within a 1000-way cluster almost certainly share a narrow topic, making the clusters valid building blocks. Quality pruning removes blocks that are thematically coherent but useless (spam, boilerplate, extremely low-quality content that happens to be topically consistent). Centroid merging aggregates the remaining pure topic-blocks into ~20 domains, where the aggregation respects semantic similarity. This pipeline is not arbitrary — each step addresses a specific failure mode of simpler approaches. Clustering directly into 20 groups would produce heterogeneous clusters (biology and medicine and chemistry all smashed together by k-means' coarse partitioning). Skipping quality pruning would retain clusters of pure-but-useless content (advertising copy, automated directory pages). Skipping merging would leave the search space at 240 dimensions, making mixture optimization infeasible with the proxy training budget.

The empirical validation of this approach is indirect but powerful: the clusters discovered by the automated pipeline produce mixture weights that make intuitive sense when analyzed (Appendix D.2, D.7, D.8) — clusters relevant to reasoning tasks get high weights, clusters with high similarity to downstream tasks are important but not exclusively so, and the optimal mixture balances relevant clusters with diverse ones. If the automated clustering were producing arbitrary or meaningless groupings, the downstream optimization would not discover sensible mixtures, and the final model would not outperform the SOTA. The fact that it does — by 2.0% over Llama-3.2-1B across 12 benchmarks (Table 2) — is evidence that the automated domain discovery is producing a structurally valid partition.

Significance: This removes the single largest barrier to applying data mixture optimization to the largest, most widely used pre-training corpora — the lack of domain labels. It means that any organization with a large web crawl can run CLIMB's pipeline and obtain a domain-structured corpus suitable for mixture optimization, without the enormous manual effort that went into datasets like The Pile. It also makes domain adaptation possible on proprietary or private corpora where human annotation would be impractical, expensive, or privacy-violating. The finding that the automated clusters are semantically interpretable (Table 4) further means that practitioners can inspect and understand the discovered domain structure, building trust in an otherwise opaque automated process.

Innovation 3: The Empirical Finding That Domain Relevance and Domain Diversity Are Partially Decoupled — Similarity to the Target Is Neither Necessary Nor Sufficient

A widely held intuition in data selection — one that motivates methods like DSIR (importance resampling to match a target distribution), CRISP (cluster sampling proportional to target domain frequency), and gradient-based selection (choose data with high gradient similarity to the target task) — is that the best data for improving performance on domain X is data that looks like domain X. If you want better math reasoning, you should train on more math. If you want better code generation, you should train on more code. This intuition is so natural that many methods effectively encode it as an axiom.

CLIMB's experiments provide concrete counterevidence to this intuition, making the decoupling of domain similarity from domain usefulness one of the paper's most practically significant findings. Appendix D.2 analyzes the relationship between cluster similarity to downstream tasks (measured by cosine similarity between cluster embeddings and task embeddings) and the mixture weights that CLIMB's optimization actually converges to. The results reveal a nuanced picture that defies the "more similar = more important" heuristic:

High similarity does not guarantee high importance. Cluster 21 shows high similarity to general reasoning (Figure 6) but "provides limited benefits to downstream performance, leading to a gradual decrease in its importance" as the iterative search progresses (Appendix D.2). The optimizer discovers that this cluster — despite looking relevant — doesn't actually help.
Low similarity does not preclude high importance. Cluster 8 initially appears out-of-domain for general reasoning but "becomes increasingly important with further iterations" (Appendix D.2), ultimately receiving the second-highest weight in the final mixture (13% in Iteration 3, Figure 7a). Something about this cluster's content — which is described as "Biology, Genetics, Astronomy, Climate Science" (Table 4) — transfers positively to reasoning tasks despite having low embedding similarity to those tasks.
Within-domain clusters can be redundant. The paper notes that "when clusters are highly similar, incorporating only one of them may suffice" (Appendix D.2), suggesting that similarity-based methods that sample all similar clusters proportionally would waste training budget on redundant signals.

The correct picture that emerges is that an optimal data mixture balances relevance AND diversity, where these two dimensions are only partially correlated. Some clusters are relevant (their content directly transfers to downstream skills) but also redundant with each other, so the optimizer selects a subset. Some clusters are apparently irrelevant but contain latent reasoning or language patterns that transfer, so the optimizer upweights them even though a similarity-based method would deprioritize them. The final mixture for general reasoning (Figure 7a) is highly sparse — only 4 clusters (C8, C9, C18, C19) account for the majority of the weight — but the selected clusters span diverse topics: from biology/genetics to history/economics to general science. The optimizer is implicitly performing a complementarity selection: picking a set of clusters whose combined training signal covers the skills needed for reasoning, rather than just picking the most similar clusters.

This finding is significant because it challenges the conceptual foundation of a large family of data selection methods. If "similar to target = good for target" were true, then simple embedding-similarity-based filtering would be sufficient, and methods like DSIR or CRISP would saturate the achievable performance. CLIMB's iterative search discovers mixtures that are better than what similarity-based sampling would produce — the 5% improvement over random sampling for Social Sciences (Figure 5c) and the 2.0% gain over Llama-3.2-1B (Table 2) are achievable precisely because the optimizer is not constrained to the "similarity heuristic" and can discover beneficial but non-obvious cluster combinations.

Significance: This is a negative result with positive implications — it tells the field that a widely used heuristic is unreliable, which redirects research effort away from better similarity metrics and toward methods that can empirically evaluate the actual contribution of data subsets. It also has practical implications for practitioners: when building a domain-specialized model, do not simply throw more in-domain data at it. Run an empirical search over a diverse data pool — you may discover that your coding model benefits from history texts, or your medical model benefits from philosophy discussions, through transfer effects that similarity-based methods would systematically miss.

Innovation 4: Computational Budget Allocation as a First-Class Design Variable in Data Mixture Search

Prior work on data mixture optimization treated the total computational budget (number of proxy model trainings) as a constraint — something you work within — but not as a design variable with an internal structure that could be optimized. DoReMi uses a fixed proxy training run; RegMix uses a fixed number of uniformly sampled configurations; the question of how to allocate a given evaluation budget across rounds, or whether multiple rounds help, was not asked because the methods were conceptually one-shot.

CLIMB makes the internal allocation of the search budget a central object of study. The ablation in Table 3 ("Abl.comp" and "Abl.allo") investigates two distinct aspects of budget allocation:

Scaling the total budget (Abl.comp): What happens when you increase the total number of proxy evaluations from 100% (112 evaluations) to 150% (168) or 200% (224)? The answer is that performance continues to improve — 60.41% → 60.72% → 61.12% average accuracy for the 1B target model — suggesting that the search has not saturated and that more extensive search would yield further gains. This is practically important because it tells practitioners: if you have more compute for the search phase, spend it — the returns are positive.
Allocating a fixed budget across iterations (Abl.allo): Given 112 evaluations, should they be spent as 96:16 (two iterations), 64:32:16 (three iterations), or 28:28:28:28 (four iterations)? The answer is that the 4:2:1 allocation (three iterations, decreasing budgets) is optimal — 60.41% vs. 60.05% and 60.14%. This means the shape of the allocation matters, not just the total.

This second finding is the more intellectually significant one. It demonstrates that the search process has an internal structure with a genuine depth-breadth tradeoff — analogous to the depth-width tradeoffs in neural architecture design or the exploration-exploitation tradeoffs in reinforcement learning and Bayesian optimization. Too few iterations (6:1) means the predictor only gets one major refinement; its initial errors — from being trained on a broad, mostly mediocre sample — are not corrected because the second round of sampling is still relatively uninformed. Too many iterations (2:2:1:1) means each round gets too few evaluations to meaningfully improve the predictor; with only 28 new data points per round, the predictor cannot substantially refine its understanding of the promising region, and the search stagnates.

The optimal 4:2:1 allocation implements a specific philosophy: early iterations should be broad (many evaluations, high exploration) because the predictor is inaccurate and the promising region is poorly characterized; later iterations should be narrow (fewer evaluations, higher exploitation) because the predictor is accurate and each new evaluation in the known-promising region provides proportionally more value. This is not an arbitrary schedule — it is a structured response to the uncertainty dynamics of the surrogate model. When the predictor is unreliable (iteration 1), you want many evaluations to cover the space and reduce uncertainty broadly. When the predictor is reliable (iteration 3), you want few, targeted evaluations to precisely locate the optimum.

This conceptualizes data mixture search as a meta-optimization problem where the hyperparameters of the search process itself — number of iterations, evaluations per iteration, sampling strategy per iteration — are design choices that can be tuned. The paper does not claim to have optimized these meta-parameters; the 4:2:1 split was found by a modest sweep, not by a principled adaptive method. But by demonstrating that these choices matter and that they interact non-trivially (increasing both depth and breadth independently does not help; you need the right balance), the paper opens a new dimension of research: what is the compute-optimal search process for data mixture optimization, analogous to how neural architecture search has studied the efficiency of different search strategies?

Significance: This changes the conversation from "how many proxy trainings can you afford?" to "what is the optimal way to structure the proxy trainings you can afford?" It implies that two groups with the same total compute budget for mixture search could achieve different final model quality depending on how they structure their search — a finding with direct practical implications for teams building LLMs. It also suggests that future work on automated data mixture optimization should report not just the total search cost but the allocation structure, since the latter is a confound for the former.

5. Experimental Analysis

Evaluation Methodology

Dataset. The source data for clustering and mixture search is the highest-quality bucket of Nemotron-CC (Su et al., 2024), a large dataset filtered from Common Crawl. Hierarchical clustering of this subset produces approximately 800B tokens distributed across 21 clusters. For the NEMOTRON-CLIMBLAB and NEMOTRON-CLIMBMIX releases (Section 6), the source data also includes smollm-corpus (Ben Allal et al., 2024). The downstream evaluation uses 12 benchmarks: PIQA, ARC_C, ARC_E, HellaSwag, WinoGrande, SIQA, MMLU, OBQA, BoolQ, RACE, LAMBADA, and TruthfulQA. MMLU is evaluated in a 5-shot setting; all others use 0-shot evaluation via the LM-Evaluation Harness (Gao et al., 2024).
Base model(s). Three sizes of standard Transformer decoder-only models are used: 62M, 350M, and 1B parameters. All three undergo an initial phase-1 pre-training on 10 trillion tokens of a DCLM + TxT360 mix using the warmup-stable-decay (WSD) learning rate schedule before any mixture experiments begin. The 350M model serves as the primary proxy model for mixture search; the 1B model serves as the main target model for reporting final performance. The authors state this model family is chosen because these model sizes are "computationally efficient for exploring data mixture configurations" while being large enough that results plausibly transfer to larger scales (Section 3, Model description).
Metrics. The primary metric is average accuracy (%) across the 12 downstream benchmarks. For the mixture search optimization target, the metric is average validation accuracy on only three tasks: PIQA, ARC_E, and HellaSwag. The paper also reports perplexity on WikiText and LAMBADA (Table 1) as a measure of language modeling quality separate from downstream task performance. All numbers in Tables 1–3 represent test-set performance after 40B tokens of continuous pre-training on the evaluated mixture, except Table 2 where models are trained on 400B tokens for comparison with state-of-the-art models.
Baselines. The paper compares against four baselines:
- Random: Each cluster is assigned equal, uniform weight; data is sampled proportionally.
- DoReMi (Xie et al., 2023): Trains a small proxy model using Group DRO to dynamically reweight domains by training loss, then uses the learned weights to resample data for the target model. The paper uses a 350M proxy model for DoReMi (Table 1).
- RegMix (Liu et al., 2024): Uniformly samples mixture configurations from the simplex, trains a proxy model on each, fits a LightGBM regressor on the (mixture, performance) pairs, and selects the predicted-optimal mixture. This is the closest prior method to CLIMB and can be viewed as CLIMB with a single iteration. The paper uses a 350M proxy model for RegMix.
- CLIMB-Best@N (Section 5, Figure 5): For domain-specific experiments, this baseline directly searches for the best configuration from randomly sampled mixtures using the target model itself (not a proxy). The number of searches is reduced for the 1B model to ensure comparable search compute.
Generation budget / compute accounting. The primary unit of search compute is the number of proxy model trainings. The default "100% compute" budget corresponds to 112 total proxy evaluations allocated across three iterations as 64, 32, and 16. A single 350M proxy model training on 40B tokens costs approximately 45 GPU-hours on 256 NVIDIA H100 GPUs (Appendix C.4). The total search cost is therefore ~5,040 GPU-hours for the default budget. Target model training (1B parameters, 40B tokens) costs ~6,400 GPU-hours — roughly 142× more than a single proxy evaluation, which is the fundamental economic motivation for the proxy-based search. For the ablation experiments (Table 3), compute budgets are scaled to 150% (168 evaluations) and 200% (224 evaluations) by proportionally increasing evaluations per iteration. For compute allocation ablations, the total budget remains fixed at 112 but is redistributed across different numbers of iterations (6:1, 4:2:1, 2:2:1:1).
Cross-validation / statistical protocol. The paper does not report error bars, confidence intervals, or significance tests. The authors state that they "performed relatively large-scale training (≥100B tokens) so that the task performance is stable across runs" (Checklist item 7). There is no cross-validation protocol described for the mixture selection process — the optimal mixture is selected by the final predictor trained on all evaluated configurations, and this mixture is then used to train the target model whose performance is reported. For the domain-specific experiments (Figure 5), the optimization target is validation-set accuracy on MMLU domain subsets, and the reported numbers are test-set accuracy on the same MMLU domains. There is no held-out mixture-level validation; the predictor's internal validation uses a random split of the (mixture, score) pairs for early stopping.

Main Quantitative Results

Comparison with Data Mixture Baselines (Table 1)

The headline result: CLIMB outperforms all baseline data mixture methods when training both 350M and 1B target models on 40B tokens of continuous pre-training. For the 350M target model, CLIMB achieves an average accuracy of 54.83%, compared to Random (52.17%), DoReMi (53.38%), and RegMix (53.78%). For the 1B target model, CLIMB achieves 60.41%, compared to Random (57.93%), DoReMi (59.16%), and RegMix (59.37%). The margin over the best baseline (RegMix) is 1.05 percentage points for the 350M model and 1.04 percentage points for the 1B model.

Several patterns in the per-task breakdown are noteworthy (Table 1). For the 1B model, CLIMB's gains over Random are not uniform: the largest absolute improvements are on HellaSwag (+3.11 percentage points: 66.01 vs. 62.90), ARC_E (+2.73: 72.97 vs. 70.24), and ARC_C (+3.86: 40.98 vs. 37.12). Improvements on PIQA (+1.73) and SIQA (+0.89) are more modest, and perplexity on WikiText actually improves (lower is better: 15.96 vs. 17.82 for Random), suggesting the optimized mixture does not degrade general language modeling capability while boosting reasoning. DoReMi achieves the best perplexity on WikiText for the 1B model (15.78 vs. CLIMB's 15.96) but trails on downstream accuracy, consistent with the observation that perplexity and task performance can be partially decoupled.

A methodological detail: DoReMi and RegMix both use 350M proxy models, meaning the comparison is fair in terms of proxy model size (all three methods use the same scale for their search phase). The Random baseline represents equal-weight sampling from the same 21 clusters, so the comparison isolates the effect of the mixture optimization rather than the effect of clustering itself.

The paper highlights that "Although the optimization objective is confined to the validation sets of PIQA, ARC_E, and HellaSwag, we observe that the resulting performance gains carry over to all the benchmark tasks" (Section 4.1). This is visible in Table 1: for the 1B model, CLIMB improves over Random on all six reported benchmarks, not just the three that were optimized. The average gain on the optimized tasks (PIQA, ARC_E, HellaSwag) is approximately 2.87 points; on the unoptimized tasks (ARC_C, WinoGrande, SIQA), the average gain is approximately 2.20 points. The generalization is strong but slightly attenuated, as expected.

Comparison with State-of-the-Art Language Models (Table 2)

Scaling up to 400B tokens of training on the CLIMB-optimized mixture, the 950M-parameter CLIMB model is compared against five other models in the sub-1.2B parameter class: Qwen2.5 (490M), SmolLM (360M), TinyLlama (1.1B), AMD-OLMo (1.2B), and Llama-3.2 (1.2B). Table 2 reports performance on 12 benchmarks plus their average.

The central claim: CLIMB achieves the highest overall average score of 53.54%, surpassing Llama-3.2-1B (51.56%) by 2.0 percentage points. This is the source of the "exceeds Llama-3.2-1B by 2.0%" claim in the abstract. The gap to the next-best model (AMD-OLMo at 49.93%) is 3.61 points.

The per-benchmark breakdown reveals where CLIMB's advantage concentrates. CLIMB substantially outperforms Llama-3.2 on ARC_C (40.96 vs. 36.26, +4.70), ARC_E (73.57 vs. 65.49, +8.08), HellaSwag (66.90 vs. 63.67, +3.23), and SIQA (43.55 vs. 42.99, +0.56). It is competitive or slightly weaker on PIQA (75.46 vs. 74.59, +0.87), WinoGrande (63.54 vs. 60.69, +2.85), MMLU (36.47 vs. 35.40, +1.07), OBQA (41.20 vs. 37.20, +4.00), BoolQ (66.02 vs. 63.98, +2.04), and RACE (36.65 vs. 37.80, -1.15). On LAMBADA and TruthfulQA, CLIMB trails Llama-3.2 (59.05 vs. 62.99 and 39.06 vs. 37.67, respectively). The pattern suggests CLIMB's mixture particularly strengthens commonsense and multi-step reasoning (ARC, HellaSwag) while providing smaller gains on factuality and reading comprehension tasks.

Critically, this is not a fully controlled comparison: CLIMB is trained on its custom 400B-token mixture, while baseline models are trained on their own (different) datasets with their own (different) total token budgets and preprocessing pipelines. The models being compared differ in total training tokens, data sources, architecture details, and tokenizer, not just data mixture optimization. The 2.0% claim should therefore be understood as "CLIMB's full pipeline (10T foundation + 400B CLIMB mixture) produces a model that outperforms Llama-3.2-1B on these benchmarks," not as "CLIMB's mixture optimization alone is worth 2.0% over Llama-3.2's training recipe." The paper is transparent that the comparison is end-to-end — Table 2's models are state-of-the-art reference points, not controlled baselines.

The sub-500M comparison is similarly strong: CLIMB-350M achieves 48.93% average, outperforming Qwen2.5-490M (48.14%) and SmolLM-360M (47.78%). The margins are smaller (0.79 and 1.15 points, respectively), which is expected at smaller model scales where capacity constraints may limit the benefits of improved data composition.

Domain-Specific Optimization on MMLU (Figure 5)

Figure 5 reports accuracy on MMLU domain subsets when CLIMB optimizes specifically for STEM, Humanities, or Social Sciences (rather than general reasoning). The optimization objective is validation-set accuracy within each domain. Results are shown for both 350M and 1B target models, across CLIMB iterations (Iter1, Iter2, Iter3), compared against CLIMB-Best@N and Random baselines.

The headline result for domain adaptation: for the 1B model on Social Sciences, CLIMB-iter3 achieves 41.79% accuracy, compared to CLIMB-Best@N at 40.66% (+1.13) and Random at 36.69% (+5.10). The abstract's claim of "a 5% improvement over random sampling" for Social Sciences is conservative — the actual improvement is approximately 5.1 percentage points (an absolute, not relative, measure), which corresponds to a ~14% relative improvement over the Random baseline of 36.69%.

Key patterns across domains:

STEM (Figure 5a): For the 1B model, CLIMB-iter3 achieves approximately 32.2% (reading off the figure), compared to CLIMB-Best@N at approximately 31.8% and Random at approximately 26.0%. The improvement is roughly +6.2 points over Random. The 350M model shows a similar trend: CLIMB-iter3 at approximately 28.7% vs. Random at approximately 26.5% (+2.2 points). Iterative improvement is visible: CLIMB-iter1 to CLIMB-iter3 increases from approximately 31.0% to 32.2% for the 1B model.
Humanities (Figure 5b): For the 1B model, CLIMB-iter3 achieves approximately 32.5%, compared to CLIMB-Best@N at approximately 31.2% and Random at approximately 27.5%. Improvement is approximately +5.0 points over Random. The 350M model shows CLIMB-iter3 at approximately 29.6% vs. Random at approximately 27.2% (+2.4 points).
Social Sciences (Figure 5c): This is the strongest result. The 1B model shows CLIMB-iter3 at 41.79%, with a clear iterative progression: Iter1 ≈ 40.18%, Iter2 ≈ 41.0%, Iter3 = 41.79%. The 350M model shows CLIMB-iter3 at 39.36%, compared to Random at 34.87% (+4.49 points).

A methodologically important detail in Figure 5: CLIMB-Best@N uses proxy models of the same size as the target model (350M proxy → 350M target; 1B proxy → 1B target), whereas the iterative CLIMB runs use a 350M proxy model for both target sizes. Despite using a smaller proxy model (350M) to optimize for the 1B target, the iterative CLIMB outperforms CLIMB-Best@N, which uses the more expensive 1B proxy. This demonstrates that the iterative refinement process can compensate for proxy model size — a 350M proxy with three iterations of refinement outperforms a 1B proxy with a single round of random sampling, even though each individual 1B proxy evaluation is more informative than each 350M evaluation.

Scaling Behavior with Training Tokens (Figure 1)

Figure 1 (in the paper's introduction) plots average performance on 12 benchmarks as training progresses from 32B to 400B tokens, comparing a 1B model trained on NEMOTRON-CLIMBMIX against models trained on five other datasets: CLIMBLAB-Random (the same 20 clusters but with uniform weights), Nemotron-CC-HQ, SmolLM, DCLM-baseline, and FineWeb-Edu. This is a pre-training from scratch experiment, not continuous pre-training — the models are trained from initialization on the respective datasets with no 10T-token foundation phase.

The CLIMBMIX curve (in blue) dominates all other curves at every token count from 80B onward. At 400B tokens, CLIMBMIX achieves approximately 51.97% average accuracy, compared to CLIMBLAB-Random at approximately 46.36% (+5.61 points), FineWeb-Edu at approximately 48.28% (+3.69 points), Nemotron-CC-HQ at approximately 50.67% (+1.30 points), SmolLM at approximately 51.35% (+0.62 points), and DCLM-baseline at approximately 44.00% (+7.97 points). The gap between CLIMBMIX and the next-best dataset (SmolLM) widens slightly over training, suggesting the optimized mixture enables better scaling — the model continues to benefit from additional tokens rather than saturating.

A notable detail: CLIMBLAB-Random (uniform weights over the same clusters) substantially underperforms CLIMBMIX, confirming that the mixture optimization — not just the clustering and quality filtering — is responsible for the gains. The gap of ~5.6 points between CLIMBMIX and CLIMBLAB-Random at 400B tokens represents the value added by the iterative mixture search over simply clustering and filtering the data.

Ablation Studies and Robustness Checks

All ablation results reported in this section use the 1B target model trained on 40B tokens, with the 350M proxy model unless otherwise specified. Results are from Table 3 and Figure 5.

Search compute budget (Abl.comp in Table 3): Increasing the total number of proxy evaluations from 100% (112 evaluations, achieving 60.41% average) to 150% (168 evaluations, achieving 60.72%) and 200% (224 evaluations, achieving 61.12%) yields monotonically improving downstream accuracy. The gains are not saturated at 200% (+0.71 over 100%), suggesting that the search has not yet converged and that further compute investment would continue to improve the mixture. The marginal benefit per additional evaluation is diminishing — the first 50% increase yields +0.31 points, the second yields +0.40 points — but remains clearly positive. This finding is practically important: if a practitioner has extra GPU-hours available for the search phase, spending them on additional proxy evaluations is beneficial rather than wasted.

Compute allocation across iterations (Abl.allo in Table 3): Three allocation strategies are compared, all using exactly 112 total proxy evaluations: 6:1 (roughly 96:16, two iterations, achieving 60.05%), 4:2:1 (64:32:16, three iterations, achieving 60.41%), and 2:2:1:1 (28:28:28:28, four iterations, achieving 60.14%). The 4:2:1 allocation is clearly optimal — +0.36 over 6:1 and +0.27 over 2:2:1:1. This validates the paper's design choice of three iterations with decreasing per-iteration budgets and demonstrates that both under-iteration (too few refinement rounds) and over-iteration (too little data per round to meaningfully update the predictor) degrade performance. The effect size is modest but consistent across all individual benchmarks.

Proxy model size (Abl.proxy in Table 3): Using proxy models of 62M, 132M, and 350M parameters for the search (all optimizing mixtures for the 1B target model), the resulting average accuracies are 60.11%, 60.19%, and 60.41%, respectively. Larger proxies yield better results, but the gains are modest — the 132M proxy nearly matches the 350M proxy (60.19 vs. 60.41), and even the 62M proxy (5.6× smaller than 350M) achieves 60.11%, only 0.30 points below the best. This is a practically significant finding: relatively small proxy models can effectively guide mixture optimization, making the approach feasible even for teams with limited computational resources. Table 5 (Appendix D.9) shows the 62M proxy model achieving strong results for domain-specific optimization on Social Sciences: CLIMB-iter3 reaches 41.72% for the 1B target model, compared to CLIMB-Best@N (using a 1B proxy) at 40.66%. The 62M-guided search actually exceeds the 1B-guided random sampling baseline.

Number of clusters (Abl.clus in Table 3): Two dimensions of clustering are ablated: K_init (initial k-means clusters before pruning/merging) and K_enhanced (final super-clusters after merging). For K_init, values of 48, 64, 100, 1000, and 2000 are tested (all with K_enhanced = 21). Performance improves from 59.90% at K_init = 48 to 60.44% at K_init = 100, peaks at 60.41% for the default K_init = 1000, and declines to 60.24% at K_init = 2000. The method is fairly robust — the range is only 0.54 points across a 40× variation in K_init — but there is a clear signal: too-coarse initial clustering (48) loses topic resolution, and too-fine initial clustering (2000) creates clusters too small to be meaningful, making the subsequent merging less principled. For K_enhanced, values of 15, 21, and 30 are tested (all with K_init = 1000). Performance is 60.59% at 15 clusters, 60.41% at 21 clusters (the default), and 60.25% at 30 clusters. The 15-cluster setting actually performs slightly better than 21 clusters (+0.18), which is not discussed in the paper but suggests that the optimal number of super-clusters for this dataset and task might be lower than the default. The paper notes that increasing K_enhanced "requires more compute for sampling, increasing the overall cost of the data search process" (Appendix D.4), implying a practical preference for fewer clusters when performance is similar.

Initialization strategy (Abl.init in Table 3): Two initialization methods for the first iteration's sampling are compared: random initialization (uniform-like random vectors on the simplex) and Dirichlet initialization (parameterized by each cluster's token count, so clusters with more tokens are initially sampled at roughly their natural frequency). Dirichlet initialization achieves 60.41% vs. 60.21% for random initialization (+0.20). The small difference suggests the method is not highly sensitive to initialization — the iterative refinement process largely overcomes a suboptimal starting distribution by iteratively focusing sampling on high-performing regions. Dirichlet's slight advantage likely comes from starting the search closer to a reasonable operating point, reducing the number of evaluations wasted on extremely improbable mixtures (e.g., 90% weight on a tiny, niche cluster).

Iterative improvement on domain-specific tasks (Figure 5): Across all three MMLU domains and both model sizes, CLIMB shows consistent improvement from Iter1 to Iter3. For the 1B model on STEM: Iter1 ≈ 31.0% → Iter2 ≈ 31.7% → Iter3 ≈ 32.2%. On Humanities: Iter1 ≈ 31.0% → Iter2 ≈ 31.9% → Iter3 ≈ 32.5%. On Social Sciences: Iter1 ≈ 40.18% → Iter2 ≈ 41.0% → Iter3 = 41.79%. The pattern is monotonic improvement, with diminishing returns in later iterations — the largest jump is typically from Iter1 to Iter2, with a smaller gain from Iter2 to Iter3. This is consistent with the predictor becoming more accurate and the search focusing on finer distinctions within an already-promising region. The 350M model shows a similar pattern but with smaller absolute improvements, likely because the smaller model has less capacity to benefit from fine-grained mixture optimization.

Predictor accuracy (Appendix D.10, Figure 9): The LightGBM predictor achieves a Spearman rank correlation of 94% between predicted accuracy and ground-truth accuracy on a held-out test set of (mixture, score) pairs from the 350M proxy models. The scatter plot in Figure 9 shows the predicted vs. true accuracy pairs clustered around the diagonal, with predictions ranging from approximately 62.0% to 64.5% and true values spanning a similar range. This high correlation validates that the predictor is learning meaningful structure in the mixture-performance relationship — it is not simply memorizing noise in the proxy evaluations. However, the 94% correlation is computed on proxy model evaluations, not target model performance; the predictor is trained to predict proxy model scores, and the implicit assumption is that proxy model scores correlate well with target model scores (which the paper demonstrates indirectly through the final target model results, but does not quantify with a direct correlation measurement).

CLIMB with 62M proxy models (Appendix D.9, Tables 5 and 6): Full experiments are replicated using a 62M proxy model instead of 350M. For general reasoning (Table 6), the 1B target model achieves 60.11% with the 62M-guided CLIMB-iter3, compared to 60.41% with the 350M-guided search (a 0.30-point gap). For Social Sciences (Table 5), the 62M-guided CLIMB-iter3 achieves 41.72% for the 1B target, compared to 41.79% with the 350M-guided search (Figure 5, implied). The 62M proxy model with iterative refinement actually outperforms CLIMB-Best@N using a 1B proxy model for Social Sciences (41.72% vs. 40.66%), confirming that the iterative process, not just proxy model quality, is a significant driver of search effectiveness.

Evolution of cluster weights across iterations (Figure 7 and Appendix D.7): Figure 7 shows heatmaps of mixture weights for each cluster across iterations, for general reasoning and the three MMLU domains. For general reasoning (Figure 7a), most clusters receive near-zero weight throughout, while a few clusters gain prominence: C8 increases from near-zero in Iter1 to 0.13 in Iter3; C9 increases to 0.18 in Iter3; C18 remains important throughout (~0.18); C19 is important early but declines from ~0.22 in Iter1 to ~0.11 in Iter3; C21 declines from ~0.13 to ~0.04. The final mixture is sparse — only 4 clusters (C8, C9, C18, C19) account for the majority of weight. This sparsity is partially by design: "during the sampling process, we intentionally bias towards sparse weights" (Appendix D.8), which amplifies important clusters while filtering out less significant ones. For domain-specific tasks (Figures 7b–d), different clusters dominate: C7, C11, and C19 are crucial for Humanities; C7 and C8 dominate STEM. This confirms that the search process discovers genuinely different mixtures for different downstream objectives, rather than converging to a single "generally good" mixture.

NEMOTRON-CLIMBMIX cluster weights (Figure 8): When optimizing for pre-training from scratch (Section 6), the weights differ notably from the continuous pre-training setting. The final mixture (CLIMB-Iter3) is more balanced across clusters — many clusters have non-trivial weights rather than the extreme sparsity of the continuous pre-training mixtures. The paper explains this: "since the experiments here are conducted under a pre-training-from-scratch setting, a more balanced cluster distribution is required compared to continuous pre-training. This difference arises because continuous pre-training provides a strong foundation, allowing the model to focus primarily on learning a few important domains, whereas pre-training from scratch necessitates more diverse data coverage" (Section 6). This is an important finding for practitioners: the optimal mixture depends on whether you are training from scratch or continuing from a strong foundation. The iterative search discovers this without being explicitly told — the different mixture structures emerge naturally from the different optimization contexts.

Critical Assessment

Claim 1: "CLIMB outperforms all baseline data mixture methods"

Does Table 1 support this? Yes, but with important scope limitations. For the exact experimental setup described — 350M and 1B target models, continuous pre-training on 40B tokens, Nemotron-CC source data clustered into 21 groups, optimization target being validation accuracy on PIQA + ARC_E + HellaSwag — CLIMB achieves higher average accuracy than Random, DoReMi, and RegMix. The margins are consistent across model sizes (~1.0 point over RegMix), and the per-task breakdown shows improvements on most individual benchmarks, including unoptimized ones.

However, the comparison has three important constraints that limit the strength of the claim. First, the baselines (DoReMi, RegMix) are not necessarily implemented in their optimal configurations. DoReMi was originally designed for and evaluated on The Pile's 22 domains, not on 21 embedding-derived clusters from Common Crawl. RegMix was evaluated on different datasets and model scales. The paper gives these baselines the same proxy model size and cluster structure as CLIMB, but other design choices (DoReMi's Group DRO hyperparameters, RegMix's predictor regularization) were presumably set to reasonable defaults rather than being extensively tuned per baseline — which is standard practice but means the comparison is between "CLIMB as carefully designed" and "DoReMi/RegMix as adapted to this setting," not between all methods at their respective optima.

Second, the baseline methods were originally designed for datasets with naturally occurring domain divisions (The Pile's 22 hand-curated domains). CLIMB's 21 clusters are algorithmically derived from embedding similarity. It is possible — and the paper does not investigate — that the performance ordering (CLIMB > RegMix > DoReMi > Random) would change if the baselines were given hand-curated domain labels instead of CLIMB's clusters. CLIMB might benefit from its tight integration with the clustering pipeline: the clusters are produced by embedding, and the mixture search operates over those same embedding-derived clusters, creating a favorable inductive bias. If RegMix were given the same 21 clusters, it underperforms CLIMB, but if RegMix were given a different set of domain divisions (e.g., The Pile's categories), its performance relative to CLIMB on those categories is unknown.

Third, the Random baseline (uniform weights over the 21 clusters) is a weak baseline. A more informative baseline would be "train on all available data without cluster sampling" — i.e., just use the filtered corpus without any domain partitioning — which would reveal whether the clustering itself provides value beyond the filtering. The paper does not report this comparison.

Claim 2: "Our 1B model exceeds Llama-3.2-1B by 2.0%"

Does Table 2 support this? The 2.0% number refers to the 53.54% vs. 51.56% average across 12 benchmarks, and these numbers are exactly what Table 2 reports. The claim is factually accurate as a description of the table.

However, the reader must understand what this claim does not establish. The comparison is between two models that differ in: architecture details, tokenizer, total training tokens, training data sources, training data preprocessing, learning rate schedules, and batch sizes — only one of which is the data mixture optimization that CLIMB contributes. CLIMB is a data mixture search method; the 2.0% advantage of the CLIMB-trained model over Llama-3.2 reflects the entire training pipeline (10T tokens of DCLM + TxT360 foundation, then 400B tokens of CLIMB-optimized mixture, with CLIMB's specific preprocessing, model architecture, and hyperparameters). It would be incorrect to attribute all 2.0 percentage points to the mixture search algorithm itself.

The paper does not attempt to isolate how much of the 2.0% comes from CLIMB vs. from other pipeline differences. A more controlled version of this claim would require training a model on the same total tokens using the same architecture and hyperparameters, changing only whether the data mixture was optimized by CLIMB or by an alternative method. The closest the paper comes to this is Table 1 (CLIMB vs. RegMix/DoReMi/Random on the same base model and same total tokens), where the mixture-specific gain is ~1.0–1.5 points over the next-best method, not 2.0.

Additionally, Table 2 compares CLIMB against models that were not designed or optimized using the MMLU, OBQA, BoolQ, or RACE benchmarks (CLIMB's optimization target was only PIQA, ARC_E, and HellaSwag validation). If Llama-3.2's training process also optimized for some of the evaluation benchmarks (explicitly or implicitly through data selection), the comparison becomes asymmetric. The paper does not discuss the optimization objectives of the baseline models.

Claim 3: "Optimizing for a specific domain yields a 5% improvement over random sampling"

Does Figure 5 support this? Yes, for Social Sciences on the 1B model, CLIMB-iter3 (41.79%) minus Random (36.69%) equals 5.10 percentage points. The "5%" refers to an absolute percentage-point difference, not a relative improvement (which would be ~14%). The same pattern holds qualitatively for STEM and Humanities, though with varying magnitudes.

The domain-specific experiments use MMLU's pre-defined subject groupings (STEM, Humanities, Social Sciences), and the optimization target is validation accuracy within each domain. The test evaluation is on the same MMLU domain subsets. This is a direct test of whether CLIMB can target specific capabilities. The iterative improvement is clearly visible in Figure 5, supporting the paper's claim that the iterative search refines mixtures for domain-specific objectives.

A limitation: the domain definitions (STEM, Humanities, Social Sciences) are MMLU's own categorizations, which are coarse and may not align with real-world domain boundaries. The paper acknowledges this in Appendix A: "our evaluation of domain-specific benefits is based on MMLU's coarse-grained domain categories, which may not fully reflect real-world applications." A practitioner wanting to optimize for "legal reasoning" or "medical diagnosis" would need to construct their own validation benchmark; whether CLIMB's mixture search would generalize to those domains is an open question.

Claim 4: "Iterative bootstrapping outperforms one-shot regression"

Do the ablation results support this? Table 3's "Abl.allo" provides the key evidence: the 4:2:1 allocation (three iterations, the CLIMB method) achieves 60.41%, while the 6:1 allocation (closer to a one-shot approach with a small refinement round) achieves 60.05%. The difference is 0.36 points — statistically modest but directionally consistent with the claim. The paper frames RegMix as the limiting case of a single iteration (Section 2.2), and the 60.41% vs. 59.37% comparison in Table 1 (CLIMB vs. RegMix on the 1B target) shows a 1.04-point advantage for the iterative approach.

However, RegMix and the 6:1 ablation are not identical: RegMix uses uniform random sampling for its single round, while CLIMB's first iteration uses Dirichlet-based sampling. The 6:1 ablation preserves CLIMB's Dirichlet initialization but compresses iterations, so it isolates the allocation effect more cleanly than the Table 1 comparison. The consistent finding — iterative search outperforms one-shot — is supported by both comparisons, but the effect size is small enough (0.36–1.04 points) that the practical significance may depend on the application. For a production LLM where 1% accuracy improvement is worth millions of dollars in user satisfaction, the iterative approach is clearly worthwhile. For a research prototype where the engineering complexity of multi-iteration scheduling adds friction, the simpler RegMix approach might be acceptable.

Claim 5: "The predictor accurately estimates mixture quality"

Does Figure 9 and the surrounding analysis support this? The 94% Spearman rank correlation between predicted and true proxy model accuracy is strong evidence that the LightGBM predictor captures meaningful structure in the mixture-performance relationship. However, this correlation is measured on the proxy model evaluations, not on target model performance. The paper's implicit assumption is that proxy model performance correlates well with target model performance — an assumption that is not directly validated with a correlation measurement.

The paper provides indirect evidence for this assumption: mixtures selected by the predictor (which was trained on proxy model data) produce strong target model performance (Tables 1–2). But the critical measurement — the rank correlation between predicted proxy performance and actual target model performance for the same mixture — is not reported. If this correlation were, say, 0.65 rather than 0.94, the predictor would be much less useful as a surrogate, and the final selected mixture would be less reliable. The paper's decision to use a maximum-depth-4 LightGBM with strong regularization (L1 + L2 + 5-sample minimum per leaf) suggests the authors were aware of the overfitting risk when generalizing from proxy to target, but they do not quantify the proxy-to-target transfer fidelity.

Additional Concerns and Missing Experiments

Single source dataset and model family. All experiments use Nemotron-CC (and smollm-corpus for the CLIMBLAB/CLIMBMIX releases) as the source data, and all models are standard Transformer decoders in the 62M–1B range. There is no evidence about whether CLIMB's approach transfers to other web corpora (C4, FineWeb, RefinedWeb), other model architectures (non-decoder, mixture-of-experts), or larger scales (>1B parameters). The paper notes that the proxy model findings transfer across 62M–350M proxy sizes, which is encouraging but does not guarantee transfer to the 7B–70B range where most production LLMs operate. The cost of the search (~5,000 GPU-hours for the default budget) is independent of target model size (since it uses proxy models), so the approach is economically viable at any target scale — but whether the quality of the discovered mixture transfers to much larger models is unknown.

No statistical significance or variance reporting. The paper reports single numbers for each experiment (e.g., "60.41% average") without error bars, confidence intervals, or multiple random seeds. The authors state this is because the "≥100B token" training runs are stable (Checklist item 7), but this assertion is not backed by data. For the main Table 1 results, the differences between CLIMB and the next-best method (RegMix) are ~1.0 point. Without knowing the run-to-run variance, it is unclear whether this difference is statistically significant or within the noise of random initialization and data ordering. The cost of running multiple seeds (each requiring 6,400 GPU-hours for the 1B target model) would be substantial, but reporting variance for at least the proxy model evaluations (which are cheaper at 45 GPU-hours each) would have been feasible and informative.

Difficulty estimation cost not accounted for. The clustering and embedding step processes 800B tokens through a 400M-parameter encoder. The computational cost of this preprocessing is not reported, but it could be comparable to or larger than the search budget itself (embedding 800B tokens at, say, 1ms per 512-token chunk would require thousands of GPU-hours). Unlike the proxy model search cost (~5,000 GPU-hours), this cost is not compared against the target model training cost (~6,400 GPU-hours for one run). A complete accounting of CLIMB's total computational footprint would include: embedding + clustering + quality annotation (1M texts with Nemotron-340B) + fastText training + proxy model search (112 × 45 GPU-hours) + target model training (6,400 GPU-hours). Without the embedding and annotation costs, the efficiency claims are incomplete.

The 40B-token continuous pre-training protocol may not generalize to from-scratch training. The main experiments (Tables 1, 3, Figure 5) all use continuous pre-training on 40B tokens after a 10T-token foundation. The NEMOTRON-CLIMBMIX experiment (Figure 1) is the only from-scratch result, and it shows a different optimal cluster weight distribution (more balanced, less sparse). This suggests that the mixtures discovered under the continuous pre-training protocol may not be optimal for from-scratch training. A practitioner wanting to use CLIMB to optimize a mixture for training from scratch would ideally run the search in a from-scratch context, which would require proxy models trained from scratch — increasing the cost per proxy evaluation, potentially substantially.

No comparison with quality-filtering-only baselines at equal token count. The paper compares CLIMB against other data mixture methods (DoReMi, RegMix) and against other pre-training datasets (FineWeb-Edu, DCLM, etc.), but does not compare against the simpler baseline of "use the same quality filtering and clustering, but don't optimize the mixture weights — just sample uniformly from the quality-filtered clusters." CLIMBLAB-Random in Figure 1 is this baseline for the from-scratch setting and shows a 5.6-point gap, which is strong evidence. But in the continuous pre-training setting (where most experiments are conducted), this baseline is not reported separately from the "Random" baseline in Table 1 (which uses equal weights on the 21 clusters — and is effectively CLIMBLAB-Random for the continuous setting). The 60.41% vs. 57.93% gap for the 1B model in Table 1 (CLIMB vs. Random) represents the combined value of clustering, filtering, AND mixture optimization, not the incremental value of mixture optimization over clustering + filtering alone.

The optimal cluster count ablation has an interesting result that is not discussed. In Abl.clus, K_enhanced = 15 achieves 60.59% while K_enhanced = 21 achieves 60.41% — the 15-cluster setting is slightly better. The paper does not comment on this. One possible explanation: with fewer clusters, the mixture search problem is lower-dimensional (14 degrees of freedom vs. 20), making the 112 proxy evaluations more effective at covering the space. This would be an interesting finding — that there is a tradeoff between cluster granularity (more clusters = finer control over data composition) and search efficiency (fewer clusters = easier optimization) — but the paper does not explore it.

Limited exploration of the cluster-topic-to-performance relationship. Appendix D.2 provides qualitative analysis of which clusters matter for which tasks, but the analysis is post-hoc and descriptive. The paper does not attempt to predict, from cluster topic labels, which clusters will be important for which downstream tasks. This is understandable — the whole point of CLIMB is that such predictions are unreliable, and empirical search is necessary — but it also means the paper does not provide guidance on how to reduce the search space a priori based on cluster characteristics. A practitioner with a new downstream task cannot look at their clusters' topic descriptions and guess a good initial mixture; they must run the full search. Developing methods to warm-start the search based on cluster-task similarity (perhaps using the embedding similarity metrics that Appendix D.2 computes) would be a valuable extension that the paper does not pursue.

6. Limitations and Trade-offs

Limitation 1: The Computational Cost of Difficulty Estimation (Clustering + Embedding) Is Not Accounted for in the Headline Efficiency Numbers

The assumption or constraint. The CLIMB pipeline requires a one-time preprocessing phase that embeds every document in the source corpus using a 400M-parameter encoder (stella_en_400M_v5), clusters the embeddings with k-means (FAISS), annotates 1 million texts with a 340B-parameter model (Nemotron-340B) for quality scoring, trains fastText classifiers on those annotations, and then performs centroid-distance-based merging. The source corpus for the main experiments is approximately 800B tokens (Section 3.1, Appendix C.2). The paper reports the cost of the proxy model search (~5,040 GPU-hours for 112 evaluations at 45 GPU-hours each) and the cost of target model training (~6,400 GPU-hours for a 1B model on 40B tokens, Appendix C.4), but explicitly does not account for the embedding, clustering, or annotation costs in any budget calculation.

The paper acknowledges this gap implicitly, noting in Section 2.1 that the clustering pipeline uses FAISS "optimized for billion-scale similarity search on GPUs" and that the fastText classifiers are trained on "1 million texts annotated with Nemotron-340B," but it never reports the GPU-hours consumed by these steps. In Appendix A, the paper notes that "training these proxy models still incurs non-negligible costs" and suggests "parameter-efficient tuning, distillation, or zero-shot evaluation strategies" as future work, but this refers to reducing proxy model costs, not the unaccounted preprocessing costs.

The consequence. A practitioner evaluating whether to adopt CLIMB cannot estimate the true total computational cost without independently benchmarking the embedding and clustering steps. If embedding 800B tokens through a 400M-parameter encoder and running k-means on the resulting vectors consumes, say, 10,000 GPU-hours, then the true cost of CLIMB is approximately 3× the reported proxy search cost. This changes the economic calculus: the claimed 142× cost advantage of proxy-based search over direct target model evaluation (45 vs. 6,400 GPU-hours per evaluation) is accurate for the search loop itself, but the total CLIMB pipeline may be substantially more expensive than simply training a few target models on manually designed mixtures if the preprocessing overhead is large.

Moreover, the embedding and clustering cost scales with corpus size, while the proxy search cost does not (it depends only on the number of clusters, not the total tokens). For a 10T-token corpus (which is increasingly common in frontier model training), the preprocessing cost could dominate the total pipeline cost. The paper provides no scaling analysis or cost model that would allow a practitioner to estimate this preprocessing overhead for their own corpus size.

What evidence exists in the paper. None. The paper does not report GPU-hours, wall-clock time, or FLOP counts for the embedding, clustering, or annotation steps. The only cost figures provided are for proxy model training (45 GPU-hours each, Appendix C.4) and target model training (6,400 GPU-hours each). The FAISS and fastText steps are mentioned as implementation details (Section 3.1) without cost quantification.

Mitigation status. Not addressed. The paper frames the contributions to include the automated clustering pipeline as a key innovation (Section 1: "Automated Data Mixture Optimization"), but the cost of that automation is left as an unknown for anyone wanting to reproduce or deploy the method. The paper does not suggest amortization strategies (e.g., clustering once and reusing for many downstream objectives), though such amortization would substantially change the cost calculus: if the 800B-token corpus is clustered once and then used for dozens of domain-specific mixture optimizations, the preprocessing cost per optimization becomes negligible. The absence of this analysis is a significant gap for practical adoption.

Limitation 2: The Main Results Are for Continuous Pre-Training on a Short 40B-Token Window After a Massive 10T-Token Foundation — Not for Pre-Training from Scratch — and the Optimal Mixture Depends Fundamentally on This Distinction

The assumption or constraint. The entire experimental validation of CLIMB's mixture search (Tables 1 and 3, Figure 5, and all ablations in Section 5) uses a specific protocol: models are first pre-trained from scratch on 10 trillion tokens of general-purpose data (DCLM + TxT360) using a warmup-stable-decay schedule, then continuously pre-trained for an additional 40B tokens on the CLIMB-optimized mixture. The mixture search itself uses this same protocol — proxy models are evaluated by continuing their training from the 10T-token checkpoint for 40B tokens on the candidate mixture. The paper states this explicitly (Section 3, Model description): "we focus on the data mixing research in the decay stage," using the WSD schedule because it "supports resuming at any time of the stable stage."

This means that CLIMB, as validated, discovers the optimal mixture for the final 0.4% of training tokens (40B out of 10.04T total), not for the entire training run. For the 10T-token foundation phase, all models see the same fixed data mixture (DCLM + TxT360) — CLIMB does not optimize this. The paper's own results in Section 6 (NEMOTRON-CLIMBMIX, Figure 1) reveal that this distinction matters enormously: when training from scratch, the optimal cluster weights are "more balanced" than in the continuous pre-training setting because "continuous pre-training provides a strong foundation, allowing the model to focus primarily on learning a few important domains, whereas pre-training from scratch necessitates more diverse data coverage." This is not a minor difference — it is a qualitative change in the structure of the optimal mixture.

The consequence. A practitioner who reads the abstract's claim that CLIMB "discovers, evaluates, and refines data mixtures in a pre-training setting" and then applies CLIMB to optimize a mixture for training a model from scratch (as the NEMOTRON-CLIMBMIX experiment in Section 6 does) is using the method in a regime where the paper's main experimental validation provides no direct evidence. The paper demonstrates that CLIMB can work for from-scratch training (Figure 1 shows CLIMBMIX outperforming other datasets), but it does not report the search cost, the iterative refinement behavior, the predictor accuracy, or the ablation results for the from-scratch setting. The proxy model evaluations for from-scratch optimization would need to train models from scratch (or at least from an early checkpoint), which would be substantially more expensive than the 40B-token continuous training used in the main experiments. Whether the 350M proxy model at 40B from-scratch tokens correlates well with 1B target model at 400B from-scratch tokens is an open question.

Furthermore, the 10T-token foundation phase means the models have already seen enormous quantities of diverse data before the CLIMB mixture is applied. The CLIMB mixture is thus optimizing for domain refinement on top of a strong general foundation, not for building general capabilities from raw data. The finding that CLIMB's optimal mixtures are sparse (only a few clusters dominate, Figure 7a) may be an artifact of this setting — when the model already has broad knowledge, a concentrated dose of high-value data is sufficient. For from-scratch training, where the model needs to learn everything from the mixture, the sparse strategy would likely fail (as the paper's own Section 6 observation about more balanced weights confirms).

What evidence exists in the paper. The distinction between continuous pre-training and from-scratch training is documented in Section 6, where the paper observes that from-scratch mixtures require "a more balanced cluster distribution." The WSD schedule and the 10T-token foundation are described in Section 3 (Model description) and Appendix C.3. The from-scratch experiment in Figure 1 uses a 1B model trained on 400B tokens of CLIMBMIX, but the search process that produced CLIMBMIX is not described in detail — specifically, whether the iterative search was run in a from-scratch context or adapted from the continuous pre-training results.

Mitigation status. Partially addressed through the NEMOTRON-CLIMBMIX experiment (Section 6, Figure 1), which demonstrates that a CLIMB-optimized mixture can produce a strong model when trained from scratch. However, the paper does not report the search methodology, cost, or ablations for this from-scratch optimization, nor does it quantify how much worse the continuous-pre-training-optimized mixture would perform in the from-scratch setting. The paper acknowledges in Section 3 that the 10T-token phase-1 pre-training "does not strictly align with scaling laws" but argues it "does not hurt performance" — this justification addresses whether the foundation phase is detrimental, not whether the continuous-pre-training results transfer to from-scratch training. A practitioner wanting to use CLIMB for from-scratch mixture optimization would need to re-run the entire search in a from-scratch context, which would multiply the search cost by the ratio of from-scratch-tokens to continuous-training-tokens (potentially 10–100×).

Limitation 3: Verification Is Against a Small Set of Benchmarks That Serve as Both Optimization Target and Evaluation — with Unclear Generalization to Tasks Outside the Reasoning Domain or to Real-World Deployment

The assumption or constraint. CLIMB's mixture search optimizes against validation-set accuracy on three specific benchmarks: PIQA, ARC_E, and HellaSwag. These are all commonsense reasoning tasks in English. The paper demonstrates that optimizing on these three tasks generalizes to improvements on other held-out reasoning benchmarks (ARC_C, WinoGrande, SIQA, MMLU, etc.) and to language modeling perplexity (WikiText, LAMBADA). However, all evaluation benchmarks are drawn from the same general category of "reasoning tasks" — they test a model's ability to perform multi-step inference, resolve ambiguity, and apply world knowledge, but they do not test factual recall, instruction following, open-ended generation quality, multilinguality, code generation, mathematical reasoning (MATH, GSM8K), or safety-related behaviors. The paper's domain-specific experiments (Section 5, Figure 5) extend the optimization target to MMLU subdomains (STEM, Humanities, Social Sciences), but MMLU is primarily a factual knowledge and reasoning benchmark, not a test of generative capabilities.

The paper acknowledges this scope limitation in Appendix A: "our evaluation of domain-specific benefits is based on MMLU's coarse-grained domain categories (e.g., STEM, Social Sciences), which may not fully reflect real-world applications" and "we have not yet evaluated its effectiveness in high-stakes domains such as finance or healthcare, where data characteristics and requirements can differ substantially." However, the paper does not discuss the limitation of the general reasoning benchmark suite itself — that optimizing for commonsense reasoning accuracy may not translate to improvements (or may even degrade performance) on qualitatively different capabilities like code generation, long-form text coherence, or factual precision.

The consequence. A practitioner who uses CLIMB to optimize a mixture for "general capability" (as measured by the paper's 12-benchmark average) may find that the resulting model underperforms on capabilities not represented in the optimization objective. The paper provides evidence that the optimized mixture improves perplexity (Table 1: CLIMB achieves lower WikiText perplexity than Random for the 1B model, 15.96 vs. 17.82), suggesting that language modeling quality is not sacrificed. But perplexity on standard corpora is a weak proxy for generative quality, factuality, or instruction-following — all of which are critical for real-world LLM deployment and none of which are measured.

More subtly, the optimization objective itself may introduce a bias. The paper notes that the optimization target is the average of three validation accuracies. If one of these tasks (say, HellaSwag) dominates the variance in the average — because its accuracy varies more across mixtures — the search will effectively optimize for HellaSwag at the expense of PIQA and ARC_E. The paper does not report per-task optimization weights or discuss whether the three-target average produces a balanced improvement. Table 1 shows that CLIMB's largest gains for the 1B model are on HellaSwag (+3.11 over Random) and ARC_E (+2.73), with PIQA showing a smaller gain (+1.73) — consistent with HellaSwag being the primary driver of optimization. This is not necessarily a problem (HellaSwag is a challenging benchmark and improving on it is valuable), but it means the "general reasoning" optimization is implicitly weighted toward whatever benchmark has the steepest gradient in the mixture-performance landscape.

What evidence exists in the paper. The full benchmark suite is described in Section 3 and Appendix C.2. The optimization target (PIQA, ARC_E, HellaSwag validation) is specified in Section 3 (Model description). The generalization to unoptimized benchmarks is shown in Tables 1 and 2, where CLIMB improves on ARC_C, WinoGrande, SIQA, MMLU, OBQA, BoolQ, and RACE despite not optimizing for them directly. The generalization appears robust within the "reasoning" category but has no evidence outside it. The paper does not evaluate on MATH, GSM8K, HumanEval, MBPP, or any code, math, or generation benchmark.

Mitigation status. Partially addressed through the broad 12-benchmark evaluation suite, which tests generalization across multiple reasoning subtypes. The domain-specific experiments (Figure 5) show that CLIMB can be targeted to different optimization objectives (STEM, Humanities, Social Sciences), demonstrating flexibility in the objective function. However, the paper does not discuss the choice of optimization targets, the risk of overfitting to the validation sets of the three target tasks, or the possibility that optimizing for commonsense reasoning could degrade other capabilities. The suggestion in Appendix A that future work should evaluate CLIMB on "high-stakes domains such as finance or healthcare" acknowledges the domain gap but not the capability-type gap (reasoning vs. generation vs. factuality).

Limitation 4: The Entire Pipeline Is Validated on a Single Source Dataset and Model Family, with No Evidence of Transfer to Other Corpora, Architectures, or Scales Beyond 1B Parameters

The assumption or constraint. All experiments in the paper use Nemotron-CC (Su et al., 2024) as the source corpus for clustering, with smollm-corpus (Ben Allal et al., 2024) added for the NEMOTRON-CLIMBLAB and NEMOTRON-CLIMBMIX releases (Section 6). All models are standard Transformer decoder-only architectures in the 62M–1B parameter range, trained with the same WSD schedule, same AdamW optimizer, same batch size (2M tokens), and same phase-1 foundation data (DCLM + TxT360). The embedding model for clustering is fixed to stella_en_400M_v5, and the quality annotator is fixed to Nemotron-340B (Section 3.1).

The paper does not claim that these choices are universal or that CLIMB would work identically with different components. The authors state they "believe this model is representative" (though this phrasing is from the reference example; the actual paper does not make a strong representativeness claim about the model family). The paper's ablation on proxy model size (Table 3, Abl.proxy) shows that results are robust from 62M to 350M proxies, which is evidence of some scale-transfer within the tested range, but 350M to 1B is only a ~3× scale-up, and 1B is far from the 7B–70B range where most production LLMs operate.

The consequence. At least four distinct transfer questions are unanswered:

Corpus transfer: Would the same clustering hyperparameters (K_init = 1000, pruning threshold 3.0, merging distance 1.5) produce semantically coherent clusters on a different web corpus (C4, FineWeb, RefinedWeb) with different quality characteristics, topic distributions, and noise patterns? The paper's clustering pipeline involves several manually set thresholds; their sensitivity to corpus characteristics is unknown.
Architecture transfer: The mixture optimization treats the model as a black box that maps data mixtures to downstream performance. In principle, this should transfer across architectures, but the paper provides no evidence that a mixture optimized for a standard Transformer decoder is also optimal for a mixture-of-experts model, a model with different activation functions, or a model with different tokenization. Recent work has shown that optimal data composition can interact with model architecture (e.g., MoE models may benefit differently from domain specialization than dense models).
Scale transfer: The most critical practical question: if you run CLIMB with a 350M proxy model to optimize a mixture, and then train a 7B or 70B target model on that mixture, does the performance ordering of mixtures preserve? The paper shows that a 62M proxy can effectively guide a 1B target (Table 5, Appendix D.9: 62M-guided CLIMB-iter3 achieves 60.11% for the 1B target, only 0.30 points below the 350M-guided search), which is encouraging. But the jump from 1B to 7B is 7× in parameters and potentially much larger in data requirements (7B models may need qualitatively different data compositions to fully utilize their capacity). The paper does not discuss scale transfer or provide any evidence beyond the 62M→1B result.
Embedding model transfer: The choice of stella_en_400M_v5 as the embedding model is stated without ablation. If a different embedding model produced different cluster assignments, would the downstream mixture optimization converge to a different (worse? better?) set of clusters? The quality of the discovered mixture depends on the quality of the clustering, which depends on the embedding model's ability to capture semantically meaningful distinctions. A weaker embedding model might produce clusters that mix unrelated topics, making the mixture search less effective; a stronger embedding model might produce finer-grained distinctions that improve the achievable performance but also increase the search space dimensionality. Neither scenario is explored.

What evidence exists in the paper. The proxy model size ablation (Table 3, Abl.proxy) provides evidence of scale transfer within the 62M–350M range. The cluster count ablation (Table 3, Abl.clus) shows robustness to K_init from 48 to 2000, which is evidence that the clustering pipeline is not exquisitely sensitive to hyperparameters. The model family and source dataset are held constant across all experiments, so no transfer evidence exists for those dimensions.

Mitigation status. Not addressed. The paper does not discuss corpus, architecture, or scale transfer as limitations. Appendix A mentions that "further reducing the computational burden... remains an important direction for future work" and that "we have not yet evaluated its effectiveness in high-stakes domains," but these refer to computational cost and domain generalization, not to the fundamental transferability of the method across different base components. The public release of NEMOTRON-CLIMBLAB (1.2T tokens, 20 clusters) and NEMOTRON-CLIMBMIX (400B tokens) partially mitigates the corpus limitation by providing a standardized benchmark for future work to test CLIMB variants on, but the paper itself does not perform cross-corpus validation.

Limitation 5: The Predictor Is Validated Only Against Proxy Model Performance, Not Target Model Performance — the Critical Link in the Surrogate Optimization Chain Is Unmeasured

The assumption or constraint. CLIMB's entire search strategy rests on a surrogate model: a LightGBM regressor f(α) is trained to predict the performance that a proxy model would achieve when trained on mixture α, and the mixture that maximizes this predictor is then used to train the target model. The method works only if the ranking of mixtures by proxy model performance correlates strongly with the ranking by target model performance. If this correlation is weak, the predictor may confidently identify a mixture that is optimal for the 350M proxy model but mediocre for the 1B target model, and CLIMB provides no mechanism to detect or correct this failure.

The paper reports a 94% Spearman rank correlation between the predictor's predictions and proxy model ground-truth performance (Appendix D.10, Figure 9). This validates that the LightGBM model can learn the mixture→proxy-performance mapping from ~100 training points. But the paper never reports the correlation between proxy model performance and target model performance for the same set of mixtures — the measurement that would directly validate the surrogate assumption. The fact that the final target model achieves strong results (Tables 1 and 2) is indirect evidence that the surrogate assumption holds in this case, but it provides no quantification of the proxy-to-target relationship and no diagnostic for when the assumption might fail.

The consequence. Without a proxy-to-target correlation measurement, a practitioner cannot assess the risk that CLIMB's selected mixture is suboptimal for the target model despite being optimal for the proxy model. This risk is not merely theoretical — there are plausible mechanisms by which proxy and target models could prefer different mixtures:

Capacity interaction: A 350M model might benefit most from a mixture that emphasizes simple, high-signal patterns (e.g., straightforward reasoning examples), while a 1B model might have the capacity to extract value from more complex or noisy data. The optimal mixture for the small model could be too narrow for the large model, or vice versa.
Phase-1 interaction: All proxy and target models share the same 10T-token foundation phase (same data, same schedule). If this foundation phase interacts differently with subsequent training for different model sizes — e.g., a 1B model might have learned more from the foundation phase and thus need different refinement — the optimal continuation mixture could differ by scale.
Optimization target interaction: The proxy model is optimized for validation accuracy on PIQA + ARC_E + HellaSwag. If the relative difficulty of these tasks differs by model scale (e.g., ARC_E might be relatively harder for a 350M model than for a 1B model), the mixture that maximizes average accuracy for the proxy might weight the tasks differently than what would maximize average accuracy for the target.

The paper's ablation on proxy model size (Table 3, Abl.proxy) provides partial evidence: the 62M proxy discovers a mixture that achieves 60.11% for the 1B target, while the 350M proxy discovers one that achieves 60.41%. The 0.30-point gap is small, suggesting that proxy size does not drastically change the selected mixture in this specific case. But this is a within-experiment comparison of two different proxy models, not a direct measurement of the correlation between proxy scores and target scores across many mixtures. It tells us that the two proxy sizes converge to similar mixtures, not that those mixtures are near-optimal for the target model.

What evidence exists in the paper. Appendix D.10 and Figure 9 show a 94% Spearman correlation between predictor predictions and proxy model ground truth. This is a validation of the predictor's accuracy, not of the proxy-to-target transfer. The final target model performance (Tables 1 and 2) is consistent with the proxy model ranking being informative, but it is a single data point (the selected mixture performs well) rather than a correlation measurement across many mixtures.

Mitigation status. Not addressed. The paper does not discuss the proxy-to-target correlation as a potential failure mode, does not measure it, and does not propose diagnostics (e.g., evaluating a handful of top predicted mixtures with the target model to validate the ranking before committing to the full 400B-token training run). The iterative search's increasing focus on high-scoring mixtures (Figure 3) would amplify any systematic bias in the proxy-to-target mapping: if the proxy systematically overestimates the value of certain clusters for the target model, the search would concentrate sampling on those clusters and the predictor would become increasingly confident in a biased optimum. The paper's success in this specific setting suggests the bias is not severe, but the absence of any measurement or discussion of this assumption is a notable methodological gap.

Limitation 6: No Statistical Significance or Variance Reporting — All Reported Numbers Are Single-Run Point Estimates from Expensive Training Runs Where Run-to-Run Variance Is Unknown

The assumption or constraint. Every result in the paper — Tables 1, 2, 3, Figure 5, Figure 1 — is reported as a single number (e.g., "60.41% average accuracy") with no error bars, confidence intervals, or multiple random seeds. The paper's checklist (item 7) states: "We performed relatively large-scale training (≥100B tokens) so that the task performance is stable across runs." This is an assertion, not evidence. Large-scale training reduces some sources of variance (batch order noise averages out over many tokens) but does not eliminate others (random initialization, data shuffle order for the 40B-token mixture phase, the specific 112 proxy model training runs, the random seed of the LightGBM predictor, the k-means initialization).

The paper's main comparisons involve margins that are modest relative to plausible run-to-run variance in language model training. For Table 1 (1B target model), CLIMB achieves 60.41% vs. RegMix at 59.37% — a gap of 1.04 points. For Table 3 (Abl.allo), the 4:2:1 allocation achieves 60.41% vs. 6:1 at 60.05% — a gap of 0.36 points. Without knowing whether the standard deviation of these measurements is, say, 0.2 points or 1.0 point, the reader cannot assess whether the observed differences are statistically reliable or within noise. The cost of a single 1B target model training run (6,400 GPU-hours, Appendix C.4) makes multiple-seed replication expensive, but the proxy model evaluations (45 GPU-hours each) could have been replicated to establish proxy-level variance, which would at least bound the uncertainty in the predictor's training data.

The consequence. At least two important claims in the paper are sensitive to unreported variance:

The optimality of the 4:2:1 allocation (Table 3, Abl.allo). The difference between 4:2:1 (60.41%) and 2:2:1:1 (60.14%) is 0.27 points. If the standard deviation of the 12-benchmark average is greater than ~0.15 points (which would not be unusual for a 40B-token training run evaluated on noisy benchmarks like TruthfulQA and RACE), this difference is not statistically significant, and the paper's conclusion that "balancing depth and breadth proves key" would be unsupported. A practitioner might reasonably choose the 2:2:1:1 allocation (which is simpler to implement, having uniform batch sizes per iteration) rather than the 4:2:1 allocation if the difference is within noise.
The superiority of CLIMB over RegMix (Table 1). The 1.04-point gap for the 1B model is based on one CLIMB run and one RegMix run. If the mixture search process has substantial variance (due to the random sampling of configurations, the random initialization of the predictor, and the randomness in proxy model training), then repeating the entire CLIMB and RegMix pipelines might produce different gaps — potentially larger, potentially smaller, potentially reversed. The paper's iterative search involves ~112 randomized decisions (which 64 configurations to sample in iteration 1, which 32 in iteration 2, etc.), and the final mixture is the predictor's single best prediction. The variance of this entire end-to-end process is completely unknown.

The paper also does not report the variance of individual benchmark scores. The 12-benchmark average in Table 2 (53.54% for CLIMB) aggregates tasks with very different score ranges and variances — PIQA typically has low variance across runs (it is a relatively easy binary-choice task), while RACE or TruthfulQA may have higher variance. An unweighted average can be dominated by improvements on high-variance benchmarks, creating an illusion of robustness.

What evidence exists in the paper. None. The paper explicitly declines to report statistical significance measures in the checklist (item 7), justifying that large-scale training makes performance "stable across runs." No variance estimates, standard deviations, or confidence intervals are provided for any result.

Mitigation status. Not addressed. The checklist answer acknowledges the absence of error reporting and justifies it on the basis of training scale, but this justification conflates the stability of a single training run (which large token counts may ensure) with the reproducibility of the entire pipeline across different random seeds (which large token counts do not guarantee). The paper does not suggest that future work should establish variance estimates, nor does it provide even a partial variance analysis using the cheaper proxy model evaluations as replicates. This is a methodological weakness that limits the strength of the paper's quantitative claims, particularly for the smaller-margin comparisons that are central to the ablation analysis.

7. Implications and Future Directions

How This Work Changes the Landscape

CLIMB shifts the conversation around pre-training data from a craft — where experienced researchers manually curate domain splits and guess mixture proportions — toward a systematic optimization problem where the data composition itself becomes a first-class object of algorithmic search. This is not a paradigm shift on the scale of the Transformer architecture or the Chinchilla scaling laws, but it is a methodological reframing with immediate practical consequences: it removes the largest bottleneck — the requirement for pre-existing domain labels — that has prevented data mixture optimization from being applied to the largest, most widely used web corpora.

The reframing works at two levels. At the representation level, CLIMB demonstrates that semantic clustering over embeddings can produce domain-like structures that are functionally equivalent to (and in some ways better than) human-curated domain splits for the purpose of mixture optimization. The clusters are not merely a cheap substitute for human labels — they are an alternative organizational principle that can capture topic boundaries humans might not think to draw (e.g., Cluster 8 combining Biology, Genetics, Astronomy, and Climate Science in ways that turn out to be highly relevant for reasoning benchmarks). This makes the implicit argument that semantic proximity in embedding space is a sufficient organizing principle for pre-training data, which is a claim with implications far beyond mixture optimization — it touches on how we should think about data curation, deduplication, and quality filtering more broadly.

At the algorithmic level, CLIMB demonstrates that the one-shot regression approach embodied by RegMix leaves substantial efficiency on the table, and that replacing it with an iterative predictor-guided search — which is essentially a budgeted Bayesian optimization loop with a tree-based surrogate — recovers meaningful gains (1.04 points over RegMix on the 1B target model, Table 1) under the same proxy training budget. This is not a fundamentally new optimization algorithm (the coordinate descent structure with alternating sampling and fitting is standard in surrogate-based optimization), but it is the first clear demonstration that the structure of the search process matters for data mixture optimization — that the allocation of proxy evaluations across rounds is a design variable with measurable impact on final model quality (Table 3, Abl.allo: 60.41% for 4:2:1 vs. 60.05% for 6:1). This opens the door to a family of more sophisticated search strategies (proper Bayesian optimization with uncertainty quantification, multi-fidelity optimization, early stopping of unpromising proxy runs) that the field has not yet explored for this problem.

The paper also provides a partial reconciliation of a tension in the data selection literature. On one side, methods like DSIR and CRISP operate on the principle that data similar to the target domain is good for the target domain — select or upweight data that matches the target distribution. On the other side, practitioners have long observed that diverse data often helps in unexpected ways (code data improves reasoning, philosophy texts improve argumentation). CLIMB's cluster importance analysis (Appendix D.2) provides concrete evidence that similarity to the target is neither necessary nor sufficient: Cluster 21 shows high embedding similarity to general reasoning but receives decreasing weight through iterations because it provides limited benefit, while Cluster 8 (Biology, Genetics, Astronomy) shows low similarity but becomes increasingly important — ultimately receiving 13% of the weight in the final mixture. The paper does not resolve why this happens (it speculates about transfer effects and complementarity), but it changes the burden of proof: methods that rely purely on similarity-based selection must now explain why their approach would not miss the Cluster 8-type cases that empirically matter.

One research direction that becomes less attractive as a result of this work: developing ever-finer similarity metrics for data selection. If cosine similarity in embedding space is a weak predictor of a cluster's actual contribution to downstream performance (as Appendix D.2 shows), then improving the similarity metric — from n-grams to embeddings to fine-tuned task-specific embeddings — may hit a fundamental ceiling. The paper suggests that the relationship between data composition and model capability is mediated by complex interactions (complementarity, redundancy, transfer) that no purely observational similarity metric can capture. Empirical evaluation through proxy training — even with the computational cost that entails — may be the only reliable signal.

A research direction that becomes more attractive: applying CLIMB-style iterative search to data selection at finer granularity than whole clusters. If 21 clusters is tractable, could 100 clusters be tractable with a more sample-efficient surrogate model? Could the search operate at the per-document level with amortized inference? The paper's finding that the optimal number of clusters may be lower than the default 21 (K_enhanced = 15 achieving 60.59% vs. 21 at 60.41% in the Abl.clus ablation, though this result is not discussed in the paper) hints at a tradeoff between granularity and search efficiency that could be systematically explored.

Follow-Up Research This Work Enables

1. Proxy-to-target correlation measurement and scale-transfer diagnostics. The paper's central unvalidated assumption is that proxy model performance rankings correlate with target model performance rankings. A direct follow-up experiment would take the ~112 (mixture, proxy_score) pairs already generated during CLIMB's search and evaluate a random subset of 10–20 of those mixtures on the target model (at the 40B-token scale, costing 64,000–128,000 additional GPU-hours — expensive but feasible). The Spearman rank correlation between proxy scores and target scores would be a single number that either validates the surrogate assumption or quantifies exactly how much efficiency is lost to proxy-target mismatch. This experiment could be run across multiple proxy scales (62M, 132M, 350M, 1B) to map out how the correlation scales with proxy model size, which would directly inform practitioners about the minimum viable proxy size for their target model. If the correlation is, say, 0.65 at 62M and 0.85 at 350M, the economic calculus shifts: the 350M proxy might be worth the extra cost. If it is 0.92 at 62M and 0.94 at 350M, the smallest feasible proxy is sufficient.

2. From-scratch mixture optimization with CLIMB and cost analysis. The paper's main results are for continuous pre-training (40B tokens after a 10T-token foundation), but Section 6 shows that the optimal mixture structure differs qualitatively for from-scratch training (more balanced weights vs. sparse). A natural follow-up would run the full CLIMB pipeline in a from-scratch context: start with randomly initialized proxy models, train them on candidate mixtures for a budget-matched number of tokens (e.g., 40B from scratch for the proxy, 400B for the target), and compare the discovered mixture against the continuous-pre-training-optimized mixture when both are used to train target models from scratch. This experiment would quantify the transfer gap: how much worse is a continuous-pre-training-optimized mixture when deployed in the from-scratch setting? Given that from-scratch proxy training is more expensive (no WSD resume-from-checkpoint trick), the experiment would also establish the true cost of CLIMB for the from-scratch use case and reveal whether the method is economically viable there or whether a different search strategy (e.g., extrapolating from short training runs using scaling laws for data mixtures) is needed.

3. CLIMB applied to code generation and mathematical reasoning benchmarks. The paper's optimization target is commonsense reasoning (PIQA, ARC_E, HellaSwag), and evaluation is on reasoning tasks. Whether CLIMB's approach transfers to qualitatively different capabilities — code generation (HumanEval, MBPP) and mathematical reasoning (GSM8K, MATH) — is unknown. A direct extension would replicate the CLIMB pipeline (same clusters, same proxy models) but change the optimization target to, say, pass@1 on HumanEval for code generation. The key question is whether the iterative search discovers a qualitatively different cluster mixture for code generation (e.g., heavily upweighting Cluster 20, "Python, Code," and downweighting the biology-heavy Cluster 8) and whether the resulting model achieves domain transfer without sacrificing the original reasoning capabilities. This experiment would also test whether the 21 clusters — derived from a general web corpus — contain sufficient code-relevant data to support specialization, or whether code generation requires fundamentally different data sources beyond what the clustering pipeline captures.

4. The cluster count tradeoff: systematic Pareto analysis of granularity vs. search efficiency. The Abl.clus result (Table 3) reveals an underexplored phenomenon: K_enhanced = 15 achieves 60.59% while K_enhanced = 21 achieves 60.41%, suggesting that fewer clusters might be better, not just cheaper. A systematic study would vary K_enhanced from 5 to 50 (with a fixed total proxy budget per setting, adjusted so that the effective samples-per-dimension is controlled or varied systematically) and measure the Pareto frontier of final model performance vs. search cost. This would establish whether there is an optimal cluster granularity for a given corpus size — a measurement that would guide practitioners in choosing K_enhanced and would inform theoretical work on the bias-variance tradeoff in mixture optimization. The hypothesis: too few clusters prevents the mixture search from isolating high-value data (underfitting the data structure), while too many clusters makes the simplex too high-dimensional to search effectively with a fixed proxy budget (overfitting the search). The Pareto curve would reveal where the sweet spot lies for a given corpus.

5. Ablating the clustering pipeline components to isolate their individual contributions. The paper's clustering pipeline has four sequential components: embedding → k-means clustering → fastText quality pruning → centroid-distance merging. The contribution of each component to downstream performance is unknown. An ablation study could systematically remove or replace each stage: (a) replace k-means on embeddings with k-means on TF-IDF vectors (testing whether semantic embeddings are necessary or just helpful), (b) skip quality pruning (testing whether quality filtering matters beyond what the mixture search can compensate for by downweighting low-quality clusters), (c) use the 240 pruned clusters directly without merging (testing whether the 21-cluster merging is a necessary dimensionality reduction or whether the search could handle 240 dimensions with the same proxy budget), (d) replace centroid-distance merging with random merging of the 240 clusters into 21 groups (testing whether semantic coherence of the merged super-clusters matters or whether any partition into 21 groups would work equally well). This would transform the clustering pipeline from a monolithic "it works" into an understanding of why it works and which components are load-bearing.

6. Warm starting the iterative search with cluster-task similarity to reduce proxy evaluations. Appendix D.2 shows that embedding similarity between clusters and downstream tasks is partially informative (some high-similarity clusters are important, some are not; some low-similarity clusters become important). A follow-up would use this similarity signal to bias the initial sampling — instead of a Dirichlet distribution based on token counts (the default initialization), sample the first iteration's configurations from a distribution that upweights clusters with high task similarity. The hypothesis is that this would reduce the number of proxy evaluations needed to converge to a good mixture, since the search starts closer to the promising region. The experiment would compare CLIMB with similarity-warm-started initialization against standard CLIMB at matched proxy budgets, measuring whether the warm start provides a compute-equivalent head start. If the warm start achieves 60.41% with only, say, 80 proxy evaluations instead of 112, that is a 29% reduction in search cost with a minimal methodological change.

Practical Applications and Downstream Use Cases

Domain-specialized model production at reduced curation cost. An organization wanting to build a specialist language model for a specific domain (legal contract analysis, medical literature synthesis, financial report generation) currently has two unappealing options: (a) spend months manually curating a domain-specific pre-training corpus, or (b) fine-tune a general model on whatever in-domain data is available and hope for the best. CLIMB offers a third path: take a large general web corpus (e.g., an internal Common Crawl derivative), run the clustering pipeline once to produce ~20 semantic clusters (a one-time cost of embedding and k-means on the corpus), then run the iterative search targeting the domain's validation benchmarks using a small proxy model (~45 GPU-hours per evaluation, ~5,000 GPU-hours total for the default search budget). The resulting mixture can then be used to train the full target model. The paper shows that this process yields a 5-percentage-point improvement over random sampling for Social Sciences (Figure 5c) and a 2-percentage-point improvement over Llama-3.2-1B for general reasoning (Table 2). For a production model where a 2% accuracy gain on domain benchmarks translates to measurable business value (fewer errors in contract review, more accurate medical information retrieval), the ~5,000 GPU-hour search cost is negligible compared to the training cost of the target model (~6,400 GPU-hours for a 1B model on 40B tokens, orders of magnitude more for larger models and longer training).

Cost-efficient data mixture selection for organizations with limited compute. The paper's finding that a 62M proxy model — 5.6× smaller than the 350M default — achieves 60.11% for the 1B target model, only 0.30 points below the 350M-guided search (Table 3, Abl.proxy), is practically significant for teams with constrained computational resources. A 62M proxy model on 40B tokens of continuous pre-training can be trained on a small number of consumer GPUs or a modest cloud instance. The full search with 112 evaluations would cost approximately 112 × (45 GPU-hours / 5.6) ≈ 900 GPU-hours for the 62M proxy (assuming roughly linear cost scaling with parameters), which is feasible for an academic lab or a startup. The target model training (6,400 GPU-hours for 1B on 40B tokens) dominates the cost, making the search phase essentially free by comparison. Even if the 62M proxy finds a slightly suboptimal mixture (0.30 points below the 350M-guided optimum), the cost savings from using the smaller proxy more than compensate for the small performance gap in most practical settings. This finding democratizes data mixture optimization — it is no longer reserved for organizations that can afford to train large proxy models.

Pre-training corpus design for new data sources. When a new large-scale web corpus becomes available (a new Common Crawl snapshot, a newly licensed text collection, a domain-specific web scrape), the traditional approach to incorporating it into pre-training is manual inspection followed by ad-hoc mixing with existing data. CLIMB provides a systematic alternative: cluster the new corpus, add its clusters to the existing set (or recluster the combined corpus), and run the iterative search to discover the optimal mixture weights for the combined data pool. The paper's NEMOTRON-CLIMBLAB and NEMOTRON-CLIMBMIX releases (Section 6) demonstrate this workflow: combining Nemotron-CC with smollm-corpus, reclustering into 20 clusters, and optimizing the mixture produces a dataset that outperforms either source alone when training from scratch (Figure 1). For organizations that regularly ingest new data sources, CLIMB offers a principled way to answer "how much of the new data should we mix in, and which parts of it?" without relying on intuition or expensive trial-and-error with large models.

When to Prefer This Method

The paper positions CLIMB against named alternatives — specifically RegMix (one-shot regression), DoReMi (Group DRO-based reweighting), and manual/random mixture design — through explicit comparisons in Table 1 and the framing in Section 2.2. The tradeoffs are clear enough to articulate as a conditional preference:

Prefer CLIMB over RegMix when the proxy training budget is fixed and you want maximum downstream performance from that budget. The iterative structure recovers 1.04 additional points (1B model, Table 1) with no additional proxy evaluations — it is a purely algorithmic improvement. The cost is implementation complexity: CLIMB requires coordinating multiple rounds of sampling, predictor training, and configuration selection, while RegMix is a single round of uniform sampling followed by one regression fit with no iteration logic. For research teams with existing ML infrastructure, this complexity is minor; for quick exploratory experiments where 1 point of accuracy is not critical, RegMix's simplicity may dominate.
Prefer CLIMB over DoReMi when your downstream objective is specific task accuracy (e.g., commonsense reasoning, domain-specific MMLU scores) rather than general language modeling loss. DoReMi optimizes domain weights based on training loss dynamics, which is a proxy for — but not equivalent to — downstream task performance. CLIMB directly models the mixture → task-accuracy mapping, which is the quantity you actually care about. The evidence is in Table 1: DoReMi achieves slightly better WikiText perplexity than CLIMB for the 1B model (15.78 vs. 15.96) but trails by 1.25 points on average downstream accuracy (59.16 vs. 60.41). If perplexity is your metric of interest, DoReMi may be preferable; if task accuracy is, CLIMB is.
Prefer CLIMB over manual/random mixture design when you have an unlabeled web corpus (no domain annotations) and a clearly defined downstream validation benchmark. CLIMB's clustering pipeline automates the domain-discovery step that would otherwise require manual curation, and its iterative search discovers non-obvious cluster combinations (like the biology-heavy Cluster 8 being crucial for reasoning) that human intuition would likely miss. The 5-percentage-point gap between CLIMB and Random for Social Sciences (Figure 5c) quantifies the cost of guessing vs. searching.
Prefer CLIMB with a 62M proxy over CLIMB with a 350M proxy when computational resources are constrained and a 0.30-point accuracy gap is acceptable. Table 3 (Abl.proxy) shows the 62M proxy achieves 60.11% vs. 60.41% for the 350M proxy on the 1B target — a small enough difference that the 5.6× reduction in proxy training cost will dominate the cost-benefit calculus in most practical settings. The 62M proxy with three CLIMB iterations also outperforms a 1B proxy with a single round of random sampling (CLIMB-Best@N) on domain-specific tasks (Table 5), demonstrating that the iterative structure compensates for proxy size.
Prefer from-scratch mixture search when you are training a model from random initialization without a multi-trillion-token foundation phase. Section 6 and Figure 7 vs. Figure 8 show that the optimal mixture structure differs qualitatively between continuous pre-training (sparse weights, few clusters dominate) and from-scratch training (more balanced weights for diverse coverage). Using a continuous-pre-training-optimized mixture for from-scratch training would likely underperform — the paper does not quantify this gap, but the qualitative difference in weight distributions is a strong warning. The cost of running the search in a from-scratch context is higher (no WSD resume trick, longer proxy training runs), but the alternative of using the wrong mixture type may cost more in wasted target model training.