OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

🎯 Pitch

OPUS reveals that data selection in LLM pre-training fundamentally misaligns when it ignores the optimizer—scoring candidates in raw gradient space, as prior dynamic methods do, implicitly assumes SGD, yet modern AdamW and Muon preconditioners radically reshape effective updates. By projecting optimizer-conditioned updates onto an in-distribution target direction, OPUS achieves a 6–8× reduction in required training tokens, outperforming full-dataset training in both from-scratch and continued pre-training settings.

1. Executive Summary

This paper introduces OPUS (Optimizer-induced Projected Utility Selection), a dynamic data selection framework that scores training candidates by projecting their optimizer-conditioned effective updates onto a target direction derived from an in-distribution proxy, thereby aligning data choice with the actual update geometry induced by optimizers like AdamW and Muon rather than raw gradients. Evaluated on from-scratch pre-training of GPT-2 Large and XL on FineWeb and FineWeb-Edu (30B update tokens) and continued pre-training of Qwen3-8B-Base on SciencePedia, OPUS outperforms industrial static filters and prior dynamic selectors, achieving up to 8× reduction in computation on GPT-XL (Figure 1) and a 6× data efficiency gain in continued pre-training where OPUS with 0.5B tokens surpasses full 3B-token training. The framework incurs only 4.7% additional compute overhead through ghost gradients and CountSketch projections, while Boltzmann sampling prevents diversity collapse, establishing that optimizer-aware utility scoring improves pre-training efficiency across model scales, optimizers, and corpus quality tiers — though the strongest gains occur when the proxy direction is constructed via Bench-Proxy retrieval that aligns the selection signal with downstream benchmark distributions rather than using a standard held-out validation pool.

2. Context and Motivation

The Core Problem: Aligning Data Selection with Modern Optimization

The fundamental question this paper tackles is deceptively simple: given that LLM pre-training is shifting from "more tokens" to "better tokens," how should we select which data to train on next? This shift is driven by the impending exhaustion of high-quality public text — the so-called "Data Wall" (Villalobos et al., 2022) — which means the field can no longer simply scale data volume to improve models. Instead, every training token must count.

But the existing approaches to data selection suffer from a critical, underappreciated misalignment. Static curation methods (like FineWeb-Edu classifiers or DCLM quality filters) score documents once before training begins, assuming a sample's value is time-invariant. Dynamic methods (like GREATS) adapt during training, which is progress, but score candidates using raw gradients, implicitly assuming the model is updated via vanilla SGD. The problem is that no modern LLM is trained with SGD. Instead, practitioners use adaptive optimizers like AdamW (Loshchilov and Hutter, 2019) and increasingly Muon (Jordan et al., 2024), which apply stateful transformations — momentum, diagonal preconditioning, matrix orthogonalization — that fundamentally reshape the effective update direction.

This means that a sample whose raw gradient aligns perfectly with a validation loss improvement might actually be a poor training choice once the optimizer's preconditioner is applied. Conversely, a sample whose raw gradient looks mediocre might be highly effective through the optimizer's transformed geometry. As the paper puts it in Section 1:

"existing approaches depart from the optimizer's actual update geometry, causing unsatisfied optimization trajectory."

This is best understood through an analogy. Think of the optimizer as a lens. Raw-gradient-based selection assumes you're looking through a flat pane of glass (identity transformation). But AdamW applies coordinate-wise rescaling — some parameter dimensions get amplified, others attenuated — based on historical gradient magnitudes. Muon applies matrix orthogonalization to entire weight tensors, fundamentally mixing coordinates in ways the raw gradient doesn't capture. Scoring data through the wrong "lens" means you're optimizing a proxy objective that doesn't match what training actually does.

The paper's Figure 2 illustrates this visually: different selection methods trace different optimization trajectories. Those that score in raw gradient space deviate from the trajectory induced by modern optimizers, while OPUS aligns with it.

Why This Problem Matters: The Data Wall and Inference Efficiency

The urgency here is twofold. First, the Data Wall (detailed in Section 1) creates an economic and scientific imperative for efficient data usage. Projections suggest that readily available high-quality public text may be exhausted by 2026–2028. For organizations pre-training frontier models, this means strategies that were viable in 2020 ("just train on more tokens") will fail by 2028. The only lever left is to train on better tokens — but "better" is not a static property of text; it depends on what the model has already learned and what the optimizer's geometry makes useful. The paper frames this shift explicitly:

"In this data-wall regime, pre-training must shift from a problem of ingestion capacity to one of control: which tokens should shape the model at this specific optimizer step?"

Second, data selection has practical deployment consequences that extend beyond pre-training efficiency. The paper demonstrates in continued pre-training of Qwen3-8B-Base on SciencePedia that OPUS achieves superior performance using only 0.5B tokens compared to random selection's full 3B-token training — a 6× data efficiency gain (Section 6.4, Figure 6). For specialized domain adaptation (e.g., training a model on medical literature or legal documents), this translates directly to reduced data acquisition costs, faster training cycles, and lower compute budgets. The paper also shows that OPUS matches or exceeds 200B-token random-sampling baselines with only 30B selected tokens (Section 6.3, Table 3), representing over 6× reduction in pure data requirements.

There's also a theoretical dimension: the paper addresses a gap in our understanding of how optimization geometry interacts with data valuation. The adaptive filtering literature has developed sophisticated criteria for which samples to prioritize (influence functions, Shapley values, gradient alignment), but these criteria have almost uniformly operated in raw-gradient space. The OPUS framework introduces the concept that utility scoring should be optimizer-conditional — a sample's value is not intrinsic but depends on how the optimizer will transform its gradient into a parameter update. This has implications beyond pre-training: instruction tuning, RLHF, and continual learning all use adaptive optimizers and all involve data selection decisions.

Where Existing Approaches Fall Short

The paper identifies two categories of prior data selection methods, each with distinct limitations.

Static curation is training-agnostic. Methods like FineWeb-Edu (Penedo et al., 2024), DCLM-FastText (Li et al., 2024), QuRating (Wettig et al., 2024), DSIR (Xie et al., 2023), and UltraFineweb (Wang et al., 2025c) assign quality scores once — before any training occurs — and select a fixed subset of the corpus. These are effectively heuristic filters: they assume a sample's utility is an intrinsic, time-invariant property of its text. But consider a model early in training that needs to learn basic syntax. High-quality, syntactically simple documents (e.g., Wikipedia articles) might be maximally useful. Later, when the model has mastered syntax and needs to acquire factual knowledge, those same documents might provide diminishing returns compared to domain-specific text. A static filter cannot adjust to these shifting needs.

More subtly, static filters conflate two different notions of "quality": (a) the intrinsic properties of the text (grammatical correctness, factual accuracy, coherence) and (b) the text's utility for improving a specific model at a specific point in training. The paper's experiments on FineWeb-Edu (Section 6.3, Table 4) demonstrate this concretely: OPUS selecting dynamically from the lower-quality subset (FineWeb-Edu score 3) outperforms static baselines trained on the higher-quality subset (scores 4 and 5). For GPT-2 XL with Muon, OPUS on score-3 data achieves an average of 44.99 across benchmarks, surpassing all static baselines on score 4+5 data. This is a striking result: dynamic, optimizer-aware selection can extract more value from lower-quality text than static filtering applied to higher-quality text. It implies that the vast amount of "mid-quality" web text that static filters discard may contain significant untapped training signal — but only if you know when and how to use it.

Dynamic selection ignores optimizer geometry. Prior dynamic methods — notably GREATS (Wang et al., 2024) and perplexity-based approaches (Ankner et al., 2025) — score samples based on their immediate impact on validation loss, but compute this impact using raw-gradient inner products. GREATS, for example, uses a Taylor expansion to estimate how much adding a candidate sample reduces validation loss, and the resulting utility score is:

$U(z) \approx \langle \nabla \mathcal{L}(z; \theta_t), \nabla \mathcal{L}(\mathcal{D}_{\text{val}}; \theta_t) \rangle$

This is an inner product between the candidate's raw gradient and the validation gradient. But when the optimizer applies a preconditioner $\mathbf{P}_t$ to both gradients before updating parameters, the actual expected loss reduction is proportional to:

$\langle \mathbf{P}_t \nabla \mathcal{L}(z; \theta_t), \mathbf{P}_t \nabla \mathcal{L}(\mathcal{D}_{\text{val}}; \theta_t) \rangle$

These two inner products can produce very different rankings of candidates because the preconditioner's spectral structure amplifies some directions and suppresses others. For AdamW, $\mathbf{P}_t$ is diagonal with entries inversely proportional to the square root of historical gradient variance — parameters that have been updated with high-variance gradients receive smaller effective learning rates. A candidate whose raw gradient points strongly in such a direction would score highly in raw-gradient space but deliver a small effective update. For Muon, $\mathbf{P}_t$ is a dense, non-diagonal matrix derived from Newton-Schulz orthogonalization, which introduces much richer coordinate mixing. The paper shows this empirically (Section 6.3, Table 3): under the same Muon optimizer, GREATS (which uses raw gradients) achieves an average score of 39.41 on GPT-2 Large, only marginally better than random selection at 39.72, while OPUS (which uses optimizer-preconditioned updates) reaches 40.25. The gap widens on GPT-2 XL: GREATS drops to 39.23 vs. random's 40.29, while OPUS reaches 41.75.

The paper also identifies a subtler failure mode of prior dynamic methods: greedy top- $k$ selection with static proxy signals. GREATS (and related methods like MATES/Group-MATES) typically select the $K$ candidates with highest utility scores each step. This can lead to redundancy: the second-best candidate might be nearly identical to the best candidate in how it moves parameters, providing diminishing marginal utility. Worse, the proxy direction (computed from a held-out validation set) is a noisy estimate of the true target direction, and greedy top- $k$ can lock onto transient features of that noisy estimate. The paper addresses this with Boltzmann sampling (Section 5.3), which preserves diversity by sampling candidates with probability proportional to $\exp(U(z)/\tau)$ rather than taking the deterministic top- $K$ , and with a redundancy penalty term that explicitly subtracts the overlap between a candidate's update and already-selected updates (Equation 22).

How This Paper Positions Itself

The paper's central positioning argument is that optimizer awareness is the missing dimension in data selection. This is articulated through three requirements that a principled dynamic selection method must satisfy (Section 5):

Principled: scores are derived from an explicit objective that measures improvement on a held-out proxy distribution under the optimizer-induced update geometry.
Efficient: scoring avoids materializing per-sample gradients in high-dimensional space (the per-sample gradient for GPT-2 XL's largest layer would be a ~10M-dimensional vector; computing this for every candidate would be prohibitive).
Scalable: overhead remains modest as model dimension grows, enabling selection at every training step rather than intermittently.

The paper situates OPUS within the lineage of dynamic selection methods but positions its contribution as a qualitative shift from raw-gradient scoring to optimizer-preconditioned scoring. Rather than proposing yet another influence approximation or a new proxy construction technique (though it does introduce Bench-Proxy for proxy construction), the key insight is that the scoring function itself must be rederived from first principles under the optimizer's actual update rule. The paper formalizes this by deriving closed-form linearizations of AdamW and Muon's one-step lookahead updates (Section 4), producing an optimizer-induced preconditioner $\mathbf{P}_t$ that captures the effective update geometry at each step. These derivations are not merely descriptive — they are the mechanism by which OPUS corrects the misalignment that Figure 2 illustrates.

Methodologically, OPUS also distinguishes itself from GREATS by how it handles the proxy target direction. GREATS uses a held-out validation set directly, computing $\nabla \mathcal{L}(\mathcal{D}_{\text{val}}; \theta_t)$ at each step. This is computationally expensive and introduces noise because the validation distribution may differ from the downstream evaluation distribution. OPUS introduces Bench-Proxy (Section 5.1, detailed in Section 6.2): rather than using the validation set directly, it uses a frozen sentence embedding model to retrieve pre-training documents that are semantically similar to the target benchmark, forming an in-distribution proxy pool $\mathcal{D}_{\text{proxy}}$ that is both task-aligned and within the pre-training manifold. This is a subtle but important refinement: gradients computed on out-of-distribution validation data can be noisy or misleading, while in-distribution proxy gradients better represent the actual improvement the model would make on tasks of interest.

The paper also positions itself explicitly in the compute-efficiency conversation. Section 6.6 (Figure 7) shows that a naïve implementation of online gradient-based selection would incur over 3.5× slowdown compared to random training. By combining ghost gradients (which exploit the rank-1 outer product structure of per-sample gradients in linear layers, avoiding materialization) with CountSketch projections (which compress high-dimensional gradient vectors into low-dimensional sketch vectors for efficient inner products), OPUS reduces this overhead to only 4.7%. This is not just an implementation detail — it determines whether the method is practical. If dynamic selection added 3.5× wall-clock time, practitioners would simply train on more tokens instead. The 4.7% overhead makes OPUS viable for deployment in standard pre-training pipelines.

Finally, the paper acknowledges its relationship to prior work on influence functions and data attribution (Section 2) while drawing a clear boundary. Classical influence functions (Koh and Liang, 2017) estimate the effect of removing a training point on validation loss but require Hessian-vector products that are intractable at LLM scale. Approximations like TracIn (Pruthi et al., 2020) use gradient inner products for efficiency but operate in raw-gradient space. OPUS inherits the practitioner's goal from this line of work — approximate the utility of a training point efficiently — but rederives the approximation under the optimizer's effective update geometry rather than assuming SGD dynamics. This connects the data selection community (which has developed sophisticated batch selection algorithms) with the optimization community (which has developed adaptive preconditioners), bridging a gap that has until now been largely ignored.

3. Technical Approach

3.1 Reader Orientation

OPUS is a dynamic data selection system that runs alongside LLM pre-training and, at every optimizer step, chooses which subset of the available training data will be most valuable for updating the model. It solves the problem that existing data selection methods score candidates using raw gradients, which is misaligned with how modern optimizers like AdamW and Muon actually update parameters — OPUS instead scores candidates in the optimizer-induced update space, projecting their effective (preconditioned) updates onto a target direction that represents improvement on a high-quality proxy set, then selects a diverse batch through a stochastic process that balances utility with redundancy reduction.

3.2 Big-Picture Architecture (Diagram in Words)

OPUS operates as a per-step pipeline embedded in the training loop. The architecture has five major components:

Candidate Buffer — a pool of $N$ sequences drawn from the pre-training stream $\mathcal{D}_{\text{tr}}$ at each step $t$ , from which OPUS will select $K = \lfloor \rho N \rfloor$ sequences to form the actual update batch.
Optimizer-Induced Preconditioner $\mathbf{P}_t$ — derived from the current optimizer state (momentum buffers, second-moment estimates, Newton-Schulz matrices), this operator captures how the optimizer transforms raw gradients into effective parameter updates. It is constructed once per step before scoring begins, held fixed during selection, and differs fundamentally between AdamW (diagonal coordinate-wise rescaling) and Muon (dense matrix-based transformation).
Proxy Pool and Proxy Gradient Direction — a frozen set of documents $\mathcal{D}_{\text{proxy}}$ constructed via Bench-Proxy retrieval (semantically similar to downstream benchmarks but drawn from the pre-training corpus). At each step, a mini-batch from this pool provides the target gradient direction $\mathbf{g}_{\text{proxy}}(t)$ that OPUS aims to align with.
Scalable Utility Estimator — for each candidate $z$ , OPUS computes a scalar utility score $U_z(t)$ that estimates how much adding $z$ to the selected batch would reduce loss on the proxy distribution, measured in the optimizer-induced geometry. This estimation uses ghost gradients (exploiting the rank-1 outer product structure of per-sample gradients in linear layers) and CountSketch projections (compressing high-dimensional effective updates into low-dimensional sketch vectors for efficient inner products), avoiding materialization of full per-sample gradients.
Boltzmann Sampler with Redundancy Penalty — rather than deterministically selecting the top- $K$ candidates by utility (which can produce redundant or brittle selections), OPUS samples candidates sequentially with probability proportional to $\exp(U_z(t)/\tau)$ , where the utility includes an explicit penalty term that subtracts overlap between a candidate's update and the updates of already-selected samples.

Information flows as follows: at each training step $t$ , the optimizer state produces $\mathbf{P}_t$ → a proxy mini-batch produces the target sketch $\psi_{\text{proxy}}(t)$ → each candidate $z$ in the buffer passes through a forward/backward pass to extract ghost factors $\{\mathbf{a}_r(z), \mathbf{b}_r(z)\}$ for a set of layers $\mathcal{R}$ → these factors are preconditioned by $\mathbf{P}_t$ and projected via CountSketch into sketch features $\phi_{(t,r)}(z)$ → utility scores are computed via sketch-space dot products against the proxy direction, with a running redundancy penalty from previously selected sketches → Boltzmann sampling selects $K$ samples → the model updates on the selected subset $\hat{\mathcal{B}}_t$ .

3.3 Roadmap for the Deep Dive

First, the formal optimizer-induced utility objective (Equation 14 → 23), which defines what "valuable" means in terms of one-step proxy loss reduction under the optimizer's actual update geometry — this is the mathematical foundation for everything else.
Second, the optimizer preconditioner derivations for AdamW and Muon (Section 4 material), since these are the mechanisms by which OPUS transforms raw gradients into the effective update space where scoring happens.
Third, the Bench-Proxy construction procedure (Section 5.1 + 6.2), since the quality of the proxy gradient direction determines what kind of improvement OPUS optimizes for.
Fourth, the scalable utility estimation machinery — ghost gradients and CountSketch projections (Section 5.2) — since these are what make scoring tractable at LLM scale without per-sample gradient materialization.
Fifth, the Boltzmann sampling mechanism and the redundancy penalty (Section 5.3), since these convert utility scores into a final batch selection while preserving diversity.
Sixth, the end-to-end algorithm pseudocode walkthrough (Algorithm 1), which ties all components together into the per-step workflow.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper whose core idea is that data utility scores should be computed in the optimizer-induced update space rather than raw-gradient space, and that this principled shift — combined with efficient sketching and stochastic selection — enables practical, scalable dynamic data selection that outperforms both static curation and raw-gradient-based dynamic methods.

The Optimizer-Induced Utility Objective

OPUS defines the value of a candidate batch through a first-order approximation of validation loss reduction, but crucially computes this reduction using the optimizer's effective update direction rather than raw gradients. The derivation proceeds in stages, starting from a general definition and progressively incorporating approximations that make the objective tractable at LLM scale.

Step 1: One-step utility definition. For any candidate subset $\mathcal{S}$ , OPUS defines utility at step $t$ as the reduction in loss on a held-out validation set $\mathcal{D}_{\text{val}}$ after applying one optimizer step induced by $\mathcal{S}$ :

$U^{(t)}(\mathcal{S}) := \mathcal{L}(\mathcal{D}_{\text{val}}; \theta_t) - \mathcal{L}(\mathcal{D}_{\text{val}}; \theta_{t+1}(\mathcal{S}))$

where $\mathcal{L}(\mathcal{D}; \theta)$ is the expected negative log-likelihood over distribution $\mathcal{D}$ under parameters $\theta$ , $\theta_t$ are the current parameters, and $\theta_{t+1}(\mathcal{S}) = \theta_t + \Delta\theta_t(\mathcal{S})$ are the parameters after one optimizer step on subset $\mathcal{S}$ with effective update $\Delta\theta_t(\mathcal{S})$ .

What it computes: the raw improvement in the model's ability to predict tokens from the validation set after training on $\mathcal{S}$ for one step. A positive value means the model got better at the validation task.

Why this form: this is the most direct measure of a batch's training value — does it move the model in a direction that helps on held-out data? Alternative definitions based on gradient norm or loss magnitude don't directly measure generalization improvement. The one-step horizon is chosen both for computational tractability (multi-step lookahead requires simulating future updates) and because re-evaluating utility at every step with fresh optimizer state naturally adapts to changing training dynamics.

Step 2: Marginal gain for a single candidate. Since OPUS constructs the update batch incrementally, it needs the marginal utility of adding one candidate $z$ to an already-selected subset $\hat{\mathcal{B}}_t$ :

$U_z(t) := U^{(t)}(\hat{\mathcal{B}}_t \cup \{z\}) - U^{(t)}(\hat{\mathcal{B}}_t)$

Let $\tilde{\theta}_t(\hat{\mathcal{B}}_t) = \theta_t + \Delta\theta_t(\hat{\mathcal{B}}_t)$ denote the virtual parameters after applying the selected subset's update. Adding $z$ induces an additional update $\Delta\theta_t(\{z\})$ , so:

$U_z(t) = \mathcal{L}(\mathcal{D}_{\text{val}}; \tilde{\theta}_t(\hat{\mathcal{B}}_t)) - \mathcal{L}(\mathcal{D}_{\text{val}}; \tilde{\theta}_t(\hat{\mathcal{B}}_t) + \Delta\theta_t(\{z\}))$

What it computes: the additional loss reduction on the validation set obtained by also training on $z$ , beyond what was already obtained from the previously selected candidates. This is a classic marginal gain — it captures the incremental value of $z$ , accounting for the fact that some of $z$ 's benefit may overlap with (be redundant with) samples already selected.

Why this form: naively scoring each candidate independently (without considering what's already selected) would produce redundant batches where the second, third, and $K$ -th best candidates provide diminishing marginal returns because their gradient directions largely overlap. The marginal formulation explicitly models this interaction. It is conceptually equivalent to greedy subset selection under a submodular objective, where each addition provides the maximum remaining benefit given previous choices.

Step 3: First-order Taylor approximation. Computing the exact loss at $\tilde{\theta}_t(\hat{\mathcal{B}}_t) + \Delta\theta_t(\{z\})$ requires a forward pass through the entire validation set at a new parameter configuration, which is prohibitively expensive. OPUS linearizes the validation loss around the current virtual parameters:

$\mathcal{L}(\mathcal{D}_{\text{val}}; \tilde{\theta}_t(\hat{\mathcal{B}}_t) + \Delta\theta_t(\{z\})) \approx \mathcal{L}(\mathcal{D}_{\text{val}}; \tilde{\theta}_t(\hat{\mathcal{B}}_t)) + \nabla_\theta \mathcal{L}(\mathcal{D}_{\text{val}}; \tilde{\theta}_t(\hat{\mathcal{B}}_t))^\top \Delta\theta_t(\{z\})$

Substituting this into the marginal gain expression:

$U_z(t) \approx -\nabla_\theta \mathcal{L}(\mathcal{D}_{\text{val}}; \tilde{\theta}_t(\hat{\mathcal{B}}_t))^\top \Delta\theta_t(\{z\})$

What it computes: the negative inner product between the validation gradient at the virtual parameters and the parameter update induced by candidate $z$ . If the update direction $\Delta\theta_t(\{z\})$ is aligned with the negative validation gradient (the direction of steepest validation loss decrease), the inner product is large and positive, meaning $z$ is valuable. If the update is orthogonal or opposed to the validation gradient, the utility is small or negative.

Why this form: the first-order Taylor expansion is the standard approximation for local sensitivity in optimization — it's accurate for small step sizes (which is the operating regime for LLM training with well-tuned learning rates) and transforms an expensive "evaluate-at-new-parameters" operation into a cheap "compute-gradient-dot-product" operation. The approximation error is $O(\|\Delta\theta_t(\{z\})\|^2)$ , which is controlled by the learning rate.

Step 4: Substituting the optimizer-induced effective update. This is where OPUS departs from prior work. Rather than using the raw gradient $\nabla_\theta \mathcal{L}(z; \theta_t)$ as the update direction, OPUS uses the optimizer-preconditioned effective update:

$\mathbf{u}_z(t) := \mathbf{P}_t \nabla_\theta \mathcal{L}(z; \theta_t)$

where $\mathbf{P}_t$ is the optimizer-induced preconditioner at step $t$ , capturing momentum, adaptive learning rates, and (for Muon) matrix orthogonalization. The parameter update from a single candidate is then $\Delta\theta_t(\{z\}) = -\eta_t \mathbf{u}_z(t)$ , where $\eta_t$ is the learning rate. Substituting:

$U_z(t) \approx \eta_t \langle \mathbf{u}_z(t), \nabla_\theta \mathcal{L}(\mathcal{D}_{\text{val}}; \tilde{\theta}_t(\hat{\mathcal{B}}_t)) \rangle$

What it computes: the alignment (scaled by learning rate) between the optimizer-preconditioned update of candidate $z$ and the validation gradient at the virtual parameters. This is fundamentally different from the raw-gradient alignment $\langle \nabla\mathcal{L}(z), \nabla\mathcal{L}(\mathcal{D}_{\text{val}}) \rangle$ used by GREATS — OPUS scores how the model will actually move, not how the raw gradient points.

Why this form: the preconditioner $\mathbf{P}_t$ can dramatically reshape the effective update direction. For AdamW, parameters with historically high-variance gradients receive smaller effective updates (the diagonal preconditioner attenuates them). A candidate whose raw gradient points strongly in such directions would score highly under raw-gradient alignment but produce a small effective update — OPUS correctly down-weights it. For Muon, the Newton-Schulz orthogonalization completely restructures the update matrix, and a raw-gradient alignment would be essentially random relative to the actual parameter movement.

Step 5: Approximating the virtual validation gradient. The expression still requires $\nabla_\theta \mathcal{L}(\mathcal{D}_{\text{val}}; \tilde{\theta}_t(\hat{\mathcal{B}}_t))$ — the validation gradient evaluated at the virtual parameters after the already-selected subset's update. Computing this after every selection step would require an additional backward pass on $\mathcal{D}_{\text{val}}$ . OPUS linearizes around the current parameters $\theta_t$ instead:

$\nabla_\theta \mathcal{L}(\mathcal{D}_{\text{val}}; \tilde{\theta}_t(\hat{\mathcal{B}}_t)) \approx \mathbf{g}_{\text{val}}(t) + \mathbf{H}_{\text{val}}(t) \Delta\theta_t(\hat{\mathcal{B}}_t)$

where $\mathbf{g}_{\text{val}}(t) := \nabla_\theta \mathcal{L}(\mathcal{D}_{\text{val}}; \theta_t)$ is the validation gradient at the current parameters, and $\mathbf{H}_{\text{val}}(t) := \nabla_\theta^2 \mathcal{L}(\mathcal{D}_{\text{val}}; \theta_t)$ is the Hessian of the validation loss.

Since the accumulated update from the selected subset is $\Delta\theta_t(\hat{\mathcal{B}}_t) = -\eta_t \sum_{z_j \in \hat{\mathcal{B}}_t} \mathbf{u}_{z_j}(t)$ , this expands to:

$U_z(t) \approx \eta_t \langle \mathbf{u}_z(t), \mathbf{g}_{\text{val}}(t) \rangle - \eta_t^2 \langle \mathbf{u}_z(t), \mathbf{H}_{\text{val}}(t) \sum_{z_j \in \hat{\mathcal{B}}_t} \mathbf{u}_{z_j}(t) \rangle$

What it computes: a two-term decomposition of the marginal utility. The first term (the "Alignment" term) measures how well $z$ 's effective update aligns with the current validation gradient — if $z$ moves parameters in a direction that directly reduces validation loss, this term is positive. The second term (the "Redundancy Penalty") subtracts the overlap between $z$ 's update and the Hessian-weighted sum of already-selected updates — if $z$ 's update is similar to what the batch already provides (as measured through the lens of the validation loss curvature), this penalty reduces its score, discouraging redundant selections.

Why this form: the Hessian term is the key mechanism for avoiding redundancy. Without it, the marginal gain would simply be proportional to alignment with $\mathbf{g}_{\text{val}}(t)$ , which is independent of what has already been selected. Adding the Hessian term means that a candidate whose effective update lies in the same Hessian-eigenspace as already-selected samples gets penalized, while a candidate probing an unexplored direction (low Hessian overlap) retains its full alignment score. This is the batch-level equivalent of avoiding duplicate gradient directions.

Step 6: Isotropic Hessian approximation. Materializing $\mathbf{H}_{\text{val}}$ is completely intractable at LLM scale (it's a $d \times d$ matrix where $d$ is hundreds of millions to billions). Following GREATS, OPUS adopts the simplest possible approximation: $\mathbf{H}_{\text{val}} \approx \mathbf{I}$ , the identity matrix. This yields the practical scoring rule:

$U_z(t) \approx \eta_t \langle \mathbf{u}_z(t), \mathbf{g}_{\text{val}}(t) \rangle - \eta_t^2 \langle \mathbf{u}_z(t), \mathbf{G}(t) \rangle$

where $\mathbf{G}(t) := \sum_{z_j \in \hat{\mathcal{B}}_t} \mathbf{u}_{z_j}(t)$ is the accumulated effective direction from already-selected samples.

What it computes: the alignment of $z$ 's effective update with the validation gradient, minus a penalty proportional to the alignment with the sum of already-selected effective updates. The penalty term is essentially a dot-product between the candidate's update and the batch's aggregate update — candidates parallel to the batch are penalized more than orthogonal candidates.

Why this form: the isotropic Hessian approximation is a strong simplifying assumption — it treats all directions in parameter space as having equal curvature, which is certainly false. However, the paper argues (following GREATS) that this approximation has empirical merit for two reasons: (1) it correctly captures the sign of the redundancy effect (adding a sample similar to existing ones should provide less marginal benefit), even if it gets the magnitude wrong in highly curved directions; and (2) the alternative of using a diagonal Hessian approximation (e.g., Fisher information) would require storing per-parameter curvature estimates and would interact problematically with the AdamW preconditioner (which already performs diagonal rescaling). The identity approximation can be seen as applying the redundancy penalty in the optimizer's own geometry — which is precisely the space where the effective updates live.

Step 7: Proxy gradient substitution with Bench-Proxy. The validation gradient $\mathbf{g}_{\text{val}}(t)$ is replaced by a proxy gradient $\mathbf{g}_{\text{proxy}}(t)$ computed from a mini-batch of $K_{\text{proxy}}$ samples drawn from the Bench-Proxy pool $\mathcal{D}_{\text{proxy}}$ :

$\mathbf{g}_{\text{proxy}}(t) = \frac{1}{K_{\text{proxy}}} \sum_{k=1}^{K_{\text{proxy}}} \nabla_\theta \mathcal{L}(\tilde{z}_k; \theta_t)$

The final scoring rule becomes:

$U_z(t) \leftarrow \eta_t \langle \mathbf{u}_z(t), \mathbf{g}_{\text{proxy}}(t) \rangle - \eta_t^2 \langle \mathbf{u}_z(t), \mathbf{G}(t) \rangle$

What it computes: the same alignment-minus-redundancy score, but with the target direction coming from an in-distribution proxy set rather than the raw validation set. The proxy is constructed to be both task-aligned (semantically similar to downstream benchmarks) and in-distribution (drawn from the pre-training corpus), avoiding the distribution shift that can make raw validation gradients noisy or misleading.

Why this form: the choice of proxy direction is critical. Using the raw validation set (as GREATS does) has two problems: (1) the validation distribution may differ from the downstream evaluation distribution, so reducing validation loss may not translate to benchmark improvement; (2) computing the validation gradient at every step is expensive and introduces noise from stochastic mini-batching. Bench-Proxy addresses both: by retrieving pre-training documents similar to the benchmark, it creates a proxy that is aligned with the target task while remaining within the model's training distribution, producing more stable and task-relevant gradient signals. The paper demonstrates empirically (Table 7) that the benchmark-matched proxy improves average accuracy from 41.03 to 41.75 compared to a standard proxy.

Optimizer Preconditioner Derivations

The utility objective in Equation 23 depends on the effective update $\mathbf{u}_z(t) = \mathbf{P}_t \nabla_\theta \mathcal{L}(z; \theta_t)$ . The form of $\mathbf{P}_t$ differs fundamentally between optimizers, and OPUS derives closed-form approximations for both AdamW and Muon by linearizing their one-step lookahead updates at fixed optimizer state.

AdamW Preconditioner

AdamW maintains two exponential moving averages: $\mathbf{m}_t$ (first moment, tracking gradient direction with momentum) and $\mathbf{v}_t$ (second moment, tracking element-wise gradient variance). The standard update for a batch $\mathcal{S}$ at step $t$ is:

$\mathbf{m}_t(\mathcal{S}) = \beta_1 \mathbf{m}_{t-1} + (1 - \beta_1) \mathbf{g}_t(\mathcal{S})$

$\mathbf{v}_t(\mathcal{S}) = \beta_2 \mathbf{v}_{t-1} + (1 - \beta_2) \mathbf{g}_t(\mathcal{S})^{\odot 2}$

where $\mathbf{g}_t(\mathcal{S}) := \frac{1}{|\mathcal{S}|} \sum_{z \in \mathcal{S}} \nabla_\theta \mathcal{L}(z; \theta_t)$ is the average gradient over subset $\mathcal{S}$ , and $\odot 2$ denotes element-wise squaring. After bias correction ( $\hat{\mathbf{m}}_t = \mathbf{m}_t / (1 - \beta_1^t)$ , $\hat{\mathbf{v}}_t = \mathbf{v}_t / (1 - \beta_2^t)$ ) and decoupled weight decay, the parameter update is:

$\theta_{t+1}(\mathcal{S}) = \theta_t - \alpha_t \frac{\hat{\mathbf{m}}_t(\mathcal{S})}{\sqrt{\hat{\mathbf{v}}_t(\mathcal{S})} + \epsilon} - \alpha_t \lambda \theta_t$

OPUS's linearized view. At a fixed step $t$ , OPUS freezes the optimizer state — $\alpha_t$ , $\beta_1$ , $\beta_2$ , $\epsilon$ , $\lambda$ , and the history buffers $\mathbf{m}_{t-1}$ and $\mathbf{v}_{t-1}$ are treated as constants with respect to $\mathcal{S}$ . The bias-corrected first moment is then an affine function of the batch gradient:

$\hat{\mathbf{m}}_t(\mathcal{S}) = \frac{\beta_1}{1 - \beta_1^t} \mathbf{m}_{t-1} + \frac{1 - \beta_1}{1 - \beta_1^t} \mathbf{g}_t(\mathcal{S})$

The first term is independent of $\mathcal{S}$ and contributes a constant offset. The second term is linear in $\mathbf{g}_t(\mathcal{S})$ .

Freezing the RMS geometry. The denominator $\sqrt{\hat{\mathbf{v}}_t(\mathcal{S})} + \epsilon$ depends on $\mathcal{S}$ through $\mathbf{v}_t(\mathcal{S}) = \beta_2 \mathbf{v}_{t-1} + (1 - \beta_2) \mathbf{g}_t(\mathcal{S})^{\odot 2}$ . Evaluating this for every candidate subset would require recomputing the element-wise square and square root, which interacts nonlinearly with the gradient. OPUS approximates this by dropping the $\mathcal{S}$ -dependence entirely, using only the historical component:

$\hat{\mathbf{v}}_t(\mathcal{S}) + \epsilon \approx \bar{\mathbf{v}}_t + \epsilon, \quad \bar{\mathbf{v}}_t := \frac{\beta_2 \mathbf{v}_{t-1}}{1 - \beta_2^t}$

This means the per-coordinate learning rate scaling (the adaptive part of AdamW) is frozen during selection — it uses whatever scaling was computed from previous steps' gradient history, not the current candidates' gradients. This is an approximation, but the paper argues it's necessary for tractable scoring and justified because the second-moment estimate changes slowly (controlled by $\beta_2 = 0.95$ ).

Resulting linearized update. Substituting these approximations and isolating the $\mathcal{S}$ -dependent terms:

$\Delta\theta_t(\mathcal{S}) := \theta_{t+1}(\mathcal{S}) - \theta_t \approx -A_t \mathbf{D}_t \mathbf{m}_{t-1} - \alpha_t \lambda \theta_t - C_t \mathbf{D}_t \mathbf{g}_t(\mathcal{S})$

where $\mathbf{D}_t := \text{Diag}(1 / \sqrt{\bar{\mathbf{v}}_t + \epsilon})$ is a diagonal matrix of per-parameter inverse RMS scaling factors, $A_t := \alpha_t \beta_1 / (1 - \beta_1^t)$ , and $C_t := \alpha_t (1 - \beta_1) / (1 - \beta_1^t)$ .

The first two terms ( $-A_t \mathbf{D}_t \mathbf{m}_{t-1}$ and $-\alpha_t \lambda \theta_t$ ) are independent of $\mathcal{S}$ — they represent the momentum-driven and weight-decay components of the update that happen regardless of which batch is selected. Since OPUS ranks candidates by relative utility, these constant offsets do not affect the ranking and can be dropped. The data-dependent component is:

$\Delta\theta_t(\mathcal{S}) \approx -\mathbf{P}_t^{\text{AdamW}} \mathbf{g}_t(\mathcal{S}) + \text{const}, \quad \mathbf{P}_t^{\text{AdamW}} := C_t \text{Diag}\left(\frac{1}{\sqrt{\bar{\mathbf{v}}_t + \epsilon}}\right)$

What it computes: the AdamW preconditioner is a diagonal matrix that rescales each coordinate of the gradient by $C_t / \sqrt{\bar{v}_{t,i} + \epsilon}$ , where $\bar{v}_{t,i}$ is the historical second-moment estimate for parameter $i$ . Coordinates that have historically received large gradients (high $\bar{v}_{t,i}$ ) get attenuated — they receive smaller effective learning rates because the optimizer "trusts" their direction less, having seen high variance. Coordinates with small historical gradients get amplified, receiving larger effective steps.

Why this form: the diagonal structure is computationally crucial for OPUS's efficiency. When the preconditioner is diagonal (as with AdamW), it preserves the coordinate-wise separable structure of the per-sample gradient's outer-product form in linear layers. Specifically, for a linear layer with weights $\mathbf{W}_r \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$ , the per-sample gradient factorizes as $\nabla_{\mathbf{W}_r} \mathcal{L}(z; \theta_t) = \mathbf{a}_r(z) \otimes \mathbf{b}_r(z)$ , where $\mathbf{a}_r(z) \in \mathbb{R}^{d_{\text{in}}}$ is the input activation and $\mathbf{b}_r(z) \in \mathbb{R}^{d_{\text{out}}}$ is the output gradient. Applying a diagonal preconditioner $\mathbf{P}_t^{\text{AdamW}}$ to this outer product simply scales rows and columns independently — it multiplies each element $(i,j)$ of the outer product by a factor depending on $i$ and $j$ — which can be done without ever materializing the full $d_{\text{in}} \times d_{\text{out}}$ matrix. This enables the CountSketch projection to operate at cost $\mathcal{O}(d_{\text{in}} + d_{\text{out}})$ rather than $\mathcal{O}(d_{\text{in}} d_{\text{out}})$ .

The paper uses standard AdamW hyperparameters: $\beta_1 = 0.8$ , $\beta_2 = 0.95$ , $\epsilon = 10^{-8}$ , weight decay $\lambda = 0$ (for GPT-2 experiments; $\lambda = 0.01$ for Qwen3-8B continued pre-training). The momentum coefficient $\beta_1 = 0.8$ is lower than the typical default of $0.9$ — this reduces the influence of historical momentum, making the update more responsive to recent gradients, which may be beneficial when the data distribution changes rapidly under dynamic selection.

Muon Preconditioner

Muon is fundamentally different from AdamW: it targets matrix-shaped parameters, applies momentum to accumulate a direction, and then orthogonalizes the accumulated matrix using Newton-Schulz iterations (an approximation to the matrix sign function, related to SVD-based orthogonalization). For a linear weight matrix $\mathbf{W}_{\ell} \in \mathbb{R}^{o \times i}$ , Muon maintains an EMA momentum on the mini-batch gradient $\mathbf{g}_{t,\ell}(\mathcal{S}) := \frac{1}{|\mathcal{S}|} \sum_{z \in \mathcal{S}} \nabla_{\mathbf{W}_{\ell}} \mathcal{L}(z; \theta_t)$ :

$\mathbf{m}_{t+1,\ell}(\mathcal{S}) = \mu \mathbf{m}_{t,\ell} + (1 - \mu) \mathbf{g}_{t,\ell}(\mathcal{S})$

Muon then forms a "double-smoothed" direction $\mathbf{q}_{t+1,\ell}(\mathcal{S})$ that the paper derives as:

$\mathbf{q}_{t+1,\ell}(\mathcal{S}) := (1 - \mu) \mathbf{g}_{t,\ell}(\mathcal{S}) + \mu \mathbf{m}_{t+1,\ell}(\mathcal{S}) = \mu^2 \mathbf{m}_{t,\ell}(\mathcal{S}) + (1 - \mu^2) \mathbf{g}_{t,\ell}(\mathcal{S})$

This double-smoothed direction is fed into the Newton-Schulz orthogonalizer $\mathcal{O}_{t,\ell}(\cdot)$ , which produces an approximately orthogonal matrix (specifically, an approximation to the matrix zeroth power, which maps singular values to $\pm 1$ while preserving singular vectors). The parameter update is then:

$\Delta \mathbf{W}_{t,\ell}(\mathcal{S}) := \mathbf{W}_{t+1,\ell}(\mathcal{S}) - \mathbf{W}_{t,\ell} = -\eta_t \mathcal{O}_{t,\ell}(\mathbf{q}_{t+1,\ell}(\mathcal{S}))$

OPUS's linearized view. The Newton-Schulz operator is nonlinear (it involves matrix multiplications and polynomial expansions), making direct scoring intractable. OPUS linearizes it by constructing $\mathcal{O}_{t,\ell}$ from a reference direction $\bar{\mathbf{q}}_{t,\ell}$ available at the start of step $t$ (from the current optimizer buffer or a proxy batch) and reusing this frozen operator for all candidates. Under this approximation, Newton-Schulz induces an approximately linear left-multiplication map:

$\mathcal{O}_{t,\ell}(\mathbf{Z}) \approx \mathbf{S}_{t,\ell} \mathbf{Z}, \quad \mathbf{S}_{t,\ell} = a \mathbf{I} + b \mathbf{A}_{t,\ell} + c \mathbf{A}_{t,\ell}^2$

where $\mathbf{A}_{t,\ell} := \tilde{\mathbf{q}}_{t,\ell} \tilde{\mathbf{q}}_{t,\ell}^\top$ with $\tilde{\mathbf{q}}_{t,\ell} := \bar{\mathbf{q}}_{t,\ell} / \|\bar{\mathbf{q}}_{t,\ell}\|_F$ , and $a, b, c$ are fixed coefficients of the Newton-Schulz polynomial.

Substituting the double-smoothed direction into the update and using the linearized $\mathcal{O}_{t,\ell}$ :

$\Delta \mathbf{W}_{t,\ell}(\mathcal{S}) \approx -\eta_t \mathbf{S}_{t,\ell} \left(\mu^2 \mathbf{m}_{t,\ell} + (1 - \mu^2) \mathbf{g}_{t,\ell}(\mathcal{S})\right)$

$= -\eta_t \mu^2 \mathbf{S}_{t,\ell} \mathbf{m}_{t,\ell} - \eta_t (1 - \mu^2) \mathbf{S}_{t,\ell} \mathbf{g}_{t,\ell}(\mathcal{S})$

As with AdamW, the first term ( $-\eta_t \mu^2 \mathbf{S}_{t,\ell} \mathbf{m}_{t,\ell}$ ) is independent of $\mathcal{S}$ and can be dropped for scoring. The data-dependent term is:

$\Delta \mathbf{W}_{t,\ell}(\mathcal{S}) \approx -\mathbf{P}_{t,\ell}^{\text{Muon}} \mathbf{g}_{t,\ell}(\mathcal{S}) + \text{const}, \quad \mathbf{P}_{t,\ell}^{\text{Muon}} := \eta_t (1 - \mu^2) \mathbf{S}_{t,\ell}$

The overall Muon preconditioner is block-diagonal across layers, with each layer $\ell$ receiving its own dense preconditioner $\mathbf{P}_{t,\ell}^{\text{Muon}} \in \mathbb{R}^{o \times i}$ that left-multiplies the gradient matrix.

What it computes: the Muon preconditioner is a dense matrix that reshapes the gradient by (1) scaling by $\eta_t (1 - \mu^2)$ (effective learning rate with momentum coupling), and (2) left-multiplying by $\mathbf{S}_{t,\ell}$ , which is a quadratic polynomial in $\mathbf{A}_{t,\ell} = \tilde{\mathbf{q}}_{t,\ell} \tilde{\mathbf{q}}_{t,\ell}^\top$ . Since $\mathbf{A}_{t,\ell}$ is a rank-1 projection onto the normalized reference direction $\tilde{\mathbf{q}}_{t,\ell}$ , the operator $\mathbf{S}_{t,\ell}$ applies a combination of identity scaling, projection, and higher-order projection effects. The net effect is to orthogonalize and rescale the gradient matrix relative to the reference direction's singular vectors — it's essentially a cheap approximation to SVD-based orthogonalization applied along the left singular space.

Why this form (and why it matters for efficiency): the dense structure of $\mathbf{P}_{t,\ell}^{\text{Muon}}$ has a critical computational consequence: it mixes coordinates, destroying the separability that makes AdamW-efficient projection possible. For a linear layer with outer-product gradient $\mathbf{a}_r(z) \otimes \mathbf{b}_r(z)$ , applying a dense left-preconditioner means computing $\mathbf{P}_{t,\ell}^{\text{Muon}} (\mathbf{a}_r(z) \otimes \mathbf{b}_r(z)) = (\mathbf{P}_{t,\ell}^{\text{Muon}} \mathbf{a}_r(z)) \otimes \mathbf{b}_r(z)$ . The CountSketch projection of this preconditioned outer product now costs $\mathcal{O}(d_{\text{in}} d_{\text{out}})$ rather than $\mathcal{O}(d_{\text{in}} + d_{\text{out}})$ because the left-preconditioner destroys the coordinate-wise separable structure. The paper explicitly notes this in Section 5.2: "for optimizers with dense preconditioners such as Muon, coordinate mixing destroys this separability, resulting in a projection cost of $\mathcal{O}(d_{\text{in}} d_{\text{out}})$ ."

The paper uses Muon with momentum $\mu = 0.95$ , Newton-Schulz quintic iteration for 5 steps in BF16, and a matrix-size-dependent learning rate rescaling: $\eta_{\text{eff}} = \eta \cdot \max(1, \sqrt{m/n})$ for a matrix of dimensions $m \times n$ , which stabilizes training across differently-shaped weight matrices. Muon is applied only to matrix-shaped parameters inside Transformer blocks (those with ndim >= 2), with all other parameters (embeddings, LM head, biases, normalization weights) optimized by AdamW — a hybrid setup that the paper explicitly documents in Table 1.

Bench-Proxy Construction

The proxy gradient direction $\mathbf{g}_{\text{proxy}}(t)$ in the utility score (Equation 23) is computed from a pool $\mathcal{D}_{\text{proxy}}$ constructed via the Bench-Proxy procedure (Section 5.1, detailed in Section 6.2). This procedure addresses a fundamental tension in proxy construction: using the raw benchmark validation set directly introduces distribution shift (benchmark text comes from a different distribution than pre-training text, so gradients computed on it may be noisy or unrepresentative), while using a random held-out subset of the pre-training corpus provides stable gradients but may be misaligned with the downstream tasks we ultimately care about.

Bench-Proxy resolves this by creating a retrieved proxy pool that is simultaneously task-aligned and in-distribution. The construction has two stages:

Stage 1: Similarity scoring. Each document in the pre-training corpus is assigned a relevance score with respect to the target benchmark validation set $\mathcal{D}_{\text{val}}$ . Concretely:

A frozen sentence embedding model (Arctic-Embed-L v2; Yu et al., 2024a) encodes both (i) each benchmark sample and (ii) each pre-training document into a shared embedding space.
For each document, cosine similarities are computed against all benchmark samples.
The document's final score is the maximum cosine similarity over all benchmark samples — this captures whether the document is strongly aligned with any benchmark instance, rather than requiring similarity to all of them.

The maximum-reduction strategy is a deliberate design choice: it ensures the proxy pool contains documents covering the full diversity of the benchmark, since different documents may align with different benchmark subdomains. An average-reduction would favor documents that are moderately similar to many benchmark samples, potentially missing specialized content.

Stage 2: Proxy pool construction. Documents are sorted by their benchmark relevance scores in descending order. The top documents are greedily accumulated (by token count) until reaching a fixed token budget of 30M tokens. This yields a compact but benchmark-aligned proxy shard $\mathcal{D}_{\text{proxy}}$ containing roughly 30M tokens (small compared to the full 3T-token pre-training corpus, but large enough to provide stable gradient estimates when mini-batched).

During training, OPUS samples a proxy mini-batch of size $K_{\text{proxy}} = 8$ from $\mathcal{D}_{\text{proxy}}$ at each step, computes the average gradient, and uses this as the target direction. The proxy pool is refreshed every step to provide a stochastic estimate, preventing overfitting to a fixed proxy batch.

Why retrieval over direct benchmark use: directly computing gradients on benchmark validation data would produce gradients that are (a) out-of-distribution relative to the model's training manifold, leading to noisy utility estimates, and (b) potentially very different in scale and direction from gradients on pre-training data, causing the alignment term in Equation 23 to be uninformative. By retrieving pre-training documents that are semantically similar to the benchmark, Bench-Proxy produces gradients that live in the same distribution as training gradients but are steered toward benchmark-relevant content. The paper validates this empirically in Table 7: the benchmark-matched proxy achieves an average of 41.75 versus 41.03 for a standard proxy (a hold-out validation set without retrieval), confirming that proxy quality matters.

Relationship to the previous section's derivation: note that the proxy gradient $\mathbf{g}_{\text{proxy}}(t)$ enters the utility score in Equation 23 unpreconditioned — it is the raw gradient from the proxy batch. Only the candidate's gradient gets preconditioned by $\mathbf{P}_t$ . This asymmetry is intentional: the proxy gradient represents the target direction in raw parameter space, and we want to measure how much the optimizer-preconditioned candidate update aligns with this target. If both were preconditioned, the alignment would be measured in a doubly-transformed space that loses the connection to the original loss landscape.

Scalable Utility Estimation via Ghost Gradients and CountSketch

The utility score in Equation 23 requires computing inner products between high-dimensional effective update vectors $\mathbf{u}_z(t)$ and the proxy gradient $\mathbf{g}_{\text{proxy}}(t)$ , plus the accumulated redundancy direction $\mathbf{G}(t)$ . For GPT-2 XL, the total number of parameters is $d \approx 1.5 \times 10^9$ , and materializing full per-sample gradients for a buffer of $N = 32$ candidates would require storing and operating on $32 \times 1.5 \times 10^9 \times 4$ bytes $\approx 192$ GB of memory — completely infeasible at scale.

OPUS avoids this through two synergistic techniques: ghost gradients, which exploit the low-rank structure of per-sample gradients in linear layers to avoid materialization, and CountSketch projections, which compress high-dimensional vectors into low-dimensional sketch vectors while preserving inner products in expectation.

Ghost Gradients

The key observation exploited by ghost gradients (following GREATS; Wang et al., 2024) is that for a linear layer with weights $\mathbf{W}_r \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$ , the per-sample gradient with respect to the weights factorizes as the outer product of two much smaller vectors:

$\nabla_{\mathbf{W}_r} \mathcal{L}(z; \theta_t) = \mathbf{a}_r(z) \otimes \mathbf{b}_r(z)$

where $\mathbf{a}_r(z) \in \mathbb{R}^{d_{\text{in}}}$ is the input activation vector to layer $r$ for sample $z$ , and $\mathbf{b}_r(z) \in \mathbb{R}^{d_{\text{out}}}$ is the output gradient vector (the error signal backpropagated from the loss through layer $r$ ). The outer product $\otimes$ produces the full $d_{\text{in}} \times d_{\text{out}}$ gradient matrix: element $(i, j)$ is $a_i \cdot b_j$ .

What this enables: instead of storing the full $d_{\text{in}} \times d_{\text{out}}$ gradient for each sample (which would cost $\mathcal{O}(d_{\text{in}} d_{\text{out}})$ memory), OPUS stores only the factor vectors $\mathbf{a}_r(z)$ and $\mathbf{b}_r(z)$ (costing $\mathcal{O}(d_{\text{in}} + d_{\text{out}})$ memory). These factors are available during the standard forward/backward pass — $\mathbf{a}_r(z)$ from the forward pass (the activations entering layer $r$ ), and $\mathbf{b}_r(z)$ from the backward pass (the gradients flowing out of layer $r$ ).

Why this works for linear layers: the factorization $\nabla_{\mathbf{W}_r} \mathcal{L}(z) = \mathbf{a}_r(z) \otimes \mathbf{b}_r(z)$ is an exact algebraic identity for linear transformations under standard backpropagation — it derives from the chain rule: $\frac{\partial \mathcal{L}}{\partial \mathbf{W}_r} = \frac{\partial \mathcal{L}}{\partial \mathbf{y}_r} \cdot \frac{\partial \mathbf{y}_r}{\partial \mathbf{W}_r}$ , where $\mathbf{y}_r = \mathbf{W}_r \mathbf{a}_r(z)$ is the layer output, $\frac{\partial \mathcal{L}}{\partial \mathbf{y}_r} = \mathbf{b}_r(z)$ , and $\frac{\partial \mathbf{y}_r}{\partial \mathbf{W}_r}$ yields the $\mathbf{a}_r(z)$ factor. This factorization is exact for any linear layer (dense, convolutional with appropriate reshaping, embedding lookups), but does not hold for nonlinear operations like attention or activation functions.

OPUS's usage. OPUS applies the ghost technique over a set of layers $\mathcal{R}$ , which includes "linear and embedding matrices" (Section 5.2). The paper does not enumerate exactly which layers, but based on the optimizer assignment in Table 1, $\mathcal{R}$ likely includes all Transformer block projection matrices (attention QKV, output, MLP expansion/contraction) for GPT-2, and the corresponding matrices for Qwen3-8B. Importantly, OPUS concatenates the proxy batch and candidate batch within a single forward/backward pass to collect $\{\mathbf{a}_r(z), \mathbf{b}_r(z)\}$ for all samples simultaneously, then discards these factors layer-by-layer after computing the projected scores to maintain low memory overhead.

Scoring-only forward passes. A critical implementation detail (Section 6.1): OPUS scores candidates using only the first $L_{\text{score}} = 512$ tokens of each sequence, even though the actual training sequences are much longer ( $L_{\text{train}} = 24,576$ for GPT-2 Large, $6,144$ for GPT-2 XL). This is a deliberate efficiency tradeoff: the ghost factors from 512 tokens capture the sample's gradient direction approximately (since gradient direction is largely determined by the type of content, not its full length), while computing factors for the full 24K-token sequence would be 48× more expensive. After selection, the model performs a full forward/backward update on the selected sequences at the full training length — the scoring forward pass is pure overhead.

CountSketch Projection

Even with ghost factors, computing the inner products $\langle \mathbf{u}_z(t), \mathbf{g}_{\text{proxy}}(t) \rangle$ and $\langle \mathbf{u}_z(t), \mathbf{G}(t) \rangle$ would require forming the full effective update vectors, which are still $d$ -dimensional. OPUS avoids this by projecting everything into a low-dimensional sketch space using CountSketch (Cormode and Muthukrishnan, 2005).

CountSketch definition. A CountSketch map $\Pi: \mathbb{R}^d \to \mathbb{R}^m$ (with $m \ll d$ ) is defined by two hash functions applied to each coordinate index $i \in \{1, ..., d\}$ :

A bucket hash $h: [d] \to [m]$ that maps each coordinate to one of $m$ sketch buckets.
A sign hash $s: [d] \to \{-1, +1\}$ that assigns each coordinate a random sign.

The sketch of a vector $\mathbf{v} \in \mathbb{R}^d$ is computed as: for each coordinate $i$ , add $s(i) \cdot v_i$ to bucket $h(i)$ . Formally, $\Pi(\mathbf{v})_j = \sum_{i: h(i) = j} s(i) v_i$ for each bucket $j$ .

Key property. CountSketch provides an unbiased estimator of inner products:

$\mathbb{E}[\langle \Pi(\mathbf{v}_1), \Pi(\mathbf{v}_2) \rangle] = \langle \mathbf{v}_1, \mathbf{v}_2 \rangle$

with variance inversely proportional to $m$ . This means OPUS can compute inner products between high-dimensional effective updates by instead computing inner products between their low-dimensional sketch vectors — reducing the cost from $\mathcal{O}(d)$ to $\mathcal{O}(m)$ per inner product.

OPUS's application to linear layers. For each linear layer $r \in \mathcal{R}$ , OPUS constructs a per-layer sketch feature $\phi_{(t,r)}(z) \in \mathbb{R}^m$ by:

Forming the ghost outer-product gradient $\mathbf{a}_r(z) \otimes \mathbf{b}_r(z)$ .
Applying the optimizer preconditioner $\mathbf{P}_{t,r}$ to this outer product — yielding the effective update contribution from layer $r$ .
Flattening the result into a $d_r$ -dimensional vector (where $d_r = d_{\text{in}} \cdot d_{\text{out}}$ for layer $r$ ) and applying CountSketch $\Pi_r$ to produce $\phi_{(t,r)}(z) \in \mathbb{R}^m$ .

Crucially, steps 2 and 3 are interleaved — the CountSketch projection is applied on-the-fly while preconditioning, without ever materializing the full $d_r$ -dimensional effective update. The efficiency of this interleaving depends on the preconditioner structure:

AdamW (diagonal preconditioner): Since $\mathbf{P}_{t,r}^{\text{AdamW}}$ is diagonal, it preserves the coordinate-wise separable structure. The preconditioned outer product element $(i,j)$ is simply $p_{ij} \cdot a_i \cdot b_j$ for some diagonal weight $p_{ij}$ . CountSketch can stream over these elements in $\mathcal{O}(d_{\text{in}} d_{\text{out}})$ time if the diagonal weights are stored, but the paper states the projection cost under AdamW is $\mathcal{O}(d_{\text{in}} + d_{\text{out}})$ — this suggests an implementation optimization where the diagonal structure is further exploited to avoid enumerating all $d_{\text{in}} \cdot d_{\text{out}}$ elements explicitly.
Muon (dense preconditioner): Since $\mathbf{P}_{t,r}^{\text{Muon}}$ is dense, it mixes coordinates. The preconditioned outer product becomes $(\mathbf{P}_{t,r}^{\text{Muon}} \mathbf{a}_r(z)) \otimes \mathbf{b}_r(z)$ , which costs $\mathcal{O}(d_{\text{in}} d_{\text{out}})$ to project. This is the computational penalty for Muon's richer preconditioning — the paper acknowledges this explicitly and accepts the cost because Muon is only applied to a subset of parameters (matrix-shaped ones in Transformer blocks).

Proxy and redundancy sketches. The proxy gradient direction $\mathbf{g}_{\text{proxy}}(t)$ is sketched layer-wise as:

$\psi_{\text{proxy}}^{(t,r)} := \Pi_r\left(\frac{1}{K_{\text{proxy}}} \sum_{k=1}^{K_{\text{proxy}}} \mathbf{a}_r(\tilde{z}_k) \otimes \mathbf{b}_r(\tilde{z}_k)\right)$

Note that the proxy gradient is not preconditioned — it represents the target direction in raw-gradient space, as discussed earlier. The redundancy accumulator $\mathbf{\Phi}^{(t,r)}$ is the running sum of selected candidate sketches: $\mathbf{\Phi}^{(t,r)} = \sum_{z_j \in \hat{\mathcal{B}}_t} \phi_{(t,r)}(z_j)$ .

Final utility computation in sketch space. The utility score from Equation 23 is approximated by summing sketch-space dot products across all layers in $\mathcal{R}$ :

$U_z(t) \approx \eta_t \sum_{r \in \mathcal{R}} \langle \phi_{(t,r)}(z), \psi_{\text{proxy}}^{(t,r)} \rangle - \eta_t^2 \sum_{r \in \mathcal{R}} \langle \phi_{(t,r)}(z), \mathbf{\Phi}^{(t,r)} \rangle$

The first sum is the sketch-space estimate of the alignment term; the second sum is the sketch-space estimate of the redundancy penalty.

Sketch dimension and compression ratio. The paper sets the CountSketch dimension to $m = 8192$ with a fixed random seed of 42 (Section 6.1, "Random projection configuration"). For GPT-2 XL, the largest matrix gradient dimension is approximately 10.24M (for a weight matrix of shape, say, $1600 \times 6400$ for the MLP expansion). The sketch dimension of 8192 provides an effective compression ratio of roughly $1250\times$ — the 10.24M-dimensional effective update is represented by an 8192-dimensional sketch while preserving inner product information in expectation.

Hyperparameter sensitivity. Table 8 reports sensitivity to sketch dimension: $m = 4096$ yields an average of 39.63, $m = 8192$ yields 40.25, and $m = 16384$ yields 39.34. The non-monotonic behavior (16384 performing worse than 8192) is notable — the paper does not explain this, but it may reflect a bias-variance tradeoff where the larger sketch dimension provides lower variance per inner product but the increased computational cost forces other approximations (e.g., reduced buffer size or proxy batch size due to memory constraints) that hurt overall performance.

Boltzmann Sampling with Redundancy Penalty

The utility scores $U_z(t)$ computed by Equation 25 rank candidates by their estimated marginal value. A naive approach would select the $K$ candidates with highest scores (deterministic greedy top- $k$ ), as done by GREATS and most prior dynamic selection methods. OPUS instead uses sequential Boltzmann sampling for two reasons:

Reason 1: Robustness to proxy noise. The proxy direction $\psi_{\text{proxy}}^{(t,r)}$ is a stochastic estimate from a small mini-batch (size 8) — it contains sampling noise. Greedy top- $k$ can lock onto candidates whose high scores are partially due to alignment with transient noise features in this particular proxy batch. Boltzmann sampling (sampling with probability proportional to $\exp(U_z(t) / \tau)$ ) maintains non-zero selection probability for candidates with slightly lower scores, preventing overfitting to proxy noise. The temperature $\tau$ controls the exploration-exploitation tradeoff: $\tau \to 0$ approaches greedy selection, $\tau \to \infty$ approaches uniform random, and $\tau = 0.9$ (the paper's default) provides moderate stochasticity.

Reason 2: Sequential selection with redundancy updating. Rather than computing all scores once and sampling independently (which would ignore interactions between selections), OPUS selects candidates sequentially: at each of $K$ selection steps, it computes the redundancy penalty using the already-selected sketches $\mathbf{\Phi}^{(t,r)}$ (which is updated after each selection), then samples one candidate from the remaining pool using the current Boltzmann distribution. This means the $j$ -th selection's utility score incorporates the redundancy penalty from the previous $j-1$ selections — if the first few selections have already "covered" a certain gradient direction, the penalty term for subsequent candidates that point in that same direction increases, reducing their selection probability.

The sampling distribution at each sub-step is:

$p_t(z) \propto \exp(U_z(t) / \tau), \quad U_z(t) \text{ computed with current } \mathbf{\Phi}^{(t,r)}$

Why this is needed despite the redundancy penalty: one might ask: if the redundancy penalty in Equation 25 already subtracts overlap, why do we also need stochastic sampling? The answer is that the redundancy penalty relies on an isotropic Hessian approximation ( $\mathbf{H}_{\text{val}} \approx \mathbf{I}$ ), which is a crude model of true parameter-space curvature. In reality, some directions have much higher curvature (sharp valleys in the loss landscape) where redundancy is more harmful, while others are flat where redundancy is less problematic. The identity approximation cannot capture this heterogeneity. Boltzmann sampling adds an additional robustness layer: even if the redundancy penalty imperfectly models curvature, stochastic sampling prevents the selector from deterministically committing to a narrow set of directions that might be suboptimal under the true Hessian.

Ablation evidence. Table 7 compares greedy selection to Boltzmann sampling: on GPT-2 XL with Muon on FineWeb, greedy achieves an average of 40.49 while Boltzmann (full OPUS) achieves 41.75 — a 1.26 percentage point improvement. This is substantial given that the only difference is the selection rule, not the utility scoring. The paper also sweeps temperature $\tau$ in Table 8: $\tau = 0.8$ achieves 39.88, $\tau = 0.9$ achieves 40.25, $\tau = 1.0$ achieves 39.87. The optimum at $\tau = 0.9$ confirms that moderate stochasticity outperforms both near-greedy ( $\tau = 0.8$ ) and near-uniform ( $\tau = 1.0$ ) sampling.

End-to-End Algorithm Walkthrough

Algorithm 1 in the paper specifies OPUS's per-step workflow. Here I walk through the algorithm line-by-line, connecting each step to the mathematical derivations above.

Initialization (lines 1-2). The CountSketch operators $\Pi_r$ are initialized once with hash functions $h: [d_r] \to [m]$ and $s: [d_r] \to \{-1, +1\}$ for each layer $r \in \mathcal{R}$ . These are deterministic (fixed seed 42) and reused for all training steps.

Step 1: Batch sampling (line 4). A candidate buffer $\mathcal{B}_t = \{z_1, ..., z_N\}$ of $N$ sequences is read from the pre-training data stream $\mathcal{D}_{\text{tr}}$ . For GPT-2 experiments, $N = 32$ . Each sequence is a document of $L_{\text{train}}$ tokens, but only the first $L_{\text{score}} = 512$ tokens are used for scoring.

Step 2: Preconditioner computation (line 5). The optimizer-induced preconditioner $\mathbf{P}_t$ is constructed from the current optimizer state. For AdamW-updated parameters, this is the diagonal matrix $\mathbf{P}_t^{\text{AdamW}} = C_t \text{Diag}(1 / \sqrt{\bar{\mathbf{v}}_t + \epsilon})$ . For Muon-updated parameters, this is the dense per-layer matrix $\mathbf{P}_{t,\ell}^{\text{Muon}} = \eta_t (1 - \mu^2) \mathbf{S}_{t,\ell}$ where $\mathbf{S}_{t,\ell}$ is computed via a frozen Newton-Schulz operator constructed from the current optimizer buffer's reference direction.

Step 3: Proxy feature generation (line 6). A proxy mini-batch of $K_{\text{proxy}} = 8$ samples is drawn from $\mathcal{D}_{\text{proxy}}$ . The ghost factors $\{\mathbf{a}_r(\tilde{z}_k), \mathbf{b}_r(\tilde{z}_k)\}$ are extracted for these samples via a forward/backward pass. For each layer $r$ , the average outer product $\frac{1}{K_{\text{proxy}}} \sum_{k=1}^{K_{\text{proxy}}} \mathbf{a}_r(\tilde{z}_k) \otimes \mathbf{b}_r(\tilde{z}_k)$ is computed and projected via CountSketch to produce $\psi_{\text{proxy}}^{(t,r)} \in \mathbb{R}^m$ . Note: this proxy gradient is not preconditioned — it represents the target direction in raw-gradient space.

Step 4: Candidate feature generation (line 7). For each candidate $z \in \mathcal{B}_t$ , the ghost factors $\{\mathbf{a}_r(z), \mathbf{b}_r(z)\}_{r \in \mathcal{R}}$ are extracted. For each layer $r$ , the preconditioned effective update $\mathbf{P}_{t,r} (\mathbf{a}_r(z) \otimes \mathbf{b}_r(z))$ is computed and projected via CountSketch to produce $\phi_{(t,r)}(z) \in \mathbb{R}^m$ . The "implicit" computation means the full outer product is never materialized — preconditioning and sketching are interleaved.

Step 5: Sequential Boltzmann sampling (lines 8-15). This is the core selection loop, running $K = \lfloor \rho N \rfloor$ times. For GPT-2 experiments, $\rho = 0.5$ (selection ratio), so $K = 16$ from a buffer of $32$ . The selected set $\hat{\mathcal{B}}_t$ starts empty, and the per-layer redundancy accumulator $\mathbf{\Phi}^{(t,r)}$ starts at $\mathbf{0}$ .

At each sub-step $j$ , for each remaining candidate $z \in \mathcal{B}_t \setminus \hat{\mathcal{B}}_t$ :

$U_z(t) \leftarrow \eta_t \sum_{r \in \mathcal{R}} \langle \phi_{(t,r)}(z), \psi_{\text{proxy}}^{(t,r)} \rangle - \eta_t^2 \sum_{r \in \mathcal{R}} \langle \phi_{(t,r)}(z), \mathbf{\Phi}^{(t,r)} \rangle$

The first term is the alignment with the proxy target. The second term is the redundancy penalty — the dot product between the candidate's sketch and the accumulated sketch of all previously selected samples. As $\mathbf{\Phi}^{(t,r)}$ grows with each selection, the penalty for candidates similar to previous ones increases.

A candidate $z^*$ is sampled from the multinomial distribution with probabilities $p_t(z) \propto \exp(U_z(t) / \tau)$ , where $\tau = 0.9$ is the temperature. This candidate is added to $\hat{\mathcal{B}}_t$ , and the redundancy accumulator is updated: $\mathbf{\Phi}^{(t,r)} \leftarrow \mathbf{\Phi}^{(t,r)} + \phi_{(t,r)}(z^*)$ for all $r$ .

Step 6: Parameter update (line 16). The model is trained on the selected batch $\hat{\mathcal{B}}_t$ using the standard optimizer $\mathcal{O}$ (AdamW or Muon+AdamW hybrid). Crucially, the full training sequence length $L_{\text{train}}$ is used here (not the scoring length $L_{\text{score}} = 512$ ), and the token budget is counted against $L_{\text{train}}$ . The additional forward computation for scoring is tracked separately as overhead (4.7% of total training compute, per Section 6.6 and Figure 7).

What makes this work at scale. Three design choices collectively make the algorithm practical: (1) $L_{\text{score}} \ll L_{\text{train}}$ reduces the cost of scoring forward/backward passes; (2) ghost factors avoid storing per-sample gradients, requiring only $\mathcal{O}(d_{\text{in}} + d_{\text{out}})$ memory per layer per sample rather than $\mathcal{O}(d_{\text{in}} d_{\text{out}})$ ; (3) CountSketch projections reduce inner products from $\mathcal{O}(d)$ to $\mathcal{O}(m)$ where $m = 8192 \ll d$ , with the projection itself amortized by interleaving with preconditioning.

A note on the selection ratio $\rho$ . The paper fixes $\rho = 0.5$ for all GPT-2 experiments. This means OPUS discards half of the candidate buffer at each step — the model trains on 16 of 32 available sequences. This may seem wasteful, but it reflects a deliberate tradeoff: the buffer overhead (storing ghost factors for 32 candidates, computing projections) scales with $N$ , while the training throughput depends only on $K$ . Setting $\rho = 0.5$ means OPUS pays the scoring cost for $N = 32$ but trains on $K = 16$ , enabling it to be selective while keeping the overhead manageable. The paper does not explore other selection ratios, which could be an interesting axis for future work — higher $\rho$ (less selective) would reduce relative overhead but potentially reduce per-token training efficiency.

Buffer size sensitivity. Table 8 reports buffer size sweeps: $N = 16$ achieves 39.76 average, $N = 32$ achieves 39.75, and $N = 64$ achieves 40.25 (with the default $\tau = 0.9$ , $m = 8192$ , on GPT-2 Large with Muon on FineWeb). The jump from 39.75 to 40.25 when increasing $N$ from 32 to 64 suggests that a larger candidate pool enables more selective filtering, but the paper's default of 32 (or 64, depending on the experiment — the ablation uses 64 but the main results at 32 for GPT-2) balances overhead with selectivity.

4. Key Insights and Innovations

Innovation 1: Optimizer Geometry as the Missing Dimension in Data Valuation

The paper's most fundamental conceptual move is the recognition that data utility is not an intrinsic property of a sample but is relative to the optimizer's update geometry. Prior dynamic selection methods — GREATS (Wang et al., 2024), perplexity-based scoring (Ankner et al., 2025), and the broader influence-function literature (Koh and Liang, 2017; Pruthi et al., 2020) — all score candidates using raw-gradient inner products, implicitly assuming SGD dynamics. This assumption is not just approximate; it is structurally wrong for modern LLM training, where AdamW applies coordinate-wise rescaling based on historical gradient variance and Muon applies matrix orthogonalization that fundamentally mixes gradient directions.

The significance of this insight is that it reframes data selection from a model-dependent problem (what does the current model need?) to an optimizer-dependent problem (what will the optimizer actually do with this sample's gradient?). A sample whose raw gradient perfectly aligns with the validation gradient might be nearly useless if the optimizer's preconditioner attenuates those parameter directions, while a sample with mediocre raw alignment might deliver a large effective update. Prior work missed this entirely because it treated the optimizer as a transparent implementation detail rather than as a first-class participant in the utility calculation.

This is a fundamental reframing rather than an incremental refinement. It does not improve an existing scoring function — it changes the space in which scoring happens. The evidence that this matters is stark in Table 3: under the Muon optimizer, GREATS (raw-gradient scoring) achieves 39.41 on GPT-2 Large, barely above random at 39.72, while OPUS (preconditioned scoring) achieves 40.25. The gap widens with scale: on GPT-2 XL, GREATS drops to 39.23 (below random's 40.29), while OPUS reaches 41.75. This pattern — where raw-gradient scoring degrades relative to random as model scale increases — suggests that larger models have more complex optimizer geometry, making the cost of scoring in the wrong space progressively higher.

The paper also derives this insight from first principles by linearizing the one-step lookahead updates of AdamW and Muon (Sections 4.2–4.3), producing closed-form preconditioners $\mathbf{P}_t$ that capture the effective update geometry. These derivations are not merely descriptive — they are the mechanism that operationalizes the insight. But the conceptual contribution is the recognition that optimizer-induced geometry is not a second-order correction to data valuation but the primary determinant of which samples produce effective training signals. This changes how researchers should think about data selection: the question is no longer "which samples reduce validation loss?" but "which samples will the optimizer effectively move the model toward better validation loss?"

Innovation 2: Retrieval-Based Proxy Construction That Bridges Task Alignment and Distributional Fidelity

Prior dynamic selection methods use one of two strategies for the target gradient direction: either a random held-out validation set (GREATS, MATES) or the raw benchmark validation data directly (some influence-function approaches). Both have fundamental problems. A random hold-out set provides stable, in-distribution gradients but is misaligned with downstream tasks — reducing loss on a random subset of web text may not translate to benchmark improvement. Raw benchmark data is task-aligned but produces gradients that are out-of-distribution relative to the model's training manifold, introducing noise and potentially unstable utility estimates.

OPUS's Bench-Proxy construction (Section 5.1, detailed in Section 6.2) resolves this tension through a retrieval-based approach: use a frozen sentence embedding model to encode both benchmark samples and pre-training documents, then retrieve the top pre-training documents by semantic similarity to form the proxy pool. This produces a target gradient that is simultaneously task-aligned (semantically close to downstream benchmarks) and in-distribution (drawn from the pre-training corpus, so gradients live in the same manifold as training gradients). The retrieval uses maximum similarity over benchmark samples (not average), ensuring the proxy pool covers the full diversity of benchmark subdomains rather than converging on a single "most representative" content type.

This is an incremental but practically significant refinement. It does not change the scoring formula — the proxy gradient enters Equation 23 the same way regardless of how it is constructed — but it substantially improves the quality of the target signal. The empirical evidence in Table 7 supports this: the benchmark-matched proxy achieves an average of 41.75 versus 41.03 for a standard proxy (a hold-out set without retrieval), a 0.72 percentage point improvement. While this is smaller than the gain from optimizer-aware scoring itself, it demonstrates that proxy quality matters independently and that retrieval is an effective way to improve it.

The broader significance is methodological: Bench-Proxy establishes a pattern for constructing evaluation-aligned training signals without introducing distribution shift. This pattern could apply beyond pre-training — instruction tuning, RLHF reward modeling, and continual learning all face the same tension between task alignment and distributional fidelity. The retrieval-based approach provides a simple, scalable solution that avoids the brittleness of using out-of-distribution benchmarks directly while providing more targeted guidance than random hold-out sets.

Innovation 3: Diversity-Preserving Selection via Joint Redundancy Penalty and Boltzmann Sampling

Most prior dynamic selection methods use deterministic greedy top- $k$ : compute a score for each candidate, pick the $K$ with highest scores, update. This has two failure modes that OPUS addresses jointly. First, geometric redundancy: the second-best candidate might point in nearly the same gradient direction as the first, providing minimal marginal utility. GREATS and related methods do not explicitly penalize this, leading to batches that explore few directions in parameter space. Second, brittleness to proxy noise: the utility scores depend on a stochastic proxy gradient estimate (from a mini-batch of size 8), and greedy top- $k$ can lock onto candidates whose high scores are partially noise artifacts.

OPUS addresses the first failure mode through the redundancy penalty term in Equation 22–23: $-\eta_t^2 \langle \mathbf{u}_z(t), \mathbf{G}(t) \rangle$ , where $\mathbf{G}(t)$ is the accumulated effective update of already-selected samples. This explicitly subtracts the overlap between a candidate's effective update and the batch's aggregate update, discouraging redundant selections. The derivation comes from a second-order Taylor expansion of the validation loss reduction (the Hessian term), approximated isotropically as $\mathbf{H}_{\text{val}} \approx \mathbf{I}$ for tractability. While this approximation is crude — it treats all parameter-space directions as having equal curvature, which is false — it captures the correct sign of the redundancy effect, and the alternative of diagonal Hessian approximations would interact problematically with AdamW's diagonal preconditioner.

The second failure mode is addressed through Boltzmann sampling: rather than deterministically selecting the top- $K$ , OPUS samples candidates with probability proportional to $\exp(U_z(t) / \tau)$ at each of $K$ sequential selection steps. This maintains non-zero selection probability for candidates with slightly sub-optimal scores, preventing deterministic overfitting to transient proxy noise. The sequential nature of sampling (recomputing the redundancy penalty after each selection) also interacts with the penalty term: the $j$ -th selection's score depends on the previous $j-1$ selections, so the sampling naturally avoids clusters of similar candidates.

These are incremental refinements individually — redundancy penalties exist in subset selection literature (e.g., determinantal point processes), and Boltzmann sampling is a standard exploration mechanism — but their joint application with optimizer-preconditioned scoring is novel. The ablation evidence in Table 7 teases apart their contributions: greedy selection (with the redundancy penalty active but deterministic top- $k$ ) achieves 40.49, while Boltzmann sampling (full OPUS) achieves 41.75 — a 1.26 point gain. This suggests that the redundancy penalty alone captures some of the diversity benefit, but stochastic sampling adds robustness beyond what the isotropic Hessian approximation can model.

The broader significance is that OPUS demonstrates a practical, scalable alternative to greedy top- $k$ that works at LLM scale. Prior diversity-promoting selection methods (e.g., determinantal point processes, submodular maximization) typically require computing pairwise similarity matrices over all candidates — $\mathcal{O}(N^2)$ complexity — which is infeasible for large buffers. OPUS achieves similar effects through a combination of a running penalty accumulator ( $\mathcal{O}(K)$ per step) and stochastic sampling, both of which are linear in the buffer size. This efficiency is what makes diversity-preserving selection deployable at the scale of LLM pre-training.

Innovation 4: Empirical Proof That Dynamic Selection from Lower-Quality Data Can Outperform Static Selection from Higher-Quality Data

Perhaps the most striking experimental result in the paper is not that OPUS outperforms baselines on equal footing, but that OPUS selecting dynamically from lower-quality data outperforms static methods trained on higher-quality data. In the FineWeb-Edu experiments (Table 4), OPUS selects from the mid-quality subset (FineWeb-Edu score 3) while static baselines are trained on the high-quality subset (scores 4 and 5). Despite this disadvantage in raw data quality, OPUS achieves the best compute-matched average in multiple settings: on GPT-2 XL with Muon, OPUS on score-3 data reaches an average of 44.99, surpassing all static baselines on score 4+5 data (the best being DCLM-FastText at 42.24 and FineWeb-Edu at 42.18). On GPT-2 Large with Muon, OPUS on score-3 data achieves 41.72 versus the best static baseline on score 4+5 data at 41.43 (GREATS).

This finding is not merely a performance improvement — it is a refutation of the static filtering paradigm's central assumption: that data quality is an intrinsic, time-invariant property that should be used to permanently filter the corpus before training. If dynamic, optimizer-aware selection can extract more value from lower-quality text than static filtering can from higher-quality text, then the vast amount of "mid-quality" web content that static pipelines discard (billions of tokens scored as 3 on a 5-point scale) may contain significant untapped training signal. The limiting factor is not the raw quality of the text but the intelligence of the selection mechanism that decides when and how to use it.

This has significant practical implications that the paper only partially explores. Organizations investing in data curation pipelines (classifier training, deduplication, quality filtering) may be over-investing in static quality assessment and under-investing in dynamic selection. The optimal strategy may be a lightweight static filter that removes only the most egregious noise (spam, malformed text, toxic content) combined with a dynamic selector that adaptively chooses from the remaining mid-to-high-quality pool based on the model's current needs. The paper demonstrates this implicitly: OPUS on FineWeb-Edu score 3 (mid-quality) outperforms OPUS on score 4+5 (high-quality) in some settings (GPT-2 XL Muon: 44.99 vs. 42.42), suggesting that the larger pool of mid-quality data provides more diversity for dynamic selection to exploit.

The result is also a strong existence proof for dynamic selection's value proposition. Skeptics might ask: does dynamic selection provide enough benefit to justify the additional complexity and compute overhead? The FineWeb-Edu result answers: dynamic selection from worse data beats static selection from better data. The benefit is not marginal — it is large enough to overcome a quality gap in the input corpus. This shifts the burden of proof: rather than dynamic methods needing to justify their overhead against a strong static baseline, static methods must now explain why they permanently discard data that a dynamic method could use productively.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary pre-training corpus is FineWeb (Penedo et al., 2024), a 3T-token pool derived from Common Crawl. For robustness evaluation on higher-quality data, the paper uses FineWeb-Edu (Penedo et al., 2024), partitioned into a 120B-token mid-quality bucket (score-3 documents) and an 80B-token high-quality bucket (scores 4 and 5). For continued pre-training, a 3B-token pool is constructed from SciencePedia (SciencePedia Team, 2025). All from-scratch experiments use a fixed optimization budget of 30B update tokens, with an additional non-compute-matched reference at 60B update tokens for contextualizing convergence speed.
Base model(s). Two GPT-2 variants are trained from scratch: GPT-2 Large (774M parameters, 36 layers, hidden size 1280) and GPT-2 XL (1.5B parameters, 48 layers, hidden size 1600). These are chosen as representative architectures at scales where comprehensive ablation and comparison against many baselines is computationally feasible while remaining large enough that data selection overhead matters. For continued pre-training, Qwen3-8B-Base (Yang et al., 2025) is used — a 36-layer, 8B-parameter model adapted on SciencePedia to test OPUS in a realistic domain-specialization scenario.
Metrics. The primary metric across all experiments is accuracy on a suite of 10 benchmarks: MMLU (zero-shot, full-answer log-likelihood following FineWeb-Edu protocol), ANLI, HellaSwag, PIQA, SIQA, WinoGrande, ARC-Easy, ARC-Challenge, CommonsenseQA, and WSC. For these benchmarks, accuracy is computed via multiple-choice perplexity scoring (select the option with lowest negative log-likelihood conditioned on the prompt), except MMLU which uses log-likelihood comparison of full answer strings and WinoGrande which uses a log-likelihood variant. For continued pre-training, accuracy is reported on OlympicArena (zero-shot) and SciAssess (3-shot with chain-of-thought, stochastic decoding at temperature 0.6, top-p 0.95, top-k 20). Validation loss curves are reported on a held-out FineWeb-Edu split. Domain-wise perplexity is evaluated on 10 domains (Health, Business, Politics, Education, History, Lifestyle, Science, Arts & Lit., Entertainment, Computing) with 1000 held-out samples per domain following the WebOrganizer protocol (Wettig et al., 2025). Out-of-distribution generalization is measured on BBH (3-shot, subsampled to tasks where base models produce non-degenerate outputs), RACE-Middle, RACE-High, AX-b, AX-g, and StoryCloze (all zero-shot).
Baselines. The paper compares against eight methods. Static baselines: QuRating (Wettig et al., 2024) — scalar quality ratings from pairwise preferences; DSIR (Xie et al., 2023) — importance resampling in reduced feature space; DCLM-FastText (Li et al., 2024) — standardized corpus filtering; FineWeb-Edu Classifier (Penedo et al., 2024) — document-level quality scoring; UltraFineweb Classifier (Wang et al., 2025c) — lightweight classifier-based filtering. Dynamic baselines: High-PPL (PPL), which selects highest-loss sequences under the current model following Ankner et al. (2025); GREATS (Wang et al., 2024), which selects samples via raw-gradient alignment with a proxy direction. Random selection at both 30B and 60B update tokens serves as the primary non-selective reference.
Generation budget / compute accounting. All compute-matched comparisons fix the number of update tokens — the total tokens on which parameter updates are actually performed — at 30B for from-scratch experiments. This means all methods perform forward/backward passes on exactly 30B tokens worth of training data. The additional computation used for scoring (forward passes on the first 512 tokens of candidate sequences during OPUS selection, proxy gradient computation, etc.) is treated as overhead and measured separately in PFLOPs and wall-clock time (Figure 7). For continued pre-training, the budget varies across checkpoints (0.5B, 1.0B, 1.5B, 3.0B update tokens). The selection ratio in OPUS is fixed at ρ = 0.5, meaning from a candidate buffer of N = 32 (or 64 in some ablations), K = 16 (or 32) sequences are selected for the actual update.
Cross-validation / statistical protocol. No cross-validation is applied to the from-scratch experiments — each configuration is trained once and evaluated on the full benchmark suite. This is standard practice at this scale given the computational cost of pre-training. The paper does not report confidence intervals or standard errors on benchmark scores, which is a limitation for interpreting the narrower margins. For continued pre-training, checkpoints are evaluated every 0.5B tokens, providing a trajectory rather than a single point estimate.

Main Quantitative Results

From-Scratch Pre-training on FineWeb

The central result for standard web-scale pre-training appears in Table 3. Across all four configurations (GPT-2 Large/XL × Muon/AdamW), OPUS achieves the best compute-matched average among all methods at 30B update tokens:

GPT-2 XL with Muon: OPUS achieves an average of 41.75, compared to random at 40.29 (+1.46), the best static baseline FineWeb-Edu at 40.74 (+1.01), and GREATS at 39.23 (+2.52, with GREATS actually underperforming random). The largest individual benchmark improvements over random are WSC (48.08 vs. 36.54, +11.54 points) and ARC-E (39.68 vs. 38.98, +0.70). Notably, OPUS at 30B tokens (41.75) exceeds Random at 60B tokens (41.29), demonstrating more than 2× data efficiency.
GPT-2 Large with Muon: OPUS achieves an average of 40.25, compared to random at 39.72 (+0.53), the best static baseline UltraFineweb at 40.15 (+0.10), and GREATS at 39.41 (+0.84). The gap narrows at this smaller scale — OPUS narrowly edges out UltraFineweb and trails Random at 60B tokens (40.13 vs. 40.25, a -0.12 difference within noise range). The largest individual gain is ARC-C (29.15 vs. 28.14, +1.01).
GPT-2 Large with AdamW: OPUS achieves an average of 41.43, compared to random at 39.29 (+2.14). This is the largest relative gain in the table. Key drivers include MMLU (31.09 vs. 28.19, +2.90), HellaSwag (45.52 vs. 42.65, +2.87), and WinoGrande (51.62 vs. 50.12, +1.50). OPUS at 30B substantially exceeds Random at 60B (41.43 vs. 39.60, +1.83), again showing data efficiency exceeding 2×.
GPT-2 XL with AdamW: OPUS achieves an average of 41.01, compared to random at 40.36 (+0.65). The best static baseline is FineWeb-Edu at 40.43 (+0.58 below OPUS). The standout benchmark is ARC-C (29.15 vs. 24.41, +4.74 over random). Random at 60B reaches 41.29, slightly above OPUS at 30B (−0.28), suggesting that at this configuration the efficiency gain is less than 2×.

Several patterns emerge from cross-reading Table 3. First, GREATS (raw-gradient scoring) performs inconsistently — it achieves 39.67 on GPT-2 Large with AdamW (above random's 39.29) but drops to 39.23 on GPT-2 XL with Muon (below random's 40.29), the only method to degrade with scale. This supports the paper's central claim that raw-gradient scoring becomes increasingly misaligned with larger models and more complex optimizers. Second, PPL (high-perplexity selection) consistently underperforms random (e.g., 39.82 vs. 40.29 on GPT-2 XL Muon), confirming that loss magnitude alone is not a reliable utility signal. Third, static methods cluster tightly — on GPT-2 XL Muon, the five static baselines range from 39.64 (UltraFineweb) to 40.74 (FineWeb-Edu), a spread of only 1.10 points, and none dramatically outperform random (40.29). This suggests static filtering provides modest but consistent improvements over random, whereas dynamic selection (when done well) can provide larger gains.

Robustness on FineWeb-Edu: Dynamic Selection from Lower-Quality Data

Table 4 presents what is arguably the paper's most striking result: OPUS selecting dynamically from the mid-quality FineWeb-Edu subset (score 3) is compared against baselines trained on the high-quality subset (scores 4 and 5). This is a deliberately stacked comparison where OPUS starts from a disadvantage in raw data quality.

GPT-2 XL with Muon: OPUS on score-3 data achieves an average of 44.99, outperforming every baseline on score 4+5 data — the best static baseline on high-quality data is DCLM-FastText at 42.24, meaning OPUS from mid-quality data leads by +2.75 points over the best static method from high-quality data. The margin over random on high-quality data (41.92) is +3.07. Individual benchmark gains are substantial: WSC (54.81 vs. 36.54 for score 4+5 random, +18.27), HellaSwag (50.54 vs. 46.39, +4.15), and PIQA (72.20 vs. 68.72, +3.48). OPUS on score-3 data also outperforms OPUS on score 4+5 data (44.99 vs. 42.42), a surprising +2.57 advantage for the lower-quality pool — the paper does not fully explain this, but it likely reflects the larger and more diverse candidate pool in the score-3 subset enabling more effective dynamic selection.
GPT-2 Large with Muon: OPUS on score-3 data achieves an average of 41.72, compared to the best baseline on score 4+5 data at 41.43 (GREATS), a narrower but still positive margin of +0.29. The validation loss curves in Figure 4 (right) corroborate the benchmark results: OPUS on score-3 data maintains lower validation loss throughout training than all baselines on score 4+5 data, with the gap widening over the 30B-token budget. For GPT-2 XL (Figure 4, left), OPUS reaches the validation loss achieved by Random at 60B tokens (score 4+5, i.e., twice the tokens of higher-quality data) after only 17B update tokens, a 3.5× convergence speed advantage.
Generalization beyond proxy-aligned benchmarks (Table 5). On the six out-of-distribution benchmarks not included in Bench-Proxy construction (BBH, RACE-M, RACE-H, AX-b, AX-g, StoryCloze), OPUS achieves an average of 40.07, compared to FineWeb-Edu at 39.08 (+0.99), GREATS at 39.42 (+0.65), and random at 38.09 (+1.98). This is critical evidence against the concern that OPUS merely overfits to the benchmarks used to construct the proxy — the gains transfer to held-out tasks. The BBH improvement (11.02 vs. 9.87, +1.15) is particularly notable given that BBH tests multi-step reasoning, a capability quite different from the factual and commonsense benchmarks in the proxy-aligned set.
Domain-wise perplexity (Table 6). Across all four configurations (GPT-2 Large/XL on FineWeb or FineWeb-Edu subset), OPUS achieves the lowest average perplexity on the 10-domain held-out set. For GPT-2 XL with Muon on FineWeb, OPUS reaches an average PPL of 3.26, compared to random's 3.37 (−3.3% relative improvement) and the next-best dynamic method GREATS at 3.40. On the FineWeb-Edu subset, OPUS achieves 3.45 versus random's 3.60 (−4.2%). The perplexity reductions are broadly distributed across all 10 domains rather than concentrated in a few, indicating that OPUS's selections improve general language modeling capability rather than specializing narrowly.

Continued Pre-training on SciencePedia

The continued pre-training experiments (Section 6.4, Figures 5 and 6) test OPUS in a domain-adaptation setting where a general-purpose model (Qwen3-8B-Base) is specialized on scientific literature.

Aggregate data efficiency (Figure 6). On SciAssess, OPUS at 0.5B tokens already matches the performance of random selection at 3B tokens — a 6× data efficiency gain. On OlympicArena, OPUS at 0.5B tokens achieves approximately 30.5% accuracy versus random's ~28%, and OPUS at 1.5B tokens reaches ~32% versus random's ~30%. The paper states that OPUS "reaches the best performance using only 0.5B tokens and already outperforms random CPT trained for 3B tokens."
Domain breakdown (Figure 5). On OlympicArena (Figure 5a), OPUS improves over Qwen3-8B-Base and random in most domains (physics, chemistry, biology, geography, text-only, multimodal) across all three token budgets. On SciAssess (Figure 5b), OPUS shows the strongest gains on material and medicine subsets, ties the best baseline on chemistry, and leads the overall average. The gains are broadly distributed rather than driven by a single outlier domain.
Comparison to DCLM and Full CPT. At 1.5B tokens, OPUS is competitive with or surpasses DCLM (a strong static baseline) and even the Full CPT reference (trained on all 3B tokens without selection) on several benchmarks, despite using only half the data. This is the continued pre-training analog of the FineWeb-Edu result: dynamic selection extracts more value per token than training on all available data indiscriminately.

Efficiency Analysis

Figure 7 quantifies the computational overhead of OPUS compared to alternative approaches.

Wall-clock time. A naïve implementation of online gradient-based selection (computing full per-sample gradients for all candidates, no ghost or sketch optimizations) would incur over 3.5× slowdown compared to random training. With ghost gradients and CountSketch projections enabled, OPUS reduces this overhead to only 4.7% — a 98.7% reduction in overhead relative to the naïve implementation.
Total compute (PFLOPs). Static methods like QuRating require substantial one-time compute to score the entire corpus before training (visible as a large initial spike in the PFLOPs curve). OPUS's per-step overhead is small and evenly distributed. Despite the additional scoring compute, OPUS achieves the best benchmark performance, meaning the overhead is a worthwhile investment rather than pure cost.
Scoring length optimization. A key efficiency factor is that OPUS scores candidates using only the first 512 tokens of each sequence, while actual training uses the full sequence length (24,576 for GPT-2 Large, 6,144 for GPT-2 XL). This means the scoring forward pass is 48× (Large) or 12× (XL) cheaper than a full training forward pass would be, while still capturing sufficient gradient direction information for ranking.

Ablation Studies and Robustness Checks

Soft sampling vs. greedy top-k (Table 7, GPT-2 XL Muon on FineWeb): Replacing Boltzmann soft sampling with deterministic greedy top-k reduces average accuracy from 41.75 to 40.49 (−1.26). This is one of the largest single-ablation effects in the paper, confirming that stochastic selection provides diversity benefits beyond what the redundancy penalty alone captures. The greedy variant still improves over random (40.49 vs. 40.29, +0.20), indicating that the utility scoring itself carries signal even without stochastic selection, but the full diversity-preserving mechanism is necessary for substantial gains.

Benchmark-matched proxy vs. standard proxy (Table 7): Using the Bench-Proxy construction (benchmark-retrieved in-distribution proxy) achieves 41.75, compared to 41.03 for a standard proxy (presumably a random held-out validation set without retrieval). The +0.72 gain confirms that proxy quality matters independently of the scoring method. An intermediate variant — greedy selection with standard proxy — would presumably show additive degradation, though the paper does not report the full 2×2 combination.

Buffer size (Table 8, GPT-2 Large Muon on FineWeb): Sweeping the candidate buffer size N at fixed temperature τ = 0.9 and sketch dimension m = 8192: N = 16 achieves 39.76, N = 32 achieves 39.75, N = 64 achieves 40.25. The near-identical performance at N = 16 and N = 32 is unexpected — doubling the buffer without gain suggests that at N = 16, the buffer already contains sufficient diversity for the selection ratio ρ = 0.5. The jump at N = 64 (+0.50 over N = 32) indicates a threshold effect where larger buffers enable more selective filtering. The paper's default of 32 (or 64 in some settings) balances overhead with performance.

Boltzmann temperature (Table 8): τ = 0.8 achieves 39.88, τ = 0.9 achieves 40.25, τ = 1.0 achieves 39.87. The optimum at τ = 0.9 is non-trivial — it is not at the extremes of greedy (τ → 0) or uniform (τ → ∞), confirming that moderate stochasticity outperforms both. The symmetry in the drop (both lower and higher temperatures degrade by similar amounts, ~0.37–0.38) suggests a well-defined optimal exploration-exploitation tradeoff.

CountSketch projection dimension (Table 8): m = 4096 achieves 39.63, m = 8192 achieves 40.25, m = 16384 achieves 39.34. This is the most intriguing ablation result: increasing the sketch dimension from 4096 to 8192 improves performance by +0.62, but further increasing to 16384 decreases performance by −0.91. The paper does not explain this non-monotonicity. Possible explanations: (1) at m = 16384, the sketch dimension is large enough that the CountSketch hash collisions become a non-issue, but the increased computational cost forces other approximations (e.g., reduced proxy batch size or scoring length to maintain throughput) that hurt overall performance; (2) the variance of CountSketch estimates scales as 1/m, but the bias may interact with the redundancy penalty in unexpected ways at different dimensions; (3) the specific hash functions and seed (42) may produce pathological collisions at m = 16384 for the particular parameter shapes in GPT-2 Large. This is an unexpected negative result that the paper does not fully investigate.

Validation loss convergence (Figure 4): Both GPT-2 Large and GPT-2 XL show consistent validation loss improvement from OPUS throughout training, with the gap over baselines widening over the 30B-token budget rather than saturating. This suggests OPUS's benefit is not merely an early-training effect but compounds as the model learns — presumably because better data selection early leads to better representations, which in turn enable more accurate utility estimation, creating a virtuous cycle. The convergence speed analysis (OPUS reaching Random-60B validation loss at 17B tokens for GPT-2 XL) is computed from the intersection of the OPUS validation loss curve with the horizontal line at the Random-60B final validation loss.

Domain-level consistency (Table 6): Across all 10 domains in the perplexity analysis and all four model/optimizer/corpus configurations, OPUS achieves lower perplexity than random in 40 out of 40 domain comparisons (10 domains × 4 configurations). There are no domain-specific regressions, indicating that OPUS's selection does not systematically neglect any content category. The relative improvement is roughly uniform across domains, with no single domain showing dramatically outsized gains — this is consistent with the Bench-Proxy construction covering diverse benchmark subdomains through its maximum-similarity retrieval.

Negative result: GREATS degrades with scale and optimizer complexity. While not presented as a formal ablation, the GREATS performance pattern across Table 3 is an important robustness check for the paper's central thesis. On GPT-2 Large with AdamW, GREATS achieves 39.67 vs. random's 39.29 (+0.38). On GPT-2 XL with Muon, GREATS drops to 39.23 vs. random's 40.29 (−1.06). This interaction — where a raw-gradient method degrades both with model scale (Large → XL) and optimizer complexity (AdamW → Muon) — provides empirical support for the claim that optimizer-induced geometry becomes more important at larger scales and with more sophisticated optimizers. If raw-gradient scoring were merely imprecise but unbiased, we would expect it to maintain a consistent margin over random regardless of scale. The fact that it underperforms random at the larger scale suggests structural misalignment, not just noise.

Critical Assessment

Claim: OPUS achieves 8× reduction in computation on GPT-XL using FineWeb dataset (Figure 1). This claim requires careful interpretation. The "8× reduction" refers to the comparison in Figure 1 between OPUS at 30B tokens and Random at a higher token budget (likely 200B or 240B, though the exact number is stated in the abstract as "full 200B-token training"). The paper's main experiments only report Random at 30B and 60B tokens (Tables 3 and 4), not 200B or 240B, so the 8× figure presumably comes from additional experiments or extrapolation not fully detailed in the main results tables. The closest in-table evidence is the FineWeb-Edu result where OPUS on score-3 data (GPT-2 XL Muon, 44.99) substantially exceeds Random on score 4+5 data at twice the tokens (60B: 42.59). For the FineWeb result specifically, the 8× computation reduction is claimed relative to a baseline trained on 200B tokens with random selection — a comparison point not directly shown in the main experimental tables but referenced in the abstract. The reader should note that this 8× figure is the most aggressive efficiency claim and has weaker direct tabular support than the 2–6× claims.

Claim: OPUS outperforms industrial-level baselines and full 200B-token training. The "outperforms industrial-level baselines" claim is well-supported by Tables 3 and 4, where OPUS achieves the best compute-matched average across all configurations and all baseline categories (static, dynamic, random). The comparison against "full 200B-token training" requires the same caveat as above — this is referenced in the abstract but the 200B-token baseline does not appear in the main experimental tables. The 60B-token random reference in Tables 3 and 4 does show OPUS at 30B tokens often matching or exceeding random at 60B (particularly in the FineWeb-Edu setting), providing partial evidence at 2× data efficiency, but the jump to 8× (200B) is not directly tabled.

Claim: When combined with industrial-level static filters, OPUS further improves efficiency even with lower-quality data. The FineWeb-Edu experiments (Table 4) test this directly: OPUS on score-3 (mid-quality, i.e., data that would be discarded by a strict static filter) outperforms all baselines trained on score 4+5 (high-quality, i.e., data that passes a strict static filter). This holds for both GPT-2 XL (44.99 vs. 42.24 best static) and GPT-2 Large (41.72 vs. 41.43 best static). However, the paper does not test OPUS combined with a static filter as a pre-processing step — it tests OPUS instead of the static filter (on the unfiltered lower-quality pool) compared against static filtering alone (on the high-quality pool). The claim as stated in the abstract ("when combined with industrial-level static filters") slightly overstates what is tested — what Table 4 shows is that OPUS eliminates the need for aggressive static filtering, not that OPUS combined with static filtering improves over OPUS alone. The OPUS on score 4+5 rows in Table 4 provide a partial test: OPUS on score 4+5 achieves 42.42 (GPT-2 XL Muon), which is better than random on score 4+5 (41.92) but worse than OPUS on score 3 (44.99). This suggests that applying OPUS to an already-filtered pool may actually reduce performance by limiting the diversity of candidates available for dynamic selection — a nuanced finding that the paper does not fully discuss.

Claim: In continued pre-training, OPUS achieves superior performance using only 0.5B tokens compared to full training with 3B tokens. Supported by Figure 6 and the accompanying text: "OPUS reaches the best performance using only 0.5B tokens and already outperforms random CPT trained for 3B tokens." The domain breakdowns in Figure 5 provide per-domain evidence that the gains are distributed rather than driven by a single benchmark. However, the continued pre-training experiments use only a single model (Qwen3-8B-Base) and a single domain (SciencePedia), so the generality of the 6× efficiency claim across different base models and domains is untested.

Missing experiments that would strengthen the paper. Several experiments are notably absent. (1) Larger-scale from-scratch pre-training: all from-scratch experiments use GPT-2 scale models (774M–1.5B parameters). The paper does not test OPUS at the 7B+ scale where data selection overhead becomes proportionally smaller (since scoring cost scales with model size but training cost also scales) and where the benefits might be larger or smaller. (2) Longer training horizons: the 30B-token budget is modest for modern pre-training standards. It is unknown whether OPUS's benefits compound (the proxy signal gets better with better models) or saturate (the model eventually learns everything useful from any subset of the corpus). The validation loss curves in Figure 4 do not show saturation at 30B tokens, suggesting headroom, but this is not tested directly. (3) Alternative proxy construction methods: the paper compares benchmark-matched Bench-Proxy against a standard hold-out set (Table 7) but does not compare against other retrieval methods (e.g., using different embedding models, different similarity metrics, different budget sizes for the proxy pool) or against the raw benchmark validation set directly. The 30M-token proxy budget is fixed without sensitivity analysis. (4) Interaction between selection ratio ρ and buffer size N: the paper fixes ρ = 0.5 but sweeps N (Table 8). The complementary ablation — fixing N and sweeping ρ — is not reported. It is plausible that lower ρ (more selective) at larger N would outperform the default configuration, but this remains untested. (5) OPUS combined with Muon across all parameters: Table 1 shows that even in the "Muon" configuration, many parameters (embeddings, LM head, biases, norms) use AdamW. The claim that "optimizer-induced selection matters under Muon" is thus tested in a hybrid setting, not a pure-Muon setting. A pure-Muon configuration (if feasible) would provide a cleaner test. (6) Statistical significance: no confidence intervals, standard errors, or multiple-seed runs are reported. For the narrower margins (e.g., OPUS 40.25 vs. UltraFineweb 40.15 on GPT-2 Large Muon FineWeb, a +0.10 difference), it is impossible to determine whether the difference is statistically reliable or within noise.

Confounding factors in the optimizer comparison. The comparison between AdamW and Muon configurations (Table 3) is not purely about the optimizer — it is confounded by differences in learning rates, momentum coefficients, weight decay, the hybrid assignment of parameters to optimizers, and the matrix-size-dependent LR rescaling in Muon. The paper acknowledges this implicitly by providing full optimizer hyperparameters but does not discuss how these differences might affect the baseline random performance (which varies across optimizer configurations: 39.72 Muon vs. 39.29 AdamW for GPT-2 Large). The claim that "OPUS works under both AdamW and Muon" is supported by the pattern of improvement over random in both configurations, but the claim that "optimizer-induced scoring is especially important under Muon" (implied by the larger OPUS-vs-random gap under Muon: 41.75 vs. 40.29, +1.46, compared to AdamW: 41.01 vs. 40.36, +0.65) is confounded by the fact that random performance differs between configurations.

The 4.7% overhead claim requires context. Figure 7 shows that OPUS with ghost + CountSketch achieves 4.7% overhead compared to random. However, this figure is measured in total compute (PFLOPs) and wall-clock time for a specific configuration (GPT-2 XL, FineWeb, Muon). The overhead percentage depends on the ratio of scoring compute to training compute, which is affected by: (1) the scoring length (512 tokens) vs. training length (6,144 for GPT-2 XL) — changing this ratio from 1:12 to 1:6 would approximately double the overhead; (2) the buffer size N — shown in Table 8 but the overhead cost of larger N is not reported; (3) the model architecture — wider but shallower models have different ratios of linear layer parameters (where ghost + CountSketch applies) to other parameters (where full gradients may be needed). The 4.7% figure should be understood as a point estimate for the specific experimental configuration, not a universal guarantee.

Difficulty-generalization vs. benchmark-overfitting tension. The out-of-distribution benchmark results (Table 5) show OPUS outperforming baselines on held-out tasks, which suggests genuine generalization rather than narrow overfitting. However, the Bench-Proxy construction retrieves documents similar to all the in-distribution benchmarks (MMLU, ANLI, HellaSwag, PIQA, SIQA, WinoGrande, ARC-E, ARC-C, CommonsenseQA, WSC), which collectively cover a very broad range of knowledge, reasoning, and language understanding capabilities. It is possible that the proxy pool ends up covering such a broad semantic space that it approximates "high-quality diverse text" rather than being specifically tailored to the evaluation suite. In that case, the out-of-distribution benchmark gains would reflect a general improvement in data quality rather than task-specific alignment. This is not necessarily a weakness — general improvement is desirable — but it complicates the interpretation of Bench-Proxy's mechanism. An ablation comparing Bench-Proxy against a random in-distribution proxy that is equally diverse but not benchmark-aligned would help disentangle these effects, but is not reported.

Scale of the Qwen3-8B continued pre-training experiments. The continued pre-training experiments train on only 1.5B tokens (with evaluation at 0.5B, 1.0B, 1.5B). This is a very short adaptation budget for an 8B-parameter model. The 6× efficiency claim (0.5B OPUS matching 3B random) is impressive but should be contextualized: at 0.5B tokens, the model has seen only ~62.5 tokens per parameter, which is far below typical continued pre-training budgets. It is possible that the gains are largest in this extremely low-data regime and would diminish if training continued to more realistic adaptation budgets (e.g., 10–50B tokens). The paper's validation loss curves for GPT-2 models (Figure 4) show the gap over baselines widening through 30B tokens, which argues against saturation, but this is at a much higher tokens-per-parameter ratio (~38,700 for GPT-2 Large).

Overall assessment. The experiments provide strong and consistent evidence that OPUS outperforms prior static and dynamic selection methods at the tested scales (GPT-2 774M–1.5B, Qwen3-8B) and budgets (30B from-scratch tokens, 1.5B continued pre-training tokens). The FineWeb-Edu result — OPUS on lower-quality data outperforming baselines on higher-quality data — is the most convincing single experiment because it demonstrates a qualitative advantage rather than a marginal improvement. The efficiency claims of 2–6× are well-supported by within-table comparisons (30B vs. 60B token baselines in Tables 3 and 4; 0.5B vs. 3B in Figure 6). The 8× claim is referenced but less directly supported by the main experimental tables. The main weaknesses are: (1) lack of statistical error quantification, (2) absence of experiments at 7B+ scale, (3) no test of whether benefits compound or saturate at longer training horizons, (4) the confounding between optimizer type and other hyperparameters in the AdamW vs. Muon comparison, and (5) limited ablation of the Bench-Proxy construction (fixed 30M token budget, single embedding model, no comparison against raw benchmark gradients). The ablation studies are thorough for the core OPUS components (selection rule, temperature, sketch dimension, buffer size) but thinner for the proxy construction and the optimizer preconditioner design choices.

6. Limitations and Trade-offs

Scale of Experiments Falls Short of Modern Pre-Training Regimes

The assumption or constraint. All from-scratch pre-training experiments use GPT-2 scale models (774M–1.5B parameters) with a budget of 30B update tokens. The paper's boldest efficiency claim — "outperforms … even full 200B-token training" (Abstract) — extrapolates well beyond these experimental conditions. The largest model tested is Qwen3-8B-Base, but only in a continued pre-training setting with a short 1.5B-token budget, not in a from-scratch regime that would test OPUS's utility scoring across the full training trajectory.

The paper does not directly state this as a limitation, but the experimental design implicitly constrains the scope: the GPT-2 family is chosen because it is "representative of the capabilities of many contemporary LLMs" (Section 6.1), a claim that is itself untested. The 30B-token budget is a design choice rather than a hard constraint, but scaling to 200B+ tokens with models at the 7B–70B scale would require substantially more compute than the paper's experiments.

The consequence. Several failure modes could emerge at larger scales that are invisible in the current experiments. First, the proxy signal quality might degrade as the model becomes more capable — when the model can already solve most tasks in the Bench-Proxy retrieval set, the utility of further alignment with those proxy directions may vanish, and OPUS could begin selecting data that reinforces already-mastered patterns rather than addressing remaining weaknesses. This is analogous to the "easy problem saturation" observed in the test-time compute scaling paper, where search over-optimizes the verifier signal when the model already performs well. Second, the 4.7% overhead measured on GPT-2 XL (Figure 7) depends on the ratio of scoring compute to training compute, which is strongly affected by the scoring length (512 tokens) versus training length (6,144 tokens for GPT-2 XL). At larger scales, training sequence lengths are typically 4K–8K tokens, making the scoring-to-training length ratio closer to 1:8–1:16, similar to the tested range. However, the absolute memory and communication cost of collecting ghost factors across all GPUs in a large distributed training setup is not tested — at 70B scale with tensor parallelism, the CountSketch projection and score gathering would require cross-device communication that may become a bottleneck not captured in the PFLOPs-only analysis.

Third, the candidate buffer size is fixed at N = 32 (or 64 in some ablations) regardless of model scale. At 70B+ parameters with global batch sizes in the millions of tokens, a candidate buffer of 32 sequences represents a tiny fraction of each step's available data. The optimal buffer size at larger scales might be substantially larger, increasing the relative scoring overhead, or the selection ratio ρ = 0.5 might need adjustment to maintain meaningful selectivity. The paper's ablation (Table 8) shows N = 64 outperforming N = 32 by +0.50 on GPT-2 Large, and the trend with scale is unknown.

What evidence exists in the paper. The paper contains no from-scratch experiments above 1.5B parameters and no training budgets above 60B tokens (the non-compute-matched Random-60B reference). The continued pre-training experiments on Qwen3-8B provide the only signal at larger scale, but these are domain-adaptation runs (1.5B tokens, ~188 tokens per parameter) rather than general pre-training, and they do not test OPUS's utility scoring behavior through the full training trajectory. The validation loss curves in Figure 4 do not saturate at 30B tokens, which provides weak evidence that benefits might continue at longer horizons, but the extrapolation from 30B to 200B is speculative.

Mitigation status. The paper does not attempt to address this limitation. There is no discussion of how OPUS's components might scale with model size, no projection of overhead at 7B+ scale, and no experiments that vary the training budget independently of other factors. The 8× computation reduction claim in the abstract implicitly assumes that the 30B-token results generalize to 200B-token regimes, which is an untested assumption.

Difficulty Estimation and Proxy Construction Costs Are Unaccounted For in Efficiency Claims

The assumption or constraint. OPUS requires a pre-constructed proxy pool D_proxy built via Bench-Proxy retrieval (Section 5.1, detailed in Section 6.2). This construction involves: (1) embedding every pre-training document and every benchmark sample using a frozen sentence embedding model (Arctic-Embed-L v2), (2) computing cosine similarities between all document-benchmark pairs, (3) taking the maximum similarity per document, and (4) sorting and accumulating documents until a 30M-token budget is reached. For a 3T-token corpus like FineWeb, this is a substantial one-time cost — embedding 3T tokens of text, computing billions of similarity scores, and sorting the results — that is not included in any of the paper's compute efficiency calculations.

The paper is transparent about the proxy construction procedure (Section 6.2) but does not report its computational cost. The overhead analysis in Figure 7 measures only the per-step scoring overhead during training (the 4.7% figure), explicitly excluding the one-time proxy construction cost. The paper also does not discuss whether the proxy pool would need to be reconstructed for different model architectures, different pre-training corpora, or different downstream evaluation suites.

The consequence. The headline efficiency numbers — 4.7% overhead, 4×–8× compute reduction, 6× data efficiency — all measure OPUS's cost relative to training after the proxy pool is available. In a realistic deployment where the proxy pool must be constructed from scratch, the total cost would include this one-time expense. For a project training a single model, the proxy construction cost might be small relative to the total training cost (embedding 3T tokens is analogous to one pass over the corpus, whereas training involves multiple passes or updates on 30B+ tokens). But the economics change in several practical scenarios:

Multi-model training: if an organization trains multiple models from the same corpus (e.g., architecture ablations, hyperparameter sweeps), the proxy construction cost is amortized over many runs and becomes negligible. This is the favorable case.
Single-model training with limited budget: for a team that can only afford to train one model, the proxy construction cost could represent a significant fraction of the total compute budget, potentially negating the 4.7% advantage over simply training on more data with random selection.
Changing evaluation suites: if downstream benchmarks change (new tasks added, different evaluation protocols), the Bench-Proxy pool must be reconstructed, incurring the full embedding and retrieval cost again.
Different pre-training corpora: switching from FineWeb to a new corpus requires re-embedding the entire new corpus, which may be impractical for frequently-updated web-scale datasets.

The paper's difficulty estimation (Section 3.2 of the earlier test-time compute paper) has an analogous unaccounted cost — generating 2048 samples per question to estimate difficulty before allocating the inference budget. In the test-time compute paper, the authors explicitly flag this as a limitation and call for cheap difficulty prediction. OPUS faces a structurally similar accounting gap but does not discuss it.

What evidence exists in the paper. The paper provides no measurement of the Bench-Proxy construction cost. Section 6.2 describes the procedure (embedding model, similarity scoring, sorting, accumulation to 30M tokens) but does not report the embedding time, the similarity computation time, the memory requirements for storing embeddings of 3T tokens, or the total FLOPs. Figure 7 shows only per-step training overhead, not the one-time proxy construction cost. The ablation in Table 7 compares benchmark-matched vs. standard proxy but does not report the cost of constructing either.

Given the paper's detailed reporting of per-step overhead (4.7%, measured in minutes and PFLOPs in Figure 7), the absence of any proxy construction cost measurement is a notable omission. A rough estimate: embedding 3T tokens with Arctic-Embed-L v2 (a 568M-parameter model) would require approximately 3 × 10^12 tokens × 2 × 568 × 10^6 parameters × forward pass operations per parameter ≈ 3.4 × 10^21 FLOPs for the embeddings alone, plus similarity computation and sorting. By comparison, training GPT-2 XL on 30B tokens costs roughly 30 × 10^9 tokens × 6 × 1.5 × 10^9 parameters ≈ 2.7 × 10^20 FLOPs. So the proxy construction cost might be roughly 10× larger than the entire 30B-token training run, making the "4.7% overhead" figure highly misleading when proxy construction is included.

Mitigation status. The paper does not address this limitation. There is no discussion of amortization, no suggestion that a lightweight proxy construction method could reduce the one-time cost, and no measurement of the cost in any form. This is the most significant unacknowledged cost in the paper, and it directly affects the practical deployability claims.

CountSketch Projection Performance Is Non-Monotonic and Poorly Understood

The assumption or constraint. OPUS relies on CountSketch projections (Section 5.2) to approximate inner products between high-dimensional effective update vectors, enabling utility scoring without materializing full per-sample gradients. The key assumption is that CountSketch provides an unbiased estimator of these inner products with variance that decreases as the sketch dimension m increases, and that choosing m = 8192 provides sufficient accuracy for ranking candidates.

The paper specifies that CountSketch uses a deterministic hash/sign pair with a fixed seed (42) and a sketch dimension of m = 8192 for all experiments. The theoretical property — E[⟨Π(v₁), Π(v₂)⟩] = ⟨v₁, v₂⟩ — guarantees unbiasedness, but the variance of individual estimates depends on the specific hash functions, the sketch dimension, the sparsity of the vectors being sketched, and potential collisions where different coordinates hash to the same bucket.

The consequence. The most concerning experimental result for this assumption appears in Table 8: sweeping the sketch dimension m on GPT-2 Large with Muon on FineWeb yields accuracy of 39.63 at m = 4096, 40.25 at m = 8192, and then a drop to 39.34 at m = 16384. This is not merely a case of diminishing returns — performance gets substantially worse when the sketch dimension is increased by 2×. This is the opposite of what standard CountSketch theory predicts: larger sketch dimensions should reduce variance and improve estimate quality.

This non-monotonicity has several troubling implications. First, it suggests that OPUS's utility estimates may be biased in practice due to interactions between CountSketch errors and the sequential Boltzmann sampling process. Theoretically, CountSketch provides unbiased inner product estimates, but when these estimates are used in a sequential decision process (sampling the j-th candidate based on utility scores that depend on previous selections, which themselves were based on CountSketch-estimated utilities), the errors can compound in ways that are not captured by the per-estimate unbiasedness guarantee.

Second, it implies that the choice of m = 8192 is not obviously optimal — it happens to work best in this specific configuration (GPT-2 Large, Muon, FineWeb), but there is no principled way to select m for a new model scale or architecture without running expensive sweeps. For a practitioner deploying OPUS on a 70B model, the optimal sketch dimension might be entirely different, and the paper provides no guidance for extrapolating.

Third, the result raises questions about the robustness of the fixed-seed deterministic hash functions. If the m = 16384 hash configuration happens to produce pathological collisions for the specific weight matrix shapes in GPT-2 Large, the performance drop might be an artifact of that specific hash rather than a fundamental property of larger sketch dimensions. The paper does not test multiple random seeds to determine whether the non-monotonicity is robust or seed-dependent.

What evidence exists in the paper. Table 8 reports the sketch dimension sweep. There is no further analysis of why m = 16384 underperforms, no measurement of CountSketch approximation error (e.g., correlation between true inner products and sketched estimates), and no variance decomposition to understand whether the degradation comes from the proxy alignment term or the redundancy penalty term. The paper does not report the effective compression ratio at different sketch dimensions, the expected collision rate, or any diagnostic metrics that would help explain the non-monotonic behavior.

Mitigation status. The paper acknowledges the sensitivity only implicitly by reporting the sweep in Table 8, but does not discuss the non-monotonicity, does not hypothesize causes, and does not suggest diagnostics or remedies. It simply notes that "m = 8192 provides the strongest results among the tested dimensions" and adopts it as the default. There is no investigation of alternative sketch structures (e.g., Johnson-Lindenstrauss random projections, which have more predictable variance scaling) that might provide more reliable monotonic improvement with dimension.

Redundancy Penalty Relies on an Isotropic Hessian Approximation That Ignores Loss Landscape Geometry

The assumption or constraint. The redundancy penalty term in OPUS's utility score (Equation 22) is derived from a second-order Taylor expansion of the validation loss reduction, but the Hessian H_val(t) is approximated as the identity matrix: H_val ≈ I. This means the penalty treats all directions in parameter space as having equal curvature — if two candidates move parameters in similar directions, their marginal utility is reduced by the same amount regardless of whether those directions lie in sharp valleys (where redundancy is genuinely harmful) or flat plateaus (where redundancy is less problematic).

The paper justifies this approximation by noting that materializing the full Hessian is intractable (Section 5.1) and that the identity approximation "correctly captures the sign of the redundancy effect even if it gets the magnitude wrong." This is a pragmatic but strong simplification. In real loss landscapes, the curvature varies enormously across parameter directions — some directions have very high curvature (small parameter changes cause large loss changes), while others are nearly flat (large parameter changes cause negligible loss changes). Training on redundant samples in high-curvature directions can cause oscillations or slow convergence, while redundancy in flat directions is essentially harmless.

The consequence. The isotropic approximation means OPUS's redundancy penalty is equally aggressive in all directions, potentially over-penalizing candidates that are similar to existing selections in flat (harmless) directions while under-penalizing candidates that are similar in sharp (harmful) directions. This could lead to suboptimal batch composition: the Boltzmann sampler might avoid a candidate that overlaps with previous selections in a flat direction (where the overlap is actually benign), while selecting a candidate that overlaps in a sharp direction (where the overlap could cause training instability).

More subtly, the isotropic approximation interacts problematically with the AdamW preconditioner. AdamW already applies a diagonal preconditioner that rescales coordinates based on historical gradient variance — this effectively makes the loss landscape appear more isotropic in the optimizer's effective update space (the space where OPUS computes utilities). By additionally applying an identity Hessian approximation, OPUS is effectively assuming that after AdamW's diagonal rescaling, the remaining curvature is isotropic — an assumption that may be reasonable for well-conditioned problems but has no theoretical justification and is not empirically validated.

The consequence is that the redundancy penalty is a heuristic rather than a principled curvature-aware term. It is directionally correct (similar candidates are penalized) but may be quantitatively wrong in ways that affect batch composition, especially when the loss landscape has strong anisotropic curvature, as is typical in early training or during distribution shifts.

What evidence exists in the paper. The paper provides no direct evidence about the quality of the isotropic Hessian approximation. There is no measurement of how well H_val ≈ I captures true loss curvature (e.g., by comparing predicted vs. actual loss reduction when adding a sample to a batch), no comparison against a diagonal Hessian approximation (e.g., using the Fisher information diagonal, which would be expensive but informative), and no sensitivity analysis of how the penalty term weight (controlled by η_t^2) affects performance. The comparison between greedy selection (no stochasticity, but still using the redundancy penalty with H_val ≈ I) and full OPUS (Table 7: 40.49 vs. 41.75) provides indirect evidence — the +1.26 improvement from adding Boltzmann sampling suggests that the isotropic penalty alone is insufficient for optimal diversity, which is consistent with the penalty being an imperfect curvature model. But this is weak evidence; the improvement could equally be explained by noise robustness rather than curvature mis-estimation.

Mitigation status. The paper acknowledges the approximation is crude (calling it the "simplest possible approximation" in the walkthrough) but does not discuss its limitations or suggest alternatives. The GREATS paper (Wang et al., 2024) uses the same identity approximation, and OPUS inherits it without further analysis. No future work is suggested on improving the redundancy penalty's curvature model, which is a notable gap given how central the penalty term is to OPUS's diversity preservation.

No Evaluation Across Diverse Domains, Languages, or Data Mixtures — Only Monolingual English Web Text and Science

The assumption or constraint. All experiments use English-language text from a single web corpus (FineWeb/FineWeb-Edu) or a single scientific domain (SciencePedia). The pre-training setup is a standard autoregressive language modeling objective on token sequences. The paper does not test on multilingual corpora, code-heavy datasets, heavily structured data (e.g., tables, JSON), or data mixtures that reflect the heterogeneous composition of frontier LLM pre-training (e.g., mixing web text, code, books, Wikipedia, scientific papers in different proportions).

The Bench-Proxy construction uses benchmarks that are exclusively English-language and focused on knowledge, commonsense reasoning, and language understanding. There is no evaluation of how OPUS would perform if the target proxy direction represented multilingual capabilities, code generation, mathematical reasoning, or other domains where the relationship between pre-training data and downstream performance may differ from the knowledge-and-reasoning benchmarks tested.

The consequence. LLM pre-training at frontier scale uses carefully curated data mixtures — for example, the Llama 3 and Qwen technical reports describe mixing web text, code, mathematical content, and multilingual data in specific ratios to achieve balanced capabilities. OPUS, as a dynamic selector, would face a fundamentally different challenge in these mixtures: the utility scores depend on the proxy gradient direction, which encodes preferences for certain types of data. If the proxy is constructed from English-language benchmarks, OPUS might systematically down-weight non-English or code data, leading to catastrophic capability regression in untargeted domains. The paper's experimental design cannot detect this failure mode because it operates in a single-domain setting (English web text or English science text).

More subtly, even within the English web text domain, the "quality" signal that Bench-Proxy encodes may interact with data mixture decisions in unpredictable ways. For example, if the pre-training corpus contains a substantial fraction of code (which is high-quality but syntactically different from natural language), Bench-Proxy retrieval might assign low similarity scores to code documents (since the benchmarks are natural language tasks), causing OPUS to systematically under-select code data. In a standard static mixture, this could be compensated by oversampling code during pre-training, but OPUS as configured would fight against this compensation, potentially degrading code capabilities relative to a static approach.

What evidence exists in the paper. The out-of-distribution benchmark results (Table 5) provide some evidence of generalization — OPUS improves on BBH, RACE, and StoryCloze, which were not used in Bench-Proxy construction. However, these benchmarks are still English-language, knowledge-and-reasoning tasks that are distributionally similar to the in-domain benchmarks. They do not test code generation, multilingual translation, mathematical theorem proving, or other capabilities that frontier models are expected to possess. The continued pre-training experiments on SciencePedia test domain transfer within English, but not cross-lingual or cross-modality transfer. There are no experiments with data mixtures and no analysis of whether OPUS's selections over- or under-represent specific content types relative to a random baseline.

Mitigation status. The paper does not discuss this limitation. The conclusion (Section 7) mentions extending OPUS to "richer training regimes, such as data mixtures" as a natural next step, but does not pre-register the concern that OPUS might fail in mixture settings without modifications. A practitioner attempting to deploy OPUS in a mixed-corpus pre-training pipeline would have no guidance on how to construct the proxy pool to balance multiple capability objectives, how to prevent capability regression in untargeted domains, or how to calibrate the utility scoring when different data types have different gradient norms and structures.

OPUS Never Combines the AdamW and Muon Preconditioner Paths in a Unified Selection — Hybrid Optimizer Assignment Creates Unmodeled Interactions

The assumption or constraint. In the Muon experiments, OPUS applies Muon preconditioning to matrix-shaped parameters in Transformer blocks and AdamW preconditioning to all other parameters (embeddings, LM head, biases, normalization weights), as specified in Table 1. The utility score (Equation 25) sums contributions across all layers in R, with each layer receiving its own preconditioner (diagonal for AdamW-updated layers, dense for Muon-updated layers). This summation implicitly assumes that utility contributions are additive across layers with different optimizer geometries — that the benefit of an aligned update in the embedding layer (AdamW-preconditioned) combines linearly with the benefit of an aligned update in an attention projection layer (Muon-preconditioned).

This linear combination is not theoretically justified. The two preconditioners induce fundamentally different update spaces — AdamW rescales coordinates, Muon orthogonalizes entire matrices. The effective step in parameter space is the concatenation of these differently-transformed updates, and the validation loss reduction depends on their joint effect, which may include cross-layer interactions that the per-layer additive utility decomposition cannot capture. For example, a candidate that produces a strongly aligned update in an AdamW layer but an orthogonal update in a Muon layer would receive a high utility score (since the AdamW contribution dominates), but the actual loss reduction might be smaller than predicted if the Muon layer's update pushes the model in a direction that partially cancels the AdamW layer's benefit.

The consequence. The utility scores in the Muon experiments are hybrids of two different optimizer geometries, with no mechanism for modeling their interaction. This creates two potential problems. First, score miscalibration: the relative scaling of AdamW-layer contributions vs. Muon-layer contributions in the utility sum depends on the learning rate, the per-layer parameter count, the gradient norms in each layer, and the spectral properties of the Muon preconditioner. The paper does not introduce any normalization or weighting scheme to balance these contributions — they are simply summed with the same η_t and η_t^2 coefficients. If one type of layer dominates the utility sum (e.g., because Muon-updated layers have more parameters and contribute larger dot products), the selection may be effectively driven by only one optimizer's geometry, negating the claimed benefit of optimizer-awareness.

Second, the ablation interpretation is confounded: when comparing OPUS under AdamW-only vs. Muon+AdamW hybrid (Table 3), the difference in performance reflects both (a) the effect of using Muon for matrix parameters and (b) the effect of OPUS scoring in a hybrid geometry space. Is the larger OPUS-vs-random gap under Muon (GPT-2 XL: +1.46) compared to AdamW (GPT-2 XL: +0.65) due to OPUS's optimizer-awareness being more important under Muon, or due to the hybrid scoring geometry producing systematically different (and perhaps serendipitously better) selections? The paper cannot disentangle these effects without a pure-Muon baseline or an ablation where OPUS scores in raw-gradient space under the Muon optimizer (i.e., GREATS under Muon, which is reported and performs poorly, dropping below random on GPT-2 XL Muon: 39.23 vs. 40.29). The GREATS comparison provides some evidence that optimizer-awareness specifically helps under Muon, but the hybrid scoring introduces an unmodeled confound.

What evidence exists in the paper. The paper reports the full optimizer assignment in Table 1 and describes the hybrid setup in Section 6.1. The ablation studies in Table 8 use GPT-2 Large with Muon optimizer, meaning all reported ablations are in the hybrid scoring regime. There is no ablation that compares OPUS's per-layer utility decomposition against a unified scoring approach (e.g., computing a single preconditioner that operates on the concatenated parameter vector, which would be intractable but could be approximated). The paper does not report the per-layer contribution distribution to the total utility score, so the reader cannot assess whether the scores are dominated by Muon-preconditioned or AdamW-preconditioned layers.

Mitigation status. The paper does not discuss the additivity assumption or the hybrid scoring confound. The optimizer assignment is presented as a typical configuration for Muon training (which is true — hybrid Muon+AdamW is the standard usage), but the implications for OPUS's scoring mechanism are not analyzed. A practitioner implementing OPUS for a custom optimizer configuration would have no guidance on how to balance contributions from differently-preconditioned parameter groups or whether the additive utility decomposition remains valid.

7. Implications and Future Directions

How This Work Changes the Landscape

OPUS makes a conceptual shift rather than a paradigm revolution: it demonstrates that data selection scores must be computed in the optimizer-induced update geometry, not raw-gradient space. This is not an incremental refinement of existing scoring functions — it is a change in where scoring happens, with downstream consequences for how every dynamic selection method should be designed. Prior work (GREATS, MATES, perplexity-based selection) assumed SGD-like dynamics where the effective update direction equals the raw gradient. OPUS shows this assumption is not just approximate but structurally wrong for modern training, and correcting it yields consistent improvements across model scales, optimizers, and data quality tiers.

The magnitude of the shift is best understood through two empirical facts. First, GREATS (raw-gradient scoring) degrades with model scale and optimizer complexity: on GPT-2 Large with AdamW it marginally beats random (39.67 vs. 39.29, +0.38), but on GPT-2 XL with Muon it falls substantially below random (39.23 vs. 40.29, −1.06). This means raw-gradient methods become counterproductive at larger scales — a finding that should cause any practitioner using dynamic selection with modern optimizers to reexamine their scoring mechanism. Second, OPUS on lower-quality data (FineWeb-Edu score 3) with Muon achieves 44.99 average accuracy on GPT-2 XL, compared to the best static baseline on higher-quality data (DCLM-FastText, scores 4+5) at 42.24. This +2.75 point gap refutes the core assumption of static filtering — that data quality is an intrinsic property that should determine permanent inclusion or exclusion from the corpus. The limiting factor is not data quality but the intelligence of the selection mechanism.

The paper reconciles a subtle contradiction in the dynamic selection literature. Prior work showed both that influence-based selection helps (GREATS improved over random in some settings) and that simple heuristics like perplexity-based selection can be competitive or better (Ankner et al., 2025). OPUS resolves this by showing that the geometry of scoring matters more than the complexity of the scoring function: GREATS uses a more principled objective than perplexity-based selection (one-step validation loss reduction vs. loss magnitude), but GREATS computes that objective in the wrong space. When the same principled objective is computed in the optimizer's geometry, it substantially outperforms both raw-gradient GREATS and simple heuristics. The lesson is not that principled objectives are unnecessary — it is that principled objectives must be aligned with the optimizer that actually trains the model.

The work also redirects research attention in two ways. More attractive: improving verifier/proxy quality becomes a higher-priority investment than inventing more sophisticated scoring functions. The paper's ablation (Table 7) shows that switching from a standard proxy to a benchmark-matched proxy improves performance by +0.72 points, comparable to the gain from optimizer-aware scoring over raw-gradient scoring (+0.84 over GREATS on GPT-2 Large Muon). This suggests that proxy construction — the questions of what target direction to align with, how to construct it, and how to keep it stable — is at least as important as the scoring mechanism itself. Less attractive: developing increasingly complex influence-function approximations (e.g., Hessian-based corrections, Shapley value decompositions) may be misallocated effort if those approximations continue to operate in raw-gradient space. The +0.84 gain from optimizer awareness on GPT-2 Large Muon (OPUS vs. GREATS) likely exceeds what could be obtained from a more accurate Hessian approximation under raw-gradient scoring, given the poor scaling behavior of GREATS at larger model sizes.

Perhaps the most practically significant shift is the finding that dynamic selection can extract more value from lower-quality data than static filtering can from higher-quality data (Table 4). This reframes the data wall conversation: rather than asking "how do we find more high-quality text before we run out?", organizations should ask "how do we build selectors that can productively use the mid-quality text we already have in abundance?" The paper estimates that high-quality public text may be exhausted by 2026–2028. If dynamic selection can make score-3 data (typically hundreds of billions to trillions of tokens in web corpora) produce training signals comparable to or better than score 4–5 data under static filtering, the effective data wall recedes substantially — the constraint shifts from data availability to selection intelligence.

Follow-Up Research This Work Enables

Scaling OPUS to 7B–70B parameter from-scratch pre-training with 200B+ token budgets. The paper's from-scratch experiments stop at GPT-2 XL (1.5B parameters, 30B tokens). The validation loss curves in Figure 4 show the gap between OPUS and baselines widening through 30B tokens — saturation is not reached. A natural follow-up would train a 7B model on 200B–500B tokens with OPUS selection, measuring whether the efficiency gains compound (the proxy signal improves as the model becomes more capable, enabling even better selection) or saturate (at some point, all useful data in the corpus has been consumed, and further selection provides diminishing returns). This experiment would also stress-test the 4.7% overhead claim at scale: as model dimension grows, the CountSketch projection cost scales with the number of linear layer parameters, while the training cost scales similarly — but communication overhead for gathering scores across GPUs in a large distributed setup may become a bottleneck not captured in the paper's single-node PFLOPs analysis. A strong follow-up would report wall-clock time overhead at 7B and 70B scales with realistic distributed training configurations, measuring whether the overhead percentage remains in the single digits.

Measuring and mitigating CountSketch non-monotonicity. Table 8 reveals an unexplained and concerning result: doubling the sketch dimension from m = 8192 to m = 16384 causes performance to drop from 40.25 to 39.34 on GPT-2 Large with Muon. Standard CountSketch theory predicts monotonic improvement (lower variance) with larger sketch dimensions. A diagnostic follow-up would: (1) compute the actual correlation between true inner products (computed exactly on a subset of parameters) and CountSketch estimates at different sketch dimensions, measuring whether m = 16384 produces higher or lower correlation than m = 8192; (2) test multiple random seeds for the hash functions to determine whether the performance drop at m = 16384 is seed-specific (pathological collisions in the specific hash assignment) or systematic; (3) compare CountSketch against alternative sketching methods (Johnson-Lindenstrauss random projections, which have more predictable variance scaling but higher computational cost for dense projections) to determine whether the non-monotonicity is specific to CountSketch's sparse hash structure or a general property of low-dimensional projection under sequential Boltzmann sampling. If the non-monotonicity is systematic, it raises fundamental questions about whether sketch-based utility estimation is reliable for sequential decision processes with feedback loops (where the j-th selection's utility depends on sketched estimates of previous selections).

Understanding and addressing proxy construction cost. Section 6.2 describes Bench-Proxy construction (embed all pre-training documents with Arctic-Embed-L v2, compute pairwise similarities with benchmark samples, sort, accumulate to 30M tokens) but reports no cost measurement. A critical follow-up would: (1) measure the total FLOPs and wall-clock time for proxy construction on the 3T-token FineWeb corpus, comparing it against the 30B-token training cost to determine the amortization break-even point (how many models must be trained from the same corpus before the one-time proxy cost is negligible?); (2) test cheaper proxy construction methods — e.g., using a much smaller embedding model, subsampling the corpus before embedding, or using a pre-computed document embedding index if one exists — measuring whether proxy quality (as reflected in downstream benchmark performance) degrades gracefully with construction cost; (3) evaluate whether a randomly-selected in-distribution proxy of the same size (30M tokens) performs comparably to the retrieval-based proxy, which would eliminate the embedding and similarity computation cost entirely if true. The ablation in Table 7 compares benchmark-matched vs. standard proxy (both 30M tokens) and shows a +0.72 gap, but "standard proxy" is described as the validation split, not a random in-distribution subset — the comparison that would isolate the retrieval benefit vs. the in-distribution benefit is not reported.

Bench-Proxy for multi-objective data mixtures. Frontier LLMs train on mixtures of web text, code, mathematics, multilingual data, and scientific text. OPUS as currently configured uses a single Bench-Proxy pool constructed from English-language knowledge and reasoning benchmarks, which would systematically under-weight non-English and code data. A follow-up would construct multi-objective Bench-Proxy pools by retrieving documents aligned with different capability categories (e.g., coding benchmarks, multilingual benchmarks, math benchmarks) and combining the proxy gradients via a weighted sum, where the weights reflect the desired capability balance. The critical experiment: train a 7B model on a realistic 5-way data mixture (e.g., 50% web text, 20% code, 15% math, 10% multilingual, 5% science) with OPUS selection, measuring downstream performance on benchmarks for all five categories against both a random-sampling baseline and a static-mixture baseline. If OPUS with a balanced multi-objective proxy maintains or improves all five capability axes, it demonstrates that dynamic selection can replace hand-tuned data mixtures. If it improves some axes at the cost of regressing others, it reveals a fundamental tension between single-proxy utility optimization and multi-capability balance that would require new proxy aggregation methods.

Dynamic proxy construction that adapts during training. OPUS constructs D_proxy once before training and holds it fixed. But as the model learns, the optimal proxy direction likely changes — early in training, alignment with basic syntax and factual knowledge proxies might be most valuable; later, alignment with reasoning and inference proxies might become more important. A follow-up would implement dynamic proxy pools that are periodically reconstructed (e.g., every 5B tokens) using the current model's own representations (rather than a frozen embedding model) to retrieve documents, exploiting the fact that the model's internal similarity function becomes more task-relevant as it trains. The risk is that using the model's own representations creates a feedback loop — the proxy pool reinforces whatever the model has already learned — so the follow-up would need to measure whether dynamic proxy construction maintains proxy diversity or gradually collapses to a narrow set of "easy" documents that the model already predicts well. A strong experiment would compare fixed proxy (OPUS default) against periodically-refreshed proxy at multiple refresh intervals, measuring both aggregate benchmark performance and per-benchmark performance to detect whether certain capabilities are systematically neglected over time.

Testing the isotropic Hessian approximation against diagonal alternatives. The redundancy penalty in Equation 22 uses H_val ≈ I, which treats all parameter-space directions as having equal loss curvature. This is known to be false — some directions have much sharper curvature than others — and the penalty's effectiveness depends on how badly this approximation distorts the true marginal utility. A follow-up would implement a diagonal Hessian approximation using the Fisher information diagonal (which is computable as the expected squared gradient, already estimated by AdamW's second-moment buffer v_t) and compare it against the identity approximation in an ablation. If the Fisher-diagonal penalty outperforms the identity penalty, it suggests that curvature awareness matters and that the AdamW preconditioner does not fully compensate for anisotropic curvature. If the identity penalty performs comparably or better, it suggests that the AdamW preconditioner already normalizes the effective curvature enough that the identity approximation is sufficient — which would be a practically important null result validating OPUS's current design. The experiment is computationally feasible because the Fisher diagonal is already maintained by AdamW (it is v_t before bias correction), and the only change is replacing I with Diag(v_t) in the redundancy penalty term.

Practical Applications and Downstream Use Cases

Cost-efficient pre-training with fixed token budgets. For organizations with a fixed token budget (e.g., 100B tokens of training compute available), OPUS provides a concrete recipe for maximizing downstream performance without increasing total FLOPs. On GPT-2 XL with Muon on FineWeb, OPUS at 30B tokens achieves 41.75 average accuracy compared to random at 30B (40.29) and even random at 60B (41.29). This means OPUS with a 30B-token budget exceeds a 60B-token budget with random selection — a 2× data efficiency gain at equal or better performance. For a team deciding between (a) training on 100B tokens with random selection or (b) training on 100B tokens with OPUS selection, the OPUS option would (extrapolating from the 30B results) yield model quality equivalent to training on 200B+ tokens with random selection. The tradeoff is the 4.7% per-step overhead plus the one-time proxy construction cost. If the team trains multiple models from the same corpus (e.g., architecture ablations), the proxy cost is amortized and OPUS becomes strictly preferable. If they train a single model, the proxy construction cost must be weighed against simply training on more tokens — the break-even depends on the ratio of proxy construction FLOPs to training FLOPs, which the paper does not measure.

Domain adaptation with severely limited data. For specialized domain adaptation where target-domain data is scarce and expensive to acquire (e.g., medical literature requiring expert curation, legal documents with licensing restrictions), OPUS's 6× data efficiency gain on Qwen3-8B-Base (0.5B OPUS tokens matching 3B random tokens on SciAssess) translates directly to reduced data acquisition costs. If an organization can only obtain 500M tokens of in-domain text, OPUS would (based on the SciencePedia results) achieve downstream accuracy equivalent to training on 3B tokens with random selection — a value that would otherwise require purchasing or curating 6× more data. This is particularly impactful for low-resource languages or specialized technical domains where the total available in-domain text is limited to a few billion tokens: OPUS could make effective pre-training feasible on corpora that would be too small for standard random-selection approaches.

Mid-quality data utilization in web-scale pre-training. Static filtering pipelines (FineWeb-Edu, DCLM) typically assign quality scores and discard documents below a threshold — for FineWeb-Edu, score-3 documents are often filtered out in favor of scores 4–5. The paper's FineWeb-Edu result (OPUS on score-3 data: 44.99 for GPT-2 XL Muon vs. best static on score 4+5: 42.24) demonstrates that this filtering discards valuable training signal. Organizations with existing static filtering pipelines can adopt OPUS to recover value from their discarded mid-quality pools — instead of training only on the high-quality subset, they can include the mid-quality subset and let OPUS dynamically decide when and how to use it. This effectively increases the usable corpus size without requiring new data acquisition. For a corpus like FineWeb-Edu where the score-3 bucket is 120B tokens and the score 4+5 bucket is 80B tokens, switching from static filtering (80B tokens) to OPUS on the combined pool (200B tokens) could increase effective training data by 2.5× while maintaining or improving data quality through intelligent selection.

When to Prefer This Method

Prefer OPUS over static filtering when:

The pre-training corpus has a substantial mid-quality fraction (e.g., 40%+ of documents fall into "usable but not excellent" quality tiers) that static filters would discard.
Downstream evaluation targets are known in advance and span diverse capabilities, making Bench-Proxy construction valuable (retrieval aligns proxy with tasks of interest).
Training from scratch with a fixed token budget, and the goal is maximizing benchmark performance rather than minimizing wall-clock time (the 4.7% overhead is acceptable).
Multiple models will be trained from the same corpus, amortizing the one-time proxy construction cost.

Prefer OPUS over other dynamic methods (GREATS, MATES) when:

Training with modern optimizers — especially Muon, where the gap between raw-gradient scoring and optimizer-aware scoring is largest (GREATS drops from 39.23 to 40.29 random on GPT-2 XL Muon, while OPUS reaches 41.75).
Model scale exceeds ~1B parameters, where the GREATS pattern suggests raw-gradient scoring degrades relative to random (this threshold is approximate; the paper only tests 774M and 1.5B).

Prefer static filtering or random selection when:

The total training token budget is extremely large relative to the proxy construction cost, and proxy construction has not been performed (the one-time cost may negate the per-step efficiency gain for single-model training).
The downstream evaluation suite is unknown or changes frequently, making Bench-Proxy construction targetless or requiring frequent reconstruction.
Training must optimize for wall-clock time rather than token efficiency, and the 4.7% overhead is unacceptable (e.g., tight iteration cycles during architecture search).
The pre-training corpus is already heavily filtered and homogeneous, leaving little room for dynamic selection to improve over random sampling (the FineWeb result showed smaller gains than FineWeb-Edu, suggesting OPUS benefits more from heterogeneous corpora with variable quality).