Fantastic Pretraining Optimizers and Where to Find Them

ArXiv: 2509.02046

🎯 Pitch

Most claimed optimizer speedups over AdamW vanish once you simply tune AdamW's learning rate—an overlooked 2× improvement on standard baselines. Even properly tuned, the top alternatives like Muon see gains shrink from 1.4× to just 1.1× as models scale to 1.2B parameters.

1. Executive Summary

This paper systematically benchmarks ten deep learning optimizers against AdamW for large language model pretraining, evaluating across four model scales (0.1B–1.2B parameters) and four data-to-model ratios (1–8× Chinchilla) using a Llama 2 architecture and OLMo 2-style data mixture. The core contribution is the finding that proper hyperparameter tuning — using coordinate descent over scaling-sensitive hyperparameters (learning rate, weight decay, warmup, batch size) rather than transferring hyperparameters between optimizers — eliminates most reported speedups, with the best alternatives (matrix-based methods like Muon and Soap) providing at most 1.4× speedup over well-tuned AdamW on small models, diminishing to only 1.1× at 1.2B parameters (matching AdamW's 8× Chinchilla loss within roughly 10% fewer tokens). The paper establishes that matrix-based optimizers consistently outperform scalar-based optimizers (Muon, Soap, and Kron all achieving ~1.3× speedup over AdamW at 130M parameters, versus scalar-based methods like NAdamW and Mars converging to similar performance), but that this advantage reverses under high data-to-model ratios — Soap and Kron surpass Muon at 16× Chinchilla — and a fitted scaling law predicts Muon may underperform AdamW at 7B parameters, demonstrating that optimizer rankings are neither universal nor stable across pretraining regimes.

2. Context and Motivation

The Core Problem: We Don't Know Which Optimizers Actually Work

The fundamental question this paper tackles is simple to state but difficult to answer: if someone claims a new optimizer delivers 2× faster pretraining than AdamW, should you believe them? Prior to this work, the answer was unclear. Since 2023, at least a dozen papers have introduced novel optimizers for LLM pretraining — Sophia (Liu et al., 2024a), Soap (Vyas et al., 2025), Muon (Jordan et al., 2024; Liu et al., 2025a), MARS (Yuan et al., 2025), Cautious AdamW (Liang et al., 2025), and others — all claiming speedups ranging from 1.4× to over 2× compared to AdamW. Yet, as the paper notes with pointed understatement, "these optimizers have not yet been widely adopted in real-world pretraining" (Section 1), with the singular exception of Kimi K2's use of Muon-clip (Team et al., 2025). This gap between claimed performance and practical adoption signals a deeper problem: the evaluation methodology for pretraining optimizers is broken, and the field lacks confidence in reported speedups.

This matters enormously because pretraining dominates the cost of modern LLM development. The paper cites DeepSeek V3 (DeepSeek-AI et al., 2025b), where pretraining accounts for over 95% of the total training budget, with the additional RL training cost in DeepSeek R1 being "comparatively much smaller" (Section 1). Even a genuine 1.3× speedup in pretraining — if real and scalable — could translate to millions of dollars saved per training run for frontier models. But adopting an optimizer based on inflated claims risks worse outcomes: wasted engineering effort, regressions in final model quality, or training instability at scale. The paper is motivated by the conviction that the optimizer evaluation literature needs the same kind of rigorous re-examination that Jiang et al. (2019) brought to generalization measures and that Schmidt et al. (2021) brought to smaller-scale optimizer comparisons — but adapted for the specific demands of LLM pretraining at scale.

Two Methodological Failures Driving the Credibility Gap

The paper identifies two specific, recurring problems in how new optimizers are evaluated (Section 1), both of which inflate reported speedups:

Problem 1: Unequal hyperparameter tuning favors new optimizers over AdamW. Most prior work performs limited or asymmetric hyperparameter optimization. The common pattern is that the proposed optimizer receives careful tuning — perhaps because the authors naturally invest more effort in making their method work well — while AdamW receives cursory treatment, often using default or legacy hyperparameters. The paper demonstrates the devastating impact of this asymmetry with a striking result (Figure 1, top left): starting from the GPT-3 recipe (Brown et al., 2020) with a peak learning rate of 6e-4 — the configuration used as the AdamW baseline in multiple optimizer papers including Sophia, MARS, Cautious AdamW, FOCUS, and others — simply tuning the learning rate alone yields a 2× speedup for AdamW. That is, the entire claimed advantage of many new optimizers can be recovered just by properly tuning a single AdamW hyperparameter. As the paper puts it (Section 2, discussing prior comparisons), "unequal tuning can account for most of the claimed speedup."

The problem extends beyond the learning rate. The paper shows that weight decay — a hyperparameter sometimes fixed at 0.1 across optimizers — has dramatically different optimal values for different methods: approximately 0.1 for AdamW but approximately 0.6–0.7 for Lion (Figure 1, top right). Kron's optimal weight decay is approximately 0.5, far from the conventional 0.1. Blindly fixing weight decay across optimizers would penalize Lion and Kron by roughly the same magnitude as the claimed speedup over AdamW. This is not a hypothetical concern — the paper notes that several prior works "keep common hyperparameters such as learning rate and weight decay fixed across optimizers" (Section 1, for example Liu et al., 2025a), a practice that the results show is fundamentally unfair.

Problem 2: Experiments at small scale and limited regimes leave scaling behavior unknown. Most optimizer evaluations are conducted on models under 1B parameters and at a single data-to-model ratio (typically 1× Chinchilla, meaning the number of training tokens equals roughly 20× the parameter count). However, real-world pretraining operates across a much wider range of regimes. Depending on the deployment context, data-to-model ratios can vary from 1× to more than 50× Chinchilla, and important scaling trends may only become visible at larger model sizes or under heavy overtraining.

The paper specifically critiques prior Muon evaluations (Liu et al., 2025a) for considering "only 1× Chinchilla regime," and notes that the original Soap paper (Vyas et al., 2025) "investigate[s] model sizes up to 0.6B parameters" (Section 1, footnote 1). Without testing across regimes, an optimizer that excels at 1× Chinchilla on a 130M model might underperform at 8× Chinchilla on a 1.2B model — and indeed, the paper finds exactly this pattern for Muon, which is overtaken by Soap and Kron when data-to-model ratios increase. Even more concerning, the paper's fitted scaling law projects that Muon will produce higher loss than AdamW at 7B parameters in the 1× Chinchilla regime (Appendix B.1) — a reversal that would be invisible from experiments at typical evaluation scales. This means that an organization adopting an optimizer based on 130M–500M results at 1× Chinchilla could be making a decision that becomes actively harmful at production scales.

Prior Approaches and Their Shortcomings

The paper situates itself within a long history of optimizer development for deep learning (Section 2), tracing three major threads:

Scalar-based adaptive methods. Starting from Adam (Kingma and Ba, 2017) and AdamW (Loshchilov and Hutter, 2019), a series of improvements targeting different aspects — incorporating Nesterov momentum (Dozat, 2016), reducing update variance (Yuan et al., 2025; Liu et al., 2021), reducing memory usage (Shazeer and Stern, 2018; Zhang et al., 2025b; Chen et al., 2023) — have all operated within the paradigm of scalar preconditioning: each parameter receives its own adaptive learning rate, computed element-wise. This family of optimizers is mature and well-understood, but the paper finds that after proper tuning, all scalar-based methods converge to similar performance, with "an average speedup ratio of less than 1.2×" over AdamW (Section 4.1). The design space for scalar-based methods may be largely saturated.

Matrix-based preconditioners. A distinct line of work, starting from Preconditioned SGD (Li, 2018a) and Shampoo (Gupta et al., 2018), uses matrix operations to precondition gradients — multiplying the gradient matrix by learned or computed matrices on the left and right, rather than dividing by a scalar vector element-wise. This leverages the inherent matrix structure of neural network parameters and can capture correlations between parameters in ways scalar methods cannot. Variants include Muon (Jordan et al., 2024), which uses Newton-Schulz iterations to approximately orthogonalize the gradient, Soap (Vyas et al., 2025), which combines Shampoo-style preconditioning with Adam's adaptive learning rates, and Kron (PSGD; Li, 2018a, 2022), which maintains Kronecker-factored preconditioners. The paper finds that all three matrix-based methods deliver approximately 1.3× speedup at 130M parameters, despite their very different update rules — suggesting that the matrix structure itself, rather than any specific algorithm, is the source of the advantage.

The missing link: rigorous cross-optimizer evaluation. The paper specifically calls out Schmidt et al. (2021) as a precursor in spirit — that work re-evaluated optimizers (mostly pre-Adam methods) and found that "which optimizer is optimal is problem-specific" and that "rigorous hyperparameter tuning is required." However, Schmidt et al. (2021) predates both the LLM pretraining era and the current generation of optimizers. The paper positions itself as extending that methodological rigor to modern LLM-scale pretraining with current optimizers, arriving at the same uncomfortable conclusion: "rigorous and fair hyperparameter tuning is still not the norm, but rather the exception in optimizer design research" (Section 2).

How This Paper Positions Itself

The paper's framing is methodological rather than algorithmic: it does not propose a new optimizer, but instead provides the first systematic, multi-scale, multi-regime benchmarking study of modern pretraining optimizers with rigorously equalized hyperparameter tuning. This positions the work as an empirical audit of the optimizer evaluation literature, analogous to how Jiang et al. (2019) audited generalization measures — establishing which claims hold up under proper experimental controls and which do not.

The methodological innovation is the three-phase hyperparameter tuning framework (Section 3): Phase I performs exhaustive coordinate descent on all hyperparameters for all optimizers across six settings (three model sizes × multiple Chinchilla ratios), identifying which hyperparameters are scaling-sensitive versus scaling-insensitive. Phase II then focuses coordinate descent only on scaling-sensitive parameters (typically learning rate, warmup, and optimizer-specific knobs like Lion's β₂ or Kron's learning rate), extending to additional Chinchilla ratios. This produces 12 near-optimal configurations per optimizer without the combinatorial explosion of full grid search at every scale. Phase III fits a hyperparameter scaling law to extrapolate to 1.2B parameters — critically, the paper validates this extrapolation by running a full Phase I sweep at 1.2B and confirming the predicted hyperparameters yield loss within 0.003 of the optimal configuration (Section 3.4).

This design is deliberately comparative rather than competitive. Every optimizer receives the same investment in tuning effort (approximately 6 settings × N sweeps per hyperparameter in Phase I, plus another 6 × M in Phase II), preventing the asymmetry that inflates claimed speedups. The paper even acknowledges when its own experimental choices might disadvantage certain optimizers — for instance, its Soap implementation "performs blocking of weight in order to reduce the memory footprint and further uses bfloat16 for the momentum in the 1.2B experiments" (Appendix G), which "may lead to slightly lower step-wise performance" compared to the original paper's implementation.

Reconciling Conflicting Results with Concurrent Work

A particularly illuminating aspect of the paper's positioning is its discussion of differences with concurrent work by Semenov et al. (2025), which independently benchmarked optimizers for pretraining (Section 2). The two papers agree on several high-level points — non-zero weight decay and decaying to a small learning rate are essential; variance-reduced AdamW variants like Mars show non-trivial speedups — but disagree on the relative performance of matrix-level optimizers. Semenov et al. (2025) found that AdEMaMix and Mars outperform Muon, while this paper finds Muon achieves significant speedups over Mars.

The paper's investigation of this discrepancy is itself a methodological contribution. It attributes the difference primarily to batch size: Semenov et al. (2025) used batch sizes of only 0.02M–0.1M tokens in their most extensively tuned 130M experiments, while this paper uses tuned batch sizes no smaller than 0.4M tokens (a consequence of running on 128 TPU-v5lite chips versus the 1–8 H100 GPUs in Semenov et al., 2025). In the small-batch regime, gradient noise dominates, and variance-reduction methods like Mars and AdEMAMix shine. In the larger-batch regime, this advantage diminishes and "matrix-level optimizers become more competitive." This explanation is plausible and testable, and it highlights a subtle but crucial point that the paper makes repeatedly: there is no universal best optimizer — the ranking depends on hardware regime, batch size, model scale, data budget, and tuning methodology. Any paper claiming otherwise is likely overclaiming based on a narrow experimental window.

The paper also notes differences in tuning methodology with Semenov et al. (2025) at larger scales: this paper conducts more extensive learning rate sweeps for 520M models and finds higher optimal values (4e-3 to 8e-3 versus 1e-3 to 2e-3), and for Muon separately tunes learning rates for embedding layers and matrix weights, which improves performance. These details reinforce the central message: optimizer comparisons are fragile, and small variations in experimental protocol can flip conclusions.

Broader Implications

The paper's motivation extends beyond merely correcting inflated speedup claims. It touches on a deeper issue in machine learning research: the difficulty of establishing trustworthy empirical results when evaluation protocols are not standardized. The optimizer literature exemplifies a pattern where new methods appear to dramatically outperform baselines, but the apparent gains evaporate when baselines receive equal experimental investment. This pattern has been documented in other subfields — reinforcement learning (Henderson et al., 2018), graph neural networks (Shchur et al., 2018), and indeed earlier optimizer benchmarking (Schmidt et al., 2021) — and the paper implicitly argues that the pretraining optimizer literature has fallen into the same trap.

By establishing a reproducible, fair comparison protocol and making all code, configurations, and WandB runs publicly available (Section 3), the paper aims to create a reference benchmark that future optimizer designers can use to honestly assess their methods. This is a deliberately community-service role: rather than proposing the next faster optimizer, the paper provides the measurement infrastructure to determine whether a proposed optimizer is actually faster. In a field where pretraining costs dominate and the difference between adopting a genuinely better optimizer versus one with inflated claims can be millions of dollars, this infrastructure is arguably as valuable as a new optimizer itself.

3. Technical Approach

3.1 Reader Orientation

This paper does not propose a new optimizer — it builds a systematic benchmarking framework that fairly compares eleven deep learning optimizers for LLM pretraining. The problem it solves is that prior optimizer evaluations suffer from two fatal flaws: the baseline (AdamW) is almost always undertuned relative to proposed methods, and experiments are confined to narrow scaling regimes that fail to capture how optimizer rankings shift with model size and data budget. The solution is a three-phase hyperparameter optimization protocol that (1) performs exhaustive coordinate descent to find near-optimal hyperparameters for every optimizer at every scale, (2) identifies which hyperparameters genuinely need re-tuning across regimes, and (3) fits scaling laws to extrapolate hyperparameter choices to larger models.

3.2 Big-Picture Architecture (Diagram in Words)

The benchmarking framework has five major components:

Training infrastructure — A fixed Llama 2 architecture (130M, 300M, 520M, 1.2B parameters), a fixed pretraining corpus (DCLM-baseline + StarCoder V2 + ProofPile 2, ~4.1T tokens total), and a fixed evaluation metric (C4/EN validation loss), implemented in JAX on TPU v5 hardware. This provides the substrate on which every optimizer runs.
Eleven optimizer implementations — Each optimizer conforms to a shared interface (receives gradients, maintains internal state, produces parameter updates with weight decay), but differs in its internal update logic: scalar-based methods (AdamW, NAdamW, Mars, Cautious, Lion, Adam-mini, Sophia), matrix-based methods (Muon, Scion, Kron, Soap), and Hessian-approximation methods (Sophia). Detailed algorithms are provided in Appendix A.
Phase I: Full coordinate descent — For each optimizer and each of 6 small-scale regimes (130M/300M/520M at 1× Chinchilla, 130M at 2×/4×/8× Chinchilla), performs iterative one-at-a-time grid search over all hyperparameters until convergence, identifying coordinate-wise local optima and revealing which hyperparameters are sensitive to scale.
Phase II: Focused descent on scaling-sensitive hyperparameters — Extends the coordinate descent to additional regimes (300M/520M at 2×/4×/8× Chinchilla) but only sweeps the subset of hyperparameters that Phase I showed vary with scale (e.g., learning rate, warmup, weight decay), freezing insensitive parameters (e.g., β₁, β₂, ε for most optimizers) at their Phase I optima.
Phase III: Hyperparameter scaling laws and 1.2B extrapolation — Fits parametric functions that predict optimal hyperparameter values as a function of model size and data budget, validates the prediction against a full Phase I sweep at 1.2B (confirming prediction error < 3e-3 in loss), then uses the predicted hyperparameters to evaluate 1.2B models with AdamW, NAdamW, Muon, and Soap at 1–8× Chinchilla, plus 16× Chinchilla for 130M and 300M models.

Information flows sequentially: the fixed architecture and data are shared across all optimizers → each optimizer enters Phase I where all its hyperparameters are independently optimized → scaling-sensitive parameters are identified → Phase II extends optimization to additional regimes using only sensitive parameters → near-optimal configurations for 12 settings per optimizer are collected → a speedup metric is computed by fitting AdamW's loss scaling law $\hat{L}_N(D) = \alpha_N D^{-B_N} + \beta_N$ and mapping each optimizer's achieved loss to the equivalent AdamW data budget $D_{\text{AdamW}}$ → Phase III fits hyperparameter scaling laws for the most promising optimizers and evaluates them at 1.2B → final rankings and scaling trends are extracted.

3.3 Roadmap for the Deep Dive

First, the fixed experimental substrate (architecture, data, evaluation metric) — this is the "laboratory" in which all comparisons occur, and its choices constrain what conclusions can be drawn.
Second, the optimizer classes and their key algorithmic differences — understanding what each optimizer does differently is prerequisite to understanding why their hyperparameter sensitivities differ.
Third, the Phase I coordinate descent protocol in detail — this is the core methodological innovation, and understanding its mechanics (one-at-a-time sweeps, convergence criteria ∆₁, scaling-sensitivity identification) is essential to interpreting all subsequent results.
Fourth, the Phase II focused descent and the speedup metric — how the paper extracts a single number (speedup ratio) from a matrix of loss measurements across optimizers, model sizes, and Chinchilla ratios, and why the fitting procedure matters.
Fifth, the Phase III hyperparameter scaling laws and validation — how the paper extrapolates hyperparameters to 1.2B without running another full coordinate descent, and the empirical check that this extrapolation works.
Sixth, the comparison with prior work setup differences — not a methodological component per se, but an analysis that explains why reported speedups differ and what implementation details matter.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical benchmarking paper whose core idea is that fair optimizer comparison requires each optimizer to receive equal hyperparameter optimization effort, that hyperparameter sensitivities vary dramatically across optimizers (making blind transfer unfair), and that optimizer rankings shift with model scale and data budget (making narrow evaluations misleading).

3.4.1 Fixed Experimental Substrate

The paper constructs an experimental environment that is held constant across all optimizer comparisons. Every design choice here has downstream consequences for the generality of the conclusions.

Architecture. All experiments use a Llama 2-style architecture with 32 transformer layers and a sequence length of 4096 tokens. Model sizes are scaled by varying three parameters: hidden dimension, intermediate dimension, and number of attention heads. The specific configurations are given in Table 2: the 130M model uses hidden dimension 512, intermediate dimension 2048, and 8 attention heads; the 300M uses 768/3072/12; the 520M uses 1024/4096/16; the 1.2B uses 1536/6144/24. The choice of 32 layers is fixed following MobileLLM (Liu et al., 2024b), meaning scaling affects only width, not depth.

Why this choice: Fixing the number of layers while scaling width is a design choice that simplifies the hyperparameter scaling problem — optimizers that are sensitive to depth (through gradient propagation through many layers) would need a different treatment, but by holding depth constant, the paper isolates the effect of parameter count on optimizer behavior. This is a reasonable simplification for models in the 0.1–1.2B range, where depth-related optimization pathologies (e.g., gradient vanishing) are less severe than at much larger scales.

Data mixture. The pretraining corpus combines three publicly available datasets, tokenized with the Llama3 tokenizer: DCLM-baseline (3.8 trillion tokens; Li et al., 2025b), StarCoder V2 Data (0.25 trillion tokens; Lozhkov et al., 2024), and ProofPile 2 (55 billion tokens; Azerbayev et al., 2024). This mixture follows OLMo 2's recipe (OLMo et al., 2025) and is designed to be representative of modern high-quality pretraining data — a deliberate choice to avoid the confound that the original Sophia paper used PILE, which the authors note is "lower quality compared to the current pretraining dataset" (Appendix G) and which may have contributed to different optimal hyperparameters.

Chinchilla ratios. For each model, the Chinchilla-optimal number of training tokens is computed as 20 times the non-embedding parameter count, following Hoffmann et al. (2022). The paper tests Chinchilla ratios of 1×, 2×, 4×, and 8×, meaning that a 130M model at 8× Chinchilla sees approximately 20.8B tokens (130M × 20 × 8). For the 16× Chinchilla experiments in Phase III, this becomes 41.6B tokens for the 130M model. The range 1–8× spans from compute-optimal training to moderate overtraining; 16× represents heavy overtraining, which the paper shows can flip optimizer rankings.

Evaluation metric. The primary metric is validation loss on the C4/EN (English) split (Raffel et al., 2023), measured at the end of training. The paper invokes Bhagia et al. (2024) to justify this as "a known proxy for downstream performance" (Section 3.1), and validates this by tracking downstream accuracy on 10 benchmarks (ARC Easy/Challenge, BoolQ, COPA, CommonsenseQA, HellaSwag, LAMBADA, OpenBookQA, PIQA, WSC273, Winogrande) — Tables 5–35 in Appendices B.4 show that loss improvements consistently translate to downstream accuracy improvements. However, at 1.2B scale (Table 5), the downstream improvements largely vanish even when loss improvements persist, suggesting a potential decoupling that the paper does not deeply investigate.

Hardware and precision. Training uses JAX on TPU v5 hardware with mixed precision (parameters in float32, activations in bfloat16). The batch size is one of the hyperparameters tuned per optimizer per regime, with typical optima in the range 128–512 tokens per batch (the paper uses "batch size" to mean sequences per device, with 4096 tokens per sequence, so a batch size of 128 corresponds to approximately 0.5M tokens). The paper notes that Semenov et al. (2025) used much smaller batches (0.02–0.1M tokens) in their most tuned experiments, and attributes divergence in results partly to this difference — a crucial point about hardware-optimizer interaction that is often overlooked.

Tokens as the unit of comparison. The paper "primarily compare[s] algorithms by the number of tokens needed to reach a given loss" (Section 1), not by wall-clock time or FLOPs. This choice is motivated by the observation that "the step-wise computation overhead of matrix-based optimizers can be reduced to under 10% through proper implementation" (Section 1, citing Vyas et al., 2025; Liu et al., 2025a). In other words, if you can make Muon's per-step cost only 10% higher than AdamW's, then a 20% token efficiency gain translates to an 18% wall-clock speedup. The paper does not verify this overhead reduction itself — it relies on prior claims — and does not measure actual wall-clock time.

Why these choices matter for generality. The fixed architecture (32 layers, Llama 2 style) means the paper cannot speak to how optimizer rankings might differ for architectures with different depth/width ratios, for mixture-of-experts models, or for non-autoregressive training objectives. The TPU v5 + JAX stack means that certain implementation tricks (e.g., ZeRO-style sharding) that interact with optimizer memory usage are not tested. These are reasonable scope limitations for a systematic study, but they mean the results are strongest within the Llama 2 + TPU ecosystem and may not transfer perfectly to, say, GPT-style architectures on GPU clusters.

3.2 Optimizer Taxonomy and Algorithmic Differences

The paper groups the eleven optimizers into five conceptual families, and understanding the algorithmic differences is essential because those differences drive different hyperparameter sensitivities — which in turn drive the need for the three-phase tuning protocol.

Class 1: Baseline AdamW (Algorithm 1). AdamW maintains two exponential moving averages: $m_t$ (first moment, momentum of gradients) and $v_t$ (second moment, momentum of squared gradients). The update at step $t$ is:

$w_{t+1} = w_t - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon} - \eta \lambda w_t$

where $\eta$ is the learning rate, $\lambda$ is the weight decay coefficient, $\epsilon$ is a numerical stability constant, and $\hat{m}_t = m_t / (1 - \beta_1^t)$ and $\hat{v}_t = v_t / (1 - \beta_2^t)$ are bias-corrected moments with exponential decay rates $\beta_1$ and $\beta_2$ .

What it computes: For each parameter, AdamW divides the smoothed gradient (which points roughly downhill) by the square root of the smoothed squared gradient (which captures the typical magnitude of recent gradients in that direction). This produces an adaptive per-parameter learning rate: parameters with consistently large gradients get smaller steps, while parameters with consistently small gradients get larger steps. The weight decay term $-\eta \lambda w_t$ directly shrinks the weights toward zero, decoupled from the gradient normalization.

Why AdamW is the baseline and not SGD: SGD with momentum lacks per-parameter adaptivity, which is critical for transformers where different parameter matrices (attention projections, feedforward layers, embeddings) have vastly different gradient scales. AdamW's moment estimates provide this adaptivity without requiring second-order curvature information. The decoupled weight decay (Loshchilov and Hutter, 2019) is important because it prevents the adaptive learning rate from interfering with regularization — in standard Adam, weight decay is implemented via L2 regularization added to the loss, which means the effective weight decay strength depends on the adaptive learning rate, making the two knobs entangled. Decoupling them allows independent control.

Class 2: Variance-reduced AdamW variants — NAdamW (Algorithm 2) and Mars (Algorithm 5). Both modify how the first moment is computed to incorporate more accurate gradient estimates.

NAdamW replaces the standard momentum $\hat{m}_t$ with a Nesterov-style lookahead: $\tilde{m}_t = \beta_1 m_t + (1 - \beta_1) g_t$ , then uses $\tilde{m}_t / (1 - \beta_1^t)$ in the update. Intuitively, standard momentum computes an exponentially weighted average of past gradients and uses that as the current direction. Nesterov momentum says: instead of taking the step in the direction of the average, first take a step in the averaged direction, then compute the gradient at that lookahead point, and use that lookahead gradient. This is known to reduce oscillation and improve convergence rates for convex problems, and NAdamW adapts it for the AdamW framework.
Mars (Yuan et al., 2025) goes further: it computes a "corrected gradient" $c_t = g_t + \gamma \frac{\beta_1}{1-\beta_1}(g_t - g_{t-1})$ where $\gamma$ is a correction strength hyperparameter. The term $\frac{\beta_1}{1-\beta_1}(g_t - g_{t-1})$ is an approximation to the gradient change that accounts for the momentum-induced delay — standard momentum smooths gradients, which means the effective update direction lags behind the true gradient. Mars attempts to compensate for this lag by adding a scaled version of the gradient difference. The corrected gradient $c_t$ is then used in place of $g_t$ in the standard AdamW update.

Why variance reduction helps in theory: Both methods try to reduce the noise in the first-moment estimate. In stochastic optimization, the gradient at each step is a noisy estimate of the true batch gradient. Averaging over past steps (momentum) reduces variance but introduces bias (the average is centered on old gradients, not the current one). Nesterov and Mars corrections reduce this bias, providing a better estimate of the current gradient direction. The paper notes (Semenov et al., 2025 discussion) that these methods are "advantageous in their noise-dominated small-batch regime," which explains why their benefits diminish when the paper uses larger batch sizes — larger batches already provide lower-variance gradient estimates, leaving less room for variance reduction to help.

Class 3: Memory-efficient optimizers — Lion (Algorithm 3) and Adam-mini (Algorithm 6). These reduce the memory required by AdamW, which stores two buffers (m and v) per parameter, each the same size as the model.

Lion (Chen et al., 2023) keeps only a single momentum buffer $m_t$ (no second moment). Its update is sign-based:

$w_{t+1} = w_t - \eta \cdot \text{sign}(\hat{m}_t) - \eta \lambda w_t$

where $\hat{m}_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$ and $m_{t+1} = \beta_2 m_{t-1} + (1-\beta_2)g_t$ . The two decay rates $\beta_1$ and $\beta_2$ are for the sign computation and the momentum update respectively — a design discovered by symbolic optimization (Chen et al., 2023). The $\text{sign}$ operation reduces the update to only $\pm 1$ per parameter, which makes the optimizer update itself extremely cheap to compute, and the single momentum buffer halves the memory overhead relative to AdamW.

Adam-mini (Zhang et al., 2025b) maintains a full-size first moment $m_t$ but drastically reduces the second moment storage by partitioning parameters into blocks (according to a fixed partitioning scheme described in the original paper) and maintaining only one scalar $v_{t,b}$ per block — the mean of the squared gradients across all parameters in that block. The output is a scalar per block rather than per parameter, reducing the second-moment memory from O(parameters) to O(number of blocks), which can be 100–1000× smaller.

Why memory efficiency matters and what it trades off: For large models, AdamW's optimizer state (2 × model parameters in float32) can be 8× larger than the model parameters themselves when the parameters are stored in float16 — a 10B parameter model requires ~80GB for optimizer state alone. Lion and Adam-mini reduce this to ~20–40GB, enabling larger models to fit on the same hardware. The tradeoff is potential loss in optimization quality — Lion lacks adaptive per-parameter step sizes (only the sign matters), and Adam-mini uses a coarse-grained second moment that may poorly approximate per-parameter variance for parameters in the same block with very different gradient scales.

Class 4: Matrix-based optimizers — Muon (Algorithm 8), Scion (Algorithm 9), Kron/PSGD (Algorithm 10), Soap (Algorithm 11). These share a common insight: neural network parameters are organized as matrices (e.g., weight matrices in linear layers), and their gradients have matrix structure. Rather than treating each element independently (scalar preconditioning), they apply matrix operations to the gradient.

Muon (Jordan et al., 2024) applies Newton-Schulz orthogonalization to the gradient matrix (for all matrices except embeddings, the LM head, and LayerNorm, which use standard AdamW). The key operation is:

$u = \text{NewtonSchulz}(\beta m_t + g_t, \text{steps}=5)$

where $m_t$ is a momentum buffer (single, not two), $\beta m_t + g_t$ is a Nesterov-style momentum combination, and NewtonSchulz is an iterative algorithm that approximately orthogonalizes a matrix — it finds the matrix $O$ with orthonormal rows that is closest (in a specific sense) to the input, specifically approximating $\arg\max_{\|O\|_{\text{op}}=1} \text{Tr}(O^\top M)$ for input $M$ . The orthogonalized matrix is then scaled by $\sqrt{\max(1, \text{rows}(w)/\text{cols}(w))}$ to account for rectangular shapes. The final update is:

$w_{t+1} = w_t - \eta \cdot u - \eta \lambda w_t$

What this does in plain language: Instead of updating each weight independently, Muon looks at the entire weight matrix and asks: what is the best rotation of the current weight space that moves in the direction of the gradient? This captures correlations between weights in the same matrix — if several weights should all increase together, a matrix operation can express this efficiently, whereas a scalar update treats them independently and may move each one at a different rate.

Why orthogonalization? The Newton-Schulz iteration approximately computes the matrix square root, which, when applied to the gradient, produces an update that is "balanced" across all directions in the weight space — no direction dominates simply because it has larger gradient magnitudes. This is analogous to what AdamW does per-element (dividing by the square root of the second moment normalizes gradient magnitudes), but Muon does it per-matrix, which can capture cross-parameter structure. The authors of Muon argue that this leads to more efficient optimization because the update better respects the geometry of the parameter space.

Scion (Pethick et al., 2025) uses the same Newton-Schulz update as Muon for transformer layer matrices, but uses a simpler sign-based SGD update for the embedding and LM head parameters ( $w_{t+1} = w_t - \eta_{\text{SignGD}} \cdot \text{sign}(\hat{m}_t)$ ), removing the second moment entirely for those parameters. This hybrid approach recognizes that embedding layers have different optimization characteristics (they are typically used for lookup, not transformation) and may not benefit from the full matrix preconditioning.
Kron / PSGD (Li, 2018a, 2022) maintains Kronecker-factored preconditioners: for each parameter tensor (possibly with more than 2 dimensions), it maintains a set of square matrices $Q_i$ , one per dimension. The update first applies these preconditioners to the gradient via a sequence of tensor contractions:

$G^{(0)} = \hat{\mu}_t, \quad G^{(i)} = \text{fold}_i\left(Q_i \cdot \text{unfold}_i(G^{(i-1)})\right)$

where $\text{unfold}_i$ flattens all dimensions except the i-th into a single dimension, and $\text{fold}_i$ reverses this. The preconditioners $Q_i$ are themselves updated online using a "conjugate sketch" method: random probes are passed through the current preconditioners and the gradient to estimate the loss curvature, then the preconditioners are adjusted to better approximate the inverse Hessian. This is a second-order method in spirit (it tries to approximate Newton's method) but uses Kronecker factorization to keep the preconditioner size manageable — instead of a full $d_1 d_2 \times d_1 d_2$ matrix, it stores $d_1 \times d_1$ and $d_2 \times d_2$ matrices whose Kronecker product approximates the full preconditioner.

Soap (Vyas et al., 2025) combines Shampoo-style preconditioning with Adam-style adaptivity. It partitions all parameters into $\text{block\_size} \times \text{block\_size}$ blocks (where the paper uses block_size=512 for experiments, optionally with blocking to reduce memory). For each block, it maintains left and right preconditioners $Q_A$ and $Q_B$ , plus Adam-style first and second moments $m_t$ and $v_t$ . The update is:

$w_{t+1} = w_t - \eta_t \cdot Q_A^\top \left(\frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}\right) Q_B^\top$

The preconditioners are updated every $k$ steps (where $k$ is a hyperparameter) using the gradient outer products:

$G_A = \mu G_A + (1-\mu)(g_t g_t^\top + \epsilon I)$ $G_B = \mu G_B + (1-\mu)(g_t^\top g_t + \epsilon I)$

and then $Q_A = \text{QR}(G_A Q_A)$ and $Q_B = \text{QR}(G_B Q_B)$ , where QR decomposition provides an orthogonal basis. The $\text{block\_size}$ parameter controls the tradeoff between approximation quality and memory: smaller blocks give better approximation but more overhead.

Why matrix-based methods work: The unifying principle is that neural network gradients often exhibit "spectral decay" — the gradient matrix, when decomposed into singular values, has a few large singular values and many small ones. A scalar optimizer like AdamW treats every element independently and cannot exploit this structure: it applies the same adaptive scaling to every element regardless of how they co-vary. Matrix preconditioning can rotate the gradient space so that the optimization proceeds more efficiently along correlated directions. The paper's finding that Muon, Soap, and Kron all achieve similar 1.3× speedup at 130M despite very different update rules suggests that the matrix structure itself — not any particular algorithm for exploiting it — is the source of the gain.

Class 5: Hessian-approximation optimizers — Sophia (Algorithm 4). Sophia (Liu et al., 2024a) uses a Hutchinson-style Hessian diagonal estimator to approximate second-order curvature. Every $k$ steps, it estimates the diagonal of the Hessian via:

$\hat{h}_t = r \odot (\nabla_\theta (g_t \cdot r))$

where $r \sim \{\pm 1\}^d$ is a random sign vector, and $g_t \cdot r$ is a scalar whose gradient with respect to $\theta$ is backpropagated. The product $r \odot \nabla_\theta(g_t \cdot r)$ gives an unbiased estimate of the Hessian diagonal. The estimate is then smoothed with exponential averaging ( $h_t = \beta_2 h_{t-k} + (1-\beta_2) \hat{h}_t$ ). Updates use a clipped Newton-style step:

$\theta_{t+1} = \theta_t - \eta_t \cdot \text{clip}\left(\frac{m_t}{\max(\gamma h_t, \epsilon)}, 1\right)$

where the clip prevents any single parameter update from exceeding $\eta_t$ in magnitude, and $\gamma$ controls how much the Hessian estimate influences the step size.

Why this fails in the paper's regime: Appendix B.2 reports that "Sophia tends to underperform AdamW in smaller compute regimes and eventually slightly outperforms AdamW when either the model size or the data size increases" (Figure 7). The paper attributes Sophia's previously reported 2× speedup to the use of a low-quality data shuffling implementation in the original Sophia codebase (Levanter), which "did not fully randomize the order of the data" and caused the optimal AdamW learning rate to be significantly smaller than with full randomization. With proper random shuffling and a well-tuned AdamW baseline, Sophia's advantage largely disappears for models under 0.5B.

3.4.3 Phase I: Full Coordinate Descent Protocol

This is the methodological core of the paper — the procedure that ensures every optimizer receives equal tuning effort. The full algorithms are in Appendix C.

Starting point. For each optimizer, the paper begins with "a initial hyperparameter configuration similar to the hyperparameter configuration provided in the original paper proposing the optimizer" (Section 3.2). For AdamW, this is the GPT-3 recipe (Brown et al., 2020): learning rate 0.008, weight decay 0.1, warmup 1000 steps, batch size 128, β₁=0.9, β₂=0.98, ε=10⁻¹⁰, gradient norm clipping=1, minimum learning rate ratio=0. For other optimizers, the starting configuration mirrors whatever the original authors reported as optimal.

Coordinate descent mechanism. For a given optimizer in a given regime (e.g., AdamW on 130M at 1× Chinchilla), the procedure works as follows:

Define a discrete grid for every hyperparameter. For AdamW, the swept hyperparameters include learning rate, weight decay, $\beta_1$ , $\beta_2$ , $\epsilon$ , gradient norm clipping, batch size, warmup steps, and minimum learning rate ratio. The grids are specified in Appendix C (e.g., for AdamW on 130M at 1×: learning rate in {0.004, 0.008, 0.016, 0.032}, weight decay in {0, 0.1, 0.2}, batch size in {128, 256, 512, 1024}, warmup in {500, 1000, 2000, 4000}, etc.).
Start from the initial configuration. In each iteration (called a "round"), hold all but one hyperparameter fixed at the current best values, then search the whole grid for that parameter. Train a model for the full Chinchilla budget with each value, record the final C4/EN validation loss, and accept the new value if the loss improves by more than $\Delta_1 = 3 \times 10^{-3}$ .
Repeat passes over all hyperparameters until "no parameter update yields further significant gain" — that is, one full pass through all hyperparameters with no accepted changes.

A concrete example. Table 3 shows AdamW on 130M at 1× Chinchilla (though the paper mistakenly labels it 300M — the loss values correspond to 130M scale). The initial configuration produces loss 3.298. Round 1: only warmup is varied, with warmup=2000 improving loss to 3.282 (a change of -0.016 > ∆₁, so accepted). Round 2: batch size is varied, with batch size=128 improving loss to 3.263 (another significant gain, accepted). The procedure terminates at loss 3.263 with the configuration learning rate=0.008, weight decay=0.1, min lr ratio=0, warmup=2000, max grad norm=2, batch size=128. Round 3 then tests gradient clipping — gnorm=2 is accepted over gnorm=1, with essentially unchanged loss, showing this parameter is nearly flat at the optimum.

What this guarantees and what it doesn't. By the end of Phase I, for each optimizer in each of 6 regimes, the paper has identified a coordinate-wise local optimum: no single hyperparameter can be changed (with all others held fixed) to improve loss by more than ∆₁. This is a weaker guarantee than a full grid search joint optimum — there could exist a combination where simultaneously changing two hyperparameters yields a larger improvement than either individually — but full grid search over 8+ hyperparameters with even 2–4 values each would require 2⁸ to 4⁸ = 256 to 65,536 runs per optimizer per regime, which is computationally infeasible. Coordinate descent with iterative refinement is the standard practical compromise.

The six Phase I regimes. The paper runs this procedure for every optimizer on: 130M at 1×, 2×, 4×, 8× Chinchilla; 300M at 1× Chinchilla; and 520M at 1× Chinchilla. This choice spans model sizes to detect scaling trends within Phase I, while the 130M Chinchilla sweep captures data budget effects. The total computational cost is substantial: for AdamW alone, the 130M 1× sweep in Table 36 required 19 training runs (initial guess + 18 ablations), with each run training a 130M model for ~2.6B tokens (~20B FLOPs). Extrapolating across 6 regimes × 10 optimizers (excluding AdamW) yields roughly 1000+ training runs in Phase I alone, each costing tens of billions of FLOPs.

Identifying scaling-sensitive hyperparameters. After Phase I, the paper defines the set of "approximately optimal" configurations for each regime $r$ :

$C_r = \{ c : L(c) \leq L^*_r + \Delta_2 \}$

where $L^*_r$ is the best-observed loss in regime $r$ and $\Delta_2 = 6.4 \times 10^{-3}$ (a looser threshold than ∆₁, capturing configurations that are nearly as good as the optimum). A hyperparameter $c_h$ is called scaling-insensitive if there exists a single value $v_h$ such that, for every regime $r$ in Phase I, there is some configuration $c \in C_r$ with $c_h = v_h$ . In other words, the same value works well across all model sizes and data budgets tested. Otherwise, $c_h$ is scaling-sensitive — its optimal value depends on scale.

What this test captures and its assumptions. The definition operationalizes an empirical question: can I pick one value for this hyperparameter and forget about it, or do I need to re-tune it for every new model size? The threshold ∆₂ allows some tolerance — a hyperparameter isn't considered sensitive just because the absolute optimum shifts by a tiny amount; the shift must be large enough that choosing the wrong value measurably hurts. The choice of $\Delta_2 = 6.4 \times 10^{-3}$ (roughly 2× the Phase I convergence threshold) balances the risk of false positives (flagging as sensitive when the effect is noise) against false negatives (missing genuine sensitivity because the grid didn't spend enough time exploring).

Results of the sensitivity analysis (Table 4). The paper finds that:

Scaling-insensitive for most optimizers: $\beta_1$ , $\beta_2$ , $\epsilon$ (moment decay rates and numerical stability) — these can be fixed at a single value across all scales without significant penalty. This is expected from theory: $\beta_1$ and $\beta_2$ control the effective memory length of the optimizer, which should be a property of the optimization landscape geometry rather than model size.
Scaling-sensitive across optimizers: learning rate (for ALL optimizers), warmup steps (for AdamW, NAdamW, Adam-mini, Mars, Soap), weight decay (for AdamW, Adam-mini), and various optimizer-specific parameters: $\beta_2$ for Lion, $\beta_1$ for Mars and Scion, batch size for AdamW and Cautious. For Soap, block size is scaling-sensitive (the optimal block size changes with model size), for Kron the learning rate is sensitive but notably many of its specific knobs (blocking strategy, normalization, preconditioner update probability) are not.

Why learning rate is universally scaling-sensitive: Larger models typically require smaller learning rates to maintain stability — this is a well-known phenomenon in deep learning, related to the scaling of gradient norms and the curvature of the loss landscape with width. Similarly, data budget affects the optimal learning rate because longer training requires more gradual learning rate decay; the paper uses a linear warmup + cosine decay schedule, and the peak learning rate interacts with the total number of steps (which varies with Chinchilla ratio).

3.4.4 Phase II: Focused Descent on Scaling-Sensitive Parameters

Phase II extends the coordinate descent to additional regimes (300M and 520M at 2×, 4×, 8× Chinchilla) but only sweeps the scaling-sensitive hyperparameters identified in Phase I. All scaling-insensitive parameters are frozen at their Phase I optimal values.

Why this reduces cost. If an optimizer has 8 hyperparameters but only 3 are scaling-sensitive, Phase II requires only 3 degrees of freedom to sweep per regime rather than 8, reducing the search cost roughly proportionally. The grids are also typically smaller for Phase II because Phase I has already narrowed the interesting range — for example, the paper already knows that AdamW's learning rate is near 0.008 for 130M and 0.004 for 520M, so Phase II only tests a few values around the Phase I optimum rather than the full grid from scratch.

The 12 regimes after Phase II. Combining Phase I (6 regimes) and Phase II (6 regimes) yields near-optimal hyperparameter-loss pairs for 12 settings: 130M, 300M, 520M, each at 1×, 2×, 4×, 8× Chinchilla. These 12 loss measurements per optimizer form the dataset from which speedup ratios are computed and scaling trends are extracted.

Speedup ratio computation. The paper does not simply compare raw loss values — it computes a speedup ratio that answers the question: "how many fewer tokens does optimizer X need to reach the same loss as AdamW?" The procedure:

Fit a parametric scaling law for AdamW's validation loss as a function of data budget $D$ for each model size $N$ independently:

$\hat{L}_N(D) = \alpha_N D^{-B_N} + \beta_N$

where $\alpha_N$ , $B_N$ , and $\beta_N$ are fitted parameters. This functional form assumes that loss decreases as a power law in data ( $D^{-B_N}$ ) approaching an irreducible loss $\beta_N$ .

For each optimizer and each regime, take the optimizer's achieved loss $L_{\text{optimizer}}$ at its own data budget $D_{\text{optimizer}}$ (which is just the Chinchilla-specified number of tokens for that model × ratio).
Solve $\hat{L}_N(D_{\text{AdamW}}) = L_{\text{optimizer}}$ for $D_{\text{AdamW}}$ — the number of tokens AdamW would need to reach the same loss.
The speedup ratio is $D_{\text{AdamW}} / D_{\text{optimizer}}$ . A ratio of 1.3 means the optimizer needs only $1/1.3 \approx 0.77\times$ the data of AdamW to match its performance.

Why fit a scaling law rather than directly comparing loss at the same budget? Direct comparison would require running every optimizer at every budget and comparing loss values, but some optimizers achieve better-than-baseline loss at the same budget, while others might achieve the same loss at a lower budget — these are different notions of "speedup" and the paper's method unifies them. The scaling law interpolation allows computing the equivalent AdamW budget even when the optimizer's loss falls between two AdamW data points.

Figures 3 and 4 reading guide. Figure 3 plots $D_{\text{AdamW}}$ (y-axis) versus $D_{\text{optimizer}}$ (x-axis) in units of Chinchilla ratios. A line parallel to the diagonal $y = x$ (which would be a 1.0 speedup line) indicates no advantage; lines below the diagonal indicate speedup (the optimizer needs fewer tokens, so $D_{\text{AdamW}}$ at that loss is larger than $D_{\text{optimizer}}$ , making the ratio > 1); lines above the diagonal indicate slowdown. The colored bands indicate speedup ranges (1.0–1.2×, 1.2–1.3×, 1.3–1.4×). Figure 2 shows the raw loss values — each point is the optimal loss at the given Chinchilla ratio for each optimizer, and lower is better.

Key hyperparameters held constant across all optimizers in Phase II. The paper's learning rate schedule uses a linear warmup followed by cosine decay to a minimum learning rate (typically zero, $\eta_{\min}=0$ ). The warmup length is tuned per optimizer. Gradient clipping by norm is applied (tuned per optimizer, typically 1.0 or 2.0). Weight decay is applied with decoupled formulation. These shared conventions ensure that the comparison is between optimizer update rules themselves, not between different learning rate schedule philosophies.

3.4.5 Phase III: Hyperparameter Scaling Laws for Extrapolation

Phase III addresses the practical problem: to evaluate at 1.2B parameters, the paper cannot afford another full coordinate descent sweep for every optimizer. Instead, it fits functions that predict the optimal value of each scaling-sensitive hyperparameter as a function of model size and data budget, then validates the prediction on AdamW at 1.2B.

The parametric form. For each scaling-sensitive hyperparameter $h$ :

$h(N, D) = \alpha N^{-A} D^{-B} + \beta$

where $N$ is the non-embedding parameter count, $D$ is the data budget, and $\alpha$ , $A$ , $B$ , $\beta$ are learned coefficients fitted via non-linear least-squares on the 12 observed $(N, D, h)$ triples from Phase I + II.

What this form encodes: It assumes that the optimal hyperparameter value follows a power-law decay (or growth, if $\alpha$ is negative or $A$ , $B$ are negative) in both model size and data budget, approaching an asymptotic value $\beta$ . This is a strong assumption — it presupposes monotonic trends and smooth scaling — but is consistent with the broader scaling laws literature (Hoffmann et al., 2022; Yang et al., 2022) where power-law relationships emerge naturally from random matrix theory and infinite-width limits.

Validation of the extrapolation. The paper runs a full Phase I sweep for AdamW at 1.2B and 1× Chinchilla (Table 160, Appendix E), comparing the hyperparameters predicted by the fitted scaling law against the actual optimal configuration found by coordinate descent. The predicted hyperparameters yield a final loss within $3 \times 10^{-3}$ of the optimal configuration (Section 3.4: "our predicted hyperparameters yield a final loss within 3e-3 of the optimal configuration"). This error is comparable to $\Delta_1$ , the convergence threshold, meaning the prediction is as good as one would get from an additional round of tuning. This validates that hyperparameter scaling laws can be practically useful for optimizer transfer across model sizes.

What Phase III evaluates. Using the predicted (or validated) hyperparameters:

1.2B models at 1–8× Chinchilla: AdamW, NAdamW, Muon, and Soap are trained at 1.2B parameters for four data budgets (1×, 2×, 4×, 8× Chinchilla), producing the scaling curves in Figure 4 (left and middle). The hyperparameters used are in Appendix E (Tables 160–175).
130M and 300M at 16× Chinchilla: To test the high-data-budget regime where Phase I + II (max 8×) could not directly observe, the paper trains additional models at 16× Chinchilla for 130M (AdamW, NAdamW, Soap, Muon) and 300M (same four). The hyperparameters for these are in Appendix F (Tables 176–183). The results in Figure 4 (right) and Figure 8 show that at 16× Chinchilla, Soap overtakes Muon at both 130M and 300M scales.
Scaling law fitting for loss extrapolation: The paper fits a joint scaling law for AdamW and Muon loss as a function of model size and data budget using the form:

$L(N, D) = \alpha N^{-A} + \beta D^{-B} + \gamma$

with fitted coefficients (Appendix B.1): for AdamW, $\alpha=21.4289$ , $A=0.1555$ , $\beta=276.4235$ , $B=0.2804$ , $\gamma=1.7324$ ; for Muon, $\alpha=32.7458$ , $A=0.1864$ , $\beta=59.0221$ , $B=0.2074$ , $\gamma=1.8063$ . The RMS fitting errors are $3 \times 10^{-3}$ (AdamW) and $5 \times 10^{-3}$ (Muon).

The 7B projection. Using these fitted scaling laws, the paper extrapolates to 7B parameters at 1× Chinchilla and finds that "Muon will actually result in a higher loss compared to AdamW" (Appendix B.1) — a reversal from all observed scales where Muon is consistently better. This projection comes with the caveat that it's an out-of-distribution extrapolation (the fitted data goes up to 1.2B parameters; 7B is ~6× larger), but the trend it extrapolates — Muon's exponent $A=0.1864$ is larger than AdamW's $A=0.1555$ , meaning Muon's loss degrades faster with model size — is genuinely present in the fitted data. The prediction is a warning, not a certainty, but it underscores the paper's core message: optimizer rankings are not scale-invariant.

3.4.6 Differences in Experimental Setup with Prior Work

Appendix G provides a detailed accounting of why previously reported speedups differ from this paper's measurements. This is not a methodological component per se, but it illuminates the practical decisions that distinguish rigorous from inflated evaluations.

Sophia (claimed 2×, measured ~1.0× at 130M–300M, slight improvement at 520M). The original Sophia paper (Liu et al., 2024a) used a peak learning rate of 6e-4 for AdamW, which this paper shows is severely undertuned — simply raising the learning rate to 8e-3 recovers a 2× speedup for AdamW alone (Figure 1, top left). The paper attributes this to two factors: (1) Sophia was trained on PILE, a lower-quality dataset than the OLMo 2 mixture, and (2) Sophia's codebase (Levanter) used data sharding that correlated examples across compute nodes rather than fully randomizing them. This non-random ordering made training less stable, forcing a smaller learning rate for AdamW. When Sophia is evaluated with proper data shuffling and a well-tuned AdamW baseline, its advantage largely disappears (Figure 7).

Mars (claimed 2×, measured ~1.1–1.2×). The original Mars paper (Yuan et al., 2025) used the same low-learning-rate AdamW baseline as Sophia. The paper notes that "in the first version of the paper, the authors also reported that increasing the learning rate of AdamW to 3e-3 can significantly improve the performance of AdamW" (Appendix G, citing Figure 6 of the arXiv v1). This is a form of self-correction: the Mars authors themselves observed that their baseline was undertuned but did not update their main speedup claims accordingly.

Soap (claimed 1.4×, measured 1.2–1.3×). The discrepancy is relatively small and the paper acknowledges its Soap implementation differs in two ways that may explain the gap: (1) the paper uses parameter blocking (partitioning large weight matrices into blocks to reduce memory, following Anil et al., 2021) which can slightly degrade preconditioner quality, and (2) the 1.2B experiments use bfloat16 for momentum storage rather than float32, trading precision for memory. These implementation details mean the paper's Soap results may be a slight underestimate relative to the original paper's configuration.

Muon (claimed 2× by Liu et al., 2025a, measured 1.1–1.4×). The paper is most critical of the Kimi Muon reproduction (Liu et al., 2025a), which claimed 2× speedup. The issues: (1) Kimi used a notably low learning rate for AdamW (8e-4 to 9e-4 for models 400M–1.5B), far below the 4e-3 to 8e-3 that this paper finds optimal; (2) Kimi's comparison on MoE models used intermediate checkpoints before learning rate decay was complete, which the paper shows (Figure 5, right) can misleadingly favor Muon — Muon often shows faster early descent but AdamW catches up during the decay phase; (3) An independent reproduction by Essential AI (AI et al., 2025) also found Muon's token efficiency is only 1.1–1.2×, consistent with this paper's findings at larger scales.

Cautious AdamW, Block-wise LR Adam, FOCUS (claimed 2×, measured ~1.0–1.1×). All three papers used the same undertuned AdamW baseline (GPT-3 recipe with lr=6e-4) as Sophia and Mars. The paper's message is clear: these claimed 2× speedups are artifacts of baseline undertuning, not genuine optimizer improvements.

SWAN, DION (claimed 2–3×, not fully tested). The paper notes that these comparisons were conducted at less than 1× Chinchilla regimes, where the speedup of matrix-based optimizers tends to be larger (as shown in Figure 3, where speedup ratios increase with data budget for matrix methods). Additionally, DION's primary contribution is reducing communication cost, which the paper does not evaluate.

SPlus (claimed 2×, not directly tested). The paper notes this work used "an atypical setup where the model is trained with a constant learning rate" (Appendix G), which is not representative of standard pretraining practice and makes the comparison difficult to interpret.

The broader lesson from Appendix G. The pattern is consistent: claimed speedups of 2× or more systematically come from papers that used the GPT-3 recipe's learning rate of 6e-4 as their AdamW baseline. Those that performed independent tuning for the baseline (most notably Soap) reported speedups much closer to what this paper measures. The paper estimates that 8e-3 is roughly optimal for AdamW in the 130M–300M range on their data mixture with full random shuffling — about 13× larger than the GPT-3 recipe's value. This single hyperparameter difference accounts for approximately 2× in apparent speedup, which explains why so many proposed optimizers appeared to be breakthroughs.

3.4.7 What the Three-Phase Protocol Achieves and Its Limits

Achievements. The protocol ensures:

Equal investment in tuning: Every optimizer receives the same number of coordinate descent rounds (1 full pass over all hyperparameters in Phase I, 1 pass over sensitive ones in Phase II). No optimizer can win simply because the authors spent more time tuning it.
Scale-aware tuning: By running Phase I across three model sizes and four Chinchilla ratios, the paper captures hyperparameter trends before they matter — a hyperparameter that appears fixed across 130M–300M might drift at 520M, and Phase I would detect this.
Empirical, not theoretical, identification of sensitivity: The scaling-sensitivity classification is based on observed loss changes, not on theoretical properties of the optimizer. This avoids assuming that, e.g., β₁ is insensitive — it measures whether varying β₁ across scales changes loss.
Principled extrapolation: Phase III provides a validated method for choosing hyperparameters at scales too expensive for full tuning, with the 1.2B AdamW validation providing an empirical error bar on the extrapolation quality.

Limitations.

Coordinate-wise, not joint, optimality: The procedure finds local optima along individual axes. There may exist hyperparameter combinations that are jointly better than any coordinate-wise optimum, though in practice, hyperparameters in optimizers tend to have relatively weak interactions (learning rate and weight decay interact, but learning rate and β₂ often do not strongly interact). The paper does not test for interactions.
The ∆₁ = 3e-3 threshold: This is set empirically and not justified theoretically. A smaller threshold would provide tighter convergence but require more sweeps; a larger threshold would be faster but coarser. The choice is reasonable for a study where loss differences between optimizers at the same scale are typically 0.01–0.05, so 3e-3 is comfortably below the signal level.
Phase II assumes Phase I sensitivity generalizes: A hyperparameter classified as insensitive based on 130M–520M at 1×–8× Chinchilla might become sensitive at 1.2B, but Phase II would not discover this because it freezes that parameter. The paper partially addresses this by validating Phase III hyperparameters for AdamW against a full sweep, but for other optimizers at 1.2B, the hyperparameters used are predictions without empirical verification.
Grid granularity: The discrete grids tested (typically 2–4 values per hyperparameter) may miss a true optimum that lies between grid points, though the $\Delta_1$ threshold provides some robustness — if the loss changes by less than 3e-3 when moving one grid step, the exact optimum is not critical.
Computational cost is untallied: The paper states that Phase I involves 6 regimes but never gives a total run count. From Appendix C, AdamW alone required roughly 120 training runs across all Phase I regimes (19 + 19 + 17 + 19 + 17 + 17 ≈ 108, plus some duplicates). With 10 non-AdamW optimizers requiring similar counts, Phase I likely involved over 1000 runs, each training a model to 1–8× Chinchilla (2.6B–21B tokens for 130M models). This is an enormous computational investment — the paper was supported by Google TPU Research Cloud and NSF grants — and replicating it independently would be prohibitive. The code and WandB runs are open-sourced, partially mitigating this.

4. Key Insights and Innovations

Innovation 1: Speedup Claims Are Mostly an Artifact of Baseline Undertuning — and This Is a Systematic Pattern, Not a One-Off Error

The paper's most intellectually distinctive contribution is not a new optimizer or a tuning algorithm — it is a diagnostic finding about how the optimizer evaluation literature systematically produces inflated claims. The core discovery is that the GPT-3 pretraining recipe (Brown et al., 2020), which served as the AdamW baseline for at least six major optimizer papers (Sophia, MARS, Cautious AdamW, FOCUS, Block-wise LR Adam, and indirectly Muon/Kimi), contains a single hyperparameter choice — a peak learning rate of 6e-4 — that is approximately 13× smaller than optimal for a modern data mixture with proper random shuffling. The paper demonstrates (Figure 1, top left) that simply tuning this one number recovers a 2× speedup for AdamW itself, meaning the entire claimed advantage of many proposed optimizers can be attributed to baseline undertuning rather than algorithmic innovation.

This is a fundamentally different claim from the usual "we tuned the baseline more carefully" defense. The paper demonstrates that the phenomenon is systematic, not incidental: Appendix G traces the same undertuned baseline through paper after paper, and shows that the speedup claims are proportional to how far from optimal the baseline's learning rate was. Papers that performed independent AdamW tuning (most notably Soap, which reported 1.4×) produced speedup estimates close to what this paper measures. Papers that used the GPT-3 recipe with lr=6e-4 almost universally claimed 2×. The paper is not accusing individual authors of malfeasance — it is documenting a structural failure in evaluation methodology where the default baseline in a widely-adopted codebase (the GPT-3 recipe, distributed in frameworks like Levanter and used as a starting point by many researchers) is far from optimal for modern pretraining, and this error propagates through the literature because each new paper inherits the same weak baseline.

The significance of this finding extends beyond optimizer evaluation. It implies that the field of pretraining optimization has been overcounting progress — the apparent improvements from novel optimizers are partly real (the paper does measure genuine 1.1–1.4× speedups for matrix-based methods), but the headline 2× numbers that motivated adoption are largely explainable by a single hyperparameter bug. More broadly, it demonstrates how a measurement infrastructure failure (the lack of a properly tuned reference baseline) can distort an entire subfield's perception of progress, a pattern documented in other areas of ML (Henderson et al., 2018 on RL; Shchur et al., 2018 on GNNs) but never before quantified so precisely in the pretraining optimizer context. The paper's evidence is Figure 1 (top left) showing the 2× gap from learning rate alone, the speedup ratio analysis in Figure 3 capping real gains at 1.4×, and Appendix G's systematic comparison of each prior paper's setup with the undertuned baseline they used.

This is a fundamental reframing, not an incremental improvement. Before this paper, the dominant assumption was that each proposed optimizer's reported speedup reflected genuine algorithmic superiority, and non-adoption was attributed to engineering inertia. After this paper, the default assumption should be that any claimed speedup over 1.3–1.4× likely reflects baseline undertuning unless the baseline's hyperparameters have been independently verified as optimal through systematic sweeps.

Innovation 2: Matrix-Based Preconditioning Is a Robust Advantage Over Scalar Methods at Small Scale — But the Gain Is Algorithm-Invariant, Not Algorithm-Specific

The paper identifies a clear empirical regularity that cuts across optimizer families: all matrix-based optimizers achieve roughly the same speedup over AdamW (~1.3× at 130M), despite having completely different update rules (Muon uses Newton-Schulz orthogonalization, Soap uses Shampoo-style Kronecker preconditioning with Adam adaptivity, Kron uses online-learned Kronecker-factored preconditioners via conjugate sketches, Scion uses the same as Muon but with sign-based updates for embeddings). Figure 2 shows Muon, Soap, Kron, and Scion all achieving similar loss values at each Chinchilla ratio, consistently below all scalar-based optimizers. Figure 3 quantifies this as speedup ratios clustering in the 1.2–1.4× band for matrix methods versus 1.0–1.2× for scalar methods.

What makes this observation conceptually novel is not the existence of matrix preconditioning (which dates back to Shampoo in 2018 and PSGD in 2018), but the finding that the advantage is robust to which matrix preconditioning algorithm you use, as long as you use some matrix structure. This is an implicit ablation via landscape: the paper did not design a controlled experiment varying the matrix preconditioner while holding all else equal — instead, it observed that three fundamentally different algorithms (Muon's orthogonalization, Soap's Shampoo-Adam hybrid, Kron's online learning) converge to similar gains. This convergence across diverse implementations suggests that the matrix structure itself — the fact of multiplying gradients by matrices rather than scalars — is the operative mechanism, not the specific mathematical properties of any particular matrix decomposition.

Prior work treated matrix-based optimizers as a collection of individual algorithms, each making specific theoretical claims about why their particular preconditioner is better (e.g., Muon emphasizes orthogonalization, Soap emphasizes Shampoo's spectral properties, Kron emphasizes online curvature estimation). The paper's results suggest these theoretical distinctions may be less important than the shared architectural choice of matrix preconditioning. This matters because it redirects future research: rather than refining the specific orthogonalization algorithm (e.g., finding better Newton-Schulz variants), the priority should be understanding why matrix structure helps at a fundamental level — perhaps through better conditioning, perhaps through implicit regularization, perhaps through more efficient exploration of the loss landscape geometry — and whether this advantage can be preserved at larger scales. The fact that all three matrix methods degrade similarly with model size (from 1.3–1.4× at 130M to ~1.1× at 1.2B, Figure 4 middle) further supports the hypothesis that the advantage is a shared property of matrix conditioning, not an algorithm-specific effect.

This is an empirical reframing rather than a theoretical advance. The paper does not explain why matrix preconditioning helps — it observes that it does, consistently, across diverse algorithms, and that this advantage shrinks with scale. The value is in converting a collection of individual algorithm claims into a unified phenomenon ("matrix-based methods outperform scalar-based methods by ~1.3× at small scale") that can be studied as a single object, and in identifying the scale-dependence of this phenomenon as the key open question.

Innovation 3: Optimizer Rankings Are Not Scale-Invariant — and This Matters for Practical Decision-Making

The paper provides the first systematic evidence that which optimizer is "best" depends on both model size and data budget, and these dependencies can reverse rankings. The three specific reversals documented are:

Muon is best at 1–4× Chinchilla but overtaken by Soap and Kron at 8–16× Chinchilla (Figure 2, Figure 4 right, Figure 8). On 130M models, Muon achieves the lowest loss at 1× Chinchilla, but at 8× and 16×, Soap and Kron pull ahead. At 300M, the same pattern: Soap surpasses Muon at 16× Chinchilla.
Matrix-based speedups decay with model size (Figure 1, bottom left; Figure 4, middle): Muon's speedup over AdamW drops from ~1.4× at 130M to ~1.1× at 1.2B, and the fitted scaling law projects that Muon will underperform AdamW at 7B parameters (Appendix B.1).
Sophia underperforms AdamW at small scales but becomes competitive at larger scales (Figure 7): Sophia is noticeably worse at 130M and 300M, achieves parity around 520M, and the trend suggests it might surpass AdamW at larger models — the opposite of how matrix-based methods scale.

These reversals are not minor reorderings — they would lead an organization to make actively wrong decisions if they chose an optimizer based on 130M-scale experiments at 1× Chinchilla and then deployed it at production scale. The projected Muon-AdamW crossover at 7B is particularly striking because it means that an optimizer that provides a meaningful speedup at the scale where it was evaluated could become a slowdown at the scale where it would be used — a failure mode that standard optimizer evaluation protocols are completely blind to.

Prior work on optimizer scaling (Everett et al., 2024; Zhang et al., 2025a) focused primarily on how hyperparameters scale with model size under a fixed optimizer, not on how the ranking between optimizers changes with scale. The paper's key conceptual move is to treat optimizer choice as a function of scale — just as the optimal learning rate changes with model size, the optimal optimizer identity may also change. This reframes optimizer selection from a one-time decision ("which optimizer should I use?") to a scaling-aware decision ("which optimizer should I use at this specific model size and data budget?").

The practical implication is that optimizer evaluations conducted only at small scale (under 1B parameters) and a single data budget (typically 1× Chinchilla) are insufficient for decision-making — they can produce rankings that reverse at the scales that actually matter for production. The paper's methodological recommendation — evaluate across multiple scales and data budgets, and extrapolate scaling trends — is not novel in itself, but the empirical demonstration that failure to do so would produce wrong decisions is a significant cautionary finding for the field. This is a fundamental empirical contribution that changes the standards for what counts as adequate optimizer evaluation.

Innovation 4: Verifier-Like Over-Optimization Does Not Apply to Optimizers — But a Different Form of Overfitting Does

One might expect, by analogy to other areas of ML (test-time compute scaling, RLHF reward hacking), that a more powerful optimizer would eventually "over-optimize" the training loss — driving training loss lower while harming validation loss through some form of implicit regularization breakdown. The paper tests this implicitly (Figure 6, right; Appendix B.4) and finds that different optimizers have remarkably similar generalization behavior: the training loss and evaluation loss follow the same trend for all optimizers, meaning a lower training loss translates to a proportionally lower validation loss, with no evidence of generalization gap divergence. HellaSwag accuracy improvements also closely mirror validation loss improvements (Figure 2, bottom row). This is a non-result, but an important one: it tells the field that optimizer choice affects optimization speed, not the shape of the generalization curve — at least in the pretraining regime tested (up to 1.2B parameters, up to 8× Chinchilla).

However, the paper documents a different form of "overfitting" that is more subtle: optimizer rankings overfit to evaluation scale. An optimizer that looks best at 130M and 1× Chinchilla (the most common evaluation regime in prior work) may not be the best at 1.2B and 8× Chinchilla (the regime where it would actually be deployed). The paper's scaling law extrapolation makes this overfitting-to-scale visible: Muon's superior performance at small scale does not generalize to larger scale, in the same way that a model's superior performance on a narrow validation set may not generalize to deployment. The difference is that in the pretraining context, "scale" is the dimension of generalization, and the paper shows that standard evaluation practice (test on one scale, assume ranking invariance) is equivalent to training on one data distribution and testing on another without checking for distribution shift.

This framing — optimizer evaluation as a generalization problem across scales rather than a fixed-scale comparison — is a conceptual contribution that reframes how the field should think about optimizer benchmarking. It connects the paper's empirical findings to the broader scaling laws literature: just as Hoffmann et al. (2022) showed that the optimal model size and data quantity must be jointly optimized (you cannot decide on the model size first and then train it on whatever data is available), this paper shows that the optimizer choice and the training scale must be jointly considered (you cannot choose the optimizer at 130M and then assume it will remain optimal at 1.2B or 7B). This is an incremental conceptual advance — it does not invent a new theory, but it provides the empirical evidence needed to shift the field's default assumptions from "optimizer rankings are stable" to "optimizer rankings must be verified at the target scale."

The concrete evidence is Figure 4 (middle) showing speedup decay, Appendix B.1's 7B projection, and the difficulty-bin-like analysis that the paper implicitly performs by stratifying results across model sizes and Chinchilla ratios (Figures 2, 3, 4, 8). The pattern is consistent: the effect size of any optimizer innovation depends on the measurement scale, and ignoring this scale-dependence produces fragile conclusions.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the C4/EN (English) validation split of the C4 dataset (Raffel et al., 2023) as the primary evaluation metric, drawn from the same distribution as the pretraining corpus (DCLM-baseline + StarCoder V2 + ProofPile 2, ~4.1T tokens total, tokenized with the Llama3 tokenizer). The paper invokes Bhagia et al. (2024) to justify C4/EN loss as "a known proxy for downstream performance" (Section 3.1), and validates this by reporting downstream accuracy on 10 benchmarks: ARC Easy, ARC Challenge, BoolQ, COPA, CommonsenseQA, HellaSwag, LAMBADA, OpenBookQA, PIQA, WSC273, and Winogrande (Appendix B.4, Tables 5–35).
Base model(s). All experiments use the Llama 2 architecture (Touvron et al., 2023b; Grattafiori et al., 2024) at four scales: 130M, 300M, 520M, and 1.2B parameters. Each variant uses a fixed sequence length of 4096 and 32 transformer layers (following MobileLLM; Liu et al., 2024b), with model capacity scaled by varying hidden dimension (512 → 1536), intermediate dimension (2048 → 6144), and number of attention heads (8 → 24), as detailed in Table 2. The choice is motivated by representativeness — Llama 2 is a standard open architecture used in many optimizer papers — and the scale range (0.1–1.2B) is comparable to prior optimizer evaluations (Sophia tested up to 540M, Soap up to 600M, Muon/Kimi up to 1.5B).
Metrics. The primary metric is final validation loss on the C4/EN split, measured at the completion of the prescribed training budget (1–16× Chinchilla ratio, where 1× Chinchilla = 20 × non-embedding parameter count tokens, following Hoffmann et al., 2022). Speedup ratios are computed by fitting an AdamW-specific loss scaling law $\hat{L}_N(D) = \alpha_N D^{-B_N} + \beta_N$ for each model size $N$ and mapping each optimizer's achieved loss at its data budget $D_{\text{optimizer}}$ to the equivalent AdamW data budget $D_{\text{AdamW}}$ needed to achieve the same loss; the speedup ratio is $D_{\text{AdamW}} / D_{\text{optimizer}}$ (Section 3.3). Downstream evaluation tracks accuracy and bits-per-byte on the 10 benchmarks listed above.
Baselines. AdamW (Kingma and Ba, 2017; Loshchilov and Hutter, 2019) serves as the primary baseline against which all other optimizers are compared. Within specific optimizer classes, additional comparisons are made implicitly — NAdamW (Dozat, 2016), Mars (Yuan et al., 2025), and Cautious (Liang et al., 2025) are compared as variance-reduced AdamW variants; Lion (Chen et al., 2023) and Adam-mini (Zhang et al., 2025b) as memory-efficient variants; Muon (Jordan et al., 2024; Liu et al., 2025a), Scion (Pethick et al., 2025), Kron/PSGD (Li, 2018a, 2022), and Soap (Vyas et al., 2025) as matrix-based methods; and Sophia (Liu et al., 2024a) as a Hessian-approximation method (algorithms detailed in Appendix A). The paper does not directly compare across all 10 non-AdamW optimizers in every experiment — Phase III and the 16× Chinchilla case studies narrow to AdamW, NAdamW, Muon, and Soap.
Generation budget / compute accounting. The paper compares optimizers by the number of tokens needed to reach a given loss, not by wall-clock time or FLOPs. This choice is motivated by the claim that "the step-wise computation overhead of matrix-based optimizers can be reduced to under 10% through proper implementation" (Section 1, citing Vyas et al., 2025; Liu et al., 2025a), implying that token efficiency gains translate approximately 1:1 to wall-clock gains. For each model size and Chinchilla ratio, the compute budget is fixed in tokens: a 130M model at 1× Chinchilla sees ~2.6B tokens; at 8× Chinchilla, ~20.8B tokens; at 16× Chinchilla (Phase III only), ~41.6B tokens. Hyperparameter sweeps during Phase I and Phase II each consume a full Chinchilla budget per trial.
Cross-validation / statistical protocol. No cross-validation, error bars, or statistical significance testing is reported. The paper uses a single training run per hyperparameter configuration per regime, identified through the coordinate descent procedure (Sections 3.2–3.3) as the configuration achieving minimum validation loss. The convergence threshold $\Delta_1 = 3 \times 10^{-3}$ for Phase I and $\Delta_2 = 6.4 \times 10^{-3}$ for the scaling-sensitivity classification provide a form of empirical robustness — configurations within these thresholds are considered effectively equivalent — but the paper does not perform replicate runs to estimate variance. The scaling law fits (Appendix B.1) report RMS fitting errors ( $3 \times 10^{-3}$ for AdamW, $5 \times 10^{-3}$ for Muon), which capture interpolation quality but not prediction uncertainty at extrapolated scales.

Main Quantitative Results

Hyperparameter Tuning Yields a 2× Speedup for AdamW Over the Widely-Used Baseline

The paper's most immediately striking result is a single ablation: starting from the GPT-3 recipe (Brown et al., 2020) with peak learning rate 6e-4 — the AdamW baseline used in Sophia, MARS, Cautious AdamW, FOCUS, and other papers — simply tuning the learning rate to 8e-3 produces nearly a 2× speedup for a 130M model (Figure 1, top left). The figure plots loss versus training steps for three configurations: "AdamW w/ lr 6e-4" (the GPT-3 recipe), "AdamW w/ lr 8e-3" (tuned), and "Mars" and "Nesterov AdamW" for reference. The tuned AdamW reaches a loss of approximately 3.6 roughly 5000 steps earlier than the GPT-3 recipe baseline, or equivalently matches the baseline's final loss using approximately half the tokens. This single result demonstrates that the claimed 2× speedups in prior work can be explained entirely by baseline undertuning rather than algorithmic innovation.

The paper reinforces this with the weight decay ablation in Figure 1 (top right): for a 130M model, Lion's optimal weight decay is approximately 0.6, while AdamW's is approximately 0.1. Fixing weight decay at 0.1 for Lion (as done in prior work that "keep common hyperparameters... fixed across optimizers," Section 1) would penalize Lion by approximately 0.02 in final loss, enough to flip the comparison between Lion and AdamW.

Matrix-Based Optimizers Consistently Outperform Scalar-Based Optimizers at Small Scale

Figure 2 (top row) shows the C4/EN validation loss for all optimizers at 130M, 300M, and 520M across Chinchilla ratios 1–8×. Every point represents the optimal loss achieved at the corresponding Chinchilla ratio for each optimizer after Phase I and Phase II tuning. The visual pattern is clear: the three solid lines at the bottom of each plot — corresponding to Soap, Muon, and Kron — consistently lie below all dashed lines (scalar-based optimizers: AdamW, NAdamW, Mars, Cautious, Lion, Adam-mini). Scion tracks close to this group but sometimes falls in the scalar-based cluster. The scalar-based optimizers themselves cluster tightly, with NAdamW and Mars showing small but consistent advantages over AdamW, while Lion and Adam-mini track AdamW closely.

Quantitatively, at 130M and 1× Chinchilla (Table 36–93, Appendix C, D), AdamW achieves loss 3.529; Muon reaches 3.464 (a difference of 0.065); Soap reaches 3.483 (difference 0.046); Kron reaches 3.492 (difference 0.037). At 130M and 8× Chinchilla, AdamW reaches 3.262; Muon reaches 3.240 (difference 0.022); Soap reaches 3.239 (difference 0.023); Kron reaches 3.239 (difference 0.023). The gaps narrow at higher data budgets but matrix-based methods remain above.

The speedup ratio analysis in Figure 3 translates these loss differences into equivalent AdamW data budgets. For the 130M model, Muon, Soap, and Kron fall in the 1.3–1.4× speedup band at higher Chinchilla ratios (their points on the $D_{\text{AdamW}}$ vs. $D_{\text{optimizer}}$ plot lie below the diagonal, in the dark-colored band). For the 300M and 520M models, the matrix-based methods still outperform scalar-based methods but the speedup band shifts to 1.2–1.3× for most points, indicating decay with model size.

Scalar-Based Variance-Reduced Methods Provide Modest and Consistent Gains Over AdamW

Within the scalar-based family, NAdamW, Mars, and Cautious consistently outperform vanilla AdamW, but the margin is small. Figure 2 shows their loss curves track just below AdamW across all scales. At 130M, 1× Chinchilla (Appendix C, Tables 36, 59, 53, 42): AdamW achieves 3.529; NAdamW 3.531 (essentially tied); Mars 3.537 (slightly worse); Cautious 3.535 (roughly tied). At 300M, 1× Chinchilla: AdamW 3.264; NAdamW 3.248 (gap 0.016); Mars 3.249 (gap 0.015); Cautious 3.260 (gap 0.004). At 520M, 1× Chinchilla: AdamW 3.110; NAdamW 3.100 (gap 0.010); Mars 3.101 (gap 0.009); Cautious 3.100 (gap 0.010). The gap widens at higher Chinchilla ratios but never exceeds approximately 0.02–0.03 in loss, corresponding to speedup ratios of 1.1–1.2× (Figure 3).

Notably, Mars shows the strongest scaling behavior among scalar-based methods — at 130M and 8× Chinchilla, Mars achieves 3.247 vs. AdamW 3.262 (gap 0.015), and at 520M and 8× Chinchilla, Mars achieves 2.906 vs. AdamW 2.913 (gap 0.007) — but the speedup never exceeds 1.2× in the computed ratios (Figure 3, right panel). This is dramatically lower than the 2× speedup claimed in the original Mars paper (Yuan et al., 2025), which the paper attributes to the Mars authors' use of the GPT-3 recipe's learning rate of 6e-4 for their AdamW baseline.

Memory-Efficient Optimizers Track AdamW Closely, With Scale-Dependent Behavior

Lion and Adam-mini — despite their reduced memory footprints — achieve validation losses within approximately 0.02–0.03 of AdamW across all tested scales (Figure 2). At 130M, 1× Chinchilla: AdamW 3.529, Lion 3.552 (gap 0.023), Adam-mini 3.542 (gap 0.013). At 520M, 1× Chinchilla: AdamW 3.110, Lion 3.108 (essentially tied), Adam-mini 3.112 (gap 0.002). The paper notes an interesting scale-dependent trend: "the disadvantage of Lion relative to AdamW widens while the disadvantage of Adam-mini over AdamW narrows" (Section 4.1, point 3). Quantitatively, at 130M and 8× Chinchilla, Lion is at 3.252 vs. AdamW 3.262 (gap -0.010, slight advantage), but at 300M and 8× Chinchilla, Lion is at 3.046 vs. AdamW 3.043 (gap +0.003, slight disadvantage), and the trend toward relative disadvantage continues at 520M. Adam-mini shows the reverse: at 130M and 8×, it trails AdamW by 0.004; at 300M and 8×, by 0.006; at 520M and 8×, by only -0.001 (essentially tied).

This reversal is modest in magnitude but conceptually important — it demonstrates that even within the memory-efficient class, scaling behavior is not uniform, and the optimizer that minimizes memory at one scale may not be the same one that minimizes loss at another.

Speedup of All Alternative Optimizers Decays With Model Size

Figure 1 (bottom left) and Figure 4 (left and middle) provide the paper's most sobering scaling result. Figure 1 shows speedup ratios for AdamW, NAdamW, Muon, and Soap at four model sizes (130M to 1.2B) at 8× Chinchilla. AdamW is fixed at 1.0× by definition; NAdamW hugs close to 1.0–1.05×; Muon drops from ~1.3× at 130M to ~1.1× at 1.2B; Soap shows a similar decay. Figure 4 (middle) quantifies this for the 1.2B model specifically: Muon and Soap both show speedups of approximately 1.1× across Chinchilla ratios 1–8×, while NAdamW hovers around 1.05–1.1×. The fitted scaling laws in Appendix B.1 project this trend forward: "when the parameter scale reaches 7B, Muon will actually result in a higher loss compared to AdamW in 1× Chinchilla regime."

The raw loss values tell the same story (Figure 4, left): at 1.2B and 8× Chinchilla, AdamW achieves 2.752, NAdamW 2.749, Muon 2.748, Soap 2.749 — all four optimizers are within 0.004 of each other. At 1× Chinchilla, the gaps are only slightly larger: AdamW 2.905, NAdamW 2.902, Muon 2.891, Soap 2.940. Soap actually performs worst at 1.2B and 1× Chinchilla (loss 2.940 vs. AdamW 2.905), though the paper notes this may be partially due to its Soap implementation using parameter blocking and bfloat16 momentum (Appendix G).

Table 5 in the main text confirms that downstream evaluation gains also vanish at 1.2B scale: the average benchmark accuracy across all 10 tasks is 67.15 for AdamW, 66.70 for NAdamW, and 66.98 for Muon at 8× Chinchilla — differences of less than 0.5 percentage points in average accuracy, well within the noise of individual benchmark variance.

Optimizer Rankings Reverse at High Data-to-Model Ratios

Figure 4 (right) and Figure 8 present the 16× Chinchilla case studies. At 130M and 16× Chinchilla (Figure 8): AdamW achieves 3.207, NAdamW 3.200, Soap 3.191, Muon 3.192. Soap and Muon are nearly tied, but NAdamW — a simple scalar-based Nesterov variant — is only 0.008 behind the best matrix-based method, substantially closer than at 1–8× Chinchilla. At 300M and 16× Chinchilla (Figure 4, right): AdamW 3.001, NAdamW 2.998, Soap 2.990, Muon 2.991. Here Soap pulls ahead of Muon by a small but consistent margin, and the gap between the best scalar method (NAdamW) and the best matrix method (Soap) is only 0.008 — much narrower than at 8× Chinchilla (where Muon led AdamW by ~0.01 at 300M).

The paper's interpretation (Section 4.1, "High data-to-model Ratio"): "We conjecture that the second-order momentum maintained by Soap and Kron becomes more effective when the data-to-model ratio increases. In the long run, adaptivity to heterogeneity in parameter directions may lead to a larger speedup." This is plausible — Soap's Shampoo-style preconditioners accumulate curvature information over time, and with more data (16× Chinchilla = ~48B tokens for 300M), the accumulated curvature estimates become more accurate — but it is not empirically isolated as the causal mechanism.

HellaSwag Performance Mirrors Validation Loss Trends

Figure 2 (bottom row) plots HellaSwag 0-shot accuracy for the AdamW baseline, the top 2 scalar-based optimizers (NAdamW, Mars), and the top 3 matrix-based optimizers (Muon, Soap, Kron) at each Chinchilla ratio for 130M, 300M, and 520M models. The ranking is preserved: matrix-based methods achieve the highest accuracy at each point, scalar-based methods cluster below. At 130M and 1× Chinchilla: AdamW ~0.312, Mars ~0.308, NAdamW ~0.312, Muon ~0.330, Soap ~0.321, Kron ~0.323. At 520M and 8× Chinchilla: AdamW ~0.578, NAdamW ~0.580, Mars ~0.579, Muon ~0.587, Soap ~0.586, Kron ~0.582. The gaps narrow with scale, matching the validation loss trend. The full downstream evaluation tables (Appendix B.4, Tables 6–35) confirm this pattern across all benchmarks: while individual benchmark scores are noisy, the average over 10 tasks consistently tracks validation loss improvements, with the matrix-based methods at the top and scalar-based methods clustered below.

Muon Outperforms All Optimizers at 1–4× Chinchilla Across Small-to-Medium Scales

Across all Phase I and Phase II experiments at 130M–520M and 1–4× Chinchilla, Muon achieves the lowest or tied-for-lowest validation loss in every setting (Figure 2). At 130M, 1× Chinchilla: Muon 3.464, Soap 3.483, Kron 3.492, AdamW 3.529. At 300M, 1×: Muon 3.224, Soap 3.231, Kron 3.244, AdamW 3.264. At 520M, 1×: Muon 3.073, Soap 3.079, Kron 3.084, AdamW 3.110. At 130M, 4×: Muon 3.296, Soap 3.295, Kron 3.307, AdamW 3.322. This is a clear dominance — Muon is the best optimizer in this regime by a margin that, while modest (0.01–0.05 loss), is consistent across scales and ratios.

However, the paper notes that this dominance does not extend to 8× Chinchilla for all model sizes: at 130M and 8×, Muon (3.240) is tied with Soap (3.239) and Kron (3.239), and at 520M and 8×, Muon (2.906) is tied with Soap (2.899 — actually slightly better) and Kron (2.900). The crossover is even clearer at 16× Chinchilla (Figure 4, right; Figure 8), where Soap leads Muon by 0.001–0.002 at 130M and by 0.001 at 300M — narrow gaps, but directionally consistent with the trend.

Sophia Underperforms AdamW at Small Scale, Approaches Parity at 520M

Figure 7 (Appendix B.2) presents the Sophia results. At 130M and 1× Chinchilla, Sophia achieves 3.544 vs. AdamW 3.529 (gap +0.015, worse). At 300M and 1×, Sophia 3.267 vs. AdamW 3.264 (gap +0.003, essentially tied). At 520M and 1×, Sophia 3.106 vs. AdamW 3.110 (gap -0.004, slightly better). Across Chinchilla ratios for 130M (Figure 7, right): Sophia trails AdamW at 1× (3.544 vs. 3.529), 2× (3.414 vs. 3.409), 4× (3.330 vs. 3.322), and 8× (3.259 vs. 3.262). The paper attributes Sophia's previously reported 2× speedup (Liu et al., 2024a) to the non-random data shuffling in the original Sophia codebase (Levanter), which forced a smaller optimal learning rate for AdamW; with proper random shuffling and a well-tuned baseline, "Sophia doesn't offer significant speedup over AdamW for models under 0.5B" (Section 4.2).

Ablation Studies and Robustness Checks

Learning rate in the GPT-3 recipe: Tuning only the peak learning rate from 6e-4 to 8e-3 yields a ~2× speedup for AdamW (Figure 1, top left), demonstrating that the widely-used baseline is severely undertuned. This is the foundational ablation that supports the paper's central methodological critique.

Weight decay transfer across optimizers: Optimal weight decay for Lion (~0.6) and Kron (~0.5) differs substantially from AdamW (~0.1), as shown in Figure 1 (top right) and the hyperparameter tables (Appendix C, Tables 47, 70). Fixing weight decay at 0.1 would penalize Lion by approximately 0.02 in final loss at 130M and 1× Chinchilla (Table 47: λ=0.1 gives 3.562, λ=0.6 gives 3.549), enough to flip several optimizer comparisons.

Phase I coordinate descent convergence: Tables 36–93 in Appendix C document the full coordinate descent procedure for every optimizer. For AdamW on 130M at 1× (Table 36), the procedure tests 19 hyperparameter configurations, moving from initial loss 3.529 to final 3.529 (the optimum was essentially the starting point for this configuration). For Lion on 130M at 1× (Table 47), 26 configurations are tested, optimizing β₁, β₂, learning rate, batch size, warmup, and weight decay. The fact that many optimizers require 15–30 runs to converge demonstrates that hand-tuning or limited sweeps would likely miss the optimum.

Phase II focused descent on scaling-sensitive parameters: Tables 100–159 in Appendix D show the Phase II sweeps for each optimizer. These are sparser than Phase I — typically 2–4 variations per optimizer per regime (e.g., AdamW on 300M at 2× Chinchilla, Table 100, tests only 3 additional configurations beyond the Phase I optimum transferred from smaller scale). The fact that these sparse sweeps still find improved configurations (e.g., AdamW on 300M at 8×, Table 102, improves from 3.043 to 3.042 by switching learning rate to 0.004) validates the sensitivity classification from Phase I.

Hyperparameter scaling law validation at 1.2B: The paper runs a full Phase I sweep for AdamW at 1.2B and 1× Chinchilla (Table 160, Appendix E) and confirms that the hyperparameters predicted by the scaling law "yield a final loss within 3e-3 of the optimal configuration" (Section 3.4). The predicted configuration (lr=0.002, weight decay=0.2, batch size=256, warmup=2000) achieves loss 2.905, while the empirically optimal configuration from the full sweep also achieves 2.905. This is a strong validation of the extrapolation methodology, though it only validates for AdamW, not for the other optimizers whose 1.2B hyperparameters are predicted but not empirically verified.

Muon embedding layer learning rate tuning: The Phase III Muon experiments (Tables 172–175, Appendix E) separately tune $\eta_{\text{adam}}$ (learning rate for embedding, LM head, and LayerNorm parameters) from $\eta$ (learning rate for matrix parameters in transformer layers). At 1.2B and 1× Chinchilla, the configuration with $\eta_{\text{adam}}=0.0012$ and $\eta=0.004$ achieves loss 2.891; when $\eta=0.008$ is tested instead, loss increases to 2.886 (Table 172 shows both $\eta=0.004$ and $\eta=0.008$ , with the note that 0.004 gave 2.891 and 0.008 gave 2.886 — actually a slight improvement, though the paper's claimed optimal is 0.004). The paper notes that Semenov et al. (2025) did not separately tune these learning rates, which "also improves [Muon's] performance" and may partly explain their finding that Muon underperforms variance-reduced methods.

Blocked vs. unblocked Soap: The paper's Soap implementation uses parameter blocking (partitioning large weight matrices into 512 × 512 blocks, following Anil et al., 2021) to reduce memory, and bfloat16 momentum for the 1.2B experiments. The paper acknowledges this as a limitation (Appendix G: "both modifications may lead to slightly lower step-wise performance"), but the effect appears small: Soap's performance remains competitive with Muon and Kron across all scales (Figures 2, 3, 4), suggesting the blocking penalty is minor in practice.

Training vs. validation loss generalization gap: Figure 6 (right) plots training loss against validation loss for all optimizers at the 520M scale and finds that all points lie on roughly the same curve — a lower training loss corresponds to a proportionally lower validation loss, with no optimizer showing systematically better or worse generalization. This is a robustness check against the concern that more aggressive optimizers might reduce training loss at the expense of generalization; the data shows this is not the case in the tested regimes.

Gradient and weight norm trajectories: Figure 6 (middle left and middle right) shows that all optimizers exhibit the same qualitative patterns: parameter norms increase during training and then decrease during learning rate decay, and gradient norms increase during the decay phase. These phenomena, previously reported for AdamW by Defazio (2025), are shown to be universal across optimizers, suggesting they arise from the learning rate schedule rather than the optimizer update rule.

Different optimizers reach similar minima at high data budgets: Figure 1 (bottom right) and Figure 2 (top right) show that at 8× Chinchilla for the 520M model, Muon (2.906), Soap (2.899), Kron (2.900), Mars (2.906), and NAdamW (2.907) all converge within 0.008 of each other, while AdamW is at 2.913. This convergence of loss values at high data budgets suggests that with enough training, the choice of optimizer becomes less impactful — a form of "overtraining washes out optimizer differences" that has practical implications for long-duration pretraining runs.

No single hyperparameter dominates across all optimizers: The full ablation tables in Appendices C and D demonstrate that the set of scaling-sensitive parameters differs per optimizer (Table 4). For example, β₂ is scaling-sensitive for Lion (optimal changing from 0.95 at 130M to 0.98 at 300M and above, Tables 47–52) but scaling-insensitive for AdamW, NAdamW, and Mars. Batch size is scaling-sensitive for AdamW and Cautious but not for most other optimizers. This validates the paper's core methodological claim that hyperparameter optimization must be optimizer-specific.

Critical Assessment

Claim 1: "Many claimed speedups stem from insufficient hyperparameter tuning — fair sweeps eliminate most of the apparent advantages"

Assessment: Strongly supported, but the paper is measuring something narrower than "all reported speedups are false." What the paper demonstrates is that the GPT-3 recipe (peak learning rate 6e-4) produces a 2× undertuned AdamW baseline, and that this specific recipe was the AdamW baseline in at least six prior papers (Sophia, MARS, Cautious AdamW, FOCUS, Block-wise LR Adam, and implicitly Muon/Kimi). The evidence is Figure 1 (top left) showing the 2× gap, Appendix G's systematic tracing of this baseline through prior work, and the contrast with Soap — the one prior paper that performed independent AdamW tuning and reported 1.4× speedup, close to this paper's measurement of 1.2–1.3×.

However, the paper has not run a controlled experiment where it exactly replicates each prior paper's setup (architecture, data, hyperparameters, codebase) and then re-tunes AdamW to measure the true speedup. Instead, it demonstrates that in its own experimental setup, the claimed speedups do not materialize, and argues that the GPT-3 recipe explains the discrepancy. This is persuasive but indirect — a skeptic from one of the prior papers could argue that the difference arises from this paper's different data mixture (OLMo 2 vs. PILE), different architecture (Llama 2 vs. GPT-3), or different hardware (TPU vs. GPU), rather than undertuning per se. The paper's discussion of the Sophia codebase's non-random data shuffling (Section 4.2, Appendix G) partially addresses this by identifying a specific mechanism, but for other papers the attribution is correlational rather than causal.

A missing experiment would be: take the exact Sophia codebase and configuration, reproduce the claimed 2× speedup, then re-tune only AdamW's learning rate in that codebase and measure whether the speedup disappears. This would directly test the mechanism rather than demonstrating it in a different setup.

Claim 2: "Matrix-based optimizers consistently outperform scalar-based optimizers" and "the best-performing alternatives provide only modest speedups, which further diminish with model scale"

Assessment: The first half is strongly supported within the tested range (130M–520M at 1–8× Chinchilla). Figure 2 shows a clear separation: all three matrix-based methods (Muon, Soap, Kron) achieve lower loss than all scalar-based methods at every tested point, and Figure 3 quantifies the speedup at 1.2–1.4×. The consistency is impressive — this is not a borderline result that depends on cherry-picked checkpoints.

The second half (diminishing with scale) is supported but the evidence base is thinner. The paper has only 4 data points on the scaling curve: 130M (speedup ~1.3×), 300M (~1.2–1.3×), 520M (~1.2×), 1.2B (~1.1×). At 1.2B and 8× Chinchilla (Figure 4, left), the loss differences are within 0.004 — AdamW 2.752, Muon 2.748, Soap 2.749. These differences are smaller than the Phase I convergence threshold $\Delta_1 = 3 \times 10^{-3}$ , meaning they could be within the noise of the hyperparameter optimization procedure. The paper acknowledges this implicitly by noting that downstream benchmarks show no meaningful differences at 1.2B (Table 5), but does not discuss whether the 1.1× speedup at 1.2B is statistically distinguishable from 1.0×. Without replicate runs or confidence intervals, the claim that Muon maintains a 1.1× advantage at 1.2B is based on a single training run per optimizer.

The 7B projection (Appendix B.1) extrapolating Muon to underperform AdamW is provocative but should be treated as a hypothesis, not a finding. The scaling law is fitted on 4 points (130M–1.2B) and extrapolated 6× beyond the training range to 7B. The RMS error on the training data is $5 \times 10^{-3}$ for Muon, and the predicted crossover could easily be an artifact of the parametric form (which assumes power-law scaling that may not hold at larger scales). The paper itself is careful to present this as a prediction, not a conclusion, but the framing of "Muon will actually result in a higher loss" in places reads more definitively than the evidence warrants.

Claim 3: "Comparisons based on early or inconsistent evaluation can be misleading, since optimizer rankings often change over the full training trajectory"

Assessment: Supported by the qualitative evidence in Figure 5 (right), but the paper provides only two examples (AdamW vs. Muon at 520M 8×, and the weight decay ablation for Muon at 520M 1×). In Figure 5 (right), Muon with weight decay 0 shows faster early loss improvement but plateaus, allowing Muon with weight decay 0.1 to overtake it late in training. In the left panel of Figure 5, 2× the optimal learning rate causes Soap to underperform Mars early but the ranking could change (the plot cuts off before the end of training for that configuration).

The paper does not systematically quantify how often early rankings flip. A table showing, for all optimizer pairs at each scale and ratio, the fraction of training steps where the ranking disagrees with the final ranking would be much more informative than the two anecdotal examples provided. The paper's methodological recommendation — "evaluating optimizers only on the final checkpoints rather than relying on intermediate checkpoints" — is sensible but is supported by illustration rather than systematic evidence.

Additionally, the paper argues (Section 4.2) that the Muon/Kimi comparison on MoE models "compare[s] two models with not fully decayed learning rate and this may significantly favor Muon, as shown in Figure 5." This is an important observation but the supporting figure (Figure 5, right) shows a different phenomenon (weight decay effect on loss plateaus, not learning rate decay truncation). A direct experiment would test Muon vs. AdamW with truncated training schedules to see if early termination changes the apparent speedup.

Claim 4: "All the fastest optimizers... use matrices as preconditioners — multiplying gradients with matrices rather than entry-wise scalars"

Assessment: This is an empirical regularity, not a tested causal claim. The paper observes that Muon, Soap, Kron, and Scion all outperform scalar-based methods, but it has not run a controlled ablation that isolates matrix preconditioning as the mechanism. The fact that these four optimizers use completely different matrix operations (Newton-Schulz orthogonalization, Shampoo-Adam hybrid, online Kronecker learning, and Muon-like with sign-based embeddings) suggests, as the paper notes, that "the matrix structure itself — not any particular algorithm for exploiting it — is the source of the gain" (this is our framing from the Key Insights, not a direct quote, but it captures the paper's implication). However, the paper does not test this by, e.g., comparing a matrix-based method with a scalar-based method that is otherwise identical in its momentum, weight decay, and learning rate schedule.

A missing experiment would be to take Muon and "scalarize" it — replace the Newton-Schulz matrix orthogonalization with an element-wise sign or normalization that preserves the per-element update magnitude but loses the matrix structure — and measure how much of the speedup disappears. Without this, the claim that matrix preconditioning causes the speedup is an inference from correlation, not a demonstrated causal link.

What the experiments do not test:

1. Interaction between batch size and optimizer performance. The paper notes (Semenov et al., 2025 discussion) that batch size may explain discrepancies in matrix-based optimizer rankings, but it does not systematically sweep batch size as an interaction variable. The batch size is tuned per optimizer per regime, but the paper does not report how the optimal batch size changes for different optimizer classes (e.g., do matrix-based methods prefer larger batches?) or whether the speedup ratios would be different at a fixed batch size.

2. Wall-clock time or memory usage. The paper explicitly excludes these from comparison, citing prior claims that matrix-based overhead can be reduced to under 10%. But the paper does not verify these claims for its own implementations, and implementation quality varies dramatically — Kron and Soap with full preconditioners are substantially more expensive per step than Muon or AdamW, and the paper's use of blocking for Soap is an acknowledgment of this. A table showing actual step time, peak memory, and total wall-clock time for each optimizer at each scale would be valuable and is conspicuously absent.

3. Larger models beyond 1.2B. The paper's most policy-relevant claim — that speedups decay to 1.1× and may reverse — is extrapolated from up to 1.2B. Frontier pretraining operates at 7B–70B+ parameters. The 7B projection in Appendix B.1 is based on a fitted curve with 4 data points; testing at 7B directly would either confirm or refute the most important practical implication of the paper.

4. Different architectures. All experiments use a 32-layer Llama 2 architecture. Optimizer behavior might differ for deeper models (where gradient propagation issues matter more), for mixture-of-experts architectures (where sparse gradients interact with optimizer state), or for non-transformer architectures. The paper makes no claims beyond transformers, but the restriction to a single architecture is a meaningful scope limitation.

5. Interactions between optimizer and data quality. The paper uses a high-quality OLMo 2-style mixture. Prior papers using PILE (lower quality) may have had different optimal hyperparameters and optimizer rankings. The paper notes this but does not test it.

6. Statistical significance of the speedup ratios. The paper reports point estimates of loss and speedup from single runs. Given that the loss differences between the best optimizers at 1.2B are within $3 \times 10^{-3}$ (the Phase I convergence threshold), these differences may not be statistically reliable. The paper does not report standard errors, confidence intervals, or even the number of tokens used to compute the validation loss (which would determine the resolution of the measurement).

7. Combination of matrix-based preconditioning with variance reduction. The paper notes that Muon incorporates Nesterov momentum (a form of variance reduction) and Mars reduces variance in the gradient estimate — both techniques target different aspects of optimization. The paper does not test whether combining, e.g., Muon's matrix preconditioning with Mars-style gradient correction yields additive improvements, or whether the gains are subadditive (suggesting they address overlapping optimization difficulties).

8. Sensitivity to the coordinate descent procedure itself. The paper uses $\Delta_1 = 3 \times 10^{-3}$ as the convergence threshold. This is large enough that the procedure may stop at a hyperparameter configuration that is not truly optimal — making it possible that some optimizers are slightly undertuned relative to others. The paper does not test sensitivity to this threshold or report how many configurations fall within $\Delta_1$ of the claimed optimum for each optimizer.

6. Limitations and Trade-offs

Limitation 1: The Difficulty Estimation Analogue — Hyperparameter Tuning Cost Is Not Accounted for in the Speedup Numbers

The assumption or constraint. The paper's headline speedup ratios (1.3–1.4× at small scale, 1.1× at 1.2B) measure token efficiency given that each optimizer's hyperparameters have already been tuned to near-optimality. But the tuning process itself is enormously expensive — Phase I alone required roughly 1000+ training runs across 6 regimes for 10 optimizers, each consuming a full Chinchilla budget (e.g., a 130M model at 1× Chinchilla = ~2.6B tokens, or ~20B FLOPs). The paper acknowledges this implicitly in Appendix G when discussing Sophia's codebase issues, and explicitly in stating that Phase III's hyperparameter scaling law extrapolation was developed because "the paper cannot afford another full coordinate descent sweep for every optimizer" at 1.2B (Section 3.4.5 paraphrase). However, no total FLOPs budget for the tuning process is reported, and the speedup ratios compare only the final optimized runs, not the cost of finding those optima.

The consequence. If a practitioner wanted to use this paper's methodology to determine the best optimizer for their specific model architecture, data mixture, and hardware setup, they would need to replicate a substantial fraction of the Phase I + Phase II tuning procedure — hundreds of training runs — before seeing any speedup benefit. The speedup ratios are therefore best-case measurements after an expensive tuning process, not end-to-end efficiency gains. For a one-off training run (e.g., a single 7B model), the tuning cost could exceed the savings from using a slightly better optimizer. The paper's claim that the hyperparameter scaling law can predict hyperparameters at larger scales (Section 3.4) partially addresses this, but the scaling law was fitted on data from this specific architecture, data mixture, and hardware regime; transferring it to a different setting would require re-fitting, which incurs the same tuning cost.

What evidence exists in the paper. The paper provides no accounting of total tuning FLOPs. Appendix C contains the full Phase I sweep tables for each optimizer — for AdamW on 130M at 1× Chinchilla alone, 19 training runs were needed (Table 36). Extrapolating across 6 regimes × ~15–30 runs per optimizer × 10 optimizers yields well over 1000 runs. Each 130M 1× Chinchilla run costs ~2.6B tokens × 130M params × 6 FLOPs/token ≈ 2 × 10^18 FLOPs (~2 exaFLOPs for the full Phase I), though the paper never states this. The speedup ratios (Figures 3, 4) are computed from the final tuned configurations only.

Mitigation status. The paper partially addresses this through the hyperparameter scaling law in Phase III — by fitting parametric functions to the optimal hyperparameters across scales, a practitioner could theoretically extrapolate hyperparameters without running full coordinate descent. The validation at 1.2B (AdamW predicted hyperparameters within 3e-3 loss of the optimum, Section 3.4) is promising, but the scaling law was fitted on data from this specific setup and has no demonstrated transfer to different architectures, data mixtures, or hardware regimes. The paper suggests future work on better hyperparameter transfer but does not claim the current scaling law is general.

Limitation 2: No Experiments Beyond 1.2B Parameters, Yet the Core Policy Claim Extrapolates to 7B

The assumption or constraint. The paper's most consequential practical finding — that matrix-based optimizer speedups decay with model size and may reverse, with Muon projected to underperform AdamW at 7B parameters — is based on measurements up to 1.2B parameters (4 data points: 130M, 300M, 520M, 1.2B) and extrapolated ~6× beyond the training range to 7B (Appendix B.1). The paper acknowledges this directly: "This benchmarking study has the limitation that it does we haven't scaled to models larger than 1.2B parameters" (Section 5). The fitted scaling law for Muon (RMS error 5 × 10⁻³ on the 4 training points) uses the form L(N, D) = α N⁻ᴬ + β D⁻ᴮ + γ, which assumes power-law scaling that may not hold at larger model sizes.

The consequence. An organization deciding whether to adopt Muon for a 7B–70B parameter pretraining run based on this paper would be making a decision on extrapolated evidence. The paper's projection that Muon will produce higher loss than AdamW at 7B (Appendix B.1) could be correct (and the trend of decaying advantage is genuinely present in the data), but it could also be wrong — the scaling law might plateau or reverse. If the projection is wrong and Muon actually maintains a 1.1× speedup at 7B, an organization that switches to AdamW based on this paper would be leaving performance on the table. Conversely, if the projection is right, organizations that ignore it and adopt Muon at scale could see worse performance than just using well-tuned AdamW. The paper's own downstream evaluation at 1.2B already shows no meaningful accuracy differences between optimizers (Table 5: average accuracy within 0.5 percentage points across all optimizers), suggesting the loss differences at 1.2B may already be below the threshold of practical relevance.

What evidence exists in the paper. The scaling projection comes from Appendix B.1. The fitted parameters for Muon's scaling law are: α = 32.7458, A = 0.1864, β = 59.0221, B = 0.2074, γ = 1.8063 (RMS error 5 × 10⁻³). For AdamW: α = 21.4289, A = 0.1555, β = 276.4235, B = 0.2804, γ = 1.7324 (RMS error 3 × 10⁻³). The larger exponent A for Muon (0.1864 vs. 0.1555) means Muon's loss degrades faster as model size increases. The data this is fitted on is 4 optimizers × 4 Chinchilla ratios = 16 loss values per optimizer, collected from Phase I–III runs. The paper does not report prediction intervals for the 7B extrapolation — only point estimates.

Mitigation status. The paper presents the 7B projection cautiously ("This scaling law predicts that when the parameter scale reaches 7B, Muon will actually result in a higher loss compared to AdamW in 1× Chinchilla regime," Appendix B.1) and explicitly calls for future work: "Promising future work includes extending our benchmarking to larger models beyond 1.2B parameters to test whether the diminishing speedup trend persists at frontier scales" (Section 5). The 1.2B Phase III experiments provide one empirical data point showing the speedup has already decayed to ~1.1×, but this is consistent with the trend rather than a strong validation of the 7B crossing. The paper treats the extrapolation as a hypothesis to be tested, not a confirmed finding, but the prominence given to it in Section 1 ("the speedup... decreasing from 1.4× over AdamW for 0.1B parameter models to merely 1.1× for 1.2B parameter models") and the abstract could lead readers to over-weight the extrapolation.

Limitation 3: The Entire Benchmarking Framework Is Tied to a Single Architecture Family, Data Mixture, and Hardware Regime

The assumption or constraint. All experiments use a Llama 2 architecture with fixed depth (32 layers) and sequence length (4096), scaled only by varying width (hidden dimension, intermediate dimension, attention heads; Table 2). The pretraining data is a single OLMo 2-style mixture (DCLM-baseline + StarCoder V2 + ProofPile 2, ~4.1T tokens, Llama3 tokenizer; Section 3.1). Training uses JAX on TPU v5 hardware with mixed precision (parameters in float32, activations in bfloat16; Section 3.1). The paper does not test different architectures (e.g., varying depth, mixture-of-experts, different normalization schemes), different data mixtures (e.g., PILE, which was used by Sophia and other prior work; Appendix G notes this difference), or different hardware/software stacks (e.g., PyTorch on GPU with ZeRO-style sharding).

The consequence. The paper's findings may not transfer to settings that differ from this specific substrate. Several concerns are concrete:

Depth scaling: The paper scales only width, not depth. If gradient propagation through many layers is a bottleneck, optimizers that handle deep networks differently (e.g., through better gradient normalization or preconditioning) might show different relative rankings in deeper architectures. The fixed 32-layer design means the paper cannot detect this.
Data quality: Appendix G notes that Sophia was originally evaluated on PILE, which is "lower quality compared to the current pretraining dataset," and that this contributed to different optimal learning rates. The paper's finding that many optimizers show minimal gains over AdamW might be specific to high-quality data mixtures — on noisier data, variance-reduction methods like Mars might show larger relative advantages (as the paper itself hypothesizes in discussing Semenov et al., 2025: variance-reduction methods are "advantageous in their noise-dominated small-batch regime").
Hardware interactions: The paper uses batch sizes optimized per optimizer per regime, with typical optima in the range 128–512 sequences (0.5M–2M tokens per batch; Appendix C, D). The comparison with Semenov et al. (2025) — which used 0.02M–0.1M tokens per batch and found variance-reduced methods outperforming matrix-based methods — demonstrates that batch size can flip optimizer rankings. The paper's batch sizes are dictated by its TPU v5 hardware (128 chips requiring large batches for full utilization), and the findings may not hold on smaller GPU clusters or at very different batch sizes.
Software stack: The paper's implementations of Soap, Kron, and Scion used specific optimizations (Soap: parameter blocking and bfloat16 momentum; Kron: merge_small_dims and partition_grads_into_blocks; Scion: sign-based updates for embeddings). These implementation choices affect both performance and memory usage, and different implementations of the same algorithm could yield different results. Appendix G explicitly notes that the Soap implementation "may lead to slightly lower step-wise performance" compared to the original paper's implementation.

What evidence exists in the paper. The paper provides no cross-architecture, cross-data, or cross-hardware experiments. The comparison with Semenov et al. (2025) (Section 2) provides the closest thing to a different-hardware baseline — showing that batch size differences can explain diverging optimizer rankings — but this is a post-hoc diagnosis, not a controlled experiment. The paper's own Phase I sweeps include batch size optimization for each regime (Tables 36–93 in Appendix C), and the optimal batch size varies across optimizers and scales (e.g., for AdamW on 130M at 1×, batch size 128 is optimal; at 8×, 256 is optimal; for Muon on 130M at 1×, batch size 128 is optimal; for Soap on 520M at 1×, batch size 128 is optimal), confirming that hardware-optimizer interactions exist but not characterizing them systematically.

Mitigation status. The paper does not attempt to address this limitation. It acknowledges the single-architecture constraint implicitly through its choice (Section 3.1: "following MobileLLM" for the 32-layer design; "following OLMo 2's recipe" for the data mixture) but does not discuss the generalizability implications. The conclusion flags "extending our benchmarking to larger models beyond 1.2B parameters" as future work but does not mention architectural or data diversity. The open-sourcing of code and WandB runs (Section 3.4.7 paraphrase) partially mitigates this by enabling other researchers to replicate on different architectures, but the computational cost of doing so (~1000+ runs for full Phase I) makes independent replication practically difficult.

Limitation 4: The Coordinate Descent Tuning Protocol Guarantees Coordinate-Wise Optimality, Not Joint Optimality, and the Convergence Thresholds Are Set Empirically

The assumption or constraint. The Phase I procedure (Section 3.2) sweeps each hyperparameter one at a time, holding all others fixed, and accepts a new value if the loss improves by more than Δ₁ = 3 × 10⁻³. This guarantees that the final configuration is a coordinate-wise local optimum — no single hyperparameter can be changed (with others fixed) to improve loss by more than Δ₁. It does not guarantee a joint optimum — there may exist combinations where simultaneously changing two hyperparameters yields improvements beyond what coordinate-wise search can find. The choice of Δ₁ = 3 × 10⁻³ is empirical, not theoretically grounded. The scaling-sensitivity classification uses Δ₂ = 6.4 × 10⁻³ (Section 3.3), roughly 2× the Phase I threshold, to define the set of "approximately optimal" configurations.

The consequence. Two specific failure modes are possible:

Interaction blindness: If two hyperparameters have strong interactions — for example, the optimal learning rate depends on the weight decay value, and vice versa — coordinate descent can get stuck. It might find a learning rate that is optimal given the current weight decay, and a weight decay that is optimal given that learning rate, but miss a better (lr, wd) pair where both need to change simultaneously. Standard practice in hyperparameter optimization (random search, Bayesian optimization) typically samples joint configurations to avoid this, but coordinate descent exploits the fact that optimizer hyperparameters often have relatively weak interactions — a claim the paper implicitly relies on but never tests.
Tuning asymmetry between optimizers: If one optimizer's loss surface has sharper narrow minima (requiring precise hyperparameters) while another has broad flat minima (forgiving to hyperparameter choice), coordinate descent with a fixed Δ₁ might find a better configuration for the flat optimizer than the sharp one, even if both receive the "same" tuning effort. The paper's convergence criterion — stop when no parameter change improves loss by ≥ Δ₁ — means optimizers with noisy or bumpy loss surfaces might stop early at a suboptimal point, while smooth optimizers are tuned more precisely. The paper does not test for this by, e.g., plotting loss as a function of each hyperparameter to check smoothness.

What evidence exists in the paper. The full coordinate descent tables in Appendices C and D provide the raw data. For AdamW on 130M at 1× Chinchilla (Table 36), the procedure terminates after 19 runs at loss 3.529, with the final configuration (lr=0.008, wd=0.1, warmup=2000, gnorm=2, batch=128). Several hyperparameters show flat loss surfaces near the optimum: gnorm varied from 1 to 2 changes loss by < 3e-4 (round 3), and ε varied from 1e-15 to 1e-20 changes loss by ≤ 2e-3. For Kron on 130M at 1× (Table 70), the procedure requires 23 runs (more than AdamW) and the loss surface is visibly rougher: learning rate 0.001 → 0.002 changes loss by 0.009 (from 3.501 to 3.492), weight decay 0 → 0.5 changes loss by 0.028 (from 3.519 to 3.491). Kron's optimum depends more sensitively on getting hyperparameters exactly right, which coordinate descent might handle differently than for AdamW.

The paper does not perform a joint 2D sweep over any pair of hyperparameters to test for interactions. The Δ₁ and Δ₂ threshold choices are not justified with sensitivity analysis — there's no experiment showing how the final optimizer rankings change if Δ₁ were 1e-3 or 1e-2 instead of 3e-3.

Mitigation status. Not addressed. The paper treats the coordinate-wise optima as sufficiently close to true optima for the purpose of comparing optimizers, and the scaling-sensitivity classification (which drives Phase II) directly depends on the Δ₂ threshold. The paper's validation at 1.2B — where a full Phase I sweep for AdamW confirmed the scaling law prediction within 3e-3 — provides some empirical support that the procedure finds configurations close to true optima, at least for AdamW, but this doesn't test for joint-optimality failures or for other optimizers where the scaling law predictions at 1.2B are not validated against a full sweep.

Limitation 5: No Statistical Uncertainty Quantification — All Results Are Point Estimates From Single Runs

The assumption or constraint. Every loss value reported in the paper (Figures 2–4, Tables 36–183 in Appendices C–F) comes from a single training run at the identified optimal hyperparameter configuration for that optimizer-regime combination. There are no replicate runs, no error bars, no confidence intervals, and no statistical tests. The primary speedup ratios (Figures 3, 4) are computed from these single-point loss estimates. The convergence criterion Δ₁ = 3 × 10⁻³ provides a form of empirical noise floor — hyperparameter changes smaller than this are considered insignificant — but this is a threshold on the hyperparameter optimization procedure, not a measurement of run-to-run variance.

The consequence. At 1.2B and 8× Chinchilla (Figure 4, left), the loss values for the four tested optimizers are: AdamW 2.752, NAdamW 2.749, Muon 2.748, Soap 2.749. The spread among the top three optimizers is 0.001 — about 30× smaller than the Phase I convergence threshold Δ₁, and smaller than what one would expect from random initialization seed variance alone. The paper reports these as meaningful differences (Muon achieves the lowest loss, hence "maintains a ~1.1× speedup"), but without replicate runs, there's no way to determine whether Muon's 0.001 advantage over NAdamW is a real effect or noise. The paper's own scaling law fit for Muon has an RMS error of 5 × 10⁻³ (Appendix B.1), which is 5× larger than the 0.001 difference between Muon and Soap at 1.2B 8× — suggesting these differences are well within the model's fitting uncertainty.

The downstream evaluation (Table 5) reinforces this concern: at 1.2B and 8× Chinchilla, the average accuracy across 10 benchmarks is 67.15 (AdamW), 66.70 (NAdamW), 66.98 (Muon) — a spread of 0.45 percentage points. Individual benchmark scores vary by 2–4 percentage points between optimizers with no consistent winner (e.g., Muon leads on Winogrande 67.09 vs. 64.96, but NAdamW leads on LAMBADA 67.84 vs. 67.53). This is consistent with random variation rather than a systematic Muon advantage.

What evidence exists in the paper. None — the paper provides no replicate runs, no measurement of run-to-run variance, and no statistical analysis. The RMS errors on the scaling law fits (3 × 10⁻³ for AdamW, 5 × 10⁻³ for Muon; Appendix B.1) are the closest thing to uncertainty quantification, but these measure interpolation quality of the fitted parametric functions, not the variance of the underlying loss measurements. The paper's open-sourced WandB runs would enable a third party to estimate variance (by, e.g., examining loss curve noise over the last few steps), but the paper does not do this itself.

Mitigation status. Not addressed. The paper implicitly treats the coordinate descent procedure as the primary source of uncertainty — if two configurations differ by less than Δ₁, they're considered equivalent — but this doesn't account for seed variance, data ordering variance, or hardware noise. In the 1.2B regime where loss differences between the best optimizers are approximately 1e-3, the lack of statistical uncertainty means the paper's finding that "Muon and Soap still deliver speedup over AdamW, but the speedup diminishes to ≈1.1×" (Section 4.1) could equally be stated as "at 1.2B, Muon, NAdamW, Soap, and AdamW produce indistinguishable losses within the measurement noise of our experiments."

Limitation 6: The Comparison with Prior Work Is Correlational, Not Causal — the Paper Does Not Reproduce the Prior Work to Isolate the Mechanism of Error

The assumption or constraint. The paper's core methodological critique — that prior optimizer evaluations produced inflated speedups because their AdamW baseline was undertuned — is supported by demonstrating that, in the paper's own experimental setup, (a) tuning AdamW's learning rate from the GPT-3 recipe's 6e-4 to 8e-3 yields a 2× speedup (Figure 1, top left), and (b) the speedups measured for alternative optimizers against this paper's well-tuned AdamW are much smaller than previously claimed (Figure 3, Appendix G). However, the paper does not reproduce the exact experimental configurations of prior work. It does not take, for example, the Sophia codebase and data (PILE, GPT-3 architecture, Levanter framework, non-random data shuffling) and show that (i) the original 2× Sophia speedup is reproducible in that setup, but (ii) re-tuning only AdamW's learning rate in that same setup eliminates the speedup.

The consequence. A defender of a prior paper (e.g., Sophia or MARS) could argue that the differences in measured speedup are due to differences in experimental setup — different data quality (PILE vs. OLMo 2), different architecture (GPT-3 vs. Llama 2), different data shuffling (correlated vs. random), different hardware (GPU vs. TPU) — rather than baseline undertuning. The paper partially addresses this by noting specific mechanisms (Appendix G: Sophia's Levanter codebase had correlated data shuffling per node, which forced a smaller learning rate), but for other papers the attribution is based on observing that they used the GPT-3 recipe learning rate and noting that this learning rate is suboptimal in the paper's own experiments. The causal chain — "the GPT-3 recipe learning rate is suboptimal in setup A, those papers used the GPT-3 recipe learning rate in setup B, therefore those papers' baselines were suboptimal in setup B" — relies on the untested assumption that optimal AdamW hyperparameters transfer across sufficiently different experimental setups. The paper's own finding that optimal hyperparameters vary with data quality and data shuffling (Appendix G notes this for Sophia) potentially undermines this assumption, because the GPT-3 recipe's 6e-4 learning rate might have been closer to optimal on PILE with correlated shuffling than it is on the paper's OLMo 2 mixture with random shuffling.

What evidence exists in the paper. Appendix G provides the comparative analysis. For each prior paper, it lists the claimed speedup, the measured speedup in this paper's experiments, and the specific setup differences. For Sophia, the paper identifies the data shuffling issue explicitly: "in the implementation of Levanter that the authors used, the data shuffling is not completely random and instead is correlated on every compute node. Upon reproducing the results, we note that this difference can significantly impact the stability of the training process and a complete random shuffling is crucial for the usage of a large learning rate." This suggests the paper did partially reproduce the Sophia setup and identified a specific mechanism. For other papers, the analysis is more generic: "these papers use a similar baseline as Sophia and MARS" (for Cautious, Block-wise LR Adam, FOCUS), or "the comparison... is carried out on a smaller than 1× Chinchilla regime" (for SWAN, DION), or "this work considers an atypical setup where the model is trained with a constant learning rate" (for SPlus).

Mitigation status. Partially addressed for Sophia through the observation about data shuffling and the reported reproduction of the effect. Not addressed for other papers — there's no evidence the paper ran, e.g., MARS's exact setup and re-tuned AdamW to measure the true speedup in that setup. The paper's strategy is to make the argument general — "many papers used the GPT-3 recipe, which we show is 2× suboptimal in a modern setup, therefore their speedups are inflated" — without verifying each case individually. This is rhetorically effective but scientifically incomplete. A skeptic could reasonably ask: how do we know the GPT-3 recipe's learning rate was actually suboptimal for those papers' specific data, architecture, and hardware, rather than being suboptimal for this paper's setup? The Sophia data shuffling analysis demonstrates this concern is real — the optimal learning rate did depend on shuffling, and the GPT-3 recipe's value might have been closer to optimal in the specific non-random shuffling regime of Sophia's setup than in the paper's fully random regime.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally changes the conversation around pretraining optimizers from "which optimizer claims the largest speedup?" to "which speedup claims survive rigorous evaluation, and at what scales?" It accomplishes this not by proposing a new optimizer, but by demonstrating that the measurement infrastructure for optimizer evaluation has been systematically broken, and that the entire subfield has been calibrated against an undertuned baseline. This is a diagnostic contribution — akin to Jiang et al. (2019) revealing that claimed generalization measures did not actually predict generalization — rather than an algorithmic advance. The practical consequence is that the burden of proof for new pretraining optimizers has been permanently raised: any future paper claiming >1.3× speedup over AdamW must demonstrate that its AdamW baseline has been tuned with at least the rigor of this paper's coordinate descent protocol, or risk being dismissed as another victim of the GPT-3 recipe's 2× undertuning.

The paper resolves the contradiction that had puzzled the field: why do so many optimizers claim 2× speedups, yet almost none are adopted in production pretraining? The answer is not that practitioners are conservative or that implementation overhead is prohibitive — it is that the claimed speedups were largely measurement artifacts. The paper traces a single hyperparameter (peak learning rate = 6e-4 in the GPT-3 recipe) through at least six major optimizer papers (Sophia, MARS, Cautious AdamW, FOCUS, Block-wise LR Adam, and Muon/Kimi) and shows that this value is ~13× smaller than optimal for modern data mixtures with proper random shuffling (Figure 1, top left). This single number accounts for approximately 2× of apparent speedup, transforming "AdamW baseline vs. New Optimizer" comparisons from "2× advantage for the new method" to "0.7× advantage for AdamW if properly tuned, or ~1.1–1.3× advantage for the best matrix-based methods if they too receive equal tuning investment." The paper does not accuse individual authors of wrongdoing — it documents a structural failure in evaluation methodology where a widely-circulated default configuration became the de facto baseline, and no one noticed it was profoundly suboptimal until someone invested the thousands of GPU-hours needed to systematically sweep AdamW's hyperparameters across scales.

The paper's second major landscape shift is the reframing of optimizer selection as a scaling problem, not a fixed-scale comparison. Before this work, it was common to evaluate optimizers at a single model size (typically 100M–500M) and a single data budget (1× Chinchilla), then assume the ranking would generalize to production scales. The paper demonstrates that this assumption is false in at least two distinct ways. First, matrix-based speedups decay with model size: Muon's advantage drops from ~1.4× at 130M to ~1.3× at 300M to ~1.2× at 520M to ~1.1× at 1.2B (Figures 1 bottom left, 4 middle), and a fitted scaling law projects that Muon will underperform AdamW at 7B parameters (Appendix B.1). Second, optimizer rankings can reverse with data budget: Muon is the best optimizer at 1–4× Chinchilla across all tested model sizes, but is overtaken by Soap and Kron at 8–16× Chinchilla (Figures 2, 4 right, 8), and the gap between the best matrix-based method and the best scalar-based method (NAdamW) shrinks at high data budgets to the point of practical irrelevance (~0.008 loss at 300M and 16× Chinchilla). This reframing implies that optimizer evaluation protocols that test only at small scale and single data budgets are not just incomplete — they can produce actively wrong rankings that would lead practitioners to adopt optimizers that become suboptimal at their actual deployment scale.

The paper also resolves a subtle puzzle about Sophia: the original paper (Liu et al., 2024a) reported 2× speedup over AdamW, but this paper finds Sophia actually underperforms AdamW at 130M and only approaches parity around 520M (Figure 7). The resolution is twofold: (a) Sophia's AdamW baseline used the GPT-3 recipe's 6e-4 learning rate, accounting for ~1.5× of the apparent gap, and (b) Sophia's codebase (Levanter) used data sharding that correlated examples across compute nodes rather than fully randomizing them, which made training less stable and forced a smaller optimal learning rate for AdamW specifically. With proper random shuffling and a well-tuned AdamW, Sophia's Hessian-based clipping provides no measurable benefit at small scale. This finding is important because it redirects research attention: Sophia is an intelligent idea (incorporating second-order curvature information), but its benefits appear to emerge only at larger scales where curvature estimation might become more valuable — and even then, the measured advantage at 520M (~1.0×, essentially tied with AdamW) suggests the current implementation does not realize the theoretical promise.

Perhaps the most subtle landscape change is the paper's implicit demonstration that batch size regimes can flip optimizer rankings, as evidenced by the comparison with concurrent work (Semenov et al., 2025). The paper finds that matrix-based optimizers (Muon, Soap) outperform variance-reduced methods (Mars) in its large-batch regime (≥0.4M tokens), while Semenov et al. find the opposite ordering in their small-batch regime (0.02–0.1M tokens). This is not just a disagreement to be adjudicated — it is evidence that the optimal optimizer depends on the hardware budget, because hardware constraints (number of chips, memory per chip) dictate the feasible batch size range. Two organizations with different hardware setups could legitimately reach different conclusions about which optimizer is best for their use case, and both could be correct for their specific regime. This complicates the very notion of a universal optimizer benchmark — the paper's results are strongest for the specific TPU v5 + JAX + mixed-precision regime they were measured in, and transferring conclusions to different hardware requires understanding the batch-size-optimizer interaction, which the paper does not systematically characterize.

Which research directions become more attractive after this paper?

More attractive: Robust verifier/tuning infrastructure for optimizers (scaling-law-based hyperparameter prediction, cheap difficulty estimation analogues for optimizer selection). The paper's Phase III hyperparameter scaling law (Section 3.4) demonstrates that principled extrapolation can replace expensive brute-force tuning at larger scales — validating AdamW predictions within 3e-3 loss at 1.2B. This opens the door to treating hyperparameter selection as a scaling law problem that can be solved once per model family and data mixture, then reused.
More attractive: Understanding why matrix preconditioning helps and whether the benefit can be preserved at scale. The paper shows that Muon, Soap, and Kron all achieve similar ~1.3× speedup at 130M despite completely different update rules, suggesting the advantage comes from the shared architecture choice (matrix preconditioning) rather than algorithmic details. But the advantage decays with scale, and the mechanism of decay is unknown. Is it that the benefits of matrix preconditioning are largest when gradients are noisy (small scale), and diminish as gradient estimates improve with larger batches at larger scale? Or is it that the matrix structure of transformer weight matrices changes with width, making the preconditioner less effective? These questions are now empirically motivated and experimentally testable.
More attractive: Variance reduction for the small-batch regime, where the paper's comparison with Semenov et al. (2025) suggests it may be the dominant effect. The paper's Mars results (Figure 2) show small but consistent advantages over AdamW in the large-batch regime, and the discussion of batch size effects suggests these advantages might be substantially larger in small-batch settings relevant to academic research or smaller GPU clusters.
Less attractive: Designing yet another scalar-based AdamW variant. The paper shows that after proper tuning, NAdamW, Mars, Cautious, and AdamW all converge to within ~0.02 loss of each other across all tested regimes (Figure 2), with speedup ratios not exceeding 1.2× (Figure 3). The design space for scalar-based methods appears saturated — the gains from improved gradient estimation or momentum schemes are real but small, and likely not worth the additional complexity and hyperparameter surface.
Less attractive: Publishing optimizer papers with the GPT-3 recipe as the AdamW baseline. The paper's evidence is so clear (Figure 1, top left: 2× speedup from a single hyperparameter; Appendix G: systematic tracing of this baseline through prior work) that reviewers and practitioners will now expect independent AdamW tuning at least as rigorous as the proposed optimizer's tuning. A paper that still uses lr=6e-4 for AdamW without justification will be met with skepticism by default.

Follow-Up Research This Work Enables

A causal isolation experiment: "scalarize" Muon to test whether matrix preconditioning is the operative mechanism. The paper observes that Muon, Soap, and Kron all achieve similar speedups despite different algorithms, suggesting matrix structure itself provides the benefit. A direct test would modify Muon to replace the Newton-Schulz matrix orthogonalization with an element-wise operation that preserves the per-element update magnitude distribution but destroys matrix-level correlations — for instance, element-wise normalization to the same Frobenius norm, or random orthogonal projection followed by the same magnitude rescaling. If the "scalarized" Muon falls back to scalar-based performance (~1.0–1.1× speedup), this would confirm that matrix structure is causal. If it retains most of the speedup, the benefit would instead come from Muon's specific momentum scheme or learning rate configuration. The paper's Phase I–II protocol provides the tuning methodology to ensure fair comparison; this experiment could be run at 130M (where the Muon-AdamW gap is large enough to measure, ~0.06 loss) and at 1.2B (where the gap has shrunk to ~0.01) to test whether the causal mechanism changes with scale.

Hyperparameter scaling laws for optimizer transfer across architectures and data mixtures. The paper's Phase III hyperparameter scaling law was fitted and validated within a single architecture family (Llama 2, 32 layers) and data mixture (OLMo 2). A critical test of generality would be: (a) fit the scaling law on the paper's Llama 2 data (130M–520M), (b) transfer the predicted hyperparameters to a different architecture at the same parameter count (e.g., a Llama 3 architecture with different depth/width ratios, or a GPT-2 architecture), and (c) measure the loss degradation compared to a full coordinate descent sweep on the new architecture. If the degradation is small (<Δ₁ = 3e-3), hyperparameter scaling laws are architecture-transferable and the paper's methodology becomes a general tool. If the degradation is large, then the scaling law is architecture-specific and each new model family would need its own Phase I–II investment. The paper's data (Appendix C, Tables 36–93) provides the full hyperparameter grids necessary to fit alternative scaling forms or test transfer hypotheses.

Batch size × optimizer interaction sweep. The comparison with Semenov et al. (2025) reveals that batch size can flip optimizer rankings (matrix-based methods dominate at large batches, variance-reduced methods at small batches), but the paper only observes this correlation, it does not test it causally. A systematic experiment would fix a model scale (e.g., 300M) and Chinchilla ratio (e.g., 4×), then sweep batch size across an order of magnitude (e.g., 0.05M to 2M tokens) for a representative from each optimizer class: AdamW (baseline scalar), NAdamW (variance-reduced scalar), Mars (variance-reduced with gradient correction), Muon (matrix preconditioning), and Soap (matrix preconditioning with second-order momentum). For each batch size × optimizer combination, perform a full Phase I-style coordinate descent (since optimal hyperparameters likely shift with batch size). The output would be a matrix of speedup ratios as a function of batch size, directly testing the paper's hypothesis that "in the noise-dominated small-batch regime, variance-reduction methods are advantageous, whereas in the larger-batch setting these benefits diminish and matrix-level optimizers become more competitive" (Section 2). This experiment would also produce actionable guidance for practitioners: given your hardware and batch size constraints, which optimizer class should you use?

Scaling law validation at 7B parameters — the critical stress test. The paper's most provocative finding is the projection that Muon will underperform AdamW at 7B parameters (Appendix B.1), based on a fitted scaling law with 4 data points (130M–1.2B). This prediction should be tested: train a 7B Llama 2 model at 1× Chinchilla (~140B tokens) using AdamW, NAdamW, Muon, and Soap, with hyperparameters predicted by the Phase III scaling law methodology (extrapolating from the 130M–1.2B fits). A full coordinate descent at 7B is infeasible (each run would cost ~8e21 FLOPs), but the paper's methodology is designed for exactly this regime — the scaling law predicts hyperparameters. The experiment would measure whether (a) Muon's speedup has indeed decayed to ≤1.0× (confirming the projection), (b) Muon still maintains a ~1.05–1.1× advantage (falsifying the projection and suggesting the scaling law's parametric form is wrong), or (c) Muon has regained advantage at larger scale (falsifying the projection and suggesting a non-monotonic relationship). Any outcome would be informative, but outcome (c) would be particularly interesting because it would imply that the speedup decay at 300M–1.2B is a transient mid-scale phenomenon, not a terminal trend. The paper's open-sourced code and WandB logs enable reconstruction of the exact scaling law fits for independent verification.

Combine matrix preconditioning with large-batch variance reduction — are the gains additive? The paper shows that matrix-based methods (Muon) and variance-reduced methods (Mars) provide complementary benefits in different regimes (large-batch vs. small-batch). A natural question is whether combining them yields additive or subadditive gains. A concrete experiment: implement a "Mars-Muon" hybrid that uses Mars's corrected gradient $c_t = g_t + \gamma \frac{\beta_1}{1-\beta_1}(g_t - g_{t-1})$ as the input to Muon's Newton-Schulz orthogonalization, replacing the standard momentum. Train at 300M and 4× Chinchilla (where both methods show measurable advantages — Figure 2 shows Mars ~0.01 better than AdamW, Muon ~0.04 better) with a Phase I-style coordinate descent on the combined method. If the combined method achieves speedup of ~1.3–1.4× (more than either individually's ~1.2× and ~1.3×), the benefits are additive and the mechanisms are complementary (matrix preconditioning addresses a different optimization difficulty than gradient variance reduction). If the combined method achieves only ~1.3× (no improvement over Muon alone), the benefits are subadditive and likely target overlapping optimization challenges, suggesting that matrix preconditioning already implicitly handles gradient variance at large batch sizes.

Continual or dynamic optimizer switching across training phases. The paper finds that Muon is best at low data budgets (1–4× Chinchilla) while Soap and Kron catch up or surpass at high data budgets (8–16×), and that Sophia underperforms early but may improve at larger scales. This suggests that different optimizers may be optimal at different phases of a long training run. A natural experiment: train a model where the optimizer changes at a predetermined switch point (e.g., use Muon for the first 4× Chinchilla, then switch to Soap for the remaining 4× to reach 8× total). The challenge is that optimizer state (momentum buffers, preconditioners) would need to be translated or re-initialized at the switch. A simpler variant would test whether the learning rate schedule alone can capture the benefits — e.g., does a Muon-trained model at 4× Chinchilla, when continued with Soap, outperform a model trained entirely with Soap to 8×? This tests whether the benefits of fast early convergence (Muon's strength) and stable late-training refinement (Soap's strength) can be composed. The paper's Phase II data (Appendix D) provides the hyperparameter configurations for both optimizers at intermediate Chinchilla ratios, enabling direct extension to the switch experiment.

Practical Applications and Downstream Use Cases

Advising small-to-medium-scale pretraining teams on optimizer selection. For teams training models in the 130M–520M range at 1–4× Chinchilla (a common regime for academic research, domain-specific models, and on-device model development), this paper provides the most actionable guidance: use Muon. The paper shows Muon achieves the lowest validation loss across all tested configurations in this regime (Figure 2: Muon at 130M 1× = 3.464 vs. AdamW = 3.529, a 0.065 gap; at 300M 1× = 3.224 vs. AdamW = 3.264, a 0.040 gap; at 520M 1× = 3.073 vs. AdamW = 3.110, a 0.037 gap), corresponding to a 1.3–1.4× speedup (Figure 3). The paper also provides the specific tuned hyperparameters for Muon at these scales (Appendix C, Tables 82–87), so a team can adopt Muon without needing to replicate the Phase I tuning themselves — though the paper notes that optimal hyperparameters may shift with data mixture, so some transfer-tuning is recommended. The practical benefit is that a team training a 520M model to 1× Chinchilla can reach the same loss using roughly 70% of the compute budget, or reach a lower loss (~0.04 nats) within the same budget, which for a 520M model at 1× Chinchilla translates to saving approximately 2B tokens of training — roughly 3 TPU-v5-days at the paper's throughput.

Cost-efficient optimizer benchmarking protocol for organizations with unique architectures or data. The paper's three-phase hyperparameter tuning protocol (Phase I: full coordinate descent across scales; Phase II: focused descent on scaling-sensitive parameters; Phase III: scaling law extrapolation) is not just a methodology for this paper — it is a reusable recipe that any large organization with a custom architecture (e.g., mixture-of-experts, novel attention mechanism, domain-specific tokenizer) can follow to determine the best optimizer for their specific setup. The paper's Phase I identifies that only a subset of hyperparameters are scaling-sensitive (Table 4: learning rate for all optimizers, warmup for most, weight decay for AdamW and Adam-mini), meaning an organization could run a reduced Phase I at 2–3 model sizes (e.g., 100M and 300M) on their specific architecture, identify their own scaling-sensitive parameters, and then use Phase II-style focused sweeps at larger scales. The paper's open-sourced code and WandB runs provide templates for the grid definitions and convergence criteria. The concrete benefit is that organizations do not need to blindly trust the paper's conclusion that "Muon is best at 130M–520M" for their architecture — they can run their own mini-version of Phase I at 1/10 the scale to verify transferability, at a cost of roughly 20–30 training runs per optimizer candidate.

Deprecate the GPT-3 recipe as a baseline in future optimizer research — immediately. The single most actionable takeaway for the research community: if you are writing a paper proposing a new pretraining optimizer, do not use the GPT-3 recipe's peak learning rate of 6e-4 as your AdamW baseline. The paper demonstrates (Figure 1, top left, and Appendix G) that this value is ~13× smaller than optimal in modern setups, and that the 2× speedup it creates accounts for the claimed advantages of most proposed optimizers. Instead, perform at minimum a learning rate sweep for your AdamW baseline on your specific data and architecture — the paper finds optimal values in the range 4e-3 to 8e-3 for models 130M–520M on the OLMo 2 mixture, but this may differ for your setup. At a minimum, test at least the values {0.001, 0.002, 0.004, 0.008} and report the AdamW baseline at its best-performing learning rate, not at a default. Reviewers of optimizer papers should treat the use of the GPT-3 recipe without justification as a methodological red flag, akin to using an untuned SGD baseline in 2025.

Guiding hardware-provisioning decisions at inference-heavy organizations. The paper's finding that optimizer speedups decay to ~1.1× at 1.2B parameters, and that downstream benchmarks show no meaningful advantage at that scale (Table 5: average accuracy within 0.5 percentage points across AdamW, NAdamW, Muon at 1.2B 8×), has practical implications for organizations deciding between investing engineering effort in optimizer research versus other efficiency improvements. If you are training models in the 1B+ range — where frontier models increasingly operate — the paper suggests that the marginal return from switching optimizers is likely below 10% in token efficiency and may be zero in downstream performance. At this scale, other efficiency levers (data filtering, architecture improvements, quantization-aware training, better learning rate schedules) may offer larger and more reliable returns than optimizer changes. The paper does not test this directly against other efficiency interventions, but the finding that even the best optimizer (Muon) is only 1.1× better than well-tuned AdamW at 1.2B, combined with the projection that Muon may underperform at 7B, suggests that the optimizer improvement budget may be largely exhausted at current scales, and organizations with limited engineering resources should prioritize other avenues.