ArXiv: 1711.05101

🎯 Pitch

L2 regularization and weight decay are not equivalent in Adam—standard implementations apply L2 regularization, which inadvertently penalizes large-gradient parameters less than small-gradient ones, explaining much of Adam’s generalization gap relative to SGD. By simply decoupling the weight decay step from the adaptive gradient update (forming AdamW), the authors close this gap on CIFAR-10 by 15% relative improvement and make the optimal weight decay independent of the learning rate.

1. Executive Summary

This paper analyzes why adaptive gradient methods like Adam generalize worse than SGD with momentum on image classification tasks (CIFAR-10, ImageNet32x32 using ResNet architectures), diagnosing the root cause as the implicit coupling of L2 regularization into the adaptive gradient update, which unequally penalizes parameters proportionally to their gradient history rather than uniformly. The authors propose decoupled weight decay regularization (operationalized in their AdamW optimizer by subtracting a weight decay term directly from the parameters outside the gradient-based update step, with analogous SGDW for SGD), a simple algorithmic modification that restores the original weight decay formulation of Hanson & Pratt (1988) and clearly separates the roles of the learning rate and weight decay factor as independent hyperparameters. This decoupling yields a 15% relative improvement in test error on CIFAR-10 and comparable gains on ImageNet32x32 (narrowing the gap between Adam and SGD with momentum from substantial to marginal), while making the hyperparameter search space ~4× more separable (evidenced by the basin of best settings aligning with axes rather than a diagonal, so that tuning one hyperparameter does not require simultaneous retuning of the other). The paper establishes that decoupled weight decay restores Adam's competitiveness with SGD only when combined with learning rate schedules—specifically cosine annealing with warm restarts (AdamWR)—and that optimal weight decay strength depends on the total training budget, motivating a normalized weight decay formulation that scales with the square root of the batch pass count.

2. Context and Motivation

The Core Problem: Adam Generalizes Worse Than SGD, and Nobody Knew Why

By early 2019, the deep learning community faced a persistent and frustrating puzzle. Adaptive gradient methods—Adam (Kingma & Ba, 2014), RMSProp (Tieleman & Hinton, 2012), AdaGrad (Duchi et al., 2011)—had become the default optimizers for training neural networks across most application domains. Their appeal was obvious: they automatically adapt the learning rate on a per-parameter basis using historical gradient statistics, which meant practitioners spent less time tuning learning rates and often saw faster initial convergence. For feed-forward networks, recurrent architectures, and generative models, Adam was the go-to choice, and for good reason—it simply worked out of the box in many settings (Xu et al., 2015; Radford et al., 2015).

Yet, on one specific but critically important class of problems—image classification on standard benchmarks like CIFAR-10 and CIFAR-100—Adam consistently lagged behind plain SGD with momentum. The evidence for this gap had been accumulating:

  • Wilson et al. (2017) conducted a systematic study across a diverse set of deep learning tasks (image classification, character-level language modeling, constituency parsing) and concluded that adaptive gradient methods "do not generalize as well as SGD with momentum." This paper was particularly influential because it wasn't anecdotal—it tested multiple architectures, multiple datasets, and multiple optimizers, and the pattern held.
  • Gastaldi (2017) achieved state-of-the-art results of 2.86% error on CIFAR-10 using SGD with momentum combined with a custom Shake-Shake regularization technique. There was no Adam-based entry anywhere near the top of the CIFAR-10 leaderboard.
  • Cubuk et al. (2018) developed AutoAugment, a learned data augmentation policy, and their strong results were also obtained with SGD, not Adam.

The situation was genuinely puzzling because adaptive methods should have been superior in theory: they adjust step sizes per parameter, they handle sparse gradients well, they reduce the need for manual learning rate tuning, and they often converge faster in the early stages of training. Why, then, was their final generalization performance worse?

Several hypotheses had been proposed to explain this gap:

  • Sharp minima hypothesis (Keskar et al., 2016): Adam might converge to sharper local minima that generalize worse than the flatter minima found by SGD. The argument was that the noise inherent in SGD's stochastic updates acts as an implicit regularizer that biases the optimizer toward flat basins, while adaptive methods' more deterministic updates might converge to sharp, poorly-generalizing solutions. However, Dinh et al. (2017) later showed that sharp minima can generalize well under certain reparameterizations, complicating this story.
  • Inherent problems with adaptive gradient methods (Wilson et al., 2017): The marginal value paper argued that adaptive methods' per-parameter learning rate adaptation—while helping on the training objective—somehow fundamentally impaired their ability to find solutions that generalize. This was a broad claim without a precise mechanism.

The Loshchilov & Hutter paper enters this debate with a completely different diagnosis: the problem isn't with adaptive gradient methods per se—it's that the regularization being used with them is the wrong kind. Specifically, the deep learning community had been using L2 regularization and calling it "weight decay," but for adaptive optimizers, these two regularization strategies are not equivalent, and the L2 version is substantially less effective.

The Equivalence That Breaks: L2 Regularization ≠ Weight Decay for Adaptive Methods

This paper's core insight hinges on a mathematical subtlety that had been largely overlooked by the community. Let's walk through it carefully.

The standard story for SGD. For plain stochastic gradient descent (no momentum, no per-parameter scaling), there is a well-known equivalence between two regularization strategies:

  1. Weight decay (Hanson & Pratt, 1988): After computing the gradient update, shrink the weights toward zero by a factor of (1−λ)(1 - \lambda) at each step. Formally: θt+1=(1−λ)θt−α∇ft(θt)\theta_{t+1} = (1 - \lambda)\theta_t - \alpha \nabla f_t(\theta_t)

  2. L2 regularization: Add a penalty λ′2∥θ∥22\frac{\lambda'}{2} \|\theta\|_2^2 to the loss function, so the gradient becomes ∇ft(θ)+λ′θ\nabla f_t(\theta) + \lambda'\theta. Formally: θt+1=θt−α∇ft(θt)−αλ′θt\theta_{t+1} = \theta_t - \alpha\nabla f_t(\theta_t) - \alpha\lambda'\theta_t

These two update rules are identical when λ′=λ/α\lambda' = \lambda / \alpha. This mathematical equivalence led the deep learning community to treat the two terms as synonyms—deep learning frameworks (including PyTorch and TensorFlow at the time) implemented L2 regularization but called it "weight decay" in their documentation and APIs. For SGD users, this was fine; the two were truly interchangeable.

Where the equivalence breaks down. The critical observation is that this equivalence only holds when the gradient ∇ft(θt)\nabla f_t(\theta_t) enters the update rule in an unmodified form. Adaptive gradient methods like Adam fundamentally alter this: they apply a preconditioning matrix MtM_t (derived from historical gradient statistics) that scales the gradient on a per-parameter basis before applying the update. In Adam, this preconditioner is:

Mt=1v^t+ϵM_t = \frac{1}{\sqrt{\hat{v}_t} + \epsilon}

where v^t\hat{v}_t is the bias-corrected running average of squared gradients. The update then becomes:

θt+1=θt−αMt⊙∇ft(θt)\theta_{t+1} = \theta_t - \alpha M_t \odot \nabla f_t(\theta_t)

where ⊙\odot denotes element-wise multiplication.

Now, consider what happens when we add L2 regularization to the loss. The gradient of the regularizer λ′θ\lambda'\theta gets scaled by the same per-parameter preconditioner MtM_t as the loss gradient:

θt+1=θt−αMt⊙(∇ft(θt)+λ′θt)\theta_{t+1} = \theta_t - \alpha M_t \odot (\nabla f_t(\theta_t) + \lambda'\theta_t)

This means that parameters with historically large gradients receive a smaller effective regularization penalty, because their entry in MtM_t is small (large historical gradients → large vtv_t → small 1/vt1/\sqrt{v_t}). Conversely, parameters with historically small gradients get penalized more heavily. The regularization is no longer uniform across parameters—it becomes coupled with the gradient history.

In contrast, with decoupled weight decay, the weight decay step is applied directly to the parameters outside the gradient scaling mechanism:

θt+1=θt−αMt⊙∇ft(θt)−λθt\theta_{t+1} = \theta_t - \alpha M_t \odot \nabla f_t(\theta_t) - \lambda\theta_t

Every parameter is shrunk by the same factor λ\lambda, regardless of its gradient history. The weight decay is truly decoupled from the adaptive gradient mechanism.

Proposition 2 in the paper formalizes this by proving that for any adaptive gradient method with a non-trivial preconditioner (i.e., Mt≠kIM_t \neq kI for any scalar kk), there exists no L2 regularization coefficient λ′\lambda' that makes the L2-regularized update equivalent to the weight-decayed update. The two strategies produce fundamentally different dynamics.

Why this matters in practice. The paper provides intuition through Proposition 3 for a simplified case with a fixed preconditioner Mt=diag(s)−1M_t = \text{diag}(s)^{-1}: weight decay with factor λ\lambda is equivalent to a scale-adjusted L2 regularization where each parameter θi\theta_i is penalized proportionally to si\sqrt{s_i}, with sis_i being the inverse of the preconditioner for that parameter. In plain terms: parameters that historically had large gradients (small MtM_t entries, large sis_i) get regularized more under weight decay than under L2 regularization. This is the opposite of what happens with standard L2 regularization in Adam, where large-gradient parameters get regularized less.

The consequences for training dynamics are significant. Parameters that are heavily used (receiving large gradients) are precisely the ones that might be most prone to overfitting and thus most in need of regularization. Standard Adam with L2 regularization under-regularizes these parameters, potentially explaining its worse generalization. Decoupled weight decay restores uniform regularization across all parameters, matching the behavior that made weight decay effective in SGD in the first place.

The Overlooked Coupling: Learning Rate and Weight Decay Are Intertwined

Beyond the Adam-specific issue, the paper identifies a more general problem: even for SGD, the standard L2 regularization formulation creates an undesirable coupling between the learning rate α\alpha and the effective regularization strength. The equivalence condition λ′=λ/α\lambda' = \lambda / \alpha means that any change to the learning rate silently changes the effective weight decay being applied.

This coupling has practical consequences for hyperparameter tuning. If a practitioner adjusts α\alpha (e.g., trying different initial learning rates or using a learning rate schedule), they are inadvertently also changing the amount of regularization. The two hyperparameters cannot be optimized independently—the basin of good (α,λ′)(\alpha, \lambda') combinations lies along a diagonal, meaning both must be moved together to maintain performance. This contributes to the perception that SGD is "sensitive to hyperparameter settings."

By expressing the weight decay term directly (separate from the loss gradient), the paper aims to break this coupling, making α\alpha and λ\lambda independent axes in the hyperparameter space. This should make hyperparameter optimization easier and more robust.

How This Paper Positions Itself

The paper's positioning is specific and deliberately narrow in scope:

Not a new optimizer, but a fix for the current one. The paper doesn't propose AdamW as a fundamentally new optimization algorithm. It's Adam with one line changed: the weight decay is applied directly to the parameters rather than folded into the gradient. Algorithm 2 in the paper shows the modification clearly—line 12 of standard Adam becomes:

θt=θt−1−ηt(αm^t/(v^t+ϵ)+λθt−1)\theta_t = \theta_{t-1} - \eta_t \left( \alpha \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon) + \lambda \theta_{t-1} \right)

where the λθt−1\lambda\theta_{t-1} term is added after the adaptive gradient scaling, not inside it (where it would be if it were part of gtg_t as in standard L2-regularized Adam).

Piggybacking on a broader effort to improve Adam. The paper explicitly acknowledges that others have tried to improve Adam's generalization, and positions this fix as complementary. It specifically mentions normalized direction-preserving Adam (Zhang et al., 2017) as a related effort that could be combined with decoupled weight decay.

Focus on image classification as the canonical testbed. The paper uses image classification on CIFAR-10 and ImageNet32x32 with ResNet architectures as its primary experimental domain—precisely because this is where Adam's generalization gap relative to SGD had been most clearly documented. The choice is strategic: if AdamW can close the gap on these benchmarks, it provides the strongest possible evidence for the decoupled weight decay hypothesis.

Building on the Bayesian filtering interpretation. After the paper's preprint appeared, Aitchison (2018) independently developed a unified theory of adaptive gradient methods as Bayesian filtering, and noted that decoupled weight decay emerges naturally from the state transition prior in that framework. The paper incorporates this theoretical justification (Section 3), which provides a principled reason why weight decay—rather than L2 regularization—is the correct regularizer: in the Bayesian filtering view, the regularizer is part of the prior over how the optimizer moves between steps, and this prior should not depend on gradient history (which L2 regularization implicitly does through the preconditioner).

Addressing a practical, not a theoretical, problem. The paper's motivation is ultimately engineering-driven: practitioners should not need to choose between Adam (easy to tune, fast convergence) and SGD (better final generalization). If Adam can be fixed to match SGD's generalization while retaining its own advantages, the community can standardize on one optimizer, reducing the "common issue of selecting dataset/task-specific training algorithms and their hyperparameters." The paper explicitly states this as its driving goal.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a diagnostic and prescriptive work: it identifies a previously overlooked mathematical inequivalence in how adaptive optimizers apply regularization and proposes a minimal algorithmic fix with no additional computational overhead. The system being modified is the optimizer itself—specifically Adam—by changing one line of the update rule to apply weight decay directly to the weights rather than folding it into the gradient that gets adaptively scaled. The problem this solves is the systematic under-regularization of frequently-updated parameters in Adam, and the solution takes the form of a surgical modification to the optimizer's update step that restores the uniform weight decay originally proposed by Hanson & Pratt (1988), making the learning rate and weight decay factor independent hyperparameters.

3.2 Big-Picture Architecture (Diagram in Words)

The proposed method has four conceptual components, though in practice it's implemented as a change to a single line of the optimizer:

  1. The base optimizer (Adam or SGD) — the gradient-based update mechanism that computes parameter updates from mini-batch gradients. For SGD, this is simply gradient descent with momentum; for Adam, it's the full adaptive mechanism with running averages of gradients and squared gradients.

  2. The loss gradient computation — the standard forward-backward pass through the network that produces ∇ft(θt−1)\nabla f_t(\theta_{t-1}), the gradient of the mini-batch loss with respect to the parameters. Crucially, this gradient does NOT include any regularization term; it represents only the data-dependent loss.

  3. The decoupled weight decay step — a separate operation that shrinks each parameter by a factor λ\lambda (the weight decay rate) at each iteration, applied outside the adaptive gradient scaling mechanism. This is the key innovation: whereas standard Adam absorbs weight decay into the gradient (where it gets scaled by the per-parameter learning rate), AdamW applies it directly to the parameters.

  4. The schedule multiplier ηt\eta_t — an optional global scaling factor that can modulate the effective learning rate over time, enabling cosine annealing, warm restarts, or fixed schedules without affecting the weight decay strength (since weight decay is decoupled).

Information flows as follows: a mini-batch is sampled → the loss gradient ∇ft(θt−1)\nabla f_t(\theta_{t-1}) is computed (no regularization term) → the base optimizer computes its adaptive update direction (momentum and RMS scaling for Adam) → the weight decay shrinkage is applied directly to the parameters → a schedule multiplier optionally scales the learning rate. The weight decay and the gradient-based update operate on parallel tracks that never interact, which is the entire point of the decoupling.

3.3 Roadmap for the Deep Dive

  • First, the precise mathematical definition of weight decay as originally formulated by Hanson & Pratt (1988) and how it differs from L2 regularization when a preconditioning matrix is present. This establishes the formal inequivalence (Propositions 1–3) that motivates everything else.
  • Second, the specific algorithmic changes to SGD (producing SGDW) and to Adam (producing AdamW), shown as pseudocode with the critical lines highlighted. This is the "what changes" part.
  • Third, the Bayesian filtering justification (Section 3 of the paper) that provides theoretical grounding for why decoupled weight decay—rather than L2 regularization—is the correct choice from a probabilistic perspective.
  • Fourth, the normalized weight decay formulation that addresses the budget-dependence of optimal λ\lambda, enabling transfer of hyperparameters across different training durations and datasets.
  • Fifth, the integration with cosine annealing and warm restarts (producing SGDWR and AdamWR), which combines the decoupled weight decay with learning rate scheduling for improved anytime performance.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an algorithmic correction paper whose core idea is that for adaptive gradient methods, the weight decay regularization should be applied directly to the parameters as a multiplicative shrinkage, not added to the loss function as an L2 penalty that gets adaptively rescaled.

The Mathematical Inequivalence: Formal Statements

Before presenting the algorithms, the paper establishes three propositions that precisely characterize the relationship between L2 regularization and weight decay under different optimizers. These formal results are the intellectual foundation of the entire contribution.

Proposition 1 (SGD equivalence): For standard SGD without momentum or preconditioning, weight decay and L2 regularization are mathematically equivalent after a simple rescaling.

The two update rules being compared are:

Weight decay on ft(θ)f_t(\theta): θt+1=(1−λ)θt−α∇ft(θt)\theta_{t+1} = (1 - \lambda)\theta_t - \alpha \nabla f_t(\theta_t)

L2 regularization on ftreg(θ)=ft(θ)+λ′2∥θ∥22f_t^{\text{reg}}(\theta) = f_t(\theta) + \frac{\lambda'}{2}\|\theta\|_2^2: θt+1=θt−α∇ft(θt)−αλ′θt\theta_{t+1} = \theta_t - \alpha\nabla f_t(\theta_t) - \alpha\lambda'\theta_t

where λ\lambda is the weight decay rate, α\alpha is the learning rate, ftf_t is the mini-batch loss at step tt, ∇ft(θt)\nabla f_t(\theta_t) is its gradient, and λ′\lambda' is the L2 regularization coefficient.

What it computes: Both update rules produce identical parameter trajectories when λ′=λ/α\lambda' = \lambda / \alpha. The L2-regularized SGD update expands to θt+1=θt−α∇ft(θt)−αλ′θt\theta_{t+1} = \theta_t - \alpha\nabla f_t(\theta_t) - \alpha\lambda'\theta_t, which is exactly (1−αλ′)θt−α∇ft(θt)(1 - \alpha\lambda')\theta_t - \alpha\nabla f_t(\theta_t). Setting αλ′=λ\alpha\lambda' = \lambda makes this identical to the weight decay update.

Why this form matters: This equivalence is why the deep learning community conflated the two terms for years—and why the conflation was harmless for SGD users. If you implemented L2 regularization in your SGD code but called it "weight decay," the math worked out fine because the two operations produce identical updates. This proposition explains the historical confusion that the paper is now correcting.

Proposition 2 (The inequivalence for adaptive methods): For any adaptive gradient method that applies a non-trivial preconditioning matrix Mt≠kIM_t \neq kI (where kk is any scalar and II is the identity matrix) to scale gradients on a per-parameter basis, there exists no L2 regularization coefficient λ′\lambda' that makes the L2-regularized update equivalent to the weight-decayed update.

Formally, let the optimizer O\mathcal{O} have iterates:

With weight decay λ\lambda on ftf_t: θt+1=(1−λ)θt−αMt∇ft(θt)\theta_{t+1} = (1 - \lambda)\theta_t - \alpha M_t \nabla f_t(\theta_t)

With L2 regularization λ′\lambda' on ftregf_t^{\text{reg}}: θt+1=θt−αMt∇ft(θt)−αλ′Mtθt\theta_{t+1} = \theta_t - \alpha M_t \nabla f_t(\theta_t) - \alpha\lambda' M_t \theta_t

where MtM_t is the preconditioning matrix (in Adam, Mt=1/(v^t+ϵ)M_t = 1/(\sqrt{\hat{v}_t} + \epsilon) applied element-wise), λ\lambda is the weight decay rate, and λ′\lambda' is the L2 coefficient.

What it computes: For the two updates to be identical for all possible parameter vectors θt\theta_t, we would need λθt=αλ′Mtθt\lambda\theta_t = \alpha\lambda' M_t\theta_t to hold for all θt\theta_t. This requires λI=αλ′Mt\lambda I = \alpha\lambda' M_t, which can only be true if Mt=kIM_t = kI for some scalar kk—that is, if the preconditioner scales all parameters uniformly. Adaptive methods by design have per-parameter scaling, so Mt≠kIM_t \neq kI, and the equality cannot hold. No choice of λ′\lambda' can compensate for the per-parameter scaling.

Why this form matters: This is the paper's central mathematical insight. It proves that the L2-regularized Adam implemented in every deep learning library was not just an approximation of weight decay—it was computing something fundamentally different. The L2 penalty on each parameter gets divided by the square root of its historical squared gradient magnitude, so parameters with large gradients (which tend to be important, frequently-updated weights) receive less regularization than parameters with small gradients. This is precisely the opposite of what good regularization should do. The proposition establishes that this isn't a matter of degree (e.g., "L2 is slightly worse than weight decay") but of kind (the two optimizers are solving different optimization problems with different effective regularizers).

Proposition 3 (Weight decay as scale-adjusted L2 regularization for a fixed preconditioner): To build intuition for what weight decay is actually doing in adaptive methods, the paper analyzes a simplified case where the preconditioner is fixed rather than time-varying. Let Mt=diag(s)−1M_t = \text{diag}(s)^{-1} be a fixed diagonal preconditioner (so each parameter θi\theta_i has a fixed scaling factor si>0s_i > 0, with larger sis_i meaning stronger historical gradients and thus a smaller effective learning rate).

With weight decay λ\lambda on ftf_t: θt+1=(1−λ)θt−α⋅diag(s)−1⋅∇ft(θt)\theta_{t+1} = (1 - \lambda)\theta_t - \alpha \cdot \text{diag}(s)^{-1} \cdot \nabla f_t(\theta_t)

With a scale-adjusted L2 penalty on ftsreg(θ)=ft(θ)+λ′2α∥θ⊙s∥22f_t^{\text{sreg}}(\theta) = f_t(\theta) + \frac{\lambda'}{2\alpha}\|\theta \odot \sqrt{s}\|_2^2 (no weight decay): θt+1=θt−α⋅diag(s)−1⋅∇ft(θt)−αλ′θt\theta_{t+1} = \theta_t - \alpha \cdot \text{diag}(s)^{-1} \cdot \nabla f_t(\theta_t) - \alpha\lambda' \theta_t

where ⊙\odot denotes element-wise multiplication, s\sqrt{s} denotes the element-wise square root of the vector ss, and λ′=λ/α\lambda' = \lambda / \alpha.

What it computes: For the two updates to be equal, the weight decay term λθt\lambda\theta_t must equal the regularizer gradient term αλ′θt⊙s/s\alpha\lambda' \theta_t \odot s / s (after accounting for the preconditioner). The division by ss from the preconditioner and the multiplication by ss from the regularizer gradient cancel, leaving αλ′θt\alpha\lambda' \theta_t. With λ′=λ/α\lambda' = \lambda / \alpha, this matches λθt\lambda\theta_t. The effective loss function being optimized is ft(θ)+λ2α2∥θ⊙s∥22f_t(\theta) + \frac{\lambda}{2\alpha^2} \|\theta \odot \sqrt{s}\|_2^2.

Why this form matters: This proposition reveals what weight decay "looks like" when reinterpreted as a regularizer on the loss: it's an L2 penalty where each parameter θi\theta_i is weighted by si\sqrt{s_i}, meaning parameters with historically large gradients (large sis_i, which correspond to small entries in the Adam preconditioner 1/vt1/\sqrt{v_t}) get penalized more heavily. This is the opposite of standard L2 regularization in Adam, where large-gradient parameters get penalized less because their regularizer gradient is divided by si\sqrt{s_i}.

In plain language: standard Adam with L2 regularization under-regularizes the most important, most-updated weights. Decoupled weight decay over-regularizes them relative to L2, which (the paper argues) is closer to correct behavior. This aligns with the intuition that weights receiving large gradients are precisely those most prone to overfitting and most in need of regularization.

The paper notes that Proposition 3 is for a fixed preconditioner, which does not directly apply to practical Adam (where MtM_t changes at every step). However, it provides the correct intuition: weight decay effectively applies a scale-adjusted regularization where the adjustment factor depends on the gradient history, and this adjustment is in the direction of more regularization for important parameters, not less.

The Algorithmic Changes: SGDW and AdamW

The paper presents two modified optimizers: SGDW (SGD with decoupled weight decay) and AdamW (Adam with decoupled weight decay). Both are defined by making the weight decay step explicit and separate from the gradient computation.

SGDW (Algorithm 1 in the paper): The modification to SGD with momentum is straightforward. In standard SGD with momentum and L2 regularization, the gradient is computed as:

gt←∇ft(θt−1)+λθt−1g_t \leftarrow \nabla f_t(\theta_{t-1}) + \lambda\theta_{t-1}

This L2-penalized gradient is then used for the momentum update:

mt←β1mt−1+ηtαgtm_t \leftarrow \beta_1 m_{t-1} + \eta_t \alpha g_t θt←θt−1−mt\theta_t \leftarrow \theta_{t-1} - m_t

where β1\beta_1 is the momentum coefficient (e.g., 0.9), α\alpha is the base learning rate, ηt\eta_t is a schedule multiplier (from SetScheduleMultiplier(t)), and mtm_t is the momentum buffer (first moment estimate).

In SGDW, the L2 penalty is removed from the gradient computation. The gradient is computed on the loss function only:

gt←∇ft(θt−1)g_t \leftarrow \nabla f_t(\theta_{t-1})

The momentum update uses this clean gradient:

mt←β1mt−1+ηtαgtm_t \leftarrow \beta_1 m_{t-1} + \eta_t \alpha g_t

And the weight decay is applied directly to the parameters as a separate subtraction:

θt←θt−1−mt−ηtλθt−1\theta_t \leftarrow \theta_{t-1} - m_t - \eta_t \lambda \theta_{t-1}

where λ\lambda is the weight decay rate (not to be confused with the L2 coefficient λ′\lambda' from earlier).

What it computes: At each iteration, SGDW (1) computes the data-dependent gradient, (2) updates the momentum buffer using this gradient, (3) takes a gradient step using the momentum, and (4) separately shrinks all parameters toward zero by a factor ηtλ\eta_t\lambda. The weight decay term −ηtλθt−1-\eta_t\lambda\theta_{t-1} is outside the momentum mechanism, meaning the momentum buffer does not accumulate weight decay contributions—it only tracks data-dependent gradient information.

Why this form: For SGD, this change does not alter the optimization dynamics mathematically (Proposition 1 guarantees equivalence after rescaling). However, it makes the weight decay coefficient λ\lambda independent of the learning rate α\alpha in the hyperparameter space. When a practitioner changes α\alpha, they are not silently changing the effective regularization strength (as they would be with L2, since λ′=λ/α\lambda' = \lambda / \alpha needed to maintain equivalence). The two hyperparameters become orthogonal axes. As Figure 2 (top row) in the paper demonstrates, SGDW's best hyperparameter settings align with the axes of the (α,λ)(\alpha, \lambda) grid, while standard SGD's best settings lie along a diagonal, indicating coupling.

The paper uses initial learning rate α\alpha and weight decay λ\lambda as the primary hyperparameters. For the SGD experiments in Figure 2, α\alpha is swept over {1/2, 1/4, 1/8, ..., 1/1024} and λ\lambda over {1/8, 1/4, 1/2, 1, 2, 4, 8} × 0.001 (relative to some base value). The momentum factor β1\beta_1 is not explicitly stated for SGD but is typically 0.9.

AdamW (Algorithm 2 in the paper): The modification to Adam follows the same principle but is more consequential because Adam's per-parameter scaling breaks the L2/weight decay equivalence. The paper presents the full AdamW pseudocode with the key modification on line 12.

Standard Adam with L2 regularization (the "before" state in common deep learning libraries):

  1. Compute the L2-penalized gradient: gt←∇ft(θt−1)+λθt−1g_t \leftarrow \nabla f_t(\theta_{t-1}) + \lambda \theta_{t-1}

  2. Update biased first moment estimate (momentum): mt←β1mt−1+(1−β1)gtm_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t

  3. Update biased second moment estimate (RMS): vt←β2vt−1+(1−β2)gt2v_t \leftarrow \beta_2 v_{t-1} + (1 - \beta_2) g_t^2 (where gt2g_t^2 denotes element-wise squaring)

  4. Compute bias-corrected estimates: m^t←mt/(1−β1t)\hat{m}_t \leftarrow m_t / (1 - \beta_1^t) v^t←vt/(1−β2t)\hat{v}_t \leftarrow v_t / (1 - \beta_2^t)

  5. Apply the parameter update: θt←θt−1−ηtαm^t/(v^t+ϵ)\theta_t \leftarrow \theta_{t-1} - \eta_t \alpha \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)

where β1=0.9\beta_1 = 0.9 is the first moment decay rate, β2=0.999\beta_2 = 0.999 is the second moment decay rate, ϵ=10−8\epsilon = 10^{-8} prevents division by zero, and α=0.001\alpha = 0.001 is the default learning rate. Note that the λθt−1\lambda\theta_{t-1} term enters through gtg_t in step 1, which means it contributes to both mtm_t (through the gradient itself) and vtv_t (through the squared gradient), and both get bias-corrected and scaled by the adaptive learning rate m^t/(v^t+ϵ)\hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon).

AdamW (the proposed fix):

Steps 1–4 remain identical except that step 1 computes the gradient without the L2 penalty: gt←∇ft(θt−1)g_t \leftarrow \nabla f_t(\theta_{t-1})

The only change is in step 5, where the weight decay is applied directly: θt←θt−1−ηt(αm^t/(v^t+ϵ)+λθt−1)\theta_t \leftarrow \theta_{t-1} - \eta_t \left( \alpha \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon) + \lambda \theta_{t-1} \right)

where all hyperparameters (α=0.001\alpha = 0.001, β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ϵ=10−8\epsilon = 10^{-8}) retain their standard Adam defaults. The λ\lambda parameter is the decoupled weight decay coefficient (not the L2 coefficient).

What it computes: At each iteration, AdamW (1) computes the loss gradient on the data alone (no regularization), (2) updates the biased first and second moment estimates using this clean gradient, (3) applies bias correction, (4) computes the adaptive gradient step αm^t/(v^t+ϵ)\alpha \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon), and (5) separately applies the weight decay shrinkage −ηtλθt−1-\eta_t \lambda \theta_{t-1} directly to the parameters. The critical difference is that λθt−1\lambda\theta_{t-1} is added to the outer update rather than being folded into gtg_t, so it bypasses the moment estimates and adaptive scaling entirely.

Why this form: Separating the weight decay from the adaptive mechanism has three practical consequences:

  1. Uniform regularization: Every parameter is shrunk by the same factor λ\lambda per step, regardless of its gradient history. The weight-decayed AdamW iterates are: θt+1=(1−ηtλ)θt−1−ηtαm^t/(v^t+ϵ)\theta_{t+1} = (1 - \eta_t\lambda)\theta_{t-1} - \eta_t\alpha \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon) The shrinkage (1−ηtλ)(1 - \eta_t\lambda) applies equally to all parameters. In standard L2-regularized Adam, parameters with large v^t\hat{v}_t entries (historically large gradients, so small 1/v^t1/\sqrt{\hat{v}_t}) get the L2 penalty scaled down, receiving less regularization.

  2. Decoupled hyperparameters: α\alpha controls the magnitude of data-dependent updates, and λ\lambda controls the regularization strength. Changing one does not silently change the other. Figure 2 (bottom right) of the paper shows that AdamW's best hyperparameter settings are aligned with the axes of the (α,λ)(\alpha, \lambda) grid, unlike standard Adam (Figure 2, bottom left) where the best settings lie on a diagonal. This makes hyperparameter tuning more robust: even if α\alpha is suboptimal, tuning λ\lambda alone can still find a good configuration.

  3. Compatibility with learning rate schedules: Because λ\lambda is external to the gradient computation, applying a learning rate multiplier ηt\eta_t (for cosine annealing, step decay, etc.) modulates the data-dependent update size without affecting the regularization strength. This is not true for standard Adam, where changing the effective learning rate through ηt\eta_t would also change how the L2 penalty is applied (since the penalty gradient is scaled by the adaptive mechanism, which is itself scaled by ηtα\eta_t\alpha).

The Bayesian Filtering Justification

After the initial arXiv posting of the paper, Aitchison (2018) independently developed a unified theory of adaptive gradient methods as Bayesian filtering and noted that decoupled weight decay emerges naturally from the state transition prior in that framework. Section 3 of the paper summarizes this theoretical justification.

The Bayesian filtering view (Aitchison, 2018): Stochastic optimization of nn parameters θ1,…,θn\theta_1, \ldots, \theta_n is cast as a Bayesian filtering problem where the goal is to track a slowly-changing optimal parameter distribution given noisy observations from mini-batches. At each time step tt, one has:

  • A state transition prior P(θt+1∣θt)P(\theta_{t+1} | \theta_t) that models how the optimal parameters drift between steps, independent of data.
  • A likelihood P(yt+1∣θt+1)P(y_{t+1} | \theta_{t+1}) derived from the mini-batch at step t+1t+1.
  • A posterior P(θt+1∣y1:t+1)P(\theta_{t+1} | y_{1:t+1}) computed by applying Bayes' rule after marginalizing over θt\theta_t.

Aitchison assumes a Gaussian state transition of the form: P(θt+1∣θt)=N((I−A)θt,Q)P(\theta_{t+1} | \theta_t) = \mathcal{N}((I - A)\theta_t, Q)

where AA is a regularizer that prevents parameter values from growing unboundedly over time (if A=0A = 0, the expected value remains θt\theta_t at each step, with no drift penalty), and QQ is the covariance of the Gaussian perturbations. The mean of the filtering posterior then updates as: μpost=μprior+Σpost×g\mu_{\text{post}} = \mu_{\text{prior}} + \Sigma_{\text{post}} \times g

where gg is the gradient of the log-likelihood from the current mini-batch, and Σpost\Sigma_{\text{post}} is the posterior uncertainty, which acts as a preconditioner. Adam and RMSprop emerge as special cases where Σpost\Sigma_{\text{post}} is approximated by a diagonal matrix based on historical gradient magnitudes.

How weight decay enters: When A=λIA = \lambda I (a scalar multiple of the identity), each parameter's prior mean is shrunk by a factor (1−λ)(1 - \lambda) at each step: E[θt+1∣θt]=(1−λ)θt\mathbb{E}[\theta_{t+1} | \theta_t] = (1 - \lambda)\theta_t. This is exactly decoupled weight decay as defined in Equation 1 and Algorithm 2. Crucially, this regularization is part of the prior over parameter drift, not part of the likelihood. It does not depend on the mini-batch data or on the uncertainty Σpost\Sigma_{\text{post}} for each parameter.

Why this favors decoupled weight decay over L2 regularization: In the Bayesian filtering framework, L2 regularization would correspond to adding a penalty to the likelihood (the data-dependent term), which would then get scaled by Σpost\Sigma_{\text{post}} (the per-parameter uncertainty). This means parameters we are more uncertain about (which in Adam corresponds to those with small historical gradients, large Σpost\Sigma_{\text{post}} entries) would get regularized more, and parameters we are certain about would get regularized less. This is the opposite of what a sensible prior should do: the regularizer should reflect our belief that parameters tend to decay toward zero regardless of how much data we have about them. Weight decay, as a state transition prior, applies uniformly to all parameters, which is the correct inductive bias.

The paper presents this as a post-hoc theoretical validation rather than a motivation for the method (the method was developed empirically first), but it provides a satisfying principled explanation for why decoupled weight decay works better: it corresponds to the correct placement of regularization in a probabilistic model, as a prior over parameter dynamics rather than as a data-dependent penalty.

Normalized Weight Decay

The paper observes empirically that the optimal value of the weight decay coefficient λ\lambda depends strongly on the total training budget (number of epochs or batch passes). More training → smaller optimal λ\lambda, because the regularization has more steps to accumulate its effect. This makes hyperparameter tuning burdensome: λ\lambda values optimized for a short exploratory run cannot be directly transferred to a longer production run.

The budget-dependence problem: If you run SGD or Adam for TT epochs with batch size bb on a dataset of size BB, the total number of weight updates is BT/bBT/b (assuming one update per batch). Over the course of training, the weight decay applies a per-step shrinkage factor of (1−λ)(1 - \lambda) to each parameter. The total accumulated shrinkage over all BT/bBT/b steps is approximately (1−λ)BT/b(1 - \lambda)^{BT/b}. For this total shrinkage to remain constant across different training durations TT, λ\lambda must scale inversely with TT. The paper's experiments (SuppFigure 3, first row) confirm this: the optimal raw λ\lambda varies significantly between 100, 300, 900, and 2700 epochs.

The proposed normalization: To make hyperparameter values transferable across budgets, the paper introduces a normalized weight decay λnorm\lambda_{\text{norm}} and sets the per-step λ\lambda as:

λ=λnormbBT\lambda = \lambda_{\text{norm}} \sqrt{\frac{b}{BT}}

where bb is the batch size, BB is the total number of training points (e.g., 50,000 for CIFAR-10), and TT is the total number of epochs (or the number of epochs in the current restart, in the warm restart setting). For warm restarts with AdamWR and SGDWR, TT is the length of the current restart period TiT_i.

What it computes: λnorm\lambda_{\text{norm}} can be interpreted as the weight decay that would be used if training lasted only one batch pass (BT/b=1BT/b = 1). The per-step λ\lambda is then scaled down by the square root of the total number of batch passes. This square root scaling was chosen empirically (the authors note it "is merely one possibility informed by few experiments; a more lasting conclusion we draw is that using some normalization can substantially improve results" — Appendix B.1).

Why this form: The square root scaling reduces the dependence of optimal hyperparameters on runtime. SuppFigure 3 (second row) shows that with normalized weight decay, the optimal λnorm\lambda_{\text{norm}} values are very similar across different numbers of epochs (100 through 2700) on CIFAR-10. SuppFigure 3 (third and fourth rows) also shows that the same normalized values transfer well to ImageNet32x32 (where an epoch is 24× longer than on CIFAR-10, with B≈1.2B \approx 1.2 million vs. 50,000). Without normalization, the raw λ\lambda that worked for CIFAR-10 would be "roughly 5 times too large for ImageNet32x32" (Appendix D). The optimal λnorm\lambda_{\text{norm}} values were also similar across SGDW and AdamW (e.g., λnorm=0.025\lambda_{\text{norm}} = 0.025 and λnorm=0.05\lambda_{\text{norm}} = 0.05).

The practical benefit is that a practitioner can tune λnorm\lambda_{\text{norm}} on short runs and directly use the same value for much longer runs or different datasets with reasonable confidence.

Cosine Annealing and Warm Restarts (AdamWR and SGDWR)

The paper combines decoupled weight decay with the cosine annealing and warm restarts technique from Loshchilov & Hutter (2016) to produce SGDWR and AdamWR, improving anytime performance—the ability to get good results quickly without waiting for full convergence.

Cosine annealing schedule: Within each restart period ii, the schedule multiplier ηt\eta_t follows a cosine decay from ηmax(i)\eta^{(i)}_{\text{max}} to ηmin(i)\eta^{(i)}_{\text{min}}:

ηt=ηmin(i)+0.5(ηmax(i)−ηmin(i))(1+cos⁡(πTcur/Ti))\eta_t = \eta^{(i)}_{\text{min}} + 0.5(\eta^{(i)}_{\text{max}} - \eta^{(i)}_{\text{min}})(1 + \cos(\pi T_{\text{cur}} / T_i))

where TiT_i is the total length of the ii-th restart period (in epochs), TcurT_{\text{cur}} tracks how many epochs have been performed since the last restart (updated at each batch, so it takes non-integer values), and ηmax(i)\eta^{(i)}_{\text{max}} and ηmin(i)\eta^{(i)}_{\text{min}} define the range of the multiplier. For simplicity, the paper uses ηmax(i)=1\eta^{(i)}_{\text{max}} = 1 and ηmin(i)=0\eta^{(i)}_{\text{min}} = 0 for all ii, simplifying to:

ηt=0.5+0.5cos⁡(πTcur/Ti)\eta_t = 0.5 + 0.5 \cos(\pi T_{\text{cur}} / T_i)

What it computes: ηt\eta_t starts at 1 when Tcur=0T_{\text{cur}} = 0 (the learning rate multiplier is at maximum at the beginning of each restart), smoothly decreases to 0 when Tcur=TiT_{\text{cur}} = T_i (the learning rate is effectively zero at the end of each restart period), following a cosine-shaped curve that spends more time at intermediate values than a linear decay would.

Warm restart procedure: The restart periods grow geometrically. The initial period T0T_0 is set to a small fraction of the total training budget (e.g., T0=100T_0 = 100 epochs). At each restart (when Tcur=TiT_{\text{cur}} = T_i), the period length is multiplied by a factor TmultT_{\text{mult}} (e.g., Tmult=2T_{\text{mult}} = 2):

Ti+1=Ti×TmultT_{i+1} = T_i \times T_{\text{mult}}

and TcurT_{\text{cur}} is reset to 0. The optimizer's state (θt\theta_t, mtm_t, vtv_t) is not reset—the "warm restart" only resets the learning rate schedule, not the parameters or momentum buffers. This means later restarts begin optimization from a good solution found in the previous period and explore further at a high learning rate, potentially escaping local minima that the annealed optimizer got stuck in.

Why this form: The warm restart strategy improves anytime performance because it produces a sequence of solutions (snapshots at the end of each restart period, when ηt=0\eta_t = 0) that get progressively better. A practitioner can stop training early and use the most recent snapshot, getting strong results without waiting for the full budget. The geometric growth of TiT_i (e.g., 100, 200, 400, 800 epochs) means the optimizer spends more time refining in later restarts. SuppFigure 1 shows an example schedule with T0=100T_0 = 100 and Tmult=2T_{\text{mult}} = 2, where restarts occur at epochs 100, 300, 700, and 1500.

Compatibility with decoupled weight decay: A key design point is that decoupled weight decay makes the warm restart strategy work well with Adam. In the authors' earlier attempts (before decoupling), "our initial version of Adam with warm restarts had better anytime performance than Adam, it was not competitive with SGD with warm restarts, precisely because L2 regularization was not working as well as in SGD." The decoupled weight decay fixes this: AdamWR (AdamW with warm restarts and normalized weight decay) achieves comparable performance to SGDWR on ImageNet32x32 and narrows the gap significantly on CIFAR-10 (Figure 4).

The normalized weight decay plays a crucial role in making warm restarts practical: within each restart period ii, the effective TT used in the normalization formula is TiT_i (the length of the current restart), not the total training budget. This ensures that the weight decay strength is appropriate for each restart period's duration without requiring manual per-period tuning.

Design Choices and Their Justifications (Summary)

  • Decoupling weight decay from the gradient (rather than fixing L2): The inequivalence proof (Proposition 2) shows that no L2 coefficient can reproduce weight decay for adaptive methods. The only way to get uniform per-parameter regularization is to apply the decay directly to the weights, bypassing the adaptive scaling mechanism.

  • Applying weight decay as a separate subtraction (rather than modifying the preconditioner): Adding the weight decay term outside the adaptive update is computationally free (one extra vector subtraction per iteration) and avoids changing the well-tuned Adam momentum and RMS mechanisms. Alternative approaches that modify the preconditioner to account for regularization would require re-deriving the entire optimizer.

  • Square root normalization of weight decay (rather than linear or no normalization): Chosen empirically based on experiments on CIFAR-10 and verified on ImageNet32x32. Linear scaling (λ=λnorm⋅b/(BT)\lambda = \lambda_{\text{norm}} \cdot b/(BT)) would make the optimal λ\lambda too small for long runs; no normalization would require per-budget retuning. The authors acknowledge this is one possible choice and that "even better scaling rules are likely to exist."

  • Cosine annealing with geometric restart growth (rather than fixed learning rate or step decay): Cosine annealing provides a smooth transition from exploration (high learning rate) to exploitation (low learning rate) within each restart, and the geometric growth ensures more time is spent refining in later restarts when the solution quality is higher. Fixed learning rates fail to converge tightly; step decay introduces discontinuous jumps that are harder to tune.

  • Keeping Adam's default hyperparameters unchanged (α=0.001\alpha = 0.001, β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ϵ=10−8\epsilon = 10^{-8}): This is deliberate: the paper wants to show that the improvement comes specifically from decoupling weight decay, not from retuning Adam's internal parameters. Practitioners can adopt AdamW with their existing Adam configurations, only needing to re-tune λ\lambda.

  • Using a 26-layer 2-branch ResNet with Shake-Shake regularization (following Gastaldi, 2017): This architecture choice establishes a strong baseline that had achieved state-of-the-art results with SGD, making AdamW's ability to match or approach those results more meaningful than if a weaker architecture were used. The 2x64d variant (11.6M parameters) is used for hyperparameter sweeps, and the larger 2x96d variant (25.6M parameters) for long-run comparisons.

4. Key Insights and Innovations

Innovation 1: The Identification of L2 Regularization as a Mechanistically Different Operation from Weight Decay in Adaptive Optimizers—Not Just a Terminology Confusion

The paper's most fundamental intellectual contribution is not the fix itself (applying decay outside the gradient computation), but the precise diagnostic move that revealed the fix was necessary in the first place. Before this work, the deep learning community operated under the assumption that "weight decay" and "L2 regularization" were synonyms—an assumption that was correct for SGD and therefore went unchallenged for years. The critical insight is that this equivalence is contingent on the absence of a per-parameter preconditioner, and that the introduction of adaptive gradient scaling in Adam silently breaks it, transforming L2 regularization into something qualitatively different from the uniform parameter shrinkage that Hanson & Pratt (1988) originally proposed.

What distinguishes this from a simple nomenclature correction is the direction of the resulting bias. Proposition 3 shows that L2 regularization in Adam effectively applies less regularization to parameters with large historical gradients (because their regularizer gradient gets divided by a large v^t\sqrt{\hat{v}_t} term), while decoupled weight decay applies more regularization to these same parameters (when reinterpreted as a scale-adjusted L2 penalty). The paper doesn't just say "these are different"—it characterizes how they differ and argues that the L2 version gets the regularization direction exactly backwards relative to what good generalization requires. Parameters that are frequently updated (receiving large gradients) are precisely those most likely to overfit and most in need of strong regularization; Adam with L2 gives them the weakest penalty.

Prior assumption vs. this work: Prior to this paper, the dominant framing—encoded in every major deep learning library's API—was that weight_decay and l2_regularization were interchangeable names for the same thing. The Equivalence Proof (Proposition 1) had been known for decades for plain gradient descent, and the community had simply never re-examined whether it survived the transition to adaptive methods. Wilson et al. (2017) had documented Adam's generalization gap but attributed it to inherent properties of adaptive gradients or to sharp-minima effects. This paper reframes the gap not as a flaw in adaptivity per se, but as a regularization mismatch: Adam can generalize well; it was just being regularized with the wrong mechanism.

Significance beyond performance: This insight is fundamentally a conceptual reframing with practical diagnostic value. It gives the community a new lens for understanding optimizer behavior: when an adaptive method underperforms SGD, one should ask not just "is the learning rate wrong?" but "is the regularization being applied correctly given the adaptive scaling?" This reframing is not an incremental finding—it identifies a first-principles mathematical error in every Adam implementation that existed at the time, and the evidence in Figure 2 (bottom rows) shows that fixing it is the difference between Adam being clearly inferior to SGD and being competitive.

Evidence anchor: Figure 2 (bottom left vs. bottom right) provides the clearest visual summary. Standard Adam with L2 regularization shows its best hyperparameter settings clustered along a diagonal, with performance that is uniformly worse than SGD with L2 (top left). AdamW (bottom right) not only improves the best accuracy substantially but produces a hyperparameter landscape where the optima align with the axes—showing that the decoupling genuinely separates the roles of α\alpha and λ\lambda. The improvement is roughly 15% relative in test error on CIFAR-10 (Figure 4), moving Adam from clearly inferior to competitive with SGD with momentum.

Innovation 2: The Decoupling of Learning Rate and Regularization Strength as Independent Hyperparameter Axes

While Innovation 1 identifies the mechanistic problem, Innovation 2 addresses a meta-optimization problem that had practical consequences for everyone training neural networks. Even for SGD—where L2 and weight decay are mathematically equivalent—the standard L2 formulation creates an undesirable coupling: λ′=λ/α\lambda' = \lambda / \alpha means that adjusting the learning rate silently changes the effective regularization, and vice versa. The paper's proposal to express weight decay as a direct parameter shrinkage (the −ηtλθt−1-\eta_t \lambda \theta_{t-1} term in Algorithms 1 and 2) makes α\alpha and λ\lambda orthogonal hyperparameters whose effects are additive rather than multiplicative on the update.

The significance of this decoupling extends beyond Adam. For SGD, the coupling might explain some of the optimizer's reputation for being "sensitive to hyperparameters": if the good region of (α,λ′)(\alpha, \lambda') space lies on a narrow diagonal, then changing either hyperparameter in isolation (as is common in grid search or manual tuning) will quickly leave the good region. A practitioner who finds a good α\alpha and then tries to tune λ′\lambda' independently will see performance degrade, leading to the conclusion that the optimizer is finicky—when the real issue is that the hyperparameter representation is entangled.

What changes conceptually: This is not just a convenience for hyperparameter tuning; it's a redefinition of what the hyperparameters mean. In the standard L2 formulation, λ′\lambda' is a coefficient on a penalty term whose effective strength is modulated by the learning rate. In the decoupled formulation, λ\lambda is a per-step shrinkage factor whose interpretation is independent of how large the gradient steps are. This semantic clarity matters for transfer learning and budget scaling: if you know that your model benefits from shrinking weights by 0.01% per step regardless of the learning rate, you can set λ\lambda directly rather than computing λ′=λ/α\lambda' = \lambda / \alpha and hoping you got the scaling right.

Evidence anchor: Figure 2 (top row) demonstrates the difference in hyperparameter landscapes. For standard SGD (top left), the best settings (black circles) lie on a diagonal spanning the grid—meaning α\alpha and λ′\lambda' must be moved together to stay in the good region. For SGDW (top right), the best settings align with grid axes, and the performance surface shows broad plateaus: even at a clearly suboptimal learning rate (e.g., α=1/1024\alpha = 1/1024), tuning λ\lambda alone can recover good performance (finding the optimum at λ=1/4×0.001\lambda = 1/4 \times 0.001). This is a qualitative change in the hyperparameter optimization landscape, not just a shift in the location of the optimum.

Innovation 3: Normalized Weight Decay as a Budget-Aware Regularization Schedule

The paper's third contribution is the empirical demonstration that optimal weight decay strength depends systematically on the total number of parameter updates, and the proposal of a simple normalization rule (λ=λnormb/(BT)\lambda = \lambda_{\text{norm}} \sqrt{b/(BT)}) to make hyperparameter values transferable across training budgets. This is not a theoretical contribution (the square root scaling is empirically chosen and the authors are explicit that better rules likely exist), but it addresses a practical scaling problem that had been largely ignored: the community routinely tuned weight decay on short runs (e.g., 100 epochs) and then used those values for much longer runs (e.g., 1000+ epochs) under the implicit assumption that the optimal regularization strength was budget-independent. The paper shows this assumption is false and provides a partial fix.

The conceptual move: The paper reframes weight decay not as a static hyperparameter but as a per-step shrinkage rate whose cumulative effect over training determines the total regularization applied. If you double the number of training steps, the per-step rate should be adjusted to keep the total shrinkage roughly constant (or, in the paper's square root scaling, to grow sublinearly with budget). This is analogous to how we think about learning rate schedules—nobody expects the same learning rate to work for 100 epochs and 1000 epochs—but the community had not extended this reasoning to regularization strength.

Significance beyond this paper: The normalized weight decay concept, while simple, opened the door to thinking about regularization schedules (analogous to learning rate schedules) where the regularization strength varies over training in a principled way. The paper's integration with warm restarts (where TT in the normalization formula is the current restart length TiT_i, not the total budget) is an early example of this: the weight decay strength adapts to each restart period's duration. This is a prototype for dynamic regularization rather than a fully-developed method, but it established the idea that regularization should be budget-aware.

Evidence anchor: SuppFigure 3 (Appendix D) provides the key evidence. The top row shows that optimal raw λ\lambda values shift dramatically across different epoch counts (100, 300, 900, 2700) for AdamW without normalization—the diagonal band of good settings moves substantially. The second row shows that with normalized weight decay, the optimal λnorm\lambda_{\text{norm}} values are nearly stationary across budgets, with the best settings (e.g., λnorm=0.025\lambda_{\text{norm}} = 0.025 and 0.050.05) remaining consistent. The third and fourth rows show cross-dataset transfer: the same λnorm\lambda_{\text{norm}} values that work on CIFAR-10 (50K images, short epochs) also work well on ImageNet32x32 (1.2M images, 24× longer epochs), which the raw λ\lambda values would not (they would be "roughly 5 times too large" without normalization, per Appendix D).

Innovation 4: The Bayesian Filtering Reinterpretation as a Post-Hoc Theoretical Grounding

The paper's fourth contribution—developed by Aitchison (2018) after the arXiv preprint appeared and incorporated into Section 3 of the published version—is the theoretical reframing of decoupled weight decay as the natural consequence of placing regularization in the state transition prior of a Bayesian filtering model rather than in the observation likelihood. This is not the paper's own theoretical development (credit goes to Aitchison), but the paper's integration of this perspective provides a principled answer to why decoupled weight decay should be preferred: it corresponds to the correct placement of our inductive bias about parameter dynamics in a probabilistic model.

The conceptual distinction: In the Bayesian filtering view, weight decay represents our prior belief that optimal parameters tend to drift toward zero over time, independently of the data we observe. This prior should apply uniformly to all parameters regardless of how much gradient information we have about them. L2 regularization, by contrast, operates through the likelihood term—meaning its effect gets scaled by the per-parameter posterior uncertainty Σpost\Sigma_{\text{post}}, which in Adam is approximated by 1/v^t1/\sqrt{\hat{v}_t}. Parameters we are uncertain about (small historical gradients, large Σpost\Sigma_{\text{post}}) get regularized more; parameters we are certain about get regularized less. This is precisely backwards: our prior belief that weights should decay toward zero should not depend on how much data we've seen about each weight.

Significance: This theoretical grounding elevates decoupled weight decay from an empirically-motivated hack to a principled design choice. It also connects optimizer design to Bayesian inference, suggesting that other optimizer components (momentum, adaptive learning rates) can be understood and potentially improved through their probabilistic interpretations. The fact that the theory was developed independently and converged on the same algorithmic modification strengthens the case that the modification is not arbitrary—it is the natural solution that emerges from two different lines of reasoning (empirical diagnosis of the L2/weight decay inequivalence and Bayesian filtering theory).

Limitations acknowledged: The paper is appropriately modest about this theoretical connection, noting that the Bayesian filtering view provides justification but was not the original motivation for the method. The theory assumes Gaussian approximations and conjugate likelihoods that may not hold in practice, and the connection to Adam specifically requires approximating the full posterior covariance Σpost\Sigma_{\text{post}} with a diagonal matrix based on squared gradients. The theoretical argument is suggestive rather than definitive, but it provides a satisfying conceptual framework that the empirical results alone would lack.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary dataset is CIFAR-10 (Krizhevsky, 2009), a 10-class image classification benchmark with 50,000 training images and 10,000 test images, each 32×32 pixels. Additional experiments use ImageNet32x32 (Chrabaszcz et al., 2017), a downsampled version of the original ImageNet dataset containing approximately 1.2 million 32×32 pixel images across 1,000 classes. The standard data augmentation procedure for CIFAR datasets is applied throughout, with a batch size of 128 in all experiments.

  • Base model(s). The primary architecture is a 26-layer ResNet with 2 residual branches and varying width, following Gastaldi (2017). Two variants are used: a 26 2x64d ResNet (11.6M parameters, with the first residual block having width 64) for hyperparameter sweeps and shorter runs, and a larger 26 2x96d ResNet (25.6M parameters) for long-run comparisons. All models employ Shake-Shake regularization (Gastaldi, 2017) in addition to the optimizer-based regularization being studied. The architecture choice is strategic: Gastaldi (2017) achieved state-of-the-art CIFAR-10 results (2.86% error) using this architecture with SGD, providing a strong baseline against which to test whether AdamW can close the generalization gap.

  • Metrics. The primary metric is test error rate (%) — specifically Top-1 error on CIFAR-10 and Top-5 error on ImageNet32x32. Training loss (cross-entropy) is also tracked to distinguish better convergence from better generalization (as in Figure 3 and SuppFigure 4). For hyperparameter landscape visualizations (Figures 1 and 2), the metric is final test error after a fixed training budget, displayed as heatmaps where darker colors indicate lower error. The paper computes relative improvement in test error (e.g., "15% relative improvement" means the error rate dropped by 15% of its original value, not 15 percentage points).

  • Baselines. The paper compares against multiple baselines, all within the same architectural framework to isolate the optimizer effect:

    • Standard Adam with L2 regularization (referred to as "Adam"): This is the Adam optimizer as commonly implemented in deep learning libraries, where the weight decay parameter is used to add an L2 penalty to the loss function. The gradient gtg_t includes the term λθt−1\lambda\theta_{t-1}, meaning the regularization enters through the adaptive gradient scaling mechanism.
    • Standard SGD with momentum and L2 regularization (referred to as "SGD"): The common SGD implementation where L2 regularization is folded into the gradient computation.
    • No regularization (λ=0\lambda = 0): Included implicitly in the heatmaps as the leftmost column or bottom row where λ=0\lambda = 0.

    The proposed methods are SGDW (SGD with decoupled weight decay), AdamW (Adam with decoupled weight decay), SGDWR (SGDW with cosine annealing and warm restarts), and AdamWR (AdamW with cosine annealing and warm restarts). All baselines use the same network architecture, batch size, and data augmentation; only the optimizer and regularization mechanism differ.

  • Generation budget / compute accounting. The paper measures training budget in epochs (full passes through the training set), which is standard for image classification. Experiments span budgets from 100 epochs (for hyperparameter sweeps in Figures 1 and 2) to 1,800 epochs (for long-run comparisons in Figure 3 and SuppFigure 4). The warm restart variants (SGDWR, AdamWR) use geometric restart schedules (e.g., T0=100T_0 = 100, Tmult=2T_{\text{mult}} = 2), with total budgets determined by how many restarts are performed. The paper does not report wall-clock time or FLOP counts; the primary cost metric is the number of epochs, which is proportional to total compute since batch size and model size are held constant. For the normalized weight decay formulation, the budget is incorporated explicitly: λ=λnormb/(BT)\lambda = \lambda_{\text{norm}} \sqrt{b/(BT)}, where BT/bBT/b is the total number of batch passes. In warm restart mode, TT in this formula is the length of the current restart period TiT_i, not the total budget.

  • Cross-validation / statistical protocol. The paper does not employ cross-validation, statistical significance testing, or multiple random seeds. This is a notable departure from standard practice in optimizer evaluation papers. All hyperparameter sweeps (Figures 1 and 2) are conducted on single training runs for each grid point; the long-run comparisons (Figures 3 and 4) also appear to use single runs. Error bars are not reported on any experimental results. The "top-10 hyperparameter settings" marked by black circles in Figure 2 represent the 10 grid points with lowest test error out of the full sweep, but no confidence intervals or variance estimates are provided for these error values. The test set is the standard CIFAR-10 test set of 10,000 images, which is large enough that sampling variance in test error is likely small, but the absence of multiple runs means that differences between optimizer configurations are not assessed for statistical significance. For a paper whose primary claim is that AdamW "substantially improves" Adam's generalization, the lack of error bars or replicate runs is a limitation—particularly for the long-run comparisons where only 7–12 hyperparameter settings are tested and small differences could be due to training noise.

Main Quantitative Results

The experimental narrative is organized around four progressively stronger tests of the decoupled weight decay hypothesis: (1) hyperparameter landscape comparisons showing improved separability, (2) short-run sweeps showing AdamW's improvement over Adam across different learning rate schedules, (3) long-run comparisons proving the generalization benefit is not just faster convergence, and (4) warm restart experiments demonstrating improved anytime performance at scale.

Hyperparameter Landscape: Decoupling α and λ Makes Optimization Easier

Figure 2 is the paper's central evidence for the claim that decoupled weight decay produces a more separable hyperparameter space. The experiment trains a 26 2x64d ResNet on CIFAR-10 for 100 epochs at each point on a logarithmic grid of initial learning rate α\alpha and weight decay/L2 factor λ\lambda. The learning rate is swept over {1/2, 1/4, 1/8, ..., 1/1024}, and λ\lambda is swept over {1/8, 1/4, 1/2, 1, 2, 4, 8} × 0.001 (where the base value 0.001 corresponds to different interpretations depending on whether L2 or decoupled weight decay is used).

SGD vs. SGDW (Figure 2, top row): The difference is visually stark. For standard SGD with L2 regularization (top left), the best test error region forms a diagonal band running from top-left to bottom-right of the heatmap. The "top-10 hyperparameter settings" (black circles) cluster along this diagonal, indicating that α\alpha and λ′\lambda' are coupled: if you change the learning rate, you must also change the L2 coefficient to stay in the good region. The paper notes that a setting at the top-left black circle (α=1/2\alpha = 1/2, λ=1/8×0.001\lambda = 1/8 \times 0.001) would be ruined by changing either hyperparameter alone—"only changing either α\alpha or λ\lambda by itself would worsen results." This is offered as evidence that the L2 formulation contributes to SGD's reputation for hyperparameter sensitivity.

For SGDW with decoupled weight decay (top right), the best settings align with the grid axes rather than a diagonal. The good region forms a broad plateau across a wide range of learning rates for the best weight decay values (particularly λ=1/4×0.001\lambda = 1/4 \times 0.001 and λ=1/2×0.001\lambda = 1/2 \times 0.001). Even at a clearly suboptimal learning rate of α=1/1024\alpha = 1/1024, tuning λ\lambda alone to 1/4×0.0011/4 \times 0.001 yields strong performance—"leaving it fixed and only optimizing the weight decay factor would yield a good value." This is the operational meaning of "decoupled": the optimal λ\lambda does not depend strongly on α\alpha, so the two can be tuned sequentially rather than jointly.

Adam vs. AdamW (Figure 2, bottom row): The story for Adam is similar but with an additional finding: Adam with L2 regularization (bottom left) not only shows coupled hyperparameters (diagonal best settings), but its absolute performance is substantially worse than both SGD variants. The paper states that "Adam's best hyperparameter settings performed clearly worse than SGD's best ones," and that "its best results obtained for non-zero L2 regularization factors were comparable to the best ones obtained without the L2 regularization, i.e., when λ=0\lambda = 0." In other words, standard Adam with L2 regularization barely benefits from regularization at all—the best regularized runs are no better than unregularized Adam.

AdamW (bottom right) both improves the absolute performance (reaching error rates competitive with SGD and SGDW) and produces a more axis-aligned hyperparameter landscape. The best settings are found at intermediate weight decay values (λ=1/4×0.001\lambda = 1/4 \times 0.001 and 1/2×0.0011/2 \times 0.001) across a range of learning rates around α=1/256\alpha = 1/256 to 1/641/64. The paper summarizes: "the results in Figure 2 support our hypothesis that the weight decay and learning rate hyperparameters can be decoupled, and that this in turn simplifies the problem of hyperparameter tuning in SGD and improves Adam's performance to be competitive w.r.t. SGD with momentum."

A subtle point: note that in Figure 2, SGDW and AdamW achieve comparable best test error (both around 3.2–3.4% based on the color scale), while standard Adam with L2 is stuck around 3.6–4.0%. The improvement from Adam to AdamW is therefore on the order of 10–20% relative reduction in error, even in this short 100-epoch regime. The longer runs in subsequent experiments show larger absolute differences.

Learning Rate Schedule Dependence: Decoupled Weight Decay Benefits All Schedules, but Cosine Annealing Is Best

Figure 1 demonstrates that the advantage of decoupled weight decay is not specific to a particular learning rate schedule—it holds across fixed, step-drop, and cosine annealing schedules—but that the magnitude of the advantage is largest with better schedules. The experiment again trains a 26 2x64d ResNet on CIFAR-10 for 100 epochs, sweeping α\alpha and λ\lambda on a grid, with results shown as heatmaps of final test error.

Three schedules tested (columns):

  • Fixed learning rate (left column): ηt=1\eta_t = 1 throughout training. No annealing.
  • Step-drop schedule (middle column): The learning rate drops by a factor at epochs 30, 60, and 80. This is a traditional schedule common in CIFAR training recipes.
  • Cosine annealing (right column): ηt\eta_t follows a cosine curve from 1 to 0 over the full 100 epochs, as in Loshchilov & Hutter (2016).

Key findings from Figure 1:

  • AdamW outperforms Adam for all three schedules (compare bottom row to top row). The improvement is visible as lower test error across the entire heatmap for AdamW.
  • The advantage is largest with cosine annealing. The right column (cosine) shows the deepest blue regions for AdamW and the largest gap relative to Adam. This makes sense: cosine annealing produces the best overall results, and the decoupled weight decay amplifies this advantage because the regularization remains effective as the learning rate drops to zero. In standard Adam with L2, as the learning rate anneals, the effective L2 penalty also anneals (since it's scaled by ηtα\eta_t \alpha), potentially leaving the model under-regularized late in training.
  • Hyperparameter separability improves most with schedules. For fixed learning rate (left column), both Adam and AdamW show somewhat diagonal best settings. For step-drop and especially cosine annealing, AdamW's best settings become more axis-aligned while Adam's remain diagonal. The paper states: "decoupled weight decay leads to a more separable hyperparameter search space, especially when a learning rate schedule, such as step-drop and cosine annealing is applied."

The figure also establishes cosine annealing as the superior learning rate schedule for both optimizers, motivating its exclusive use in the remainder of the paper's experiments. This is a deliberate choice: if the goal is to show AdamW can compete with SGD, it makes sense to use the best available learning rate schedule for both optimizers.

Long-Run Generalization: AdamW's Advantage Is Not Just Faster Convergence

Figures 3 and SuppFigure 4 address a potential confound: does AdamW only look better because it converges faster, or does it actually find solutions that generalize better at the same training loss? The experiment trains a larger 26 2x96d ResNet on CIFAR-10 and ImageNet32x32 for 1,800 epochs, comparing standard Adam (with 12 different L2 regularization settings) against AdamW (with 7 settings of normalized weight decay). The initial learning rate is fixed at α=0.001\alpha = 0.001 for both optimizers (the Adam default).

Training dynamics (Figure 3, top row; SuppFigure 4, top row):

  • The learning curves (training loss over epochs) show that Adam and AdamW often track closely for the first ~500–900 epochs—their convergence speeds are similar. However, in the second half of training, AdamW's training loss continues to decrease while Adam's plateaus or increases (indicating overfitting).
  • The test error curves (top right) show a more dramatic divergence: AdamW's test error keeps improving throughout the 1,800 epochs, while Adam's test error plateaus or even rises late in training (a classic overfitting signature). On CIFAR-10, the best AdamW runs achieve roughly 2.6–2.8% test error at epoch 1,800, while the best Adam runs plateau around 3.2–3.4%.

Generalization gap, not just convergence (Figure 3, bottom row; SuppFigure 4, bottom row):

  • The bottom-left scatter plot shows test error vs. weight decay setting at the end of training. Adam's test error is roughly flat across L2 regularization values (around 3.2–3.4%), confirming the earlier finding from Figure 2 that L2 regularization in Adam provides minimal benefit over λ=0\lambda = 0. AdamW's test error drops substantially at moderate normalized weight decay values (λnorm≈0.025–0.05\lambda_{\text{norm}} \approx 0.025–0.05), reaching roughly 2.6–2.8%.
  • The bottom-right scatter plot is the critical evidence: test error vs. training loss at the end of training. For the same training loss, AdamW consistently achieves lower test error than Adam. This rules out the hypothesis that AdamW only wins because it converges faster—even when Adam is allowed to train long enough to achieve similar or lower training loss, its test error remains worse. This is the operational definition of a generalization improvement: the model trained with AdamW finds a solution that fits the training data similarly but transfers better to unseen data.

On ImageNet32x32 (SuppFigure 4), the qualitative results are identical: AdamW achieves lower training loss (better convergence) and substantially lower test error (better generalization), with the generalization gap evident in the test-loss-vs-training-loss plot. The paper states these results "yield the same conclusion of substantially improved generalization performance."

Quantifying the improvement: The paper reports that AdamW achieves "15% relative improvement in test error compared to Adam both on CIFAR-10 and ImageNet32x32" (Section 4.4). For CIFAR-10, if Adam's best test error is roughly 3.3% and AdamW's is roughly 2.8%, the relative improvement is (3.3 - 2.8) / 3.3 ≈ 15%. This is the source of the headline number in the abstract and executive summary.

Warm Restarts: Closing the Gap with SGD

Figure 4 presents the most practically important comparison: Adam with decoupled weight decay plus cosine annealing and warm restarts (AdamWR) against SGD with the same warm restart schedule (SGDWR), evaluating both anytime performance (error vs. epochs) and final performance.

Anytime performance (Figure 4, both panels):

  • The x-axis is epochs; the y-axis is test error. The warm restart schedule (T0=100T_0 = 100, Tmult=2T_{\text{mult}} = 2) produces sharp drops in error at each restart (when the learning rate jumps back up and explores a new region), followed by gradual improvement as the learning rate anneals.
  • AdamWR (the proposed method) shows significantly faster improvement than AdamW without restarts: "AdamWR greatly sped up AdamW on CIFAR-10 and ImageNet32x32, up to a factor of 10 (see the results at the first restart)." At epoch 100 (the first restart), AdamWR has already achieved an error rate that AdamW without restarts took 1,000+ epochs to reach.
  • The comparison between AdamWR and SGDWR shows that AdamWR narrows the gap to SGDWR substantially. On CIFAR-10 (left panel), SGDWR still holds a small advantage (its curve is slightly below AdamWR's throughout), but the gap is much smaller than between standard Adam and SGD. On ImageNet32x32 (right panel), AdamWR and SGDWR show "comparable performance"—their curves largely overlap, with perhaps a slight edge to SGDWR at the very end of training.

Default learning rate results (text in Section 4.4): The paper specifically highlights that for the default learning rate α=0.001\alpha = 0.001, AdamW "achieved 15% relative improvement in test error compared to Adam both on CIFAR-10 and ImageNet32x32." AdamWR "achieved the same improved results but with a much better anytime performance." This is practically important: practitioners using the default Adam learning rate can adopt AdamW/AdamWR without retuning α\alpha, only adjusting λ\lambda.

SuppFigure 5 and SuppFigure 6 add training loss curves to the same data, revealing an additional finding: Adam and its decoupled variants "converge faster (in terms of training loss) on CIFAR-10 than the corresponding SGD variants." This is consistent with Adam's well-known advantage in convergence speed. The decoupled weight decay doesn't sacrifice this advantage—AdamWR converges fast AND generalizes well, combining the strengths of both optimizer families. The figures also show that the restart variants (AdamWR, SGDWR) achieve better generalization than their non-restart counterparts (AdamW, SGDW) at the same training loss, suggesting that the warm restart procedure itself provides a generalization benefit.

Cross-Dataset and Cross-Architecture Adoption (Section 4.5)

Section 4.5 reports on adoption of AdamW by other research groups, serving as informal external validation:

  • Wang et al. (2018) used AdamW for face detection on WIDER FACE, achieving "almost 10× faster predictions than the previous state of the art algorithms while achieving comparable performance." This is an existence proof that AdamW works on non-classification tasks with different architectures.
  • Völker et al. (2018) applied AdamW with cosine annealing to EEG signal classification. In a direct comparison provided to the authors, AdamW outperformed Adam on two architectures: Deep4Net (73.68% vs. 71.37% test accuracy) and a ResNet variant (72.04% vs. 61.34%, described as "statistically significantly higher"). The large gap on ResNet (over 10 percentage points) suggests AdamW's benefit may be architecture-dependent, with ResNet-like architectures benefiting more.
  • Radford et al. (2018) used AdamW to train Transformer architectures, obtaining "new state-of-the-art results on a wide range of benchmarks for natural language understanding." This extends AdamW's applicability beyond vision to NLP and to Transformer architectures.
  • Zhang et al. (2018) compared L2 vs. weight decay for SGD, Adam, and K-FAC on CIFAR datasets with ResNet and VGG, reporting that decoupled weight decay "consistently outperformed L2 regularization in cases where they differ." This provides independent replication of the core finding with additional architectures (VGG) and optimizers (K-FAC).

These external results address a limitation of the paper's own experiments (which only test ResNet on image classification): they show that the decoupled weight decay principle transfers to different architectures, different modalities, and different tasks. However, the reported numbers come from personal communications or preprints without detailed experimental protocols, so they should be treated as suggestive rather than definitive.

Ablation Studies and Robustness Checks

Normalized vs. raw weight decay across training budgets (SuppFigure 3): The paper tests whether the optimal weight decay value depends on the number of training epochs. Without normalization, the optimal raw λ\lambda shifts substantially across budgets (100, 300, 900, 2,700 epochs): the diagonal band of best settings in SuppFigure 3 top row moves to smaller λ\lambda values as the budget increases. With the square root normalization (λ=λnormb/(BT)\lambda = \lambda_{\text{norm}} \sqrt{b/(BT)}), the optimal λnorm\lambda_{\text{norm}} remains approximately stationary across budgets (SuppFigure 3, second row), with λnorm=0.025\lambda_{\text{norm}} = 0.025 and 0.050.05 being consistently good choices. This ablation validates the normalization scheme as a practical tool for transferring hyperparameters across budgets. The cross-dataset transfer check (third and fourth rows) shows the same λnorm\lambda_{\text{norm}} values work on ImageNet32x32, where without normalization the raw λ\lambda optimal for CIFAR-10 would be "roughly 5 times too large."

Fixed learning rate Adam for 1,800 epochs (SuppFigure 2): To test whether cosine annealing is truly necessary or whether standard Adam with L2 regularization just needs more training time, the paper runs standard Adam with fixed learning rate for 1,800 epochs on the larger 26 2x96d ResNet. The results, shown on a 4×4 logarithmic grid of (α,λ′)(\alpha, \lambda') settings, are "at best comparable to the ones obtained with AdamW with 18 times less epochs and a smaller network." This suggests that additional training time cannot compensate for the poor regularization in standard Adam—the problem is not slow convergence, but fundamentally worse generalization.

Learning rate schedule comparison (Figure 1): Already discussed in the main results, but serves as an ablation: the advantage of decoupled weight decay holds across fixed, step-drop, and cosine annealing schedules, showing the benefit is not tied to a particular schedule. However, the magnitude of the benefit is schedule-dependent: it is smallest with fixed learning rate and largest with cosine annealing.

Warm restarts vs. no restarts (Figure 4 and SuppFigures 5–6): Adding warm restarts to AdamW (producing AdamWR) improves anytime performance by up to 10× (reaching in ~100 epochs what AdamW takes ~1,000 epochs to achieve) while maintaining or slightly improving final test error. This ablation shows that the warm restart technique from Loshchilov & Hutter (2016) is orthogonal to and compatible with decoupled weight decay, and that the combination yields the best overall results.

SGDW vs. SGD comparison (Figure 2, top row): For SGD, decoupled weight decay does not improve final test error (as expected from Proposition 1—the two are mathematically equivalent), but it does make the hyperparameter landscape more separable. This is a robustness check confirming that the improvement in Adam is not simply because decoupled weight decay is "stronger" regularization in some absolute sense, but specifically because it corrects the inequivalence introduced by adaptive gradient scaling. If decoupled weight decay were universally better, SGDW would outperform SGD with L2, but it doesn't—it performs equivalently. This supports the paper's diagnostic: the problem is specific to adaptive methods.

Normalized weight decay choice validation: The paper is admirably transparent about the empirical nature of the square root scaling: "our choice of normalization is merely one possibility informed by few experiments; a more lasting conclusion we draw is that using some normalization can substantially improve results" (Appendix B.1). This is not a full ablation—the paper does not compare linear, square root, logarithmic, or other scaling rules—but the qualitative finding that normalization helps is robust within the tested regime. The cross-dataset transfer (CIFAR-10 to ImageNet32x32) provides additional validation that the specific square root rule is reasonable.

Critical Assessment

Claim 1: Decoupled weight decay substantially improves Adam's generalization performance, achieving 15% relative improvement in test error.

What was tested: The paper tests this on two image classification datasets (CIFAR-10, ImageNet32x32) using one architecture family (26-layer 2-branch ResNet with Shake-Shake regularization) at two widths (2x64d for sweeps, 2x96d for long runs). The 15% figure comes from comparing the best AdamW configuration against the best standard Adam configuration in the 1,800-epoch runs with default learning rate (Figure 3, Figure 4 text).

What supports the claim: The improvement is consistent across all tested conditions: different learning rate schedules (Figure 1), different training budgets from 100 to 1,800 epochs (Figures 1–3), and different datasets (SuppFigure 4 replicates the finding on ImageNet32x32). The generalization-gap analysis (Figure 3 bottom-right, SuppFigure 4 bottom-right) shows that the improvement is not merely faster convergence—AdamW achieves lower test error at the same training loss. External replications by Zhang et al. (2018) on VGG and K-FAC, and by Völker et al. (2018) on EEG data with different architectures, provide additional evidence that the finding is not unique to the paper's specific setup.

What weakens the claim: The "15%" number is cherry-picked from the best hyperparameter settings for each optimizer. A practitioner using default settings or a less thorough hyperparameter search might see smaller improvements. The absence of error bars or multiple random seeds means we cannot assess whether a 15% relative improvement is statistically significant or within the range of training noise—the test set is large (10,000 images), but optimizer performance can vary across random initializations. The claim also applies only to ResNet architectures with Shake-Shake regularization on image classification; the paper does not test on simpler architectures (plain CNNs, fully connected networks) or different tasks (language modeling, reinforcement learning), though external adoptions partially fill this gap.

Missing experiments that would strengthen the claim: Testing on CIFAR-100 (where overfitting is more severe and regularization matters more) would show whether the improvement scales with the need for regularization. Testing without Shake-Shake regularization would isolate the optimizer's contribution from the architecture's built-in regularization. Multiple random seeds per hyperparameter setting (even just 3–5) would allow computation of confidence intervals. A comparison against RMSProp and AdaGrad with decoupled weight decay would show whether the finding generalizes beyond Adam.

Claim 2: Decoupled weight decay decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam.

What was tested: The hyperparameter landscape experiments in Figure 2, which sweep a logarithmic grid of α\alpha and λ\lambda for SGD, SGDW, Adam, and AdamW and visualize the resulting test error as heatmaps.

What supports the claim: For SGDW (top right) and AdamW (bottom right), the best hyperparameter settings align with the grid axes—the optimal λ\lambda (weight decay) is approximately constant across a wide range of α\alpha values. For standard SGD and Adam with L2 regularization, the best settings lie on a diagonal, indicating that α\alpha and λ\lambda must be changed together. The paper's operational test—"even if the learning rate is not well tuned yet (e.g., consider the value of 1/1024 in Figure 2, top right), leaving it fixed and only optimizing the weight decay factor would yield a good value"—provides a concrete criterion for "decoupled," and SGDW/AdamW pass it while standard SGD/Adam fail.

What weakens the claim: The claim is about the shape of the hyperparameter landscape, which is inherently a continuous property, but the experiments test it on a coarse discrete grid (8 values of α\alpha, 8 values of λ\lambda). A grid-aligned optimum on a coarse grid does not guarantee true independence—the underlying continuous function could still have diagonal structure that the grid misses. The paper also only tests this at one training budget (100 epochs) and one dataset (CIFAR-10). The claim that the hyperparameters are "decoupled" would be stronger if the optimal λ\lambda were invariant to α\alpha at multiple budgets and on multiple datasets.

Missing experiments: A more rigorous test of decoupling would fix λ\lambda at its optimal value for one α\alpha, then vary α\alpha across a wide range and measure whether performance degrades. The paper's heatmaps qualitatively show this (the good region for SGDW spans many rows), but a quantitative measure—e.g., the variance in test error when fixing λ\lambda and varying α\alpha—would be more convincing. Testing at multiple budgets would show whether the decoupling holds as the optimal λ\lambda changes with training duration.

Claim 3: Adam with decoupled weight decay (AdamW) can compete with SGD with momentum on image classification datasets where it was previously outperformed.

What was tested: The direct comparison comes from Figure 4, which plots test error vs. epochs for AdamW, AdamWR, SGDW, and SGDWR on CIFAR-10 and ImageNet32x32. The warm restart variants use the same schedule (T0=100T_0 = 100, Tmult=2T_{\text{mult}} = 2).

What supports the claim: On ImageNet32x32, AdamWR and SGDWR show "comparable performance"—their curves largely overlap, with the paper describing the difference as yielding "comparable performance." On CIFAR-10, SGDWR still holds a small advantage, but the gap is substantially smaller than between standard Adam and SGD. The paper states that the improvements "closed most of the gap between Adam and SGDWR on CIFAR-10." This is a more honest characterization than "AdamW matches SGD"—it closes most of the gap, not all of it.

What weakens the claim: The claim says "compete with," which is vague. On CIFAR-10, SGDWR still wins by a small margin (visible in Figure 4 left panel). On ImageNet32x32, the curves are very close but SGDWR appears slightly better at the very end of training. The paper does not report the numerical difference in final test error or assess whether it is statistically significant. "Competitive" here means "in the same ballpark," not "indistinguishable from." Additionally, the comparison uses SGD with decoupled weight decay (SGDW/SGDWR), not standard SGD with L2 regularization. This is a fair comparison (both get the same algorithmic improvement), but it means the paper is comparing best-Adam-variant against best-SGD-variant, not AdamW against the SGD baseline that achieved state-of-the-art results in Gastaldi (2017). Gastaldi's original results used standard SGD with L2 regularization and achieved 2.86% error with the same architecture; the paper's SGDWR results (Figure 4) appear to reach roughly 2.5–2.6%. So AdamWR is competitive with an improved SGD baseline, not just the old one.

Missing experiments: A direct comparison between AdamWR and the exact SGD configuration from Gastaldi (2017) would establish whether AdamWR can actually achieve state-of-the-art results or merely gets close. Testing on CIFAR-100 would provide a more challenging benchmark where the generalization gap between optimizers is typically larger. Testing on ImageNet (full resolution) would test scalability to larger datasets.

Claim 4: Optimal weight decay depends on the total number of batch passes/weight updates, and normalized weight decay addresses this.

What was tested: SuppFigure 3 sweeps λ\lambda (raw) and λnorm\lambda_{\text{norm}} (normalized) across four training budgets (100, 300, 900, 2,700 epochs) for AdamW on CIFAR-10, and also tests the transfer of λnorm\lambda_{\text{norm}} to ImageNet32x32.

What supports the claim: The optimal raw λ\lambda clearly shifts with budget—the diagonal band of good settings in SuppFigure 3 top row moves to smaller λ\lambda values as epochs increase. With normalization, the optimal λnorm\lambda_{\text{norm}} is approximately stationary (SuppFigure 3 second row), with values around 0.025–0.05 working well across all budgets. The cross-dataset transfer works: the same λnorm\lambda_{\text{norm}} values that are optimal on CIFAR-10 are also near-optimal on ImageNet32x32, while raw λ\lambda values would be off by a factor of ~5.

What weakens the claim: The square root scaling rule (λ=λnormb/(BT)\lambda = \lambda_{\text{norm}} \sqrt{b/(BT)}) is acknowledged to be empirically chosen and not theoretically motivated. The paper explicitly states this is "one possibility informed by few experiments" and that "even better scaling rules are likely to exist." The normalization was tested only for AdamW (and SGDW in the third and fourth rows of SuppFigure 3), not for standard Adam—it's unclear whether the same normalization would help standard Adam's L2 regularization or whether the budget-dependence has a different functional form there. The normalization is also only validated on two datasets (CIFAR-10 and ImageNet32x32) at specific batch sizes; the square root scaling might not generalize to very different batch sizes or dataset sizes.

Missing experiments: Comparing linear scaling (λ∝1/T\lambda \propto 1/T) against square root scaling (λ∝1/T\lambda \propto 1/\sqrt{T}) would reveal whether the specific functional form matters or whether any reasonable budget adjustment is sufficient. Testing across a wider range of total batch passes (e.g., by varying batch size while holding epochs constant) would validate that the relevant variable is indeed the total number of updates, not the number of epochs per se. Testing the normalization on standard Adam with L2 regularization would show whether the budget-dependence is specific to decoupled weight decay.

Cross-Cutting Assessment: What the Experiments Do and Don't Show

What the experiments convincingly demonstrate:

  1. L2 regularization and decoupled weight decay produce different optimization dynamics in Adam, and the decoupled version yields better test accuracy on the tested benchmarks. The evidence for this is multi-faceted (hyperparameter landscapes, learning curves, generalization-gap analysis) and consistent across conditions.
  2. The hyperparameter coupling between learning rate and regularization strength can be reduced by applying weight decay externally to the gradient computation, making hyperparameter tuning more robust. The heatmap visualizations in Figure 2 are qualitatively compelling even without statistical tests.
  3. Adam with decoupled weight decay and cosine annealing/warm restarts (AdamWR) is a practically useful optimizer that narrows the gap to SGD on image classification. The anytime performance curves in Figure 4 show clear improvements over standard Adam.

What the experiments do not demonstrate (and the paper does not claim they do):

  1. That AdamW is universally better than Adam across all tasks, architectures, and datasets. The experiments are limited to ResNet on image classification, and external adoptions are cited but not systematically evaluated.
  2. That decoupled weight decay is the only fix needed to make Adam fully competitive with SGD. AdamWR still slightly underperforms SGDWR on CIFAR-10 in Figure 4, and the paper acknowledges this.
  3. That the square root normalization rule is optimal. The paper presents it as a practical heuristic, not a theoretical result.

The most significant experimental limitation is the absence of statistical rigor: no error bars, no multiple random seeds, no significance tests. This is common in optimizer papers from this era but weakens the quantitative claims. The 15% relative improvement is a point estimate without a confidence interval. The "comparable performance" between AdamWR and SGDWR on ImageNet32x32 (Figure 4 right) could reflect either genuine parity or a small but real difference that the single-run comparison cannot resolve. The hyperparameter landscape visualizations (Figures 1, 2) treat each grid point as a precise measurement when in fact each is a single training run subject to initialization noise and batch ordering stochasticity.

A second limitation is the narrow architectural scope. While the paper cites external adoptions using Transformers (Radford et al., 2018) and custom architectures (Wang et al., 2018; Völker et al., 2018), these are not controlled experiments. The paper's own experiments use only one architecture (26-layer ResNet with Shake-Shake) at two widths. Whether AdamW's advantage over Adam is larger or smaller on other architectures (plain CNNs, VGG, Inception, DenseNet, RNNs) is not tested. The Shake-Shake regularization in particular might interact with the optimizer's regularization in ways that don't generalize—if the architecture already has strong built-in regularization, the optimizer's regularization might matter less, and vice versa.

A third limitation is that the paper never systematically varies batch size. All experiments use batch size 128. Since the normalized weight decay depends on batch size through b/(BT)b/(BT), and since batch size is known to interact with both optimal learning rates and generalization (Keskar et al., 2016), testing across batch sizes would strengthen the claim that the normalization rule is robust.

A fourth limitation is the absence of an RMSProp or AdaGrad comparison. The paper claims that "similar results also hold for other adaptive gradient methods, such as AdaGrad and AMSGrad" (Section 5), but never tests this. Given Proposition 2's generality (it applies to any optimizer with Mt≠kIM_t \neq kI), this claim is theoretically well-founded, but empirical validation would have been straightforward and would have significantly strengthened the paper's generality.

6. Limitations and Trade-offs

6.1 Single Architecture Family and Task Domain: All Experiments Use ResNet with Shake-Shake on Image Classification

The assumption or constraint. The paper's entire empirical case for decoupled weight decay rests on experiments using one architecture family—26-layer 2-branch ResNets with Shake-Shake regularization (Gastaldi, 2017)—trained exclusively on image classification benchmarks (CIFAR-10, CIFAR-100 via external replication, and ImageNet32x32). The paper acknowledges this narrowness only obliquely, stating in Section 5 that "our results obtained on image classification datasets must be verified on a wider range of tasks, especially ones where the use of regularization is expected to be important." No experiments are conducted on recurrent networks, transformers, generative models, reinforcement learning, or any task outside supervised image classification.

The consequence. The paper's central diagnostic—that L2 regularization under-regularizes parameters with large historical gradients in adaptive methods—depends on the interaction between the optimizer's per-parameter scaling and the parameter gradient distribution. This gradient distribution is shaped by both the architecture and the task. A ResNet's gradient distribution (with its skip connections, batch normalization layers, and convolutional weight sharing) may differ systematically from that of an LSTM, a Transformer, or a fully-connected network. If ResNet parameters naturally have certain gradient magnitude patterns that make the L2/weight-decay inequivalence particularly damaging, then AdamW's advantage might be smaller (or larger) on other architectures. The external adoptions cited in Section 4.5 provide some evidence of transfer—Radford et al. (2018) used AdamW successfully for Transformers on NLP tasks, and Völker et al. (2018) used it for CNNs on EEG data—but these are not controlled comparisons. The EEG results in particular show a striking 10+ percentage point gap on ResNet (72.04% vs. 61.34%) but a much smaller ~2 point gap on Deep4Net (73.68% vs. 71.37%), suggesting the benefit may indeed be architecture-dependent.

What evidence exists in the paper. The paper's own experiments (Sections 4.1–4.4, Figures 1–4, SuppFigures 2–6) test exactly one architecture at two widths (2x64d with 11.6M parameters, 2x96d with 25.6M parameters). The external adoptions in Section 4.5 are anecdotal: no learning curves, hyperparameter sweeps, or generalization-gap analyses are shown for the face detection, EEG, or NLP applications. The EEG comparison provided by Völker et al. was a direct request from the authors, not a systematic study, and the numbers are reported without error bars or replication details.

Mitigation status. The paper acknowledges the limitation in passing (Section 5) but does not attempt to address it experimentally. The Bayesian filtering justification (Section 3) provides theoretical reason to believe the decoupling should be beneficial across architectures—it argues that weight decay naturally emerges from the state transition prior independent of the specific gradient distribution—but this theory assumes diagonal Gaussian approximations that may not hold equally well for all architectures. The paper treats external adoptions as sufficient evidence of generality, but these adoptions are self-selected (researchers only report results when AdamW works well) and are not rigorous comparisons. A controlled study across 3–4 diverse architectures (e.g., ResNet, VGG, LSTM, Transformer) on a common task would be needed to establish whether the benefit is universal or architecture-specific.

6.2 No Statistical Rigor: Single Training Runs, No Error Bars, No Significance Tests

The assumption or constraint. Every experimental result in the paper—every heatmap cell in Figures 1 and 2, every learning curve in Figures 3 and 4, every point in SuppFigures 2–6—is the outcome of a single training run. The paper does not report error bars, confidence intervals, standard deviations across random seeds, or statistical significance tests for any comparison. This is not an oversight the authors discuss; the concept of statistical uncertainty in optimizer comparisons is simply absent from the paper's methodology.

The consequence. The paper's quantitative claims—"15% relative improvement in test error" (Section 4.4), "more separable hyperparameter search space" (Section 4.2), "AdamWR... yielded comparable performance on ImageNet32x32" (Section 4.4)—are point estimates without any measure of reliability. Neural network training is stochastic: different random initializations, different batch orderings, and different data augmentation draws produce different final test errors even with identical hyperparameters. Without replicate runs, it is impossible to determine whether the observed differences between optimizers exceed the run-to-run variance of each optimizer.

This matters particularly for the long-run comparisons in Figures 3 and 4. The paper's claim that AdamWR is "comparable" to SGDWR on ImageNet32x32 (Figure 4, right panel) rests on overlapping learning curves from single runs. If the run-to-run standard deviation of test error at epoch 1800 is, say, 0.1–0.2 percentage points (which is plausible for CIFAR-scale datasets), then the visual overlap might reflect noise rather than genuine parity. Conversely, the apparently "clearly worse" performance of standard Adam relative to AdamW in Figure 3 might partially reflect an unlucky initialization for the Adam runs. The hyperparameter landscape heatmaps (Figures 1 and 2) are particularly vulnerable: each cell is a single run, so the smooth "basins" of good performance might be partly noise artifacts. A cell that appears optimal could be a lucky run, and an adjacent cell that appears worse could be an unlucky run—the landscape's shape might differ substantially if each cell were averaged over 3–5 seeds.

What evidence exists in the paper. None. The paper contains no mention of random seeds, no variance estimates, no replicate runs, and no statistical tests. The test set sizes (10,000 images for CIFAR-10, 50,000 for ImageNet32x32 validation) are large enough that sampling error in test-set accuracy is small, so differences of, say, 0.5% in test error likely represent genuine differences in model quality. But the single-run design means we cannot separate model quality differences from training-run noise. The "top-10 hyperparameter settings" marked by black circles in Figure 2 are the 10 single runs with lowest test error; their apparent clustering along axes (or diagonals) might change if runs were repeated and averaged.

Mitigation status. The paper does not acknowledge this limitation, propose any remedy, or discuss why single runs were considered sufficient. This practice was common in optimizer papers of this era (circa 2018–2019), particularly those focused on algorithm design rather than benchmark-chasing, but it weakens the strength of the quantitative claims. The most straightforward fix—running 3–5 seeds per configuration and reporting means with error bars—would have added computational cost proportional to the number of seeds (3–5× the GPU hours) but is well within the budget of a paper that already runs 1,800-epoch experiments on a 25.6M-parameter model.

6.3 No Direct Combination with Other Adam Improvements: Decoupled Weight Decay Is Tested in Isolation

The assumption or constraint. The paper tests decoupled weight decay as a standalone modification to Adam, keeping all other Adam hyperparameters at their original defaults (β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ϵ=10−8\epsilon = 10^{-8}, α=0.001\alpha = 0.001 for most runs). It does not experiment with combining decoupled weight decay with other proposed improvements to Adam, such as AMSGrad (Reddi et al., 2018), which addresses convergence issues in Adam by modifying the second-moment estimate, or normalized direction-preserving Adam (Zhang et al., 2017). The paper mentions these methods in passing (Section 5: "It would be interesting to integrate our findings on weight decay into other methods which attempt to improve Adam") but does not test any combination.

The consequence. The paper demonstrates that decoupled weight decay improves Adam from a baseline that is clearly inferior to SGD to one that is "competitive" or "comparable." But the claim that "practitioners do not need to switch between Adam and SGD anymore" (Section 1) is only supported if AdamW represents the best possible version of Adam. If combining decoupled weight decay with AMSGrad or another Adam variant yields even better performance—potentially surpassing SGDWR on CIFAR-10, where a small gap remains in Figure 4—then the paper's conclusion that AdamW, specifically, is the solution is premature. Conversely, if the other Adam improvements provide no additional benefit when decoupled weight decay is used (because the regularization fix was the dominant problem), then the paper's hypothesis—that poor regularization, not inherent flaws in adaptivity, was the root cause of Adam's generalization gap—would be strengthened. The paper leaves this question unanswered.

A particular gap is the interaction between decoupled weight decay and the ϵ\epsilon hyperparameter. In standard Adam with L2 regularization, ϵ\epsilon appears in the denominator of the adaptive scaling and thus modulates both the gradient step and the effective L2 penalty. In AdamW, ϵ\epsilon only affects the gradient step (the weight decay is applied outside). Changing ϵ\epsilon might have different effects on AdamW vs. Adam, and the optimal ϵ\epsilon might differ. The paper's decision to keep ϵ=10−8\epsilon = 10^{-8} (the default) for all AdamW experiments means this interaction is unexplored.

What evidence exists in the paper. None. The paper conducts no experiments combining decoupled weight decay with any other Adam variant or hyperparameter modification. The external adoption by Radford et al. (2018) used AdamW for Transformers, but this was not a controlled comparison against Adam+AMSGrad or other combinations. Zhang et al. (2018) compared decoupled weight decay with L2 regularization for K-FAC in addition to Adam, but this is a different optimizer, not a combination of improvements within Adam.

Mitigation status. The paper explicitly suggests this as future work (Section 5): "It would be interesting to integrate our findings on weight decay into other methods which attempt to improve Adam, e.g., normalized direction-preserving Adam (Zhang et al., 2017)." This is an honest acknowledgement but does not mitigate the limitation for the claims made in the current paper. A practitioner reading this paper in 2019 who already uses AMSGrad would not know whether switching to AdamW alone is sufficient or whether they should use AMSGrad+decoupled weight decay. The most direct experiment—AdamW vs. AdamW+AMSGrad on CIFAR-10—would require minimal additional implementation effort and would clarify whether the regularization fix subsumes or is orthogonal to the convergence fix.

6.4 Normalized Weight Decay Is Empirically Chosen and Not Theoretically Justified

The assumption or constraint. The paper's normalized weight decay formulation—λ=λnormb/(BT)\lambda = \lambda_{\text{norm}} \sqrt{b/(BT)}—is presented as a practical heuristic rather than a principled result. The authors are explicit about this in Appendix B.1: "our choice of normalization is merely one possibility informed by few experiments; a more lasting conclusion we draw is that using some normalization can substantially improve results." The square root scaling was chosen because it worked well on CIFAR-10 and transferred reasonably to ImageNet32x32, but no theoretical derivation or systematic comparison against alternative scaling rules (linear, logarithmic, constant, etc.) is provided.

The consequence. A practitioner using AdamW on a new dataset or with a substantially different training budget cannot be confident that the square root rule will produce the correct scaling. The normalization was tested across epoch counts from 100 to 2,700 on CIFAR-10 (a range of 27× in training steps) and transferred to ImageNet32x32 (where the total batch passes are ~24× larger). Outside this range—e.g., very short fine-tuning runs of 5–10 epochs, or very large-scale training with millions of batch passes—the square root rule might over- or under-correct. The paper provides no evidence about whether the square root, linear, or some other functional form is correct in general, or whether the correct form depends on architecture, dataset size, or batch size.

Furthermore, the normalized weight decay interacts with the warm restart procedure in a specific way: within each restart period ii, TT in the normalization formula is set to TiT_i (the length of the current restart), not the total training budget. This means the effective weight decay rate jumps at each restart (when TiT_i doubles, λ\lambda decreases by a factor of 1/21/\sqrt{2}). The paper never ablates this choice—would using the total budget TtotalT_{\text{total}} for all restarts work better? Would a constant λ\lambda throughout all restarts work just as well? The normalization rule for warm restarts is a design choice layered on top of an empirically-chosen base rule, with no systematic evaluation of alternatives.

What evidence exists in the paper. SuppFigure 3 is the sole evidence for the normalization rule. It shows that optimal raw λ\lambda values shift with budget (top row), that normalized λnorm\lambda_{\text{norm}} values are approximately stationary across budgets (second row), and that the same normalized values transfer from CIFAR-10 to ImageNet32x32 for both AdamW and SGDW (third and fourth rows). The experiment sweeps λnorm\lambda_{\text{norm}} over a grid and shows that values around 0.025 and 0.05 are consistently good. However, this is a validation on two datasets, not a derivation or a comparison of alternative scaling rules. The paper does not, for example, test whether λ=λnorm⋅b/(BT)\lambda = \lambda_{\text{norm}} \cdot b/(BT) (linear scaling) would have produced equally stationary optimal λnorm\lambda_{\text{norm}} values—the claim that square root is "better" than linear is untested.

Mitigation status. The paper is transparent that the normalization is a heuristic ("one possibility informed by few experiments") and that "even better scaling rules are likely to exist." This honesty is commendable and prevents practitioners from treating the square root rule as canonical. However, it also means that the normalized weight decay component of AdamW is not a reliable tool for hyperparameter transfer outside the tested regime. A practitioner using AdamW on a new task should treat λnorm\lambda_{\text{norm}} as a hyperparameter to be tuned, not as a fixed value that transfers from the paper's experiments. The paper does not provide guidance on how to choose λnorm\lambda_{\text{norm}} for new tasks beyond "try values around 0.025–0.05"—advice that may not generalize beyond CIFAR-scale image classification.

6.5 The Bayesian Filtering Justification Is Post-Hoc and Not Empirically Validated

The assumption or constraint. Section 3 of the paper presents Aitchison's (2018) Bayesian filtering interpretation as theoretical justification for why decoupled weight decay is preferable to L2 regularization. The key claim is that weight decay naturally emerges as the state transition prior A=λIA = \lambda I in the Gaussian model P(θt+1∣θt)=N((I−A)θt,Q)P(\theta_{t+1} | \theta_t) = \mathcal{N}((I - A)\theta_t, Q), and that this prior should not depend on per-parameter uncertainty (which L2 regularization implicitly does through the preconditioner). The paper presents this as a satisfying theoretical grounding for the empirical results. However, this theory was developed independently by Aitchison after the paper's preprint appeared, and the paper does not test any of its specific predictions empirically.

The consequence. The Bayesian filtering interpretation is suggestive but not validated. The theory makes specific assumptions—Gaussian state transitions, approximate conjugate likelihoods, diagonal posterior covariance approximations—that may not hold in practice. If these assumptions are violated, the theoretical argument for weight decay over L2 regularization might not apply, even if the empirical benefit remains. Conversely, the theory might predict additional modifications to Adam (e.g., to the momentum term, the ϵ\epsilon parameter, or the form of the second-moment estimate) that would further improve performance, but these are not explored.

The theory also makes a strong claim about the direction of the L2/weight-decay discrepancy: weight decay regularizes parameters with large historical gradients more than L2 does, and this is the "correct" behavior because the regularizer should not depend on parameter uncertainty. But the paper never directly tests whether this mechanism—differential regularization of large-gradient vs. small-gradient parameters—is actually what drives AdamW's improvement. The improvement could be due to other effects of the decoupling (e.g., the weight decay term no longer contributing to the adaptive moment estimates mtm_t and vtv_t, which changes the optimizer's dynamics in ways unrelated to per-parameter regularization strength). The paper provides no ablation that isolates the per-parameter regularization effect from other consequences of the decoupling.

What evidence exists in the paper. The Bayesian filtering section (Section 3) is purely theoretical and cites Aitchison (2018) as its source. The paper does not claim to have developed this theory; it "summarizes it here to shed some light on why weight decay may be favored over L2 regularization." There are no experiments designed to test predictions of the Bayesian filtering model—no measurement of whether AdamW actually produces different per-parameter regularization patterns than Adam in a way consistent with the theory, no comparison of the Gaussian transition model's predictions against observed parameter trajectories, and no test of whether alternative transition priors (e.g., non-diagonal AA) would work better.

Mitigation status. The paper is appropriately modest about the role of this theory, noting that "full credit for this theory goes to Aitchison" and that the theory "gives us a theoretical framework in which we can understand the superiority of this weight decay over L2 regularization"—framing it as explanation rather than proof. The limitation is not that the theory is wrong, but that the paper presents it as a supporting argument without testing its empirical implications. The theory's value to a practitioner deciding whether to use AdamW is limited: AdamW works empirically regardless of whether the Bayesian filtering interpretation is correct. The theory's value is primarily for researchers developing new adaptive methods who want a principled framework for where to place regularization.

6.6 The Weight Decay Factor λ Is Not Truly Independent of the Learning Rate α in Practice

The assumption or constraint. A central claim of the paper is that decoupled weight decay "decouples the optimal choice of weight decay factor from the setting of the learning rate" (abstract, Section 4.2). The evidence for this is Figure 2, which shows that the best hyperparameter settings for SGDW and AdamW align with the axes of the (α,λ)(\alpha, \lambda) grid rather than forming a diagonal. However, the paper's definition of "decoupled" is based on visual inspection of a coarse heatmap from 100-epoch runs, not on a quantitative independence metric, and the decoupling is not tested across different training budgets, architectures, or datasets.

The consequence. The claim of independence between α\alpha and λ\lambda is likely an overstatement. Even in the decoupled formulation, α\alpha and λ\lambda interact through the optimization dynamics: a larger learning rate α\alpha produces larger parameter updates, which changes the parameter values that the weight decay term −λθt−1-\lambda\theta_{t-1} operates on. If α\alpha is very large, the parameters may oscillate or diverge, and the weight decay term's effect will differ from the case where α\alpha is small and updates are smooth. The paper's own Figure 2 (bottom right) shows that AdamW's optimal λ\lambda is not completely independent of α\alpha: at very small learning rates (α=1/1024\alpha = 1/1024), the optimal λ\lambda shifts to larger values, and at very large learning rates (α=1/2\alpha = 1/2), performance degrades regardless of λ\lambda. The heatmap shows a broad plateau rather than perfect axis-alignment, which is a reduction in coupling, not its elimination.

The coupling may also re-emerge at different training budgets. The paper tests decoupling only at 100 epochs. At 1,800 epochs (Figure 3), the learning rate is fixed at α=0.001\alpha = 0.001 for all runs, so the α\alpha–λ\lambda interaction is not tested at long budgets. If a practitioner changes α\alpha for a long run (e.g., using α=0.0001\alpha = 0.0001 instead of 0.0010.001), the optimal λ\lambda might shift in ways not captured by the 100-epoch heatmaps.

What evidence exists in the paper. Figure 2 is the sole evidence for the decoupling claim. The visual argument—best settings align with axes rather than diagonals—is qualitatively convincing but not quantitative. The paper does not compute a correlation coefficient, mutual information, or any other independence metric between α\alpha and the optimal λ\lambda. It does not test whether the axis-alignment holds at 300, 900, or 1,800 epochs. It does not test whether the degree of decoupling differs between architectures or datasets.

Mitigation status. The paper's language is careful: it says the proposed modification "renders the optimal settings of the learning rate and the weight decay factor much more independent" (abstract), not "completely independent." Figure 2 supports "more independent" as a qualitative claim. However, the practical implication—that practitioners can tune α\alpha and λ\lambda sequentially rather than jointly—is only partially supported. Even with the reduced coupling, a practitioner who changes α\alpha substantially (e.g., from 0.001 to 0.01) would be wise to re-tune λ\lambda, as the optimal value may shift. The paper does not provide guidance on how much α\alpha can change before λ\lambda needs re-tuning, or a quantitative measure of the residual coupling. The Bayesian filtering interpretation (Section 3) also suggests that α\alpha and λ\lambda should remain coupled through the effective step size in the posterior update, but the paper does not explore this theoretical coupling or compare it against the empirical decoupling observed in Figure 2.

7. Implications and Future Directions

How This Work Changes the Landscape

A diagnostic reframing rather than a paradigm shift. This paper does not introduce a fundamentally new optimizer or a novel regularization technique. Its contribution is more precise and, in some ways, more impactful: it identifies a mathematical error in the standard implementation of the most widely-used adaptive optimizer, traces that error to a silent assumption (that L2 regularization and weight decay are always equivalent) inherited from plain SGD without re-examination, and provides a one-line algorithmic fix. The conceptual shift is from "Adam generalizes worse than SGD for unknown reasons related to adaptivity" to "Adam was being regularized incorrectly; fix the regularization, and the generalization gap largely closes."

This reframing matters because it redirects the research community's attention away from designing new optimizers toward auditing the assumptions embedded in existing ones. Before this paper, the dominant hypothesis for Adam's generalization gap was that adaptive gradient scaling itself was somehow harmful—that per-parameter learning rates inherently led to worse solutions. Wilson et al. (2017) had made this argument forcefully, and it motivated a wave of new optimizer proposals (AMSGrad, Padam, AdamNC, QHAdam) that modified the adaptivity mechanism. This paper demonstrates that a substantial fraction of the gap can be eliminated without touching the adaptivity at all—simply by applying the regularization outside the gradient computation. This suggests that the adaptivity itself was not the primary culprit; the real problem was a regularization implementation bug that happened to be invisible in SGD but became consequential in Adam.

A concrete resolution of a persistent contradiction. Before this work, the literature contained a genuine puzzle: Adam consistently outperformed SGD on some tasks (language modeling, GAN training, reinforcement learning) while consistently underperforming on others (image classification with standard architectures). The paper's diagnosis offers a unified explanation: on tasks where L2 regularization is minimally important (or where architectures have strong built-in regularization through batch normalization, dropout, or data augmentation), the L2/weight-decay inequivalence doesn't hurt because regularization isn't doing much work in the first place. On tasks like CIFAR-10 image classification with ResNets—where weight decay is known to be crucial for achieving state-of-the-art results—the L2 formulation silently sabotages Adam's performance by under-regularizing the most important parameters. This explains why Adam could be simultaneously excellent (on tasks insensitive to regularization) and disappointing (on tasks where regularization matters), resolving the apparent contradiction in prior findings.

Reconciliation of the Wilson et al. (2017) results. The Wilson et al. marginal-value paper had been particularly influential in establishing the narrative that adaptive methods were fundamentally flawed. The current paper does not refute Wilson et al.'s empirical results—Adam really did perform worse than SGD on the tested benchmarks—but it provides an alternative interpretation: the problem was not the adaptive gradient mechanism but the mismatch between the regularization strategy and the optimizer. AdamW narrows the gap to SGD to the point of near-parity (Figure 4, right panel, ImageNet32x32) or small remaining difference (Figure 4, left panel, CIFAR-10). This suggests that the "inherent problems of adaptive gradient methods" that Wilson et al. hypothesized may not be inherent at all—they may be artifacts of a specific implementation choice that can be fixed without changing the core adaptive algorithm.

What becomes more attractive and what becomes less so. After this paper:

  • More attractive: Research on regularization-auditing for existing optimizers—systematically checking whether standard practices (L2 regularization, gradient clipping, learning rate warmup) interact correctly with adaptive gradient mechanisms. The paper demonstrates that a seemingly innocuous equivalence can fail silently when an adaptive preconditioner is introduced, and there may be other such failures waiting to be found. Research on combining AdamW with other Adam improvements (AMSGrad, Padam, etc.) becomes a natural next step: if the regularization fix and the convergence fix address orthogonal problems, the combination should outperform either alone. Work on learning rate schedules for adaptive methods also becomes more attractive, since the paper shows that decoupled weight decay + cosine annealing produces substantially better results than fixed learning rate AdamW (Figure 1).

  • Less attractive: Research premised on the idea that adaptive gradient methods are inherently inferior to SGD for generalization. The paper does not prove this premise is always wrong—SGDWR still slightly outperforms AdamWR on CIFAR-10 (Figure 4, left), and the hardest problems in the Wilson et al. suite may still show gaps—but it substantially weakens the case. New papers claiming that adaptivity causes poor generalization now bear the burden of showing that the effect persists after decoupled weight decay is applied. Similarly, research proposing entirely new optimizer families to replace Adam must now compare against AdamW, not just standard Adam—a stronger baseline that many earlier proposals might not clear.

A practical shift in how the community implements regularization. Perhaps the most concrete landscape change is the rapid community adoption documented in Section 4.5 and the acknowledgments. Within months of the preprint's appearance, decoupled weight decay was implemented in PyTorch (via a pull request from multiple contributors), TensorFlow (as a DecoupledWeightDecayExtension), Keras, and Caffe. The AdamW optimizer became a standard option in all major deep learning frameworks, and many practitioners switched to it as their default without waiting for further validation. This is a rare case where a single paper's diagnostic finding directly and quickly changed the default behavior of the entire software ecosystem—arguably more impactful than a new optimizer that requires users to change their workflow.

The paper's main limitation as a landscape-shifting contribution: the experimental scope is narrow (one architecture family, two datasets, image classification only). The paper does not prove that decoupled weight decay is the only fix needed, or that it closes the gap on all tasks where Adam previously underperformed. The remaining gap on CIFAR-10 (Figure 4, left) suggests there may be additional issues. But as a diagnostic, the paper succeeds: it identifies one major, previously-overlooked problem, provides a clean fix, and demonstrates that fixing it produces large improvements on the canonical benchmarks where the problem was most visible. This is a model of what diagnostic ML research should look like.

Follow-Up Research This Work Enables

Systematic audit of optimizer-regularizer interactions across the adaptive optimizer family. This paper proves the L2/weight-decay inequivalence for any optimizer with a non-trivial preconditioner Mt≠kIM_t \neq kI (Proposition 2), which includes AdaGrad, RMSProp, AdaDelta, AMSGrad, and Nadam in addition to Adam. Yet the paper tests only Adam. A natural follow-up would replicate the CIFAR-10 and ImageNet32x2 experiments with RMSProp+decoupled weight decay, AdaGrad+decoupled weight decay, and AMSGrad+decoupled weight decay, comparing each against its standard L2-regularized counterpart. The experiment would answer: is the AdamW improvement specific to Adam's particular form of the preconditioner (1/v^t1/\sqrt{\hat{v}_t}), or does it generalize to the entire family as Proposition 2 predicts? The experiment would also reveal whether some adaptive methods benefit more than others—for instance, RMSProp's lack of bias correction and different momentum handling might make the L2/weight-decay discrepancy larger or smaller than in Adam. A strong study would use the identical architecture (26 2x64d ResNet), identical hyperparameter sweep design, and identical training budgets as the current paper, producing directly comparable heatmaps.

Testing whether the L2/weight-decay inequivalence is the dominant cause of Adam's generalization gap, or one of several causes. The paper shows that AdamW substantially narrows the gap to SGD on CIFAR-10, but a small gap remains (Figure 4, left panel: SGDWR is visibly below AdamWR throughout training). This remaining gap could be due to: (a) suboptimal tuning of other Adam hyperparameters (β1\beta_1, β2\beta_2, ϵ\epsilon) for the decoupled regime, (b) a genuine generalization disadvantage of adaptive gradient scaling that persists even with correct regularization, or (c) an interaction between the Shake-Shake regularization and the optimizer that favors SGD. A follow-up could distinguish these by (1) running a full hyperparameter sweep of AdamWR over β1\beta_1, β2\beta_2, and ϵ\epsilon in addition to α\alpha and λnorm\lambda_{\text{norm}} to see if retuning closes the remaining gap, and (2) testing on a simpler architecture without Shake-Shake (e.g., a plain 20-layer ResNet or a VGG variant) where optimizer differences should be more purely attributable to the optimization algorithm rather than architecture-optimizer interactions. If the gap closes fully with hyperparameter retuning, the "inherent adaptivity problem" hypothesis is dead. If the gap persists even with optimal tuning and simple architectures, there may be a second, subtler issue in adaptive methods that the community has not yet diagnosed.

Characterizing the per-parameter regularization profile of Adam vs. AdamW to directly test Proposition 3's mechanism. Proposition 3 provides a theoretical model for how AdamW differs from Adam: weight decay is equivalent to a scale-adjusted L2 penalty where each parameter θi\theta_i is regularized proportionally to si\sqrt{s_i}, with sis_i being the inverse of the Adam preconditioner entry for that parameter. Parameters with historically large gradients get more regularization under AdamW than under standard Adam. This mechanism is the paper's core explanatory story, but it is never directly tested. A follow-up could instrument the training process to measure, at each layer and each epoch, the distribution of effective per-parameter regularization strength—defined as the magnitude of the weight decay term relative to the parameter magnitude—for both Adam and AdamW. The prediction is that for AdamW, this effective regularization is relatively uniform across parameters with different gradient histories, while for standard Adam, it is strongly anti-correlated with historical gradient magnitude. If the data show this pattern, Proposition 3's mechanism is confirmed as the causal driver of improved generalization. If the pattern is absent or reversed, then the improvement must come from a different source (e.g., the weight decay term no longer contaminating the second-moment estimate vtv_t, which changes the adaptive learning rate trajectory). This would be a decisive experiment for the paper's theoretical narrative.

Systematic evaluation of normalized weight decay scaling rules. The paper's square root normalization (λ=λnormb/(BT)\lambda = \lambda_{\text{norm}} \sqrt{b/(BT)}) is explicitly acknowledged as an empirically-chosen heuristic. A follow-up could systematically compare scaling rules—constant λ\lambda, linear (λ∝1/T\lambda \propto 1/T), square root (λ∝1/T\lambda \propto 1/\sqrt{T}), and logarithmic (λ∝1/log⁡T\lambda \propto 1/\log T)—across a broad range of training budgets spanning three orders of magnitude (e.g., 10 to 10,000 epochs) on CIFAR-10, CIFAR-100, and ImageNet32x32. For each scaling rule and each budget, the optimal λnorm\lambda_{\text{norm}} would be determined by grid search. A good scaling rule would produce a stationary optimal λnorm\lambda_{\text{norm}} across all budgets (as the square root rule approximately does in SuppFigure 3 for the limited range tested). The experiment would answer whether the square root rule is genuinely close to optimal or merely adequate for the narrow range in the original paper, and whether the optimal scaling rule itself depends on dataset size, architecture, or batch size. This is a straightforward hyperparameter study that would directly improve the practical utility of AdamW by providing a more reliable transfer rule.

Combining decoupled weight decay with other Adam improvements to test whether the fixes are additive. The paper mentions AMSGrad (Reddi et al., 2018) and normalized direction-preserving Adam (Zhang et al., 2017) as candidate combination partners, but never tests them. A follow-up would implement AdamW+AMSGrad (modifying the second-moment estimate to be non-decreasing while keeping weight decay decoupled), AdamW with normalized direction preservation, and potentially other combinations (e.g., with learning rate warmup, with different β2\beta_2 values). The experiment would train all combinations on CIFAR-10 and CIFAR-100 under the same protocol as the original paper. The key question: do the improvements from decoupled weight decay and from, say, AMSGrad add linearly, super-linearly, or sub-linearly? If they add linearly or super-linearly, the combination could finally close the remaining gap to SGDWR on CIFAR-10. If they add sub-linearly or not at all, this suggests that the fixes address overlapping problems and that the regularization fix was the dominant one—which would strengthen the paper's implicit claim that L2 regularization was the primary cause of Adam's poor generalization.

Testing AdamW on tasks where Adam already matches or outperforms SGD, to check for regressions. The paper's experiments are all on image classification, where Adam was known to underperform. But Adam remains the default optimizer for many other domains—NLP (especially Transformer training), GANs, variational autoencoders, and reinforcement learning—where it often matches or exceeds SGD. A critical stress-test for AdamW is whether decoupled weight decay hurts performance on any of these tasks. If Adam performs well on these tasks partly because the L2 formulation under-regularizes certain parameters (e.g., in language models where frequent words produce large gradient magnitudes and should perhaps be regularized less), switching to AdamW might degrade performance. A follow-up would replicate standard Adam benchmarks on, say, Transformer training for machine translation (IWSLT or WMT), language modeling (WikiText-2 or Penn Treebank), and GAN training (CIFAR-10 or CelebA), comparing AdamW against standard Adam at equivalent hyperparameter settings. Finding tasks where AdamW underperforms would be as scientifically valuable as finding tasks where it improves—it would delineate the boundary conditions of the L2/weight-decay inequivalence's practical importance and provide a more complete picture than the paper's positive-results-only presentation.

Practical Applications and Downstream Use Cases

Default optimizer selection for image classification and similar regularization-sensitive tasks. The most direct practical implication of this paper is that practitioners training deep networks on tasks where weight decay matters (image classification, object detection, semantic segmentation) should replace Adam with AdamW in their training scripts. The improvement is "free" in the sense that it requires no additional computation (one extra vector subtraction per iteration, which is negligible relative to the forward-backward pass), no change to architecture or data pipeline, and only minimal hyperparameter retuning (the same α\alpha and β\beta values work; only λ\lambda needs adjusting). For a team currently using Adam on a CIFAR-like task, switching to AdamW with λ\lambda tuned on a short exploratory run (as supported by the normalized weight decay's budget-transfer property in SuppFigure 3) should yield a roughly 15% relative reduction in test error (the paper's headline number) without increasing training time. Given that AdamW is now implemented in all major frameworks, the switching cost is essentially zero—making this one of the highest-ROI changes a practitioner can make, on par with adopting batch normalization or learning rate scheduling.

Hyperparameter optimization workflows with reduced coupling. For teams that run automated hyperparameter optimization (grid search, random search, Bayesian optimization) over learning rate and weight decay, the decoupling demonstrated in Figure 2 has a practical consequence: the search can be sequential rather than joint. Because the optimal λ\lambda does not depend strongly on α\alpha, a practitioner can first tune α\alpha with λ\lambda fixed at a reasonable default (e.g., λnorm=0.05\lambda_{\text{norm}} = 0.05), then tune λ\lambda with α\alpha fixed at its optimum. This reduces the search space from a 2D grid to two 1D sweeps, cutting the number of training runs from, say, 64 (8×8) to 16 (8+8). For expensive training runs on large datasets, this 4× reduction in tuning experiments translates directly to reduced compute cost and faster iteration. The paper does not quantify this cost reduction, but it is a direct consequence of the axis-aligned hyperparameter landscape that Figure 2 demonstrates. The normalized weight decay formulation further reduces tuning cost by allowing λnorm\lambda_{\text{norm}} values to transfer across training budgets—a practitioner can tune on a short 100-epoch run and use the same λnorm\lambda_{\text{norm}} for a full 1,800-epoch production run, avoiding expensive long-tuning experiments.

Training with learning rate schedules on Adam without silent regularization decay. Before this paper, a practitioner using Adam with a cosine annealing learning rate schedule would unknowingly be annealing their L2 regularization strength along with the learning rate—because the effective L2 penalty is scaled by ηtα\eta_t \alpha, and ηt→0\eta_t \to 0 at the end of cosine annealing. The model would be effectively unregularized during the final, critical fine-tuning phase of training. With decoupled weight decay, the weight decay strength remains independent of ηt\eta_t, so the model continues to receive regularization even as the learning rate approaches zero. This is particularly important for cosine annealing schedules (which the paper shows are the best-performing schedule, Figure 1 right column), where the learning rate spends many epochs at very small values. A practitioner adopting AdamW + cosine annealing gets both the anytime-performance benefits of the schedule and the consistent regularization of decoupled weight decay, without the silent interaction that plagued standard Adam. The paper's AdamWR results (Figure 4) show this combination yields the best overall performance, and practitioners can adopt it by simply combining two standard techniques (AdamW optimizer + cosine annealing scheduler) that are now both available in framework APIs.

Budget-aware hyperparameter transfer for teams iterating on model design. A common workflow in applied deep learning is to prototype model architectures and hyperparameters on a small, fast-training dataset (or on a subset of the full data) before scaling to the full dataset for final training. The normalized weight decay formulation (λ=λnormb/(BT)\lambda = \lambda_{\text{norm}} \sqrt{b/(BT)}) enables this workflow for the weight decay hyperparameter: a λnorm\lambda_{\text{norm}} value found to work well on a 100-epoch CIFAR-10 run (where BT/bBT/b is small) can be plugged into the normalization formula to compute the appropriate λ\lambda for a full 1,800-epoch run or for an ImageNet-scale run with millions of batch passes. Without normalization, the λ\lambda optimized on the small run would be far too large for the long run, producing over-regularization and poor results. The paper demonstrates this transfer between CIFAR-10 and ImageNet32x2 (SuppFigure 3, third and fourth rows), where without normalization the optimal raw λ\lambda values differ by a factor of ~5. For teams training on proprietary datasets where hyperparameter tuning is expensive, this transferability directly reduces the number of expensive tuning runs needed when scaling up.

When to Prefer This Method

The paper positions decoupled weight decay as a replacement for L2 regularization in Adam, not as an alternative to SGD with momentum. The decision rule it implies is:

  • Prefer AdamW/AdamWR over standard Adam whenever you are currently using Adam with any non-zero weight decay/L2 regularization. There is no scenario where standard Adam+L2 is preferable: the two are identical at λ=0\lambda = 0, and AdamW strictly dominates Adam for λ>0\lambda > 0 on the tested benchmarks (Figure 2, bottom rows; Figure 3; SuppFigure 4). The fix has zero computational cost and only requires re-tuning λ\lambda.

  • Prefer AdamW/AdamWR over SGD with momentum when you value Adam's faster initial convergence (visible in SuppFigures 5–6, where Adam variants reach lower training loss faster than SGD variants) and are willing to accept a small remaining generalization gap on some tasks (Figure 4, left: SGDWR still slightly outperforms AdamWR on CIFAR-10). On ImageNet32x32, the choice is essentially neutral—AdamWR and SGDWR perform comparably.

  • Prefer SGDW/SGDWR over AdamW/AdamWR when you are operating in a regime where the generalization gap, even if small, is critical (e.g., pushing state-of-the-art on a competitive benchmark) and the faster convergence of Adam is not a bottleneck. The paper's own results (Figure 4 left) show SGDWR retains a marginal edge on CIFAR-10.

The paper does not provide evidence to make optimizer recommendations outside the image classification domain it tested. Practitioners using Adam on NLP, GANs, or RL should test AdamW against their current Adam baseline before switching, since the paper's experiments do not cover these domains. The paper also does not compare AdamW against more recent Adam variants (AMSGrad, Padam, etc.), so practitioners already using those variants cannot determine from this paper alone whether AdamW is preferable—the combination experiments remain to be done.