ArXiv: 1412.6980
🎯 Pitch
Adam shatters the need for per-parameter learning rate tuning by simply tracking running averages of gradients and their squares—and then fixing the zero-start bias. It marries the sparsity handling of AdaGrad with RMSProp's agility on shifting data, hitting a sweet spot where one default setup conquers logistic regression, deep nets, and CNNs alike.
1. Executive Summary
This paper introduces Adam, an algorithm for first-order gradient-based stochastic optimization that computes adaptive learning rates for each parameter from estimates of the first and second moments of the gradients — the mean (momentum) and the uncentered variance (RMS-like scaling), respectively. Evaluated on logistic regression, multi-layer neural networks, and convolutional neural networks using MNIST, IMDB, and CIFAR-10, Adam combines the sparse-gradient robustness of AdaGrad with the non-stationary handling of RMSProp, while introducing a bias-correction mechanism that counteracts the zero-initialization of the moment estimates (dividing by at each timestep). Empirically, Adam converges as fast as AdaGrad on sparse features, outperforms RMSProp and AdaGrad on multi-layer networks with dropout, and shows marginal improvement over SGD with momentum on CNNs, with the paper's theoretical analysis establishing an regret bound comparable to the best known results for online convex programming — though the bound requires decaying toward zero over time.
2. Context and Motivation
The Core Problem: Stochastic Optimization Requires Hand-Tuned Hyperparameters Across Heterogeneous Features
The fundamental problem this paper addresses is that first-order stochastic gradient descent, while computationally efficient, performs poorly when gradients vary systematically across parameters — and the existing solutions to this problem each suffer from their own limitations that make them unsuitable for the full range of practical machine learning workloads.
To understand what makes this hard, consider what happens in standard SGD. The update rule applies the same learning rate to every parameter. This is problematic because different parameters in a neural network typically have vastly different gradient characteristics. A weight in the first convolutional layer might receive small, dense gradients, while a weight in a rarely-activated unit might receive large, sparse gradients. Using the same learning rate for both forces a tradeoff: set large enough to make progress on parameters with small gradients, and the parameters with large gradients will oscillate wildly; set small enough to control the large-gradient parameters, and the small-gradient parameters will barely move.
This heterogeneity is not a corner case — it is the default in deep learning. Convolutional neural networks exhibit different gradient magnitudes across layers due to weight sharing (as the paper shows in Section 6.3). Models trained with dropout (Hinton et al., 2012b) have stochastic gradient patterns where different hidden units see different effective data distributions. Natural language processing models work with sparse features — bag-of-words representations where most words don't appear in any given document — creating a regime where most gradient components are zero at any given timestep, but the non-zero components carry critical signal.
The paper frames this as a problem of determining the right per-parameter learning rate automatically from data, rather than through manual tuning. The phrase "adaptive learning rates" — now ubiquitous in deep learning — was, at the time of this paper's publication (ICLR 2015), an active research frontier with competing approaches that each solved part of the problem but failed on others.
Why This Problem Matters: The Scaling Imperative in Deep Learning
The motivation for solving per-parameter learning rate adaptation is both practical and theoretical, and the paper grounds its importance in the specific demands of deep learning as practiced in 2014–2015.
Practical significance. The paper opens Section 1 by situating stochastic gradient-based optimization as "of core practical importance in many fields of science and engineering." This is not rhetorical flourish — the deep learning revolution was in full swing at the time of publication, with AlexNet (Krizhevsky et al., 2012) having demonstrated the power of large convolutional networks on ImageNet, recurrent networks achieving breakthrough speech recognition results (Graves et al., 2013), and unsupervised learning methods like variational autoencoders (Kingma & Welling, 2013) and dropout regularization (Hinton et al., 2012b) gaining traction. All of these advances relied on stochastic optimization, and all of them required careful — often painful — hyperparameter tuning.
The paper explicitly identifies four desiderata that an optimization algorithm must satisfy to be practically useful for the kinds of problems the field was tackling:
-
Computational efficiency. The algorithm should have cost per iteration comparable to computing the gradient itself. This rules out second-order methods (which require Hessian computation or storage) for high-dimensional problems. As the paper notes, "computation of first-order partial derivatives w.r.t. all the parameters is of the same computational complexity as just evaluating the function" — meaning that any overhead beyond gradient computation should be minimal.
-
Small memory footprint. Methods that require storing per-datapoint curvature matrices (like SFO; Sohl-Dickstein et al., 2014) are infeasible on GPU-constrained systems where memory is at a premium. The paper emphasizes this in Section 5, noting that SFO "has memory requirements linear in the number of minibatch partitions of a dataset, which is often infeasible on memory-constrained systems such as a GPU."
-
Robustness to sparse gradients. In NLP, recommender systems, and any setting with bag-of-words or one-hot representations, most gradient components are zero at each timestep. An optimizer that can handle this without requiring massive averaging windows is essential.
-
Robustness to non-stationary objectives. Dropout, minibatch sampling, and the natural progression of learning through different regimes during training all make the effective objective function change over time. An optimizer that assumes the statistics of the gradient are stationary will fail when they aren't.
Theoretical significance. Beyond practical utility, the paper aims to provide a regret-based convergence analysis under the online convex optimization framework of Zinkevich (2003). This is important because it provides a principled guarantee — not just empirical evidence — that the algorithm will not diverge or catastrophically fail. The bound they derive, regret with specific dependence on the gradient sparsity pattern, connects Adam to the existing theory for adaptive methods like AdaGrad and establishes that Adam inherits the theoretical benefits of per-parameter learning rate adaptation while extending them to the non-stationary setting.
Where Existing Approaches Fall Short
The paper identifies two primary predecessor algorithms — AdaGrad and RMSProp — and systematically analyzes where each one excels and where each one breaks down. Understanding these failure modes is essential to understanding why Adam's design choices are what they are.
AdaGrad: Sparse-Gradient Specialist with a Diminishing Learning Rate Problem
AdaGrad (Duchi et al., 2011) was a major advance because it demonstrated that per-parameter learning rate adaptation could be both computationally efficient and theoretically sound. The core idea is simple: accumulate the sum of squared gradients for each parameter, , and scale the learning rate for each parameter inversely proportional to :
Parameters that have received large gradients in the past get small effective learning rates; parameters that have received small or sparse gradients get large effective learning rates. This directly addresses the heterogeneous gradient problem, and as the paper's experiments confirm (Figure 1, IMDB plot), AdaGrad dramatically outperforms SGD with momentum on sparse bag-of-words features.
What AdaGrad gets wrong. The accumulation grows monotonically. There is no decay or forgetting mechanism. Over the course of training, grows without bound, meaning the effective learning rate decays toward zero whether or not the parameter has converged. In convex optimization, this annealing behavior can be benign or even helpful near the optimum. But in non-convex problems — which characterize most of deep learning — and in non-stationary settings where the gradient distribution changes over time (e.g., as the model moves through different loss landscapes, or as dropout masks change), AdaGrad's monotonically decreasing learning rate becomes a liability. The optimizer loses the ability to make significant parameter updates even when the gradients warrant them.
The paper's experiments demonstrate this failure concretely. In Figure 3 (right), AdaGrad converges substantially slower than both Adam and SGD with momentum on the CIFAR-10 convolutional network after the first few epochs. The paper diagnoses this as follows:
"We notice the second moment estimate vanishes to zeros after a few epochs and is dominated by the in algorithm 1."
By "vanishes to zeros," the authors mean that the accumulated squared gradients become extremely large, making huge, which drives the effective learning rate to near zero for all parameters — even those that still need to move. This is AdaGrad's fundamental limitation: it has no mechanism to forget the distant past.
RMSProp: Non-Stationary Specialist Without a Theoretical Foundation
RMSProp (Tieleman & Hinton, 2012) solves AdaGrad's accumulation problem by replacing the sum of squared gradients with an exponential moving average:
where controls the decay rate. When is close to 1 (e.g., the recommended 0.999), the moving average gives exponentially decreasing weight to gradients from the distant past, so old gradient information is gradually forgotten. This makes RMSProp well-suited to non-stationary objectives — as the gradient distribution changes, adapts to reflect the new distribution rather than being dominated by history.
What RMSProp gets wrong. The paper identifies two specific problems:
-
No momentum. The basic RMSProp rescales the gradient by but does not maintain a separate moving average of the gradient itself (a momentum term). While a version "with momentum has sometimes been used" (Graves, 2013), this is an ad-hoc addition rather than an integral part of the algorithm. Momentum is important empirically — it smooths the gradient estimate and allows the optimizer to build up velocity in consistent directions — and the paper's experiments show that momentum-accelerated methods outperform those without it in most settings.
-
Initialization bias. This is the more subtle and technically important issue. When is initialized to zero (as it naturally would be), the exponential moving average at timestep is:
This is a weighted sum of past squared gradients, but the weights do not sum to 1 at small . They sum to . When is close to 1 (as it must be for the moving average to serve its purpose of averaging over many gradients), is very small for small , meaning the effective magnitude of is systematically underestimated in the early steps. This bias causes the early parameter updates to be much larger than intended, because the denominator underestimates the true gradient scale.
The paper demonstrates this failure mode dramatically in Figure 4. When or (values needed for stable learning in sparse-gradient settings), training a VAE without bias correction leads to much higher loss and instability, especially in the early epochs. The paper states:
"Values of close to 1, required for robustness to sparse gradients, results in larger initialization bias; therefore we expect the bias correction term is important in such cases of slow decay, preventing an adverse effect on optimization."
This is a concrete, empirically verified failure mode: RMSProp with high (which is needed for generalization performance) diverges or performs poorly during early training because the denominator starts too small. Practitioners who encountered this might have tried to compensate by reducing the learning rate , but this would slow down learning throughout training, not just in the initial phase.
The Missing Synthesis
The paper's diagnosis of the landscape is that AdaGrad and RMSProp present a tradeoff:
- AdaGrad handles sparse gradients well and has theoretical grounding, but fails on non-stationary objectives due to its monotonically increasing denominator.
- RMSProp handles non-stationary objectives well, but lacks theoretical analysis, lacks built-in momentum, and suffers from initialization bias that destabilizes early training.
No existing method combined all three desiderata: sparse-gradient tolerance, non-stationarity tolerance, and rigorous initialization handling. The authors also note that some methods (SFO, natural Newton methods, AdaDelta) exist in the same design space but either have prohibitive memory requirements, are too expensive per iteration, or make assumptions (deterministic subfunctions in the case of SFO) that don't hold under stochastic regularization like dropout.
How Adam Positions Itself Relative to Prior Work
Adam is explicitly positioned as a synthesis — not a radical departure — that combines the proven mechanisms of AdaGrad and RMSProp while adding a principled correction for initialization bias and incorporating momentum directly into the adaptive framework. This synthetic ambition is stated plainly in the abstract and introduction:
"The method computes individual adaptive learning rates for different parameters from estimates of first and second moments of the gradients; the name Adam is derived from adaptive moment estimation."
The "moment estimation" framing is deliberate. Rather than thinking of as merely a running average of squared gradients (as in RMSProp), Adam reinterprets it as an estimate of the second raw moment (uncentered variance) of the gradient. Simultaneously, is an estimate of the first moment (the mean). This moment-based perspective unifies the two mechanisms under a common statistical framework: the optimizer is maintaining running estimates of and , and using these to compute a per-parameter signal-to-noise ratio .
This reframing is not just cosmetic. It leads directly to the bias correction derivation in Section 3, where the authors mathematically derive exactly how much the zero-initialized estimates underrepresent the true moments at timestep . By dividing by and by , Adam recovers unbiased estimates without requiring any heuristic warmup period or manual learning-rate scheduling. The bias correction is baked into the statistical model of what the algorithm is computing.
The paper also positions Adam as the natural successor to both AdaGrad and RMSProp by showing formal connections:
-
RMSProp with momentum is Adam without bias correction. The paper states this equivalence explicitly in Section 6.4, where removing the bias correction terms "results in a version of RMSProp with momentum." The experiments in Figure 4 then demonstrate that Adam (with bias correction) equals or outperforms this RMSProp variant across all hyperparameter settings tested, with the gap largest precisely when is large (the sparse-gradient regime).
-
AdaGrad is Adam with , , and an annealed learning rate . The paper derives this correspondence in Section 5:
"Note that if we choose to be infinitesimally close to 1 from below, then ."
Under this limit, the bias-corrected Adam update reduces to the AdaGrad update (with the annealing). This formal reduction is important because it means Adam can recover AdaGrad's behavior through appropriate hyperparameter settings, while generalizing beyond it with to enable forgetting in non-stationary settings.
The unifying theme is that Adam's hyperparameters and provide a continuous interpolation between extremes: gives AdaGrad-like permanent accumulation; gives RMSProp-like exponential forgetting. gives no momentum; gives momentum smoothing. The paper's recommended defaults (, ) sit at an empirically motivated middle ground that has since become the de facto standard initialization for adaptive optimization in deep learning.
Critically, the paper does not claim to invent momentum, per-parameter scaling, or exponential moving averages — all of these existed. The contribution is the integration of these components into a single, principled algorithm with bias correction, backed by a regret analysis that accounts for the decaying schedule (connecting to prior empirical findings that reducing momentum late in training helps convergence; Sutskever et al., 2013) and a sparsity-sensitive bound that matches the rate AdaGrad achieves on sparse problems. The paper essentially argues: "The pieces are all here, but they've never been assembled correctly, and the assembly matters a great deal for practical performance."
3. Technical Approach
3.1 Reader Orientation
The paper presents Adam, a first-order gradient-based optimization algorithm — an iterative update rule for the parameters of a machine learning model. The algorithm maintains running estimates of both the mean (first moment) and uncentered variance (second raw moment) of the stochastic gradient at each parameter, and uses these estimates to compute an individual, adaptive learning rate for every parameter that automatically scales updates based on the gradient's historical magnitude and stability. The problem Adam solves is that standard stochastic gradient descent uses a single global learning rate for all parameters, which fails when gradients have different scales across parameters (convolutional layers vs. bias terms, frequent vs. rare features) — Adam's solution is to estimate per-parameter gradient statistics online and divide each parameter's update by the square root of its gradient variance, effectively normalizing the step size so that parameters with consistently large gradients get smaller steps and parameters with small or sparse gradients get larger steps.
3.2 Big-Picture Architecture (Diagram in Words)
The Adam algorithm, as shown in Algorithm 1, consists of five major computational components that execute sequentially at each timestep:
-
Stochastic Gradient Computation — evaluates the gradient
$g_t = \nabla_\theta f_t(\theta_{t-1})$of the stochastic objective$f_t$(typically a minibatch loss) at the current parameter vector$\theta_{t-1}$. -
First Moment Estimator (Momentum) — maintains an exponential moving average
$m_t$of the gradient, controlled by decay rate$\beta_1$. This smooths out the noisy gradient signal and accumulates consistent gradient directions, analogous to momentum in physics. -
Second Raw Moment Estimator (Variance) — maintains an exponential moving average
$v_t$of the elementwise squared gradient, controlled by decay rate$\beta_2$. This tracks how large the gradient magnitudes tend to be for each parameter, providing the information needed to scale learning rates per-parameter. -
Bias Correction — divides
$m_t$by$1 - \beta_1^t$and$v_t$by$1 - \beta_2^t$to correct for the systematic underestimation caused by initializing the moving averages at zero. These correction factors approach 1 as$t$grows large, making the correction most important in early iterations. -
Parameter Update — computes the final update as
$\theta_t = \theta_{t-1} - \alpha \cdot \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$, where$\hat{m}_t$and$\hat{v}_t$are the bias-corrected moment estimates,$\alpha$is the global stepsize, and$\epsilon$is a small constant for numerical stability.
Information flows strictly forward through these components: gradient → both moment estimators (in parallel) → bias correction on each → parameter update. The updated parameters feed into the next iteration's gradient computation, closing the loop.
3.3 Roadmap for the Deep Dive
The explanation will proceed in order of conceptual dependence, building from the simplest component to the complete algorithm:
-
First, the parameter update rule (the core equation that defines what Adam computes). This gives the destination before we explain how each piece is built.
-
Second, the first and second moment estimators — the exponential moving averages
$m_t$and$v_t$. Understanding these requires understanding why we want them before we can understand what goes wrong with them. -
Third, the initialization bias problem and its correction. This is where Adam departs from both AdaGrad and RMSProp, and the derivation reveals exactly what bias is introduced by zero-initialization and how dividing by
$1 - \beta^t$fixes it. -
Fourth, the properties of the effective step size — what bounds the magnitude of parameter updates, why the
$\hat{m}_t / \sqrt{\hat{v}_t}$ratio behaves like a signal-to-noise ratio, and how this produces automatic annealing. -
Fifth, the hyperparameters and default settings, including the rationale for the recommended values and their relationship to algorithm behavior.
-
Sixth, the AdaMax variant, because it flows naturally from the analysis of
$v_t$as an$L^2$norm and shows how the algorithm generalizes.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily an algorithm design paper whose core idea is that individual adaptive learning rates can be derived from running estimates of the first and second moments of the stochastic gradient, and that the zero-initialization bias in those running estimates can be exactly corrected by a simple time-dependent multiplicative factor.
The Parameter Update Rule
The Adam algorithm's central computation at each timestep $t$ is:
where $\theta_t \in \mathbb{R}^d$ is the parameter vector after the update, $\theta_{t-1}$ is the parameter vector before the update, $\alpha$ is the global stepsize (a positive scalar, default 0.001), $\hat{m}_t \in \mathbb{R}^d$ is the bias-corrected estimate of the gradient's first moment (mean), $\hat{v}_t \in \mathbb{R}^d$ is the bias-corrected estimate of the gradient's second raw moment (uncentered variance), and $\epsilon$ is a small constant (default $10^{-8}$) to prevent division by zero.
What it computes: For each dimension $i$ of the parameter vector, the update subtracts a term proportional to $\hat{m}_{t,i} / \sqrt{\hat{v}_{t,i}}$. The numerator $\hat{m}_{t,i}$ is the smoothed gradient in that dimension — it tells us which direction to move and with what confidence. The denominator $\sqrt{\hat{v}_{t,i}}$ is the root-mean-square magnitude of recent gradients in that dimension — it tells us how large gradients typically are for this parameter. The ratio $\hat{m}_{t,i} / \sqrt{\hat{v}_{t,i}}$ is therefore a normalized gradient: if the current smoothed gradient is $+0.1$ but historical gradients for this parameter have RMS magnitude $0.01$, the ratio is $10$, indicating a strong signal relative to noise, and the parameter moves substantially. If the smoothed gradient is $+0.001$ but historical RMS is $0.1$, the ratio is $0.01$, indicating a weak signal, and the parameter barely moves. The $\epsilon$ in the denominator simply ensures the computation doesn't blow up if $\hat{v}_{t,i}$ happens to be exactly zero.
Why this form: This update rule unifies three mechanisms that were previously separate in the literature. First, the numerator $\hat{m}_t$ provides momentum: it smooths out gradient noise and allows the optimizer to build up velocity in consistent directions. Second, the denominator $\sqrt{\hat{v}_t}$ provides per-parameter adaptive scaling: parameters with historically large gradients receive smaller effective learning rates, and parameters with historically small or sparse gradients receive larger effective learning rates. Third, the entire ratio is multiplied by a single global stepsize $\alpha$, which the paper shows (Section 2.1) acts as an approximate upper bound on the magnitude of the parameter change $|\Delta_t|$, making it interpretable: if you know that good parameter values lie within some region, you can set $\alpha$ to be a fraction of that region's size, and Adam will respect that constraint while still allowing faster progress for parameters where the gradient signal is strong.
The alternative — standard SGD $\theta_t = \theta_{t-1} - \alpha g_t$ — uses the same $\alpha$ for all parameters regardless of their gradient scale, which means that if one parameter's gradient is 100× larger than another's, the former's update dominates training. The alternative — RMSProp $\theta_t = \theta_{t-1} - \alpha g_t / \sqrt{v_t}$ — scales per-parameter but uses the raw gradient rather than a smoothed momentum estimate, making the direction noisy. Adam integrates both by applying the per-parameter scaling to the momentum, not to the raw gradient. The paper notes that RMSProp with a separate momentum step does exist in practice (Graves, 2013), but that version applies momentum after rescaling ($m_t = \beta_1 m_{t-1} + (1 - \beta_1) (g_t / \sqrt{v_t})$), whereas Adam computes the moments separately and then takes their ratio. The Adam approach means the momentum estimate $m_t$ is in the same units as $g_t$ (it's an average of gradients), not in rescaled units, which matters for the bias correction and the theoretical analysis.
The First Moment Estimator (Momentum)
The first moment estimator is an exponential moving average (EMA) of the gradient:
with initialization $m_0 = 0$ (a vector of zeros). Here $m_t \in \mathbb{R}^d$ is the biased first moment estimate at timestep $t$, $\beta_1 \in [0, 1)$ is the exponential decay rate for the first moment (default $0.9$), $m_{t-1}$ is the estimate from the previous timestep, and $g_t = \nabla_\theta f_t(\theta_{t-1})$ is the gradient of the stochastic objective at timestep $t$.
What it computes: At each timestep, the new estimate $m_t$ is a convex combination of the old estimate $m_{t-1}$ and the new gradient $g_t$. The weight on the old estimate is $\beta_1$; the weight on the new gradient is $1 - \beta_1$. Because this recurrence applies at every step, $m_t$ can be unrolled as a weighted sum of all past gradients:
The weight assigned to gradient $g_i$ is $(1 - \beta_1) \beta_1^{t-i}$. Since $\beta_1 < 1$, gradients from further in the past receive exponentially smaller weights. With the default $\beta_1 = 0.9$, a gradient from 10 steps ago receives weight $(1 - 0.9) \cdot 0.9^{10} \approx 0.035$, while the most recent gradient receives weight $(1 - 0.9) \cdot 0.9^0 = 0.1$.
The quantity $m_t$ estimates $\mathbb{E}[g]$, the expected value of the gradient, under the assumption that the gradient distribution changes slowly relative to the EMA's time constant. The effective averaging window (the timescale over which past gradients have non-negligible weight) is approximately $1 / (1 - \beta_1)$ steps. For $\beta_1 = 0.9$, this is roughly 10 steps.
Why this form: The exponential moving average serves two purposes. First, it provides momentum: by averaging out the per-minibatch noise in $g_t$, the optimizer moves more consistently in the direction of the true expected gradient rather than oscillating due to minibatch variance. In physical terms, $m_t$ is like velocity — it accumulates contributions from consistent gradient directions and damps out random fluctuations. Second, the exponential decay (as opposed to a simple arithmetic average) gives the estimator the ability to forget old gradients, which is essential for non-stationary objectives where the gradient distribution changes over time. If the optimizer has moved to a different region of parameter space where the gradient distribution is different, the old gradient estimates from the previous region should be discounted.
The alternative — no momentum ($\beta_1 = 0$, giving $m_t = g_t$) — would make the algorithm equivalent to RMSProp with the raw gradient in the numerator. The alternative — a simple arithmetic average of all past gradients — would be AdaGrad-like and would prevent forgetting, causing the momentum to become increasingly stale in non-stationary settings. The EMA with $\beta_1 \approx 0.9$ is an established choice from the momentum literature that balances noise suppression with adaptability.
The initialization $m_0 = 0$ is the natural choice (no prior information about the gradient direction), but as Section 3 explains, it introduces a systematic bias that must be corrected — the EMA of a sequence starting from zero systematically underestimates the true mean in early steps.
The Second Raw Moment Estimator (Uncentered Variance)
The second moment estimator follows the same EMA pattern but operates on the elementwise square of the gradient:
with initialization $v_0 = 0$. Here $v_t \in \mathbb{R}^d$ is the biased second raw moment estimate at timestep $t$, $\beta_2 \in [0, 1)$ is the exponential decay rate for the second moment (default $0.999$), and $g_t^2 = g_t \odot g_t$ is the elementwise square of the gradient vector. The notation $\odot$ denotes Hadamard (elementwise) multiplication, so $g_t^2$ is a vector whose $i$-th component is $(g_{t,i})^2$.
What it computes: At each timestep, $v_t$ updates each component $i$ independently: $v_{t,i} = \beta_2 \cdot v_{t-1,i} + (1 - \beta_2) \cdot g_{t,i}^2$. Unrolled, this gives:
The quantity $v_{t,i}$ estimates $\mathbb{E}[g_i^2]$, the expected squared gradient (the second raw moment, also called the uncentered variance when $\mathbb{E}[g_i] = 0$) for parameter $i$. The effective averaging window is approximately $1 / (1 - \beta_2)$ steps. With the default $\beta_2 = 0.999$, this is roughly 1000 steps — two orders of magnitude longer than the first-moment window. This difference is intentional: the first moment (direction) can change quickly as the optimizer moves through parameter space, but the scale of gradients (magnitude) tends to change more slowly, so a longer averaging window provides a more stable estimate.
The square root $\sqrt{v_{t,i}}$ is approximately the root-mean-square (RMS) magnitude of recent gradients for parameter $i$. In the final update, dividing $\hat{m}_{t,i}$ by $\sqrt{\hat{v}_{t,i}}$ effectively normalizes the gradient by its historical magnitude, producing an update with magnitude independent of the raw gradient scale.
Why this form: The second moment estimate is what provides per-parameter adaptive learning rates. Parameters that consistently receive large gradients will have large $v_{t,i}$, so $\sqrt{v_{t,i}}$ will be large, and the effective step $\alpha \cdot \hat{m}_{t,i} / \sqrt{\hat{v}_{t,i}}$ will be small. Parameters that receive small or sparse gradients will have small $v_{t,i}$, so the effective step will be proportionally larger. This directly addresses the heterogeneous gradient problem: instead of manually tuning per-layer or per-parameter learning rates, Adam estimates the appropriate scaling from the gradient statistics themselves.
The exponential moving average (as opposed to AdaGrad's cumulative sum $\sum_{i=1}^t g_i^2$) is the critical design choice that enables Adam to handle non-stationary objectives. In AdaGrad, the denominator $\sqrt{\sum_{i=1}^t g_i^2}$ grows monotonically and eventually becomes so large that learning effectively stops. In Adam, old squared gradients are exponentially forgotten, so $v_t$ adapts to the current gradient scale. If the optimizer enters a region where gradients become larger, $v_t$ will increase to reflect this; if gradients become smaller, $v_t$ will decrease. This prevents the learning rate from decaying to zero prematurely.
The paper establishes a direct correspondence between Adam and AdaGrad in Section 5: if we take the limit $\beta_2 \to 1$ (from below), the unrolled expression for the bias-corrected estimate $\hat{v}_t$ approaches $t^{-1} \sum_{i=1}^t g_i^2$ (see the bias correction derivation below), which is exactly AdaGrad's denominator scaled by $1/t$. Combined with an annealed learning rate $\alpha_t = \alpha / \sqrt{t}$, Adam with $\beta_2 \to 1$ and $\beta_1 = 0$ recovers AdaGrad exactly. For any $\beta_2 < 1$, Adam departs from AdaGrad by forgetting old gradients, which is what makes it work on non-stationary problems.
The default $\beta_2 = 0.999$ is chosen to make the denominator stable (averaging over ~1000 steps) while still allowing adaptation. A much smaller $\beta_2$ (e.g., 0.9) would make the denominator too noisy, oscillating with each minibatch. A $\beta_2$ extremely close to 1 (e.g., 0.99999) approaches AdaGrad behavior and loses the forgetting property. The value 0.999 was determined empirically and has proven robust across a wide range of deep learning applications.
Initialization Bias and Its Correction
Both moment estimators are initialized at zero ($m_0 = 0$, $v_0 = 0$), which is the natural choice when no prior information about the gradient distribution is available. However, this zero initialization introduces a systematic underestimation of the true moments in the early timesteps, because the EMA is a weighted average where the initial zeros pull the estimate downward. The paper derives the exact form of this bias and shows that it can be perfectly corrected by dividing by $1 - \beta^t$.
Derivation for the second moment (Section 3). The derivation for the first moment is analogous; the paper presents the derivation for $v_t$ explicitly. Starting from the unrolled expression:
Take the expectation of both sides, assuming the second moment is stationary ($\mathbb{E}[g_i^2] = \mathbb{E}[g^2]$ for all $i$) or nearly so:
where $\zeta$ accounts for non-stationarity and is small when $\beta_2$ is chosen appropriately (because old gradients with changing statistics receive exponentially small weights). The key term is $(1 - \beta_2^t)$: the sum of the EMA weights is not 1 at timestep $t$, but rather $1 - \beta_2^t$. This means that on average, $v_t$ underestimates the true second moment $\mathbb{E}[g^2]$ by a factor of $1 - \beta_2^t$.
Operational meaning: At $t = 1$, $\beta_2^1 = 0.999$, so $1 - \beta_2^1 = 0.001$, and $\mathbb{E}[v_1] = 0.001 \cdot \mathbb{E}[g^2]$ — the estimate is three orders of magnitude too small. At $t = 10$, $\beta_2^{10} \approx 0.99$, so $1 - \beta_2^{10} \approx 0.01$, and the estimate is still off by two orders of magnitude. At $t = 1000$, $\beta_2^{1000} \approx 0.368$, so $1 - \beta_2^{1000} \approx 0.632$, and the estimate has mostly converged. The bias is most severe in the early steps and decays as $t$ grows.
The bias correction. To obtain an unbiased estimate, Adam divides by the correction factor:
Similarly for the first moment:
where $\beta_1^t$ and $\beta_2^t$ denote $\beta_1$ and $\beta_2$ raised to the power $t$ (not indexed parameters).
What this computes: At each timestep, the biased estimates $m_t$ and $v_t$ are scaled up by $1 / (1 - \beta^t)$ to counteract the zero-initialization effect. At $t = 1$ with $\beta_2 = 0.999$, the biased $v_1 = 0.001 \cdot g_1^2$, and the correction divides by $1 - 0.999^1 = 0.001$, yielding $\hat{v}_1 = g_1^2$ — exactly the squared gradient, which is the correct estimate when only one data point is available. At $t = 2$, the correction is $1 / (1 - 0.999^2) \approx 1 / 0.001999$, slightly smaller than $1 / 0.001$, reflecting that the bias is slightly smaller. As $t \to \infty$, $\beta^t \to 0$ and the correction factor approaches 1, meaning no correction is applied — the raw EMA is already unbiased in the limit.
Why this form: The correction is exact under the stationary assumption ($\zeta = 0$). It is not a heuristic or an approximation — it is derived directly from the expected value of the EMA. The paper contrasts this with RMSProp, which does not include any bias correction. In RMSProp, the denominator $\sqrt{v_t}$ is systematically too small in early steps, causing the effective learning rate $\alpha / \sqrt{v_t}$ to be much larger than intended. This is precisely the failure mode demonstrated in Figure 4: when $\beta_2$ is large (0.999 or 0.9999), RMSProp without bias correction produces unstable training and poor performance, particularly in the first few epochs. Adam's bias correction eliminates this problem entirely, allowing the use of large $\beta_2$ values (which are necessary for robust estimation with sparse gradients) without any early-training instability.
An important subtlety: the bias correction is applied to each moment estimate independently before the ratio is taken. The corrected estimates $\hat{m}_t$ and $\hat{v}_t$ are then used in the update $\theta_t = \theta_{t-1} - \alpha \cdot \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$. The correction factors $1 / (1 - \beta_1^t)$ and $1 / \sqrt{1 - \beta_2^t}$ do not cancel because $\beta_1$ and $\beta_2$ are typically different and the second-moment correction involves a square root. The paper also notes that the order of computations in Algorithm 1 can be rearranged for efficiency by combining the bias corrections with the learning rate into a single effective stepsize $\alpha_t = \alpha \cdot \sqrt{1 - \beta_2^t} / (1 - \beta_1^t)$, which simplifies the update to $\theta_t = \theta_{t-1} - \alpha_t \cdot m_t / (\sqrt{v_t} + \hat{\epsilon})$ at the cost of some clarity.
The paper demonstrates the practical importance of the bias correction empirically in Section 6.4 (Figure 4). Training a variational autoencoder (VAE) across a grid of $\alpha$, $\beta_1$, and $\beta_2$ values, the bias-corrected version (Adam) consistently matches or outperforms the non-bias-corrected version (equivalent to RMSProp with momentum). The gap is largest precisely when $\beta_2$ is closest to 1 (0.999 and 0.9999) — the regime where the initialization bias is most severe — and is most pronounced in the early epochs of training before the $1 - \beta_2^t$ factor has had time to approach 1 naturally.
Properties of the Effective Step Size
The paper devotes Section 2.1 to analyzing the effective step size $\Delta_t = \alpha \cdot \hat{m}_t / \sqrt{\hat{v}_t}$ (assuming $\epsilon = 0$ for the analysis). This analysis reveals several properties that explain Adam's robustness and its "automatic annealing" behavior.
Upper bound on step magnitude. The paper establishes two cases for the maximum possible magnitude of $|\Delta_t|$:
- Case 1: If
$(1 - \beta_1) > \sqrt{1 - \beta_2}$, then$|\Delta_t| \leq \alpha \cdot (1 - \beta_1) / \sqrt{1 - \beta_2}$. - Case 2: Otherwise,
$|\Delta_t| \leq \alpha$.
The default values $\beta_1 = 0.9$ and $\beta_2 = 0.999$ give $1 - \beta_1 = 0.1$ and $\sqrt{1 - \beta_2} = \sqrt{0.001} \approx 0.0316$. Since $0.1 > 0.0316$, we are in Case 1, and the upper bound is $\alpha \cdot 0.1 / 0.0316 \approx 3.16\alpha$. However, this worst-case bound is only achieved in extreme sparsity: when the gradient has been exactly zero at all previous timesteps and then becomes non-zero at the current timestep. In that scenario, $\hat{m}_t$ jumps from near-zero to capture the new gradient, while $\hat{v}_t$ still primarily reflects the (previously zero) history. For less extreme cases, the paper shows that $|\Delta_t|$ is generally bounded by approximately $\alpha$, because $|\hat{m}_t / \sqrt{\hat{v}_t}| \lessapprox 1$.
Operational meaning: The single hyperparameter $\alpha$ controls the maximum distance any parameter can move in a single update. This is in contrast to standard SGD, where the step magnitude is $\alpha |g_t|$ and can be arbitrarily large if the gradient is large. In Adam, the normalization by $\sqrt{\hat{v}_t}$ clips the update — even if the current gradient is enormous, the update is bounded because the historical gradient magnitude (captured by $\hat{v}_t$) is also enormous, and their ratio is bounded.
Interpretation as a signal-to-noise ratio (SNR). The paper calls $\hat{m}_t / \sqrt{\hat{v}_t}$ the signal-to-noise ratio with "a slight abuse of terminology." The idea is that $\hat{m}_t$ estimates the mean gradient (the signal — the direction and magnitude we should move), while $\sqrt{\hat{v}_t}$ estimates the RMS gradient magnitude (which includes both signal and noise). When the SNR is high (the mean gradient is large compared to the typical gradient magnitude), the effective step is large — the optimizer has a clear, consistent direction and moves confidently. When the SNR is low (the mean gradient is small compared to the typical magnitude), the effective step is small — the gradient direction is uncertain, so the optimizer moves cautiously.
Automatic annealing toward optima. As the optimizer approaches a minimum, the true gradient shrinks toward zero. The noise due to minibatch sampling, however, does not shrink — it remains approximately constant. This means the SNR naturally decreases near convergence: the signal (mean gradient) becomes small while the noise (gradient variance) stays roughly the same. Adam automatically reduces its effective step size in response, because $\hat{m}_t$ shrinks faster than $\sqrt{\hat{v}_t}$. This produces a form of automatic learning rate annealing without any explicit schedule — the optimizer naturally takes smaller and smaller steps as it converges, simply because the gradient signal weakens relative to the gradient noise. The paper contrasts this with AdaGrad's aggressive annealing (which forces the learning rate to zero regardless of whether convergence has been achieved) and with SGD's lack of any automatic annealing.
Invariance to gradient rescaling. A key property mentioned in the abstract and demonstrated in Section 2.1: if all gradients are multiplied by a constant factor $c$ (e.g., because the loss function is scaled by $c$), the effective step remains unchanged:
This means Adam is invariant to diagonal rescaling of the gradients. If each parameter's gradient were scaled independently (corresponding to multiplying the parameter vector by a diagonal matrix), Adam's updates would automatically compensate. This is important because many objective functions have parameters with inherently different scales (e.g., weights vs. biases, different layers), and Adam handles these differences automatically without requiring the practitioner to set different learning rates for different parameter groups.
The trust region interpretation. The paper describes the step size bound as "establishing a trust region around the current parameter value, beyond which the current gradient estimate does not provide sufficient information." This connects Adam to trust-region methods in optimization: rather than blindly following the gradient wherever it leads, Adam constrains the update to a region where the local gradient estimate is reliable. This makes $\alpha$ interpretable: if you have prior knowledge that good parameter values lie within some range (e.g., from weight initialization schemes or Bayesian priors), you can set $\alpha$ to be a fraction of that range, and Adam will not take steps larger than that fraction.
Hyperparameters and Default Settings
The paper specifies four hyperparameters for the Adam algorithm, with recommended default values that were determined empirically across a range of machine learning problems:
-
$\alpha = 0.001$: The global stepsize. This is the primary hyperparameter that practitioners should tune. The paper notes that because$\alpha$approximately bounds the magnitude of per-parameter updates, it can often be set based on prior knowledge of the parameter space's scale. In the experiments,$\alpha$is sometimes combined with a$1/\sqrt{t}$decay schedule (as in the logistic regression and CNN experiments) to match the theoretical analysis, but the default of 0.001 with no decay works well for many problems. -
$\beta_1 = 0.9$: The exponential decay rate for the first moment estimate. This controls how much momentum is applied. A value of 0.9 means the EMA effective window is roughly$1 / (1 - 0.9) = 10$steps. Higher values (e.g., 0.99) give more momentum and more stable estimates at the cost of slower adaptation to changing gradient directions. Lower values (e.g., 0.5) give less momentum and more responsive updates at the cost of higher variance. -
$\beta_2 = 0.999$: The exponential decay rate for the second moment estimate. This controls how quickly the per-parameter learning rate adaptation adapts. A value of 0.999 means the EMA effective window is roughly$1 / (1 - 0.999) = 1000$steps, providing a stable estimate of the gradient scale. This value should be close to 1 for sparse-gradient problems (where reliable variance estimation requires averaging over many steps), and the bias correction makes values this close to 1 practical by preventing early-training instability (as demonstrated in Figure 4). -
$\epsilon = 10^{-8}$: A small constant added to the denominator$\sqrt{\hat{v}_t}$for numerical stability. It prevents division by zero in the (rare but possible) case where$\hat{v}_{t,i}$is exactly zero. The paper uses$\epsilon = 10^{-8}$throughout the experiments described in Section 6. The value is small enough that it doesn't meaningfully affect the effective step size under normal operation (since typical$\sqrt{\hat{v}_{t,i}}$values are much larger), but large enough to prevent floating-point overflow.
Why these values: The defaults reflect a design philosophy of "usually good enough without tuning." The paper states that "the hyper-parameters have intuitive interpretations and typically require little tuning." The values 0.9 and 0.999 are specific and have been widely validated since the paper's publication, but the paper's analysis in Section 2.1 provides a framework for understanding how changing them affects behavior:
- Increasing
$\beta_2$closer to 1 makes the denominator adapt more slowly, behaving more like AdaGrad. This is beneficial for sparse features but requires the bias correction to avoid early instability. The paper recommends 0.999 as a balanced choice. - Decreasing
$\beta_1$reduces momentum, making the optimizer more responsive to recent gradients but noisier. The paper demonstrates in the theoretical analysis (Theorem 4.1) that decaying$\beta_1$toward zero over time (specifically$\beta_{1,t} = \beta_1 \lambda^{t-1}$) is required for the regret bound and matches empirical findings from Sutskever et al. (2013) that reducing momentum late in training helps convergence.
The paper also notes that the learning rate $\alpha$ can be combined with the bias correction factors into a single effective stepsize $\alpha_t = \alpha \cdot \sqrt{1 - \beta_2^t} / (1 - \beta_1^t)$, which makes the algorithm implementation slightly more efficient. With this reparameterization, the update becomes $\theta_t = \theta_{t-1} - \alpha_t \cdot m_t / (\sqrt{v_t} + \hat{\epsilon})$, where $\hat{\epsilon} = \epsilon \sqrt{1 - \beta_2^t}$ (the $\epsilon$ must also be adjusted for the bias correction if this reparameterization is used). The paper presents both forms but uses the clearer bias-corrected form in Algorithm 1.
AdaMax: The $L^\infty$ Norm Variant
The paper generalizes the second moment update to an $L^p$ norm and then takes the limit $p \to \infty$, yielding a variant called AdaMax (Algorithm 2). This variant is interesting both as a conceptual extension and as a practical algorithm with a simpler denominator.
Generalization to $L^p$ norm. Instead of tracking $v_t$ as the EMA of squared gradients (which corresponds to an $L^2$ norm), consider tracking an EMA of the $p$-th power of the gradient magnitude:
where the decay rate is parameterized as $\beta_2^p$ rather than $\beta_2$ (so the effective decay changes with $p$). The update would then use $v_t^{1/p}$ in the denominator, corresponding to an $L^p$ norm of the gradient history. For large $p$, this becomes numerically unstable because $|g_t|^p$ can overflow or underflow for large $p$.
Limit as $p \to \infty$. The paper shows that taking the limit $p \to \infty$ leads to a remarkably simple update. Define $u_t = \lim_{p \to \infty} (v_t)^{1/p}$. The derivation proceeds:
The last step uses the fact that the $L^p$ norm approaches the $L^\infty$ norm (the maximum) as $p \to \infty$.
The AdaMax update. The $L^\infty$ norm can be computed recursively as:
with $u_0 = 0$. The AdaMax parameter update is then:
where $m_t$ is the same biased first moment estimate as in Adam, and the learning rate $\alpha / (1 - \beta_1^t)$ incorporates the bias correction for the first moment.
What this computes: At each timestep, $u_t$ tracks the exponentially weighted maximum of the absolute gradient values. The decay factor $\beta_2^{t-i}$ ensures that old large gradients are gradually forgotten — a gradient of magnitude $|g_i|$ from $t - i$ steps ago contributes $\beta_2^{t-i} |g_i|$ to the maximum. The denominator $u_t$ is therefore the largest recent absolute gradient, discounted by age. This is simpler than Adam's $\sqrt{\hat{v}_t}$ denominator: there is no squaring, no square root, and crucially, no bias correction needed for the second moment, because $u_0 = 0$ does not bias the maximum (the max of $\{0, |g_1|\}$ is simply $|g_1|$).
Why this form: AdaMax has two advantages over Adam. First, the update bound is simpler: the paper notes that $|\Delta_t| \leq \alpha$ always holds for AdaMax (compared to the slightly more complex bound for Adam). Second, the absence of a bias correction for the second moment removes one potential source of numerical error. The tradeoff is that the $L^\infty$ norm is less smooth than the $L^2$ norm — $u_t$ is determined entirely by the single largest (discounted) gradient, whereas Adam's $\sqrt{\hat{v}_t}$ averages over all recent gradients. In practice, the paper recommends AdaMax with $\alpha = 0.002$, $\beta_1 = 0.9$, and $\beta_2 = 0.999$ as default settings, noting a slightly larger default $\alpha$ than Adam (0.002 vs. 0.001) — likely because the $L^\infty$ denominator tends to be larger than the $L^2$ denominator, so a larger $\alpha$ is needed to achieve comparable effective step sizes.
AdaMax is presented as an extension rather than a replacement, and the paper's primary empirical evaluations use Adam. The variant demonstrates that the moment-estimation framework naturally generalizes beyond the $L^2$ norm, and the elegant simplification in the $p \to \infty$ limit reinforces the theoretical coherence of the approach.
Complete Algorithm Walkthrough
Bringing all components together, here is what happens at each timestep $t$ in the Adam algorithm (Algorithm 1):
-
Increment timestep:
$t \leftarrow t + 1$. -
Compute stochastic gradient: Evaluate
$g_t = \nabla_\theta f_t(\theta_{t-1})$, the gradient of the current minibatch loss (or other stochastic objective) at the current parameter values$\theta_{t-1}$. This is a vector in$\mathbb{R}^d$. -
Update biased first moment:
$m_t = \beta_1 \cdot m_{t-1} + (1 - \beta_1) \cdot g_t$. This blends the previous momentum vector with the new gradient, producing a smoothed gradient estimate. -
Update biased second raw moment:
$v_t = \beta_2 \cdot v_{t-1} + (1 - \beta_2) \cdot g_t^2$, where$g_t^2$is the elementwise square. This updates the per-parameter variance estimate. -
Compute bias-corrected first moment:
$\hat{m}_t = m_t / (1 - \beta_1^t)$. This scales up the biased estimate to account for zero-initialization. -
Compute bias-corrected second moment:
$\hat{v}_t = v_t / (1 - \beta_2^t)$. Same correction for the second moment. -
Update parameters:
$\theta_t = \theta_{t-1} - \alpha \cdot \hat{m}_t / (\sqrt{\hat{v}_t} + \epsilon)$. This computes the per-parameter adaptive update: each parameter moves by an amount proportional to the smoothed gradient direction scaled inversely by the RMS gradient magnitude, bounded approximately by$\alpha$.
The algorithm repeats steps 1–7 until convergence (or a fixed number of iterations). The vectors $m_t$ and $v_t$ carry state between iterations, giving them memory of past gradients. The bias correction factors $1 - \beta_1^t$ and $1 - \beta_2^t$ are computed fresh each iteration from the current timestep $t$ and the fixed $\beta_1$, $\beta_2$ values — they do not require any additional state.
Computational cost: Each iteration requires computing the gradient (same as SGD), plus two elementwise vector operations for updating $m_t$ and $v_t (each a weighted average), two scalar divisions of vectors for bias correction, one elementwise square root and division for the final update, and one elementwise subtraction for the parameter update. All operations are $O(d)$ in the number of parameters and require storing only $m_t$ and $v_t$ (each size $d$) in addition to the parameters and gradient — hence the paper's claim of "little memory requirement" (specifically, $2d$ extra floats beyond the parameters and gradient themselves).
4. Key Insights and Innovations
Innovation 1: Reframing Adaptive Learning Rates as Moment Estimation
The paper's most distinctive conceptual move is recasting per-parameter learning rate adaptation — previously understood as a heuristic scaling trick — as a statistical estimation problem: maintaining running estimates of the first and second moments of the stochastic gradient and using their ratio as a normalized update direction. This reframing appears simple in retrospect, but it was not how the field thought about adaptive methods before Adam.
Prior work conceptualized the problem mechanistically. AdaGrad (Duchi et al., 2011) accumulated squared gradients in a sum — a bookkeeping operation. RMSProp (Tieleman & Hinton, 2012) replaced the sum with an exponential moving average — a forgetting mechanism. Both were described in terms of what they compute (a denominator that scales the learning rate) rather than what they estimate (the uncentered variance of the gradient). The shift from "we maintain a running average of squared gradients to normalize the step" to "we estimate $\mathbb{E}[g]$ and $\mathbb{E}[g^2]$ online and use their ratio as a signal-to-noise measure" is subtle but consequential.
Why does this reframing matter? Because it unifies momentum and adaptive scaling under a single statistical framework. In the moment-estimation view, $m_t$ and $v_t$ are not two separate mechanisms bolted together — they are two estimates of the same underlying gradient distribution, and their ratio $\hat{m}_t / \sqrt{\hat{v}_t}$ has a natural interpretation as the signal-to-noise ratio (SNR) of the gradient. When the SNR is high, the optimizer takes large steps (confident direction); when the SNR is low, it takes small steps (uncertain direction). This SNR interpretation, which the paper introduces in Section 2.1, provides a principled explanation for automatic annealing — the optimizer naturally slows down near convergence because the mean gradient shrinks while the gradient variance does not — without requiring any explicit learning rate schedule.
The moment-estimation framing also opens the door to exact bias correction. Once you think of $v_t$ as an estimator of $\mathbb{E}[g^2]$, it becomes natural to ask: is this estimator unbiased? The answer — no, because of zero-initialization — leads directly to the derivation in Section 3, where the expectation of the EMA is computed and the correction factor $1 / (1 - \beta_2^t)$ falls out algebraically. This is fundamentally different from how RMSProp's denominator was understood. In RMSProp, the fact that $v_t$ starts small was a problem to be worked around (by tuning $\alpha$ smaller or using a warmup). In Adam, it's a bias to be corrected — a statistical error with a known, computable magnitude.
This is a fundamental reframing, not an incremental improvement. It changed how the community thinks about adaptive optimization: from "we scale the learning rate by some function of past gradients" to "we maintain online estimates of gradient moments and make decisions based on the estimated SNR." The fact that AdaMax (Section 7.1) emerges naturally from generalizing the second moment to an $L^p$ norm and taking $p \to \infty$ — yielding an $L^\infty$ variant with no bias correction needed — demonstrates that the moment-estimation framework is generative: it suggests new algorithms, not just a single recipe.
Innovation 2: The Bias Correction as an Exact, Not Heuristic, Solution
The initialization bias correction — dividing $m_t$ by $1 - \beta_1^t$ and $v_t$ by $1 - \beta_2^t$ — is often described as a "bug fix" for zero-initialization. This undersells its intellectual contribution. What makes the bias correction a genuine innovation is that it is derived, not tuned — it emerges from computing the expectation of the exponential moving average under the stationarity assumption and recognizing that the sum of the EMA weights is $1 - \beta^t$, not 1. The correction is exact (up to the $\zeta$ term accounting for non-stationarity, which the paper argues is small by design) and requires no additional hyperparameters.
Before Adam, the standard approach to the zero-initialization problem was to avoid it heuristically. Practitioners using RMSProp with large $\beta_2$ values (0.999 or higher) would sometimes reduce the learning rate $\alpha$ during early training to compensate for the artificially small denominator, then increase it later — an ad-hoc warmup schedule. Alternatively, they might initialize $v_0$ to a non-zero value based on prior knowledge of gradient scales, which trades one source of bias for another. Both approaches require tuning. Adam's insight is that the bias has a known, computable magnitude — $1 - \beta^t$ — and can be exactly neutralized by division.
This is significant beyond the specific correction factor because it establishes a principle: when using exponential moving averages as online estimators, initialization bias is not an implementation detail to be papered over — it's a statistical error to be quantified and corrected. The paper makes this principle explicit by deriving the bias in closed form (Section 3) and then empirically demonstrating its importance in Figure 4. The results there show that the gap between Adam (with bias correction) and RMSProp-with-momentum (without) is largest precisely when $\beta_2$ is closest to 1 — the regime where the bias is most severe — and is most pronounced in early epochs before $1 - \beta_2^t$ has had time to naturally approach 1. This is not a small effect: at $t = 1$ with $\beta_2 = 0.999$, the uncorrected $v_1$ underestimates $\mathbb{E}[g^2]$ by a factor of 1000, making the effective step size 31.6× larger than intended. The VAE training diverges under these conditions without bias correction.
The conceptual contribution is that Adam treats the optimization algorithm itself as a statistical estimator subject to bias-variance tradeoffs, not just a procedural update rule. This perspective — the optimizer as an estimator — was not prominent in prior work on adaptive methods and represents a fundamental shift in how to design and analyze optimization algorithms for stochastic objectives. The subsequent success of Adam (and the widespread adoption of bias correction in derivative optimizers like AdamW) validates this shift.
Innovation 3: Synthesizing Sparse-Gradient and Non-Stationary Robustness in One Algorithm
The paper's most practically consequential contribution is demonstrating that a single algorithm can simultaneously handle sparse gradients (AdaGrad's strength) and non-stationary objectives (RMSProp's strength) without compromising either. Prior to Adam, these were treated as separate capabilities requiring different algorithmic choices. If you had sparse features, you used AdaGrad and accepted that learning would eventually stall. If you had non-stationary objectives (dropout, changing data distributions), you used RMSProp and accepted the initialization instability and lack of theoretical grounding. The field lacked a unified solution.
Adam achieves this unification through a specific combination of design choices that each serve a distinct purpose, none of which were new individually, but whose combination was non-obvious:
- The exponential moving average for
$v_t$(inherited from RMSProp) provides forgetting, so the denominator adapts to changing gradient scales rather than growing monotonically. This handles non-stationarity. - The large
$\beta_2$(0.999, giving a ~1000-step effective window) provides the stable, long-horizon variance estimation needed for sparse gradients — parameters that update infrequently still get meaningful denominators because the EMA remembers their history. - The bias correction makes large
$\beta_2$viable by preventing the denominator from being catastrophically small in early training. Without bias correction, large$\beta_2$causes instability (Figure 4); with it, large$\beta_2$is benign and beneficial. - The momentum term
$m_t$(with$\beta_1 = 0.9$) provides gradient smoothing that helps in all regimes but is not tied to the sparsity or stationarity properties — it's a general-purpose accelerator that the adaptive denominator makes safer by bounding per-parameter step sizes.
The intellectual contribution here is not any single mechanism but the recognition that sparse-gradient robustness and non-stationarity robustness place conflicting demands on the denominator (long memory for sparsity vs. short memory for stationarity), and that the bias correction resolves this conflict by making long memory safe. AdaGrad's denominator has infinite memory — it's robust to sparsity but fails on non-stationarity. RMSProp's denominator has finite memory — it handles non-stationarity but, without bias correction, becomes unstable when the memory is long enough for sparsity. Adam's denominator has long-but-finite memory with exact initialization correction, satisfying both requirements simultaneously.
The empirical validation of this synthesis appears across the paper's experiments: on the sparse IMDB bag-of-words task (Figure 1), Adam matches AdaGrad's convergence rate while SGD with momentum lags far behind — confirming that Adam inherits AdaGrad's sparse-gradient advantage. On multi-layer networks with dropout (Figure 2), Adam outperforms both AdaGrad and RMSProp — confirming that the synthesis beats either predecessor alone when both non-stationarity (from dropout) and parameter heterogeneity (from deep networks) are present. This is an incremental synthesis — the pieces existed — but the integration was non-obvious and the result has proven extraordinarily robust across a decade of deep learning practice.
Innovation 4: The Effective Step Size Analysis and the Trust-Region Interpretation
The analysis in Section 2.1 — establishing that the effective step size $\Delta_t = \alpha \cdot \hat{m}_t / \sqrt{\hat{v}_t}$ is approximately bounded by $\alpha$ in typical scenarios — is more than a mathematical property. It is a usability insight that changes how practitioners relate to the learning rate hyperparameter.
In standard SGD, $\alpha$ has an obscure relationship to the actual parameter changes. The update magnitude is $\alpha |g_t|$, which depends on the gradient scale, which depends on the loss function's scaling, the batch size, the network architecture, and the current point in parameter space. Tuning $\alpha$ requires trial and error because its effect is mediated by unknown quantities. The same $\alpha$ that works at initialization may be catastrophically large or uselessly small after a few epochs as the gradient scale changes.
Adam's step size analysis provides a different relationship: $|\Delta_t| \lessapprox \alpha$. The learning rate directly controls (approximately bounds) the maximum distance any parameter can move in a single update, regardless of the gradient scale, the network architecture, or the training progress. The paper calls this a "trust region" — the optimizer won't take steps larger than $\alpha$ because the gradient estimate doesn't justify moving further. This makes $\alpha$ interpretable: if you have prior knowledge that good parameters lie within some range (e.g., weight initialization schemes place initial weights in $[-0.1, 0.1]$), you can set $\alpha$ to a fraction of that range ($0.001$ is $1\%$ of $0.1$), and Adam will respect this bound.
The trust-region interpretation is not just convenient — it connects Adam to a broader optimization principle (trust-region methods) that had previously been applied mainly in second-order and quasi-Newton methods. First-order stochastic methods were not typically analyzed through this lens. By showing that the $\hat{m}_t / \sqrt{\hat{v}_t$ ratio naturally produces a trust-region effect, the paper bridges first-order stochastic optimization and classical trust-region theory, providing a conceptual anchor for why adaptive methods work.
This is a conceptual innovation rather than an algorithmic one: it doesn't change what Adam computes, but it changes how we understand what it computes and how we set its hyperparameters. The fact that the recommended default $\alpha = 0.001$ has proven remarkably robust across architectures and tasks — from the shallow networks of 2015 to modern transformers — supports the claim that this interpretation captures something real about Adam's behavior, not just a mathematical coincidence.
Innovation 5: The Regret Analysis Unifying Adaptive Methods and Sparsity
The theoretical contribution in Section 4 and the Appendix — establishing an $O(\sqrt{T})$ regret bound with explicit dependence on gradient sparsity — is significant not for the asymptotic rate (which matches known results) but for how the bound characterizes Adam's advantage over non-adaptive methods.
The key term in the regret bound (Theorem 4.1) is $\sum_{i=1}^d \|g_{1:T,i}\|_2$, the sum over parameters of the $L^2$ norm of each parameter's gradient sequence. For non-adaptive methods like SGD, the analogous regret bound depends on $\sum_{i=1}^d \sqrt{T} G_\infty = d G_\infty \sqrt{T}$, which grows linearly with the number of parameters $d$. For Adam (and AdaGrad), the dependence is through $\sum_{i=1}^d \|g_{1:T,i}\|_2$, which can be substantially smaller when gradients are sparse. Specifically, if most gradient components are zero at most timesteps, then $\|g_{1:T,i}\|_2$ is proportional to $\sqrt{T_i}$ where $T_i$ is the number of times parameter $i$ receives a non-zero gradient, not the total number of timesteps $T$. The paper notes that this can yield $O(\log d \sqrt{T})$ regret in sparse regimes, compared to $O(\sqrt{dT})$ for non-adaptive methods — an exponential improvement in the dimension dependence.
This result was known for AdaGrad (Duchi et al., 2011). What Adam's analysis adds is showing that the same sparsity benefit holds even with exponential moving averages (finite memory) rather than cumulative sums (infinite memory). This is non-trivial because the EMA weights $\beta_2^{t-i}$ mean that old gradients are forgotten, which could in principle hurt the algorithm's ability to accumulate sparse signal. The proof (in particular Lemma 10.4) demonstrates that the EMA's forgetting does not degrade the sparsity-dependent bound, provided $\beta_2$ is close enough to 1 that the effective memory is long relative to the sparsity pattern.
The theoretical analysis also requires decaying $\beta_{1,t}$ toward zero over time — specifically $\beta_{1,t} = \beta_1 \lambda^{t-1}$ with $\lambda \in (0, 1)$. This is an interesting connection to empirical practice: Sutskever et al. (2013) had observed that reducing the momentum coefficient near the end of training improved convergence, but this was an empirical finding without theoretical justification. Adam's analysis provides that justification: decaying momentum is required for the regret bound because constant momentum would cause the optimizer to overshoot and oscillate around the optimum, accumulating linear regret. This is a rare example of theory postdicting empirical practice and reinforcing it with a formal guarantee.
The limitation, which the paper acknowledges, is that the analysis assumes convexity and does not apply to the non-convex settings where Adam is primarily used. The regret bound is therefore a sanity check — a guarantee that the algorithm behaves reasonably in the tractable case — rather than a complete characterization of its behavior. This is standard for optimization theory in deep learning (most practical optimizers lack non-convex convergence guarantees), but it's important not to overstate the theoretical contribution. The value is in showing that Adam inherits the theoretical benefits of adaptive methods (sparsity-dependent convergence) while extending them to include momentum and finite memory, neither of which AdaGrad's original analysis covered.
This is a theoretical advance of moderate significance — it doesn't prove anything about Adam's deep learning performance, but it establishes that the algorithm is mathematically coherent and connects it to the existing theory of adaptive optimization in a principled way. The combination of finite-memory adaptation (for non-stationarity) with sparsity-aware regret (for high-dimensional problems) was not present in prior analyses, making this a genuine, if bounded, theoretical contribution.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. The paper evaluates on three standard benchmarks across different model types: MNIST (handwritten digit classification, 28×28 grayscale images, 10 classes) for logistic regression and multi-layer neural networks; IMDB movie reviews (Maas et al., 2011) pre-processed into bag-of-words feature vectors with the 10,000 most frequent words for sparse logistic regression; and CIFAR-10 (natural images, 32×32 color, 10 classes) for convolutional neural networks. The paper does not explicitly describe train/validation/test splits, but follows standard practice of the era: MNIST uses 60,000 training and 10,000 test images; CIFAR-10 uses 50,000 training and 10,000 test images; IMDB uses 25,000 training and 25,000 test reviews. All experiments report training cost (negative log likelihood or cross-entropy loss) rather than test-set metrics — the paper's focus is on optimization speed, not generalization.
-
Base model(s). The experiments span three model families of increasing complexity: (1) Logistic regression — a convex linear model classifying 784-dimensional MNIST pixel vectors or 10,000-dimensional IMDB bag-of-words vectors, chosen because its convexity eliminates local-minimum confounds and allows clean comparison of optimizer convergence properties independent of model architecture effects. (2) Multi-layer neural networks — two fully connected hidden layers with 1000 rectified linear units (ReLU) each, trained on MNIST, chosen as a standard non-convex benchmark that is deep enough to expose optimizer differences but small enough for rapid iteration. (3) Convolutional neural networks — an architecture with three alternating stages of 5×5 convolution filters and 3×3 max pooling with stride 2, followed by a fully connected layer of 1000 ReLU hidden units, trained on whitened CIFAR-10 images, chosen because weight sharing in CNNs naturally produces heterogeneous gradient magnitudes across layers, directly testing Adam's per-parameter adaptation claim. The paper uses "the same parameter initialization when comparing different optimization algorithms," though the specific initialization scheme (e.g., Glorot uniform, Gaussian) is not stated.
-
Metrics. The primary metric throughout is training cost (negative log likelihood for logistic regression; cross-entropy loss for neural networks), plotted against two x-axis alternatives: iterations over the entire dataset (epochs) and, in some comparisons, wall-clock time. This choice of training cost rather than test accuracy is deliberate — the paper's goal is to measure optimization speed (how fast the algorithm reduces the objective), not generalization quality. The minibatch size is fixed at 128 across all experiments, making "iterations over entire dataset" directly comparable across optimizers (one epoch = dataset size / 128 iterations). For the VAE bias-correction experiment, the metric is the loss after a fixed number of epochs (10 and 100), evaluated across a grid of hyperparameter settings.
-
Baselines. The paper compares Adam against several established and contemporary optimizers: (1) SGD with Nesterov momentum (Sutskever et al., 2013) — the strongest momentum variant of SGD available at the time, using accelerated gradient descent with a correction factor that improves convergence over standard momentum. (2) AdaGrad (Duchi et al., 2011) — the per-parameter adaptive method that accumulates squared gradients in a sum, providing the sparse-gradient baseline against which Adam's forgetting property is evaluated. (3) RMSProp (Tieleman & Hinton, 2012) — the exponential-moving-average adaptive method without momentum, representing the non-stationary baseline. For the multi-layer network experiments, (4) AdaDelta (Zeiler, 2012) is added as an additional adaptive method that addresses AdaGrad's diminishing learning rates without requiring a global learning rate. For the deterministic-cost neural network comparison, (5) Sum-of-Functions Optimizer (SFO) (Sohl-Dickstein et al., 2014) is included as a quasi-Newton minibatch method that claimed state-of-the-art performance at the time. For the bias-correction ablation, the relevant baseline is RMSProp with momentum (which is equivalent to Adam without bias correction), implemented by removing the
$1 - \beta^t$divisors from Algorithm 1. -
Generation budget / compute accounting. Compute is measured in two complementary ways: (1) Iterations over the entire dataset (epochs) — the number of full passes through the training data, which is a hardware-independent measure of algorithmic data efficiency. Since all methods use the same minibatch size of 128, one epoch represents the same number of gradient computations for all optimizers. (2) Wall-clock time — measured in seconds, reported specifically for the SFO comparison in Figure 2(b), where the paper notes that "due to the cost of updating curvature information, SFO is 5-10× slower per iteration compared to Adam." All methods use first-order gradient information only (except SFO, which approximates second-order curvature), so the per-iteration computational cost is dominated by the gradient computation itself, which is identical across SGD, AdaGrad, RMSProp, AdaDelta, and Adam. The additional cost of Adam's moment updates is
$O(d)$elementwise operations — negligible compared to the forward-backward pass that computes$g_t$. There is no generation budget in the sense of later LLM work; the relevant budget is the number of gradient evaluations (minibatch iterations). -
Cross-validation / statistical protocol. The paper states that hyperparameters "are searched over a dense grid and the results are reported using the best hyper-parameter setting" for each optimizer. For the logistic regression and CNN experiments, the learning rate
$\alpha$is decayed according to$\alpha_t = \alpha / \sqrt{t}$to match the theoretical analysis. For the bias-correction experiment (Section 6.4), a systematic grid search is performed:$\beta_1 \in [0, 0.9]$,$\beta_2 \in [0.99, 0.999, 0.9999]$, and$\log_{10}(\alpha) \in [-5, \ldots, -1]$. Results are reported for both 10 epochs and 100 epochs of VAE training. The paper does not describe cross-validation or multiple random seeds — results appear to be from single training runs with the best-found hyperparameters. This is a limitation: without error bars or multiple trials, the reliability of small performance differences (e.g., Adam vs. SGD on CNNs in Figure 3) is difficult to assess.
Main Quantitative Results
Logistic Regression: Convex Setting
The paper first validates Adam on a convex problem where convergence behavior is well-understood theoretically, using L2-regularized multi-class logistic regression on MNIST and IMDB.
MNIST (dense features). Figure 1 (left) plots training cost (negative log likelihood) against iterations over the entire dataset. The headline finding:
"Adam yields similar convergence as SGD with momentum and both converge faster than Adagrad."
Specifically, Adam and SGD with Nesterov momentum track each other closely, both reaching a training cost of approximately 0.3 after 45 epochs. AdaGrad converges more slowly, ending at a higher cost around 0.4 after the same number of epochs. This is expected behavior on dense features — AdaGrad's denominator accumulates squared gradients monotonically, causing the effective learning rate to decay even when the parameters haven't converged, while Adam's exponential forgetting prevents premature step size collapse.
IMDB (sparse features). Figure 1 (right) shows the critical sparse-gradient test. The 10,000-dimensional bag-of-words vectors are highly sparse (most words don't appear in any given review), and 50% dropout noise is applied to the input features (Wang & Manning, 2013). The results:
"Adagrad outperforms SGD with Nesterov momentum by a large margin both with and without dropout noise. Adam converges as fast as Adagrad."
After 160 epochs, AdaGrad and Adam both reach training costs around 0.22–0.23, while SGD with Nesterov momentum lags substantially at approximately 0.28. RMSProp with dropout (shown in the same figure) performs similarly to AdaGrad and Adam, consistent with its known sparse-gradient handling.
This pair of results demonstrates the synthesis claim: on dense features, Adam inherits SGD-with-momentum's good convergence (which AdaGrad lacks due to its decaying denominator); on sparse features, Adam inherits AdaGrad's robustness (which SGD lacks due to its uniform learning rate). Adam matches the best performer in each regime — it doesn't sacrifice one capability for the other.
Connection to theory. The paper notes: "Adam with $1/\sqrt{t}$ decay on its stepsize should theoretically match the performance of Adagrad." The IMDB result confirms this: the $\alpha_t = \alpha / \sqrt{t}$ schedule used in these experiments ensures that as $\beta_2 \to 1$, Adam approaches AdaGrad exactly (Section 5), and the empirical convergence rate matches.
Multi-Layer Neural Networks: Non-Convex Setting with Dropout
The paper next evaluates on 2-hidden-layer ReLU networks (1000 units each) on MNIST, a standard non-convex benchmark where optimizer differences become more pronounced. Two sub-experiments are reported: one with dropout stochastic regularization (Figure 2a) and one with a deterministic cost function (Figure 2b).
Dropout regularization (Figure 2a). This is the most comprehensive comparison, pitting Adam against AdaGrad, RMSProp, SGD with Nesterov momentum, and AdaDelta:
"Adam shows better convergence than other methods."
At 200 epochs, Adam reaches a training cost below $10^{-1}$ (the y-axis is log-scale), while SGD with Nesterov momentum converges to approximately the same level but takes roughly 50 more epochs to get there. AdaGrad, RMSProp, and AdaDelta all converge more slowly, with AdaGrad showing the expected plateau behavior. The margin is not enormous — all methods eventually reach similar cost levels — but Adam is consistently fastest.
Deterministic cost function (Figure 2b). This comparison is against SFO (Sohl-Dickstein et al., 2014), a quasi-Newton method that was state-of-the-art for training multi-layer networks at the time. Two x-axes are shown: iterations over the dataset (left/linear scale) and wall-clock time (log scale on right). The results:
"Adam makes faster progress in terms of both the number of iterations and wall-clock time."
On the iterations axis, Adam's cost drops below $10^{-2}$ by roughly 50 epochs, while SFO requires approximately 100–150 epochs. On the wall-clock axis, the gap is even larger because SFO has per-iteration overhead for curvature estimation. The paper quantifies this:
"SFO is 5-10× slower per iteration compared to Adam, and has a memory requirement that is linear in the number minibatches."
This is a practically significant result: Adam achieves better optimization in less absolute time while using constant memory ($O(d)$) versus SFO's $O(\text{minibatches} \times d)$. The paper also notes that SFO "assumes deterministic subfunctions, and indeed failed to converge on cost functions with stochastic regularization" — i.e., SFO cannot handle dropout, making it inapplicable to a major class of regularization techniques that were standard by 2015. Adam has no such limitation.
Interpretation. The multi-layer network results establish that Adam's advantages are not limited to convex problems. On non-convex objectives with stochastic regularization (dropout), Adam outperforms all tested alternatives, including the specialized quasi-Newton method SFO. The SFO comparison is particularly revealing: Adam achieves better optimization with lower computational cost and without the deterministic-subfunction assumption. This supports the paper's claim that Adam is "well-suited to a wide range of non-convex optimization problems."
Convolutional Neural Networks: Heterogeneous Gradient Scales
The CNN experiments on CIFAR-10 test Adam's claim of handling different gradient scales across layers. The architecture (three conv-pool stages followed by a fully connected layer) naturally creates heterogeneous gradients because weight sharing in convolutional layers produces gradient magnitudes that differ systematically from those in fully connected layers.
Early training (Figure 3, left). The first three epochs reveal:
"both Adam and Adagrad make rapid progress lowering the cost in the initial stage of the training"
Adam (both with and without dropout) and AdaGrad reduce the training cost from roughly 2.5 to near 1.0 within three epochs, while SGD with Nesterov momentum starts slower, reaching approximately 1.5–2.0 by epoch three. This is consistent with the per-parameter adaptation property: early in training, gradient magnitudes vary widely across layers, and adaptive methods quickly normalize them, while SGD's uniform learning rate must be small enough not to destabilize the largest-gradient layer, slowing progress on smaller-gradient layers.
Full training (Figure 3, right). Over 45 epochs, the ranking shifts:
"Adam and SGD eventually converge considerably faster than Adagrad for CNNs."
The paper diagnoses the reversal:
"We notice the second moment estimate
$\hat{v}_t$vanishes to zeros after a few epochs and is dominated by the$\epsilon$in algorithm 1."
By "vanishes to zeros," the authors mean that $\hat{v}_t$ becomes so large that $\sqrt{\hat{v}_t} \gg \epsilon$ is no longer true — rather, the effective denominator becomes dominated by the $\epsilon = 10^{-8}$ constant, making the per-parameter scaling effectively uniform. At that point, AdaGrad's monotonically accumulated denominator has grown without bound, and the effective learning rate $\alpha / \sqrt{\hat{v}_t}$ is approximately $\alpha / \epsilon$, which is tiny. This stalls learning. Adam's exponential moving average, by contrast, forgets old gradient magnitudes, allowing the denominator to adapt to the current (smaller) gradient scale and maintain a meaningful learning rate throughout training.
Adam vs. SGD on CNNs. The paper characterizes Adam's advantage as "marginal improvement over SGD with momentum." In Figure 3 (right), Adam (with dropout) reaches a training cost slightly below $10^{-1}$ by epoch 45, while SGD with Nesterov momentum (with dropout) reaches approximately the same level but slightly slower. The gap is much narrower than in the multi-layer network experiments. The paper attributes this to the fact that:
"reducing the minibatch variance through the first moment is more important in CNNs and contributes to the speed-up"
In CNNs, the primary challenge is not heterogeneous gradient scales (Adam's main strength) but gradient variance due to the small effective minibatch size relative to the complexity of the convolutional weight sharing. Momentum helps with this, and both Adam and SGD-Nesterov have momentum, so the gap narrows.
Practical benefit. The paper emphasizes a different advantage for Adam on CNNs:
"it adapts learning rate scale for different layers instead of hand picking manually as in SGD."
With SGD, practitioners often set different learning rates for convolutional layers (smaller) and fully connected layers (larger) through manual tuning. Adam eliminates this manual step — it automatically infers appropriate per-layer scales from the gradient statistics. The "marginal" accuracy improvement understates the practical value: achieving the same performance without per-layer learning rate tuning is a significant reduction in human effort.
Dropout effect. Across both Figure 3 panels, adding dropout (dashed vs. solid lines) consistently improves or maintains convergence speed for Adam and SGD, while AdaGrad with dropout shows a slight disadvantage compared to AdaGrad without dropout in the early epochs. This is consistent with Adam's design for non-stationary objectives: dropout creates stochasticity that changes the effective gradient distribution, and Adam's forgetting mechanism adapts to this better than AdaGrad's permanent accumulation.
Bias-Correction Ablation (Variational Autoencoder)
Section 6.4 directly tests the bias correction mechanism by comparing Adam (with bias correction, red lines in Figure 4) against the identical algorithm without bias correction (green lines) — which is equivalent to RMSProp with momentum — across a grid of hyperparameters when training a VAE with the same architecture as Kingma & Welling (2013): a single 500-unit hidden layer with softplus nonlinearities and a 50-dimensional spherical Gaussian latent variable.
Headline result:
"Adam performed equal or better than RMSProp, regardless of hyper-parameter setting."
This is a strong claim supported by Figure 4, which shows loss (y-axis, lower is better) for each combination of $\beta_1$ (columns: 0 and 0.9), $\beta_2$ (rows: 0.99, 0.999, 0.9999), and $\log_{10}(\alpha)$ (x-axis from -5 to -1). The red curves (with bias correction) lie at or below the green curves (without bias correction) across essentially all 24 hyperparameter combinations shown (2 $\beta_1$ × 3 $\beta_2$ × 4 $\alpha$ values per panel, with 2 epochs × 2 panels = 48 total comparisons).
Effect of $\beta_2$: The gap between bias-corrected and uncorrected is largest precisely when $\beta_2$ is closest to 1:
"Values of
$\beta_2$close to 1, required for robustness to sparse gradients, results in larger initialization bias; therefore we expect the bias correction term is important in such cases of slow decay, preventing an adverse effect on optimization."
At $\beta_2 = 0.9999$ (bottom row), the uncorrected version shows severe instability — the loss is dramatically higher than the corrected version, and the curves are noisy, indicating that training partially diverges. This is the clearest empirical evidence for the theoretical derivation in Section 3: at $t = 1$, $\beta_2^t = 0.9999$, so $1 - \beta_2^t = 0.0001$, meaning $v_1$ underestimates $\mathbb{E}[g^2]$ by a factor of 10,000. The effective step size is $\sqrt{10000} = 100$ times larger than intended — easily enough to cause the instability visible in Figure 4.
Effect of $\beta_1$: The bias correction matters more when $\beta_1 = 0.9$ (right column) than when $\beta_1 = 0$ (left column), because with momentum, the first few gradient steps have an outsized influence on the trajectory, making early-stage instability particularly damaging.
After 10 vs. 100 epochs (Figure 4a vs. 4b): The gap between corrected and uncorrected narrows somewhat after 100 epochs (Figure 4b) compared to 10 epochs (Figure 4a), particularly at moderate $\beta_2$ values (0.99, 0.999). This is expected: $1 - \beta_2^t$ approaches 1 naturally over time, so the uncorrected estimate becomes less biased. However, at $\beta_2 = 0.9999$, the gap remains substantial even after 100 epochs because $\beta_2^{100 \times \text{iterations per epoch}}$ may still be non-negligible depending on the dataset size and minibatch count. The paper's characterization:
"this was more apparent towards the end of optimization when gradients tends to become sparser as hidden units specialize to specific patterns"
suggests that the bias problem is not just an early-training issue — as hidden units specialize (producing sparser gradient patterns), the need for stable second-moment estimation with large $\beta_2$ persists or increases throughout training.
AdaMax: The $L^\infty$ Variant (No Dedicated Experiments)
Section 7.1 derives AdaMax but does not report separate experiments for it. The paper provides recommended defaults ($\alpha = 0.002$, $\beta_1 = 0.9$, $\beta_2 = 0.999$) but all empirical comparisons use standard Adam. The AdaMax extension is presented as a conceptual contribution — demonstrating that the moment-estimation framework generalizes — rather than an empirically validated improvement. This is a gap: without head-to-head AdaMax vs. Adam comparisons on the same benchmarks, the practical value of the $L^\infty$ variant remains unquantified.
Ablation Studies and Robustness Checks
The paper's ablation-like analyses are embedded within the main experiments rather than presented in a separate section. The key ones are:
-
Bias correction on/off (Section 6.4, Figure 4): Removing the
$1 - \beta^t$correction divisors from Algorithm 1 — which reduces Adam to RMSProp with momentum — causes substantial performance degradation, especially at large$\beta_2$and early in training. This is the central ablation demonstrating that the bias correction is not cosmetic. At$\beta_2 = 0.9999$, the uncorrected variant shows catastrophic instability (loss values far above the corrected variant), confirming the theoretical derivation that the initialization bias is severe when decay is slow. The effect persists across all tested$\alpha$and$\beta_1$values — no hyperparameter setting rescues the uncorrected version. -
$\alpha$decay schedule (across experiments): The paper uses$\alpha_t = \alpha / \sqrt{t}$decay for the logistic regression and CNN experiments (matching Theorem 4.1's requirement) but does not systematically ablate this choice against constant$\alpha$. The VAE experiment uses constant$\alpha$. Results are not directly comparable across these experiments because they involve different models, but the fact that Adam works well with both constant and decaying$\alpha$suggests robustness. The paper does not quantify the performance difference between the two schedules on any single task. -
Momentum decay in theory only (Theorem 4.1): The theoretical analysis requires
$\beta_{1,t} = \beta_1 \lambda^{t-1}$(decaying momentum). The paper cites Sutskever et al. (2013) as empirical support but does not run experiments with decaying$\beta_1$. This is a gap between theory and empirical validation — the$\beta_{1,t}$schedule that the regret bound requires is not tested in the experiments, which all use constant$\beta_1$. The paper is transparent about this but does not close the loop. -
Second moment aggregation:
$L^2$(Adam) vs.$L^\infty$(AdaMax): AdaMax is derived but not empirically compared to Adam. The paper does not report whether the$L^\infty$denominator improves, matches, or degrades performance relative to the$L^2$denominator on any benchmark. The claim that AdaMax produces a "simpler bound" ($|\Delta_t| \leq \alpha$always) is a theoretical observation, not an empirical finding. -
$\epsilon$value: The paper uses$\epsilon = 10^{-8}$throughout but does not ablate this choice. The CNN diagnosis — that$\hat{v}_t$becomes "dominated by$\epsilon$" — suggests that$\epsilon$plays a non-trivial role in practice and that its value could interact with the effective learning rate. An ablation over$\epsilon \in [10^{-7}, 10^{-9}, 10^{-10}]$would clarify whether the default is critical or whether Adam is robust to this choice. -
Minibatch size: All experiments use a minibatch size of 128. The paper does not investigate how Adam's performance changes with smaller or larger minibatch sizes. This matters because the gradient noise level (and thus the optimal
$\beta_1$,$\beta_2$settings) depends on the minibatch size. The claim that Adam "typically requires little tuning" would be strengthened by showing that the same hyperparameters work across minibatch sizes, but this is not tested. -
SFO failure with dropout: The paper reports that SFO "failed to converge on cost functions with stochastic regularization" without quantitative details. This is presented as a qualitative observation rather than a systematic comparison. Figure 2b shows SFO only on the deterministic cost function. The claim that SFO fails with dropout is credible (SFO assumes deterministic subfunctions) but is not supported by a figure or table.
Critical Assessment
Claim 1: Adam combines the advantages of AdaGrad (sparse gradients) and RMSProp (non-stationary objectives).
What was tested: The IMDB experiment (Figure 1, right) demonstrates that Adam matches AdaGrad on sparse bag-of-words features, while SGD-Nesterov lags substantially. The multi-layer network with dropout experiment (Figure 2a) demonstrates that Adam outperforms AdaGrad on non-stationary objectives (dropout-regularized training), confirming that Adam does not suffer from AdaGrad's diminishing learning rate problem.
What was not tested: The paper never evaluates Adam on a problem that is simultaneously sparse and non-stationary. The IMDB task has sparse features but uses logistic regression, which is convex and stationary (aside from minibatch sampling). The dropout experiments have non-stationarity from stochastic regularization but use MNIST, which has dense pixel features. The claim that Adam "combines" both advantages is supported by separate experiments showing each advantage in isolation, but not by a single experiment demonstrating both simultaneously. A natural test — training a deep model on sparse text features with dropout — is absent.
Assessment: The claim is supported in parts but not demonstrated holistically. Adam does match AdaGrad on sparse features and does outperform AdaGrad on non-stationary objectives, but the paper infers rather than demonstrates that both benefits coexist in a single training run. Given that the subsequent decade of practice has thoroughly validated this claim, this is more a limitation of the experimental design than a substantive weakness.
Claim 2: The initialization bias correction is essential for stable training with large $\beta_2$.
What was tested: Figure 4 provides a systematic grid search over $\beta_1$, $\beta_2$, and $\alpha$ comparing Adam (with correction) to RMSProp-with-momentum (without correction) on VAE training. At large $\beta_2$ (0.999, 0.9999), the uncorrected version shows substantially worse performance, especially in the first 10 epochs. The effect is consistent across all tested $\alpha$ values.
What was not tested: The VAE experiment uses a single architecture on a single dataset. The paper does not demonstrate that the bias correction matters on the MNIST, IMDB, or CIFAR-10 benchmarks used in Sections 6.1–6.3. It is possible that on those tasks, the default $\beta_2 = 0.999$ and $\alpha = 0.001$ with $1/\sqrt{t}$ decay would have worked adequately without bias correction because the learning rate decay provides a form of implicit compensation. The paper also doesn't test whether a simple heuristic — e.g., a fixed number of warmup steps with a smaller $\alpha$ — could substitute for the exact correction.
Assessment: The claim is well-supported for the specific VAE architecture tested, and the theoretical derivation (Section 3) provides a principled justification that should generalize. However, the magnitude of the practical benefit on the paper's main benchmarks is unknown, because those experiments all use Adam with bias correction and do not include an ablation. A skeptic could argue that the bias correction matters most when $\beta_2$ is very close to 1 (0.9999) and $\alpha$ is constant, and that under the $\alpha / \sqrt{t}$ decay used in most experiments, the initialization bias might be partially mitigated by the small initial $\alpha_t$. The paper does not rule this out.
Claim 3: Adam's effective step size is approximately bounded by $\alpha$, making the learning rate interpretable.
What was tested: The analysis in Section 2.1 derives the bound $|\Delta_t| \lessapprox \alpha$ from the properties of $\hat{m}_t / \sqrt{\hat{v}_t}$. The paper does not empirically verify this bound by, for example, tracking the actual maximum parameter change across training and comparing it to $\alpha$. The claim is presented as a mathematical property of the update rule, not as an empirical finding.
What was not tested: There is no experiment showing that practitioners can successfully set $\alpha$ based on prior knowledge of parameter scales, or that Adam's performance is less sensitive to $\alpha$ than SGD's is. The grid search over $\alpha$ in Figure 4 shows that Adam's performance does vary with $\alpha$ — it is not completely insensitive — but there is no direct comparison to SGD's $\alpha$-sensitivity on the same task.
Assessment: The claim is analytically correct (the bound follows from the algorithm's definition) but its practical significance is not empirically substantiated. The paper asserts that the bound makes $\alpha$ easier to set, but provides no user study, no sensitivity analysis comparing Adam and SGD across $\alpha$ ranges, and no example of a practitioner using prior knowledge to set $\alpha$. The decades of practice since 2015 have largely validated the claim — $\alpha = 0.001$ with Adam has become a standard default that works across many architectures — but the paper itself provides only the mathematical argument, not experimental evidence.
Claim 4: Adam achieves $O(\sqrt{T})$ regret with sparsity-dependent improvements over non-adaptive methods.
What was tested: This claim is purely theoretical (Theorem 4.1, Corollary 4.2). The paper does not run experiments that measure regret or test the theoretical bound's predictions. The logistic regression experiments (Figure 1) are consistent with the theory — Adam converges faster than SGD on sparse features and comparably on dense features — but do not constitute a direct test of the regret bound.
What was not tested: The analysis assumes convexity and a decaying $\beta_{1,t}$ schedule. The paper's deep learning experiments are non-convex and use constant $\beta_1$, so the theory and experiments operate in different regimes. The claim that the bound explains Adam's performance is therefore speculative — the theory provides a sanity check (the algorithm is well-behaved in the tractable case) but does not directly characterize its behavior in the settings where it is actually used.
Assessment: The theoretical claim is narrowly valid — the proof is mathematically sound under its stated assumptions — but its relevance to practice is not established. The paper acknowledges this implicitly by separating the convex experiments (Section 6.1, where theory and practice align) from the non-convex experiments (Sections 6.2–6.4, where theory does not apply). This is standard practice in optimization papers, and the theory serves its purpose as a rigorous characterization of worst-case behavior rather than a predictive model of deep learning performance. The paper does not overclaim here.
Claim 5: Adam is computationally efficient with little memory requirement.
What was tested: The paper reports that SFO is 5-10× slower per iteration than Adam (Figure 2b), confirming that Adam's per-iteration overhead is low compared to quasi-Newton methods. The memory claim follows analytically: Adam stores $m_t$ and $v_t$ (each size $d$) in addition to the parameters and gradient, so total memory is $3d$ floats vs. $2d$ for SGD with momentum (parameters + momentum buffer). The paper contrasts this with SFO's $O(\text{minibatches} \times d)$ memory.
What was not tested: No experiment directly measures Adam's memory usage against SGD's on a fixed hardware budget, or tests whether the $2d$ extra memory matters in practice. The wall-clock comparison in Figure 2b uses a single data point (SFO) on a single architecture — it does not compare Adam's per-iteration speed to SGD or AdaGrad on equal footing.
Assessment: The claim is analytically obvious (storing two additional vectors of size $d$ adds minimal overhead for typical $d$ in the millions to billions) but the empirical validation is thin. The SFO comparison demonstrates that Adam is fast relative to a known-expensive method, but doesn't establish that Adam's overhead over plain SGD is negligible — it almost certainly is (elementwise vector operations vs. forward-backward pass), but the paper doesn't quantify this.
Overall Strengths of the Experimental Design
The experiments follow a logical progression: convex → non-convex shallow → non-convex deep, with each stage testing a different aspect of the claimed advantages. The multiple x-axis choices (epochs and wall-clock time) provide complementary views of optimization speed. The hyperparameter grid search (with best results reported) is standard practice that gives each optimizer a fair chance.
The IMDB experiment is particularly well-designed: sparse bag-of-words features with dropout noise create a challenging, realistic test case that directly exercises AdaGrad's and Adam's claimed sparse-gradient advantages. Showing that Adam matches AdaGrad here while also outperforming AdaGrad on non-stationary dropout-regularized networks (Figure 2a) effectively demonstrates the synthesis claim.
The VAE bias-correction experiment (Figure 4) is the strongest single piece of evidence in the paper. The systematic grid search over three hyperparameters ($\beta_1$, $\beta_2$, $\alpha$), the comparison at two time horizons (10 and 100 epochs), and the clear visual separation between corrected and uncorrected curves make the case convincingly. The fact that the gap is largest where theory predicts (large $\beta_2$, early training) and that no hyperparameter setting rescues the uncorrected version demonstrates that the bias correction is necessary, not merely helpful.
Weaknesses and Missing Experiments
1. No test-set metrics. The paper reports only training cost, never generalization error. This is defensible for an optimization paper — the goal is to reduce the objective quickly, not to achieve state-of-the-art accuracy — but it means the paper cannot distinguish between an optimizer that genuinely finds better minima and one that overfits faster. The standard practice in later optimization papers (e.g., learning rate range tests, optimizer comparisons) is to report both training loss and test accuracy, because optimizers that converge faster on the training set sometimes generalize worse. The paper provides no evidence either way on Adam's generalization properties.
2. Single runs, no error bars. The paper does not report standard deviations, confidence intervals, or multiple random seeds. Given that stochastic optimization is inherently noisy (minibatch sampling, dropout, random initialization), the reported differences — especially the "marginal improvement" of Adam over SGD on CNNs — could be within the noise. Without error bars, it's impossible to assess whether Adam's advantage on Figure 3 (right) is statistically reliable or a single-run artifact.
3. No sensitivity analysis for $\beta_1$ and $\beta_2$ on main benchmarks. The VAE experiment sweeps $\beta_1$ and $\beta_2$, but the MNIST, IMDB, and CIFAR experiments use fixed defaults (0.9 and 0.999 respectively). The paper claims that Adam "typically requires little tuning," but this claim is not tested by varying $\beta_1$ and $\beta_2$ and showing that performance is robust. If Adam's performance degraded sharply when $\beta_1 = 0.95$ or $\beta_2 = 0.99$, the "little tuning" claim would be false. The paper simply doesn't provide the data to evaluate this.
4. AdaMax is not evaluated. A new algorithm variant (Section 7.1) is proposed with recommended hyperparameters and a theoretical analysis, but no experiments compare it to Adam. The reader cannot assess whether AdaMax is better, worse, or equivalent to Adam on any task. This is a significant omission — if AdaMax is not empirically validated, it's unclear why it's included beyond intellectual completeness.
5. No comparison to plain SGD (without momentum). All SGD baselines use Nesterov momentum. While this is a strong baseline (Nesterov momentum was the best SGD variant at the time), the paper never shows how plain SGD performs. This matters because one of Adam's claimed advantages is that it provides momentum automatically — but the experiments don't isolate the contribution of momentum vs. per-parameter adaptation by comparing Adam to both momentum and non-momentum baselines.
6. Fixed minibatch size throughout. All experiments use minibatch size 128. The gradient statistics that Adam estimates depend on the minibatch size (larger minibatches produce lower-variance gradient estimates), so the optimal $\beta_1$ and $\beta_2$ likely depend on minibatch size. The claim that the defaults are universally good is untested at other batch sizes.
7. Limited scale. The largest model tested is a 3-conv-layer CNN with one 1000-unit fully connected layer — tiny by modern standards. The paper was published in 2015 when these were reasonable experimental scales, but the claim that Adam scales to "large-scale high-dimensional machine learning problems" is not tested on truly large models (e.g., AlexNet-scale or larger). The SFO comparison hints at scalability issues for competing methods, but doesn't demonstrate Adam scaling to sizes where its claimed advantages would be most impactful.
8. No combination with learning rate schedules beyond $1/\sqrt{t}$ decay. The paper uses $\alpha_t = \alpha / \sqrt{t}$ decay for most experiments but doesn't test other common schedules (step decay, exponential decay, cosine annealing). The interaction between Adam's automatic annealing (via SNR) and explicit learning rate schedules is unexplored, leaving practitioners uncertain whether they should use a schedule with Adam or rely on the automatic mechanism.
9. SFO failure with dropout is stated but not shown. The paper claims SFO "failed to converge on cost functions with stochastic regularization" but provides no figure, table, or quantitative description. For a method that was a published, peer-reviewed optimizer (Sohl-Dickstein et al., ICML 2014), a failure claim should be substantiated with evidence. The paper's Figure 2b only shows SFO on the deterministic objective — the dropout failure is purely anecdotal.
10. No comparison to second-order methods beyond SFO. The paper mentions natural gradient descent (Amari, 1998) and the natural Newton method (Roux & Fitzgibbon, 2010) in the related work but doesn't compare to them empirically. While these methods have known scalability limitations that Adam is designed to avoid, a small-scale comparison demonstrating the performance gap would strengthen the case for Adam's design choices.
Conditional Nature of the Claims
The paper's central empirical claim — that Adam outperforms or matches the best alternative across diverse problems — must be understood with the following conditionals:
- On dense, convex problems (MNIST logistic regression): Adam matches SGD-Nesterov; both beat AdaGrad. The claim is "no worse than the best," not "strictly better."
- On sparse, convex problems (IMDB logistic regression): Adam matches AdaGrad; both beat SGD-Nesterov. Again, "no worse than the best."
- On non-convex problems with stochastic regularization (dropout MLP): Adam is strictly better than all tested alternatives. This is the strongest positive result.
- On deep convolutional networks: Adam shows marginal improvement over SGD-Nesterov; the main advantage is eliminating per-layer learning rate tuning. The accuracy claim is weak; the usability claim is plausible but not experimentally tested.
- On VAEs with large
$\beta_2$(bias correction test): Adam with correction dominates the uncorrected variant; the effect size depends on$\beta_2$and training duration. This is the most robustly demonstrated result.
The paper does not present evidence that Adam is always the best choice — rather, it shows that Adam is never the worst across a range of representative problems, and is sometimes the best by a significant margin. This "robust default" characterization is both more modest and more credible than a claim of universal superiority, and it aligns with Adam's subsequent adoption as the default optimizer for most deep learning training pipelines.
6. Limitations and Trade-offs
The Convergence Theory Does Not Apply to the Non-Convex Settings Where Adam Excels
The assumption or constraint. The regret analysis in Section 4 and Appendix 10.1 assumes that the sequence of objective functions $f_1(\theta), \ldots, f_T(\theta)$ is convex. Theorem 4.1 opens with "Assume that the function $f_t$ has bounded gradients" and the proof relies on Lemma 10.2, the standard convexity inequality $f(y) \geq f(x) + \nabla f(x)^T(y - x)$. The entire online convex optimization framework of Zinkevich (2003) that the analysis adopts is fundamentally a convex framework — the regret definition $R(T) = \sum_{t=1}^T [f_t(\theta_t) - f_t(\theta^*)]$ and the optimal comparator $\theta^* = \arg\min_{\theta \in \mathcal{X}} \sum_{t=1}^T f_t(\theta)$ only make sense when the functions are convex and a global optimum over the sum exists.
The paper is transparent about this disconnect. In Section 6.2, the authors state:
"Although our convergence analysis does not apply to non-convex problems, we empirically found that Adam often outperforms other methods in such cases."
This is an honest acknowledgment, but it means the theoretical results provide no guarantees about Adam's behavior in deep learning — the application domain where the algorithm has seen the widest adoption. The $O(\sqrt{T})$ regret bound, the sparsity-dependent improvement to $O(\log d \sqrt{T})$, and the requirement for decaying $\beta_{1,t}$ are all proven under convexity. None of these carry over to the multi-layer networks, CNNs, or VAEs tested in Sections 6.2–6.4.
The consequence. Without non-convex convergence guarantees, practitioners have no theoretical assurance that Adam will converge to a stationary point (a local minimum or saddle point), let alone a good local minimum. Standard failure modes in non-convex optimization — divergence, oscillation, getting stuck in poor local minima, or converging to sharp minima that generalize poorly — are not ruled out by the theory. The $O(\sqrt{T})$ regret bound says nothing about whether Adam reaches a point with small gradient norm in a non-convex landscape, because regret is not the appropriate metric there.
Furthermore, the theoretical requirement of decaying $\beta_{1,t} \to 0$ (specifically $\beta_{1,t} = \beta_1 \lambda^{t-1}$ with $\lambda \in (0, 1)$) is not followed in any of the paper's deep learning experiments. The experiments in Sections 6.2–6.4 all use constant $\beta_1 = 0.9$. This means that even if one were to extrapolate the convex theory to non-convex settings (which is not mathematically justified), the algorithm as actually deployed does not satisfy the conditions of its own regret bound. The theory and practice of Adam operate in different algorithmic regimes — constant momentum vs. decaying momentum — connected only by a citation to Sutskever et al. (2013) that "suggests reducing the momentum coefficient in the end of training can improve convergence." This is empirical precedent, not theoretical justification.
A specific practical concern: the theory requires $\beta_2^2 / \sqrt{\beta_2} < 1$ for the convergence proof to go through. With the default $\beta_1 = 0.9$ and $\beta_2 = 0.999$, this evaluates to $0.81 / \sqrt{0.999} \approx 0.81 / 0.9995 \approx 0.81 < 1$, so the condition holds. But the proof's other requirements — bounded gradients $\|\nabla f_t(\theta)\|_2 \leq G$, $\|\nabla f_t(\theta)\|_\infty \leq G_\infty$, and bounded parameter distances $\|\theta_n - \theta_m\|_2 \leq D$ — are not verified for the experimental settings and may not hold (particularly the bounded-gradient assumption, which can be violated with poor initialization or exploding gradients in deep networks).
What evidence exists in the paper. The gap between theory and experiments is visible in the paper's own structure: the logistic regression experiments (Section 6.1) follow the theory (convex problem, $\alpha / \sqrt{t}$ decay, and — though not shown — the $\beta_{1,t}$ decay could in principle be applied). The multi-layer network and CNN experiments (Sections 6.2–6.3) depart from the theory and are presented purely as empirical demonstrations. The paper provides no bridge — no convergence-to-a-stationary-point proof for the non-convex case, no analysis of whether Adam's updates satisfy the descent property needed for non-convex optimization, and no empirical measurement of gradient norms over training to verify that convergence to a stationary point occurs.
Mitigation status. The paper does not attempt to address this limitation. The theoretical analysis is presented as a self-contained result under convexity, and the authors explicitly mark the boundary (Section 6.2) rather than trying to extend the theory. The limitation is structural: the Zinkevich (2003) online convex optimization framework that the proof uses does not have a natural non-convex extension, and the paper does not propose one. In the subsequent literature, Reddi et al. (2018) demonstrated that Adam can fail to converge even on simple convex problems when the hyperparameters are not chosen carefully (specifically, when $\beta_{1,t}$ does not decay sufficiently), and more recent work has established non-convex convergence guarantees for corrected variants of Adam. But in this paper, the theoretical gap is unclosed. For a practitioner, this means the decision to use Adam on deep networks rests entirely on empirical validation (the paper's experiments and community experience), not on a guarantee of convergence.
The Effective Step Size Analysis and Trust-Region Interpretation Are Not Empirically Validated
The assumption or constraint. Section 2.1 derives an upper bound on the effective step size $|\Delta_t| = \alpha \cdot |\hat{m}_t / \sqrt{\hat{v}_t}|$ and argues that in typical scenarios, $|\Delta_t| \lessapprox \alpha$. The paper interprets this as "establishing a trust region around the current parameter value, beyond which the current gradient estimate does not provide sufficient information." From this, the paper draws a practical conclusion:
"This typically makes it relatively easy to know the right scale of
$\alpha$in advance. For many machine learning models, for instance, we often know in advance that good optima are with high probability within some set region in parameter space; it is not uncommon, for example, to have a prior distribution over the parameters."
The claim — that practitioners can set $\alpha$ based on prior knowledge of parameter scale and that Adam will respect this bound — is presented as a consequence of the mathematical analysis, but it is never tested empirically.
The consequence. Three separate claims are embedded here, and each lacks empirical support:
-
The bound actually holds in practice. The analysis assumes
$\epsilon = 0$and examines the ratio$\hat{m}_t / \sqrt{\hat{v}_t}$. But in the actual algorithm, the denominator is$\sqrt{\hat{v}_t} + \epsilon$, and when$\hat{v}_t$becomes very small (which the paper itself observes happening in CNN training — Section 6.3: "the second moment estimate$\hat{v}_t$vanishes to zeros after a few epochs and is dominated by the$\epsilon$"), the effective step size is$\alpha \cdot \hat{m}_t / \epsilon$, which can be much larger than$\alpha$. The trust region collapses precisely when the denominator becomes dominated by$\epsilon$— the regime where the per-parameter adaptation is most needed. -
The SNR interpretation corresponds to actual optimizer behavior. The paper calls
$\hat{m}_t / \sqrt{\hat{v}_t$a signal-to-noise ratio "with a slight abuse of terminology." But a true SNR would require$\sqrt{\hat{v}_t}$to measure noise (variance around the mean), while$\hat{v}_t$measures the second raw moment$\mathbb{E}[g^2]$, which includes both the squared mean (signal) and the variance (noise):$\mathbb{E}[g^2] = \mathbb{E}[g]^2 + \text{Var}(g)$. Near convergence, the signal$\mathbb{E}[g]$is small, so$\hat{v}_t$primarily measures noise — the SNR interpretation holds approximately. But far from convergence, when gradients are consistently large and in a consistent direction,$\hat{v}_t$is dominated by signal, and the ratio$\hat{m}_t / \sqrt{\hat{v}_t}$approaches$\pm 1$regardless of the actual noise level. This means the "automatic annealing" that the paper attributes to decreasing SNR may not engage until the optimizer is already near convergence — at which point many other learning rate schedules (step decay, cosine annealing) would also be reducing the step size. The paper provides no empirical demonstration that Adam's automatic annealing is superior to explicit scheduling. -
Practitioners can successfully set
$\alpha$based on prior knowledge. The paper recommends$\alpha = 0.001$as a default, but provides no evidence that this value was derived from prior knowledge of parameter scales rather than from a hyperparameter search. The grid search in Figure 4 covers$\log_{10}(\alpha) \in [-5, -1]$, i.e.,$\alpha \in [10^{-5}, 10^{-1}]$— a range spanning four orders of magnitude. The "best"$\alpha$varies by dataset, model, and the presence of dropout. There is no ablation showing that Adam's performance is less sensitive to$\alpha$than SGD's, or that a practitioner armed with a prior over parameters could have selected the right$\alpha$without the grid search.
What evidence exists in the paper. None. The trust-region interpretation and the SNR analysis are purely analytical. The paper does not:
- Track
$|\Delta_t|$over the course of training and compare it to$\alpha$. - Measure the SNR
$\hat{m}_t / \sqrt{\hat{v}_t}$and show that it decreases near convergence. - Compare Adam's sensitivity to
$\alpha$against SGD's sensitivity. - Provide an example of a practitioner setting
$\alpha$from prior parameter knowledge without a grid search. - Ablate
$\epsilon$to show that the bound$|\Delta_t| \lessapprox \alpha$holds when$\hat{v}_t$is not dominated by$\epsilon$.
Section 6.3 inadvertently provides counter-evidence: the observation that $\hat{v}_t$ becomes dominated by $\epsilon$ in CNN training means that the effective step size is controlled by $\alpha / \epsilon$, not by $\alpha$, and the trust region interpretation fails for that architecture. The paper does not address this contradiction.
Mitigation status. The paper does not acknowledge this as a limitation. The step size analysis is presented as a feature of the algorithm, and the interpretive claims about trust regions and SNR are stated as properties, not hypotheses requiring validation. The $\epsilon$ issue in CNNs is mentioned as an observation but its implications for the trust-region claim are not discussed. For a practitioner, this means the recommended default $\alpha = 0.001$ should be treated as an empirically found starting point, not as a value derived from the trust-region principle. The interpretive framework — while elegant — remains unvalidated.
The Empirical Validation Is Limited to Small-Scale Models and Three Datasets from 2014-2015
The assumption or constraint. The paper evaluates Adam on a specific set of benchmarks that, while diverse for their time, are small by any modern standard and are all drawn from the supervised image and text classification domains:
- Logistic regression on MNIST (784-dimensional dense features, 60K training examples) and IMDB (10,000-dimensional sparse BoW features, 25K training examples).
- Two-hidden-layer MLP (1000 units each, ~2M parameters) on MNIST.
- Three-conv-layer CNN (~1M parameters, exact count not specified but standard for the described architecture) on CIFAR-10 (50K training examples, 32×32 color images).
- Single-hidden-layer VAE (500 units, 50-dim latent) on an unspecified dataset — presumably MNIST or a similar binarized image dataset, following Kingma & Welling (2013).
The largest model tested has on the order of a few million parameters. The datasets have at most 60,000 training examples. Training runs last at most 200 epochs (MLP), 45 epochs (CNN), or 100 epochs (VAE). The abstract claims the method is "well suited for problems that are large in terms of data and/or parameters," but neither the data nor the parameter counts tested are large by the standards of 2015 (AlexNet had 60M parameters trained on 1.2M ImageNet examples), let alone by modern standards.
The paper also does not evaluate on several important problem types that were active research areas in 2015: recurrent neural networks (the paper cites Graves, 2013 on speech recognition with RNNs, but doesn't test Adam on sequence models), generative adversarial networks (which were just emerging; Goodfellow et al., 2014), or reinforcement learning. The algorithm is presented as a general-purpose optimizer, but the evidence covers only feedforward classification and one generative model.
The consequence. Several of Adam's claimed advantages may behave differently at scale:
- Sparse gradient handling. The IMDB experiment demonstrates the sparse-gradient advantage on a 10,000-dimensional BoW classifier. But in large-scale NLP or recommendation systems with millions of sparse features and embedding tables, the interaction between Adam's exponential decay (
$\beta_2 = 0.999$) and extremely sparse updates (features that appear once in millions of steps) may be different — the effective averaging window of ~1000 steps may be too short relative to the feature frequency, and the bias correction becomes negligible after ~1000 steps anyway. - Per-parameter adaptation across heterogeneous layers. The CNN experiment demonstrates this with 3 convolutional layers and 1 fully connected layer. In modern architectures with 50-100+ layers (ResNets, transformers), skip connections, layer normalization, and attention mechanisms, the gradient heterogeneity is far more extreme and the interaction with architectural choices (e.g., different learning rates for different components is standard practice) is untested.
- Memory claims. The paper's claim of "little memory requirement" (
$2d$extra floats) is uncontroversial for small$d$, but at the scale of billion-parameter models, storing$m_t$and$v_t$doubles or triples the optimizer memory footprint relative to SGD with momentum. This can be the difference between fitting the model on available hardware or not. The paper does not discuss this scaling concern. - Interaction with learning rate schedules. The paper uses
$\alpha_t = \alpha / \sqrt{t}$for most experiments but the deep learning community has since found that Adam interacts poorly with certain schedules and that learning rate warmup is often necessary with Adam in large models — a phenomenon not observed or discussed in this paper because the models are too small to exhibit it. - Generalization vs. optimization. The paper reports only training cost, not test-set performance. A well-known subsequent finding (Wilson et al., 2017; Keskar & Socher, 2017) is that adaptive methods like Adam sometimes generalize worse than SGD with momentum, particularly on image classification tasks, because they find different (sharper) minima. The paper provides no evidence on this question because it only tracks optimization speed, not generalization quality.
What evidence exists in the paper. The SFO comparison in Figure 2b provides a useful data point about per-iteration cost, but at the scale tested (~2M parameters), per-iteration cost differences are measured in milliseconds. The paper's claim that SFO is "5-10× slower per iteration" tells us about SFO's overhead, not about Adam's scalability. The CNN experiment's observation about $\hat{v}_t$ being "dominated by $\epsilon$" is specific to the tested architecture and may not hold for networks with different activation functions, normalization layers, or gradient magnitudes.
Mitigation status. The paper does not claim to have tested large-scale settings — it claims the algorithm is "well suited" for them based on its computational properties (first-order only, $O(d)$ memory, elementwise operations). This is an architectural argument, not an empirical one. The paper does not acknowledge the lack of large-scale validation as a limitation. In fairness, the computational resources required for large-scale experiments in 2014-2015 were far more constrained than today, and the paper's experimental scale was standard for optimizer papers of the era. The subsequent decade of practice has validated many of the scalability claims — Adam does work on billion-parameter transformers and large-scale recommendation systems — but this validation came from the community, not from the paper. A practitioner reading the paper in 2015 would have had to take the scalability claims on faith.
The Bias Correction Is Derived Under a Stationarity Assumption That May Not Hold
The assumption or constraint. The derivation of the bias correction in Section 3 explicitly assumes that the second moment of the gradient is stationary, or nearly so. The derivation computes $\mathbb{E}[v_t]$ as:
where "$\zeta = 0$ if the true second moment $\mathbb{E}[g_i^2]$ is stationary; otherwise $\zeta$ can be kept small since the exponential decay rate $\beta_2$ can (and should) be chosen such that the exponential moving average assigns small weights to gradients too far in the past." The paper argues that $\beta_2$ should be chosen to make $\zeta$ small, but this is a design guideline, not a guarantee. The correction factor $1 / (1 - \beta_2^t)$ is derived by factoring $\mathbb{E}[g^2]$ out of the sum, which is only valid if $\mathbb{E}[g_i^2]$ is constant for all $i = 1, \ldots, t$. If the gradient distribution changes substantially over time — which is the defining characteristic of the non-stationary objectives that Adam is designed to handle — the correction is approximate, not exact.
The consequence. In non-stationary settings, the bias correction can over-correct or under-correct. Consider a scenario where the gradient variance increases substantially over the course of training (as can happen when transitioning from a well-initialized regime to a regime with larger gradients, or when changing the data distribution). The stationarity assumption in the derivation means that $\mathbb{E}[g_i^2]$ is treated as constant when computing $\mathbb{E}[v_t]$, but it is not constant in reality. The correction factor $1 / (1 - \beta_2^t)$ scales $v_t$ up based on the assumption that all past gradients had the same expected squared magnitude as current gradients. If past gradients were actually much smaller, $v_t$ is being under-estimated, and the correction will not fully compensate; if past gradients were much larger, $v_t$ is being over-estimated, and the correction will over-compensate. The $\zeta$ term captures this discrepancy, but the paper provides no bound on its magnitude and no mechanism for estimating it online.
The practical consequence is that in highly non-stationary settings, the effective step size during early training may still deviate from the intended $\alpha \cdot \hat{m}_t / \sqrt{\hat{v}_t}$ — the bias correction guarantees unbiasedness only under stationarity. The paper's statement that "$\zeta$ can be kept small since the exponential decay rate $\beta_1$ can (and should) be chosen such that the exponential moving average assigns small weights to gradients too far in the past" is circular: it says the correction works if you choose $\beta_2$ to make the past irrelevant, but the whole point of using a large $\beta_2$ is to make the past relevant for variance estimation with sparse gradients. There is a tension between making $\zeta$ small (requires small $\beta_2$, short memory) and achieving stable variance estimates (requires large $\beta_2$, long memory). The paper does not analyze this tension.
Note also a likely typo in the paper's text: the discussion of $\zeta$ references $\beta_1$ (the first-moment decay), but the context is the second-moment derivation and the relevant decay rate should be $\beta_2$. This slip further suggests that the non-stationarity analysis was not the primary focus.
What evidence exists in the paper. The VAE bias-correction experiment (Figure 4) provides indirect evidence. The fact that the bias-corrected version outperforms the uncorrected version at both 10 and 100 epochs across all $\beta_2$ values suggests that the correction is beneficial even when the stationarity assumption is violated (the VAE objective is non-stationary due to minibatch sampling and the changing latent representations). However, this experiment does not test whether the correction is optimal — it only shows that it is better than no correction. The possibility remains that an adaptive correction that accounts for non-stationarity (e.g., by estimating $\zeta$ online and adjusting) would outperform the fixed $1 / (1 - \beta_2^t)$ factor. The paper does not investigate this.
Mitigation status. The paper acknowledges the stationarity assumption by including $\zeta$ in the derivation and arguing it can be kept small, but does not quantify $\zeta$, bound its magnitude, or validate the argument empirically. The bias correction is presented as an exact solution to the initialization problem, and the stationarity caveat is mentioned but not explored. For a practitioner, this means the bias correction should be understood as a well-motivated heuristic for non-stationary settings, not as a guarantee of unbiasedness. In practice, the correction's benefit in early training (where $t$ is small and the initialization bias dominates any non-stationarity effects) is likely substantial regardless of later non-stationarity, because the correction factor $1 / (1 - \beta_2^t)$ converges to 1 relatively quickly — within ~1000 steps for $\beta_2 = 0.999$, by which time the non-stationarity effects captured by $\zeta$ may not have had time to accumulate significantly. But this is speculation; the paper provides no analysis.
The Paper Provides No Guidance on Hyperparameter Sensitivity or the Cost of Tuning
The assumption or constraint. The paper states in the abstract that Adam's "hyper-parameters have intuitive interpretations and typically require little tuning" and recommends specific defaults ($\alpha = 0.001$, $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$). These defaults are presented as a key practical advantage — the algorithm works well out of the box without extensive hyperparameter optimization.
However, the paper's own experimental methodology contradicts this claim. Section 6 states:
"The hyper-parameters, such as learning rate and momentum, are searched over a dense grid and the results are reported using the best hyper-parameter setting."
The paper does not report what the best settings were for each experiment, how sensitive performance was to deviations from those settings, or whether the recommended defaults ($\beta_1 = 0.9$, $\beta_2 = 0.999$, $\alpha = 0.001$) were among the best settings or even competitive. The only experiment where a full grid is visible is the VAE bias-correction test (Figure 4), which sweeps $\beta_1 \in [0, 0.9]$, $\beta_2 \in [0.99, 0.999, 0.9999]$, and $\log_{10}(\alpha) \in [-5, -1]$. In that experiment, the performance varies substantially across the grid — the best settings achieve much lower loss than mediocre settings, even for Adam with bias correction. The fact that a grid search was needed to find good hyperparameters for the VAE suggests that "require little tuning" may be optimistic.
The consequence. A practitioner who adopts Adam with the recommended defaults may experience poor performance on their specific problem and have no principled guidance for how to adjust the hyperparameters. The paper's interpretive framework (Section 2.1) provides some guidance — $\alpha$ bounds the step size, $\beta_1$ controls momentum, $\beta_2$ controls the adaptation timescale — but this guidance is qualitative, not quantitative. If the defaults don't work, should the practitioner increase or decrease $\beta_2$? By how much? The paper provides no sensitivity analysis to answer these questions.
Specific unaddressed questions:
- How does the optimal
$\alpha$scale with model size? The recommended 0.001 was found on models with ~1M parameters. Does the same$\alpha$work for models with 100M parameters? 1B parameters? The paper provides no evidence. - How should
$\beta_2$be adjusted for different levels of gradient sparsity? The paper argues that$\beta_2$should be close to 1 for sparse gradients, but provides no rule for how close. Is 0.999 sufficient for NLP-scale sparsity? Is 0.9999 necessary? Does it depend on the feature frequency distribution? - How should
$\beta_1$be adjusted for different levels of gradient noise? Smaller minibatches produce noisier gradients, which might benefit from more momentum (higher$\beta_1$). The paper fixes minibatch size at 128 throughout and doesn't explore this interaction. - When should
$\epsilon$be adjusted? The CNN experiment reveals that$\hat{v}_t$can become so small that$\epsilon$dominates the denominator, effectively disabling per-parameter adaptation. Should$\epsilon$be reduced in such cases? Increased? The paper doesn't discuss tuning$\epsilon$at all.
What evidence exists in the paper. The VAE grid search (Figure 4) is the closest the paper comes to a sensitivity analysis, and it reveals substantial variation across hyperparameters even with bias correction enabled. For example, at $\beta_1 = 0.9$, $\beta_2 = 0.999$, and 100 epochs (Figure 4b, middle row, right column), Adam's loss varies from approximately 90 at $\log_{10}(\alpha) = -5$ to approximately 82 at $\log_{10}(\alpha) = -4$ — a meaningful difference. The paper's conclusion from this experiment is that "Adam performed equal or better than RMSProp, regardless of hyper-parameter setting," which is a statement about relative performance, not about absolute sensitivity. The fact that Adam beats RMSProp across the grid does not mean Adam's performance is insensitive to its own hyperparameters — it means Adam is robustly better, but still hyperparameter-dependent.
Mitigation status. The paper does not acknowledge hyperparameter sensitivity as a limitation. The claim of "little tuning required" is stated as a property of the algorithm, and the recommended defaults are offered as universally applicable. The grid search methodology — standard practice in optimizer papers — is described in Section 6 but its implications for the "little tuning" claim are not discussed. A practitioner reading the paper would not know whether the reported results depend critically on the grid search or whether the defaults would have achieved similar performance. Given that Adam has become the de facto standard optimizer partly because its defaults are remarkably robust across many problems — a fact that required a decade of community experience to establish — the paper's omission of sensitivity analysis means the claim was asserted rather than demonstrated at the time of publication.
No Comparison Against or Combination with Learning Rate Schedules
The assumption or constraint. Adam is presented as an algorithm that replaces the need for explicit learning rate scheduling through its automatic SNR-based annealing:
"a smaller SNR means that there is greater uncertainty about whether the direction of
$\hat{m}_t$corresponds to the direction of the true gradient... the SNR value typically becomes closer to 0 towards an optimum, leading to smaller effective steps in parameter space: a form of automatic annealing." (Section 2.1)
Despite this claim, the paper uses an explicit $\alpha_t = \alpha / \sqrt{t}$ learning rate decay schedule in the logistic regression experiments (Section 6.1: "The stepsize $\alpha$ in our logistic regression experiments is adjusted by $1/\sqrt{t}$ decay... that matches with our theoretical prediction") and in the CNN experiments (Section 6.3, visible in the faster early convergence). The VAE experiment uses constant $\alpha$, and the MLP experiments don't specify whether decay was used. The paper never ablates the effect of the $1/\sqrt{t}$ schedule against constant $\alpha$ on any single benchmark, and never compares Adam with a schedule against Adam without one.
This creates an ambiguity: is Adam's good performance due to the algorithm's internal mechanisms (moment estimation, bias correction, per-parameter scaling) or to the explicit learning rate decay that is applied on top? When $\alpha_t = \alpha / \sqrt{t}$ is used, the effective step size is $\Delta_t = (\alpha / \sqrt{t}) \cdot \hat{m}_t / \sqrt{\hat{v}_t}$. Both the explicit $1/\sqrt{t}$ factor and the implicit SNR factor contribute to the annealing. The paper's interpretive framework implicitly credits the SNR, but the experiments don't disentangle the two effects.
The consequence. A practitioner who adopts Adam with constant $\alpha$ (following the Algorithm 1 pseudocode, which makes no mention of a learning rate schedule) may experience worse performance than the paper reports, because the paper's best results on logistic regression and CNNs leveraged an additional decay schedule. Conversely, a practitioner who continues to use hand-tuned learning rate schedules with Adam — which is common practice in modern deep learning (cosine annealing, warmup, step decay) — is deviating from Adam's design philosophy (the algorithm is supposed to make scheduling unnecessary) but may be getting better results because of it. The paper provides no evidence to guide this choice.
The paper also never compares Adam-with-$1/\sqrt{t}$-decay against SGD-with-$1/\sqrt{t}$-decay. In the logistic regression experiments (Figure 1), both Adam and SGD-Nesterov use the same $\alpha_t = \alpha / \sqrt{t}$ schedule and perform similarly on MNIST. It's possible that the $1/\sqrt{t}$ schedule, not the per-parameter adaptation, is the primary driver of convergence on this problem. Without an ablation, this remains unclear.
What evidence exists in the paper. The VAE experiment (Section 6.4) uses constant $\alpha$ and shows that Adam works well. The MLP experiments (Section 6.2) likely use constant $\alpha$ as well (decay is not mentioned). So there is some evidence that Adam works without a schedule. But there is no direct comparison: train model X with Adam + constant $\alpha$, train the same model X with Adam + $\alpha / \sqrt{t}$, and report the difference. The paper reports Adam with schedule on one task and Adam without schedule on another, but tasks aren't comparable.
Mitigation status. The paper does not address this as a design choice requiring justification. The use of $\alpha / \sqrt{t}$ decay is mentioned in passing as matching the theoretical prediction, and the shift to constant $\alpha$ in later experiments is not commented on. The paper's Algorithm 1 pseudocode includes only a constant $\alpha$, not a schedule, implying that scheduling is optional or unnecessary. The experiments suggest otherwise. For practitioners, this means the recommended default of constant $\alpha = 0.001$ should be treated with caution — a learning rate schedule may still be needed, and the paper provides no guidance on when or which schedule to use.
7. Implications and Future Directions
How This Work Changes the Landscape
Adam's impact on deep learning practice is arguably larger than its technical novelty might suggest. The paper does not propose a fundamentally new category of optimizer — it synthesizes existing mechanisms (exponential moving averages, momentum, per-parameter scaling) and adds a principled initialization correction. But this synthesis, combined with the careful default hyperparameter settings (α = 0.001, β₁ = 0.9, β₂ = 0.999), changed how practitioners approach optimization in a way that few algorithm papers achieve. Before Adam, training a new architecture typically required an extensive hyperparameter search over the optimizer itself (which algorithm? what learning rate? what momentum? per-layer rates?), followed by tuning a learning rate schedule. After Adam, the default became: start with Adam at default settings, tune α if needed, and move on.
This is an incremental synthesis with outsized practical impact, not a paradigm shift. The intellectual pieces — exponential moving averages for gradient variance (RMSProp), momentum for gradient smoothing (SGD with momentum), per-parameter scaling (AdaGrad) — all existed. Adam's contribution was recognizing that (1) these pieces could be unified under a consistent statistical framework (moment estimation), (2) the initialization bias from zero-starting those estimates could be exactly corrected with a simple time-dependent factor, and (3) the resulting algorithm would work robustly across enough problem types to serve as a default rather than a specialized tool. The third point is critical: Adam isn't just another optimizer in the toolbox — it became the optimizer that practitioners reach for first, with alternatives considered only when Adam fails or when problem-specific knowledge suggests a better choice. This shift from "choose your optimizer per problem" to "start with Adam, deviate only with reason" is the paper's most enduring impact on the field's methodology.
The paper also resolves a specific tension in the prior literature that was blocking progress toward a unified adaptive optimizer. AdaGrad demonstrated that per-parameter scaling was possible and theoretically sound, but its monotonically growing denominator made it unusable for non-stationary objectives — the very setting where deep learning with dropout, batch normalization, and changing loss landscapes operates. RMSProp fixed the non-stationarity problem with exponential forgetting, but the zero-initialization of its denominator caused training instability that practitioners had to work around with ad-hoc warmup schedules or reduced learning rates. The field was stuck: you could have sparse-gradient robustness (AdaGrad) or non-stationarity handling (RMSProp), but not both simultaneously, because the long memory needed for sparse gradients (large β₂) amplified the initialization bias that RMSProp couldn't handle. Adam's bias correction resolves this tension directly — it makes large β₂ safe, enabling the long-memory denominator that sparse gradients require without the early-training instability that previously accompanied it. Figure 4 demonstrates this resolution empirically: at β₂ = 0.9999, the bias-corrected version trains stably while the uncorrected version diverges. The tension between sparse-gradient robustness and non-stationarity handling was not a fundamental tradeoff — it was an artifact of a missing correction.
The paper also shifts the conceptual framing of adaptive optimization from mechanism-design toward statistical estimation. Before Adam, adaptive methods were described in terms of what they compute — "we accumulate squared gradients," "we divide by the root mean square." Adam reframes this as "we are estimating the moments of the gradient distribution online, and the update is the estimated signal-to-noise ratio." This isn't just a rhetorical difference. It opens the door to analyzing adaptive optimizers through the lens of statistical efficiency, bias-variance tradeoffs, and estimator design — a perspective that has since influenced work on optimizer theory, variance reduction, and the analysis of why adaptive methods sometimes generalize worse than SGD (a line of work that examines whether the implicit bias of the √v_t denominator leads to sharp minima). The AdaMax extension (Section 7.1) demonstrates the generative power of this reframing: generalizing the second moment from L² to L^p and taking p → ∞ yields a new algorithm that would not have been obvious from the mechanistic "divide by RMS" perspective.
However, the paper leaves several important questions about the landscape unresolved. It provides no evidence on generalization — whether Adam finds minima that generalize better or worse than SGD — because all experiments report training cost only. This omission proved significant: subsequent work (Wilson et al., 2017; Keskar & Socher, 2017) found that adaptive methods can generalize worse than SGD on some image classification tasks, leading to a literature on the "adaptive vs. SGD" generalization gap that the paper does not anticipate. The paper also provides no guidance on when not to use Adam — it presents Adam as universally applicable, but the subsequent finding that SGD with momentum sometimes generalizes better means the "default" status Adam achieved needed qualification that the paper doesn't provide.
The paper also changes which research directions are attractive. Before Adam, optimizer design focused on mechanisms — can we add curvature information? can we estimate the Hessian diagonal efficiently? can we incorporate second-order information without the memory cost? Adam's success redirected attention toward robust defaults and bias correction as first-class design goals. The fact that Adam's key innovation (bias correction) is a statistical fix rather than a new mechanism, and that its main practical contribution is a set of hyperparameter defaults that work across problems, subtly shifted the field's values: an optimizer that works well out of the box on 80% of problems is more impactful than one that achieves state-of-the-art on 20% of problems after extensive tuning. This value shift influenced the design of subsequent optimizers (AdamW, Ranger, Nadam) and the evaluation methodology of optimizer papers (which now routinely include sensitivity analyses and default-setting comparisons). At the same time, the paper made research on more sophisticated adaptive mechanisms less immediately attractive — if Adam with simple moving averages works well enough, the marginal benefit of adding natural gradient information or Hessian approximations must be weighed against the loss of simplicity and robustness. This is not necessarily negative (it focused effort on high-impact problems), but it did redirect attention away from second-order and quasi-Newton methods for deep learning.
Follow-Up Research This Work Enables
Generalization gap between Adam and SGD: when and why does Adam find worse minima? The paper reports only training cost, never test-set accuracy or loss. A natural and urgent follow-up is to train identical architectures with Adam and SGD-with-momentum on classification benchmarks (CIFAR-10, CIFAR-100, ImageNet) and report both training and test metrics. If Adam reaches lower training loss but higher test error, this would reveal an overfitting tendency that the paper's optimization-speed focus misses. The specific hypothesis to test is whether Adam's per-parameter scaling leads it to converge to sharper minima (where the Hessian has larger eigenvalues) compared to SGD, which is known to have an implicit bias toward flatter minima. A well-designed experiment would compare Adam, SGD, and Adam with various β₂ values (controlling the adaptation aggressiveness) on a benchmark like CIFAR-10 with a ResNet, measuring both test accuracy and the sharpness of the found minimum (via the Hessian's largest eigenvalue or the sensitivity to parameter perturbation). If Adam generalizes worse, the natural follow-up is whether the bias correction or the √v_t denominator is the cause — ablating each component separately.
Adaptive bias correction for non-stationary settings. The paper derives the bias correction 1 / (1 - β^t) under the assumption that the gradient moments are stationary (ζ = 0), then argues that ζ can be kept small by choosing β appropriately. But there's a tension: large β (needed for sparse gradients and stable variance estimates) means old gradients remain influential longer, making ζ potentially non-negligible in non-stationary settings. A concrete research direction is to estimate ζ online and use it to adjust the bias correction adaptively. For instance, one could maintain a second, slower-moving average of v_t to detect drift in the second moment, and use the discrepancy to scale the correction factor. The experiment would compare standard Adam against this adaptive-correction variant on a deliberately non-stationary benchmark — e.g., training on a sequence of increasingly difficult data subsets, or training with dropout rates that increase over time — measuring whether the adaptive correction reduces the variance of the effective step size and improves final performance.
Scaling Adam to very large models: memory and numerical stability. The paper claims Adam "has little memory requirement" because it stores two additional vectors (m_t and v_t) of size d. For the ~1M parameter models tested, this is negligible. For a 175B-parameter model (GPT-3 scale), m_t and v_t each require 175B × 4 bytes = 700 GB in float32 — together with parameters (700 GB) and gradients (700 GB), the total is 2.8 TB, which exceeds the memory of any single GPU. The research question is whether Adam's memory footprint can be reduced without sacrificing its convergence properties. Specific approaches to test: (1) low-rank factorization of m_t and v_t (treating them as matrices for weight tensors and using truncated SVD), (2) quantization to float16 or bfloat16 for the moment buffers, measuring whether the reduced precision causes training instability, (3) sharing a single scalar v estimate across all parameters in a layer (reducing v_t from d-dimensional to a scalar per layer), and measuring the convergence penalty. A strong experiment would train a 1B+ parameter transformer with each variant and report both memory usage and final perplexity, identifying the Pareto frontier of memory vs. performance.
Interaction between Adam and learning rate schedules: is the automatic annealing sufficient? The paper claims that Adam's SNR-based step size reduction provides "a form of automatic annealing" that makes explicit learning rate schedules unnecessary. Yet the paper's own logistic regression and CNN experiments use α_t = α / √t decay, and modern practice commonly pairs Adam with cosine annealing or warmup. A systematic ablation is needed: train a suite of models (CNN on CIFAR-10, ResNet on ImageNet, transformer on language modeling) with Adam using (a) constant α, (b) α / √t decay, (c) cosine annealing, and (d) step decay. Measure both training loss and test accuracy. If the automatic annealing claim holds, constant α should match or exceed the scheduled variants. If it doesn't, the paper's interpretive framework needs revision — the SNR ratio doesn't fully substitute for explicit scheduling, and the recommended practice of using Adam without a schedule is incorrect. A deeper analysis would track |Δ_t| over the course of training for each schedule and compare it to the SNR, diagnosing whether the SNR actually decreases near convergence as predicted (Section 2.1) and whether explicit schedules are compensating for cases where it doesn't.
AdaMax head-to-head evaluation and the L^p norm family. Section 7.1 derives AdaMax as the p → ∞ limit of an L^p-norm generalization of Adam's second moment, but provides no experiments. A natural follow-up is a systematic comparison of Adam (p = 2) against AdaMax (p → ∞) and intermediate p values (p = 4, 8, 16) on the same benchmarks used in the paper (MNIST MLP, CIFAR CNN, IMDB logistic regression) plus a modern benchmark (e.g., CIFAR-10 ResNet, WikiText-2 language modeling). The hypothesis to test is whether the L^∞ norm's robustness to outliers (it tracks the maximum rather than the average) is beneficial in settings with heavy-tailed gradient distributions or occasional gradient spikes — e.g., training with large learning rates, training GANs where the discriminator and generator gradients have different statistics, or training with noisy labels. The experiment should also measure sensitivity to β₂, since AdaMax eliminates the need for bias correction on the second moment — if AdaMax is equally performant but less sensitive to β₂, it would be a strictly better default.
Adam's behavior on recurrent neural networks and sequence models. The paper evaluates Adam on feedforward architectures only (logistic regression, MLP, CNN, VAE), despite citing Graves (2013) on RNNs for speech recognition. RNNs present a fundamentally different optimization challenge: gradients are products of many Jacobians (leading to vanishing/exploding gradients), the loss landscape has long, flat valleys and steep cliffs, and the gradient distribution changes dramatically as the network learns long-range dependencies. The paper's per-parameter scaling and momentum may behave differently in this setting — for instance, v_t for recurrent weight matrices might need very different β₂ than v_t for input/output weights, but Adam applies the same β₂ globally. A specific experiment: train an LSTM on language modeling (Penn Treebank) and on a sequence prediction task with long-range dependencies (e.g., the adding problem) using Adam at various (β₁, β₂) settings, measuring both convergence speed and the network's ability to learn long-range patterns. Compare against RMSProp (the standard for RNNs at the time) and SGD with gradient clipping. If Adam matches or exceeds RMSProp for RNNs, it strengthens the "universal default" claim; if it underperforms, it reveals a problem class where the synthesis breaks down.
Practical Applications and Downstream Use Cases
Default optimizer for rapid prototyping in deep learning. The paper's most immediate practical contribution is establishing Adam with default α = 0.001, β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸ as a robust starting point for training new neural network architectures. Before Adam, a practitioner implementing a novel architecture had to simultaneously debug the architecture and tune the optimizer — learning rate schedules, momentum values, per-layer scaling — making it difficult to determine whether poor performance was due to the architecture or the optimization. With Adam, the practitioner can start with the defaults, verify that the architecture can overfit a small dataset (a standard debugging step), and only tune the optimizer if performance is clearly suboptimal. The paper's results support this workflow: across logistic regression, MLPs, CNNs, and VAEs — a broad enough range to cover most architecture types of the era — Adam with defaults (plus a 1/√t schedule where noted) was never the worst-performing optimizer and was often the best or tied for best. The 4× orders-of-magnitude grid search on the VAE (Figure 4, α ∈ [10⁻⁵, 10⁻¹]) shows that Adam's performance degrades gracefully with suboptimal α, meaning the default 0.001 is likely to land in a reasonable region even when not perfectly tuned. This shifts the cost-benefit calculation for prototyping: instead of spending hours on optimizer tuning before evaluating an architecture idea, the practitioner can spend minutes on a coarse α sweep (e.g., {0.0001, 0.001, 0.01}) and invest the saved time in architecture iteration.
Training models with sparse features without manual learning rate engineering. The IMDB logistic regression experiment (Figure 1, right) demonstrates that Adam matches AdaGrad's convergence on a sparse bag-of-words task with 10,000 features, where SGD with Nesterov momentum lags far behind. This directly translates to production scenarios involving text classification, recommendation systems, and any model trained on one-hot or TF-IDF feature representations. In these settings, feature frequencies follow a Zipfian distribution — a small number of features appear in most examples, a large number appear rarely — creating exactly the gradient sparsity pattern that AdaGrad was designed for and that Adam inherits. The practical benefit is that practitioners can use Adam as a drop-in replacement for AdaGrad on sparse models without worrying about AdaGrad's diminishing learning rate problem on the dense features (e.g., bias terms, embedding dimensions that are always active) or when continuing training on new data. The paper's key result here is not that Adam beats AdaGrad on sparsity — they match — but that Adam is no worse on sparsity while being substantially better on non-stationary and dense problems, meaning a single optimizer can serve both the sparse feature pipeline and the downstream dense network without retuning.
Training deep networks with heterogeneous layer types without per-layer learning rates. The CNN experiment (Figure 3) explicitly identifies a pain point in deep learning practice: "a smaller learning rate for the convolution layers is often used in practice when applying SGD." Because convolutional layers have weight sharing (many parameters share the same gradient contributions) and fully connected layers do not, the gradient magnitudes differ systematically, and SGD's uniform learning rate forces the practitioner to manually set different rates per layer. Adam eliminates this manual tuning: the per-parameter v_t automatically scales the learning rate down for layers with large gradients and up for layers with small gradients. The paper's results show that Adam achieves "marginal improvement" over SGD with Nesterov momentum on the CNN (Figure 3, right), but this understates the practical benefit — achieving the same accuracy without the per-layer tuning effort is a substantial reduction in practitioner time. For production teams training custom CNN architectures (e.g., for medical imaging, satellite imagery, or industrial inspection), Adam with defaults can replace a multi-day tuning process (train with per-layer SGD rates, evaluate, adjust, repeat) with a single training run, accelerating deployment timelines.
Stable training of generative models with stochastic objectives. The VAE bias-correction experiment (Figure 4) demonstrates that Adam trains stably with β₂ = 0.999 and β₂ = 0.9999, while the uncorrected variant (equivalent to RMSProp with momentum) deteriorates or diverges at these β₂ values. In generative modeling — VAEs, but also the GANs that were emerging at the time of publication — the objective is inherently stochastic (due to sampling from the latent distribution or the generator), and gradient variance can be high. A large β₂ is necessary to get stable variance estimates for the denominator, but without bias correction, the early-training denominator is catastrophically small, causing large, destabilizing parameter updates exactly when the model is most fragile (random initialization). Adam's bias correction eliminates this failure mode. For a practitioner training a VAE or GAN, this means they can set β₂ = 0.999 (or higher) to handle the stochasticity without needing a learning rate warmup or a hand-tuned β₂ schedule. The paper's grid search (Figure 4) shows that this benefit holds across a wide range of α values — Adam with bias correction trains stably where the uncorrected version fails — meaning the practitioner doesn't need to precisely tune α to avoid divergence.
