ArXiv: 1212.5701
🎯 Pitch
ADADELTA eliminates the learning rate hyperparameter entirely by scaling each parameter update using the ratio of decaying RMS averages of past updates to past gradients. On MNIST, it matches or beats tuned methods within 6 epochs, and its two built-in sensitivity parameters barely matter across four orders of magnitude.
1. Executive Summary
This paper introduces ADADELTA, a per-dimension adaptive learning rate method for stochastic gradient descent that dynamically adjusts step sizes using only first-order information with minimal computational overhead beyond vanilla SGD. The method constructs its update by taking the ratio of two exponentially decaying root-mean-square quantities — the RMS of past parameter updates in the numerator and the RMS of past gradients in the denominator — which simultaneously eliminates the need for a manually tuned learning rate, provides natural per-dimension scaling, and derives from a unit-correctness argument via a diagonal Hessian approximation. Evaluated on MNIST digit classification with a two-hidden-layer network and on a large-scale speech recognition task trained across 100–200 distributed replicas, ADADELTA achieves a 1.83% test error on MNIST after 6 epochs (matching or exceeding the 2.10% error reported by the competing Schaul et al. method) while showing insensitivity to its two hyperparameters over orders-of-magnitude variation (ρ from 0.9–0.99, ϵ from 1e−2–1e−8), and on speech data it converges faster than ADAGRAD and Momentum under both logistic and rectified linear nonlinearities. The method establishes that a learning-rate-free per-dimension update rule can be constructed from an RMS ratio that approximates second-order curvature, though on MNIST it converges near the best performance achieved by a properly tuned momentum schedule rather than surpassing it at full convergence.
2. Context and Motivation
The Core Problem: Learning Rates Are a Fragile Manual Tuning Knob
The fundamental problem this paper addresses is deceptively simple: stochastic gradient descent (SGD) requires a learning rate hyperparameter, and choosing it well is painful, brittle, and often the difference between successful training and complete failure. The paper opens by laying out the standard gradient descent update rule (Equations 1–2 in Section 1), where the parameter change at each iteration is:
Here is the gradient of the objective function with respect to the parameters at iteration , and is the learning rate — a scalar that controls the step size. This formulation is elegant in its simplicity but burdens the practitioner with a hyperparameter that is simultaneously critical and poorly understood.
The paper characterizes the learning rate selection problem in stark terms (Section 1):
"Setting the learning rate typically involves a tuning procedure in which the highest possible learning rate is chosen by hand. Choosing higher than this rate can cause the system to diverge in terms of the objective function, and choosing this rate too low results in slow learning. Determining a good learning rate becomes more of an art than science for many problems."
This is not an exaggeration. In neural network training, the learning rate sits at a knife's edge: too high and the optimization diverges explosively (the loss becomes NaN or infinity); too low and training crawls, wasting compute and potentially trapping the model in poor local minima before meaningful progress occurs. The acceptable range is often narrow — sometimes spanning less than an order of magnitude — and depends on everything from the network architecture and activation function choice to the data distribution and batch size. Finding this range typically requires grid search, which multiplies the cost of training by the number of attempts.
Why This Problem Matters: Practical and Theoretical Dimensions
Practical impact. The learning rate tuning burden scales with the ambition of the model. In 2012, when this paper was published, the deep learning community was rapidly scaling up network sizes and deploying distributed training systems (e.g., the DistBelief framework described by Dean et al., 2012, which the paper cites as [4] and uses in its experiments). Each training run on a large distributed cluster costs thousands of machine-hours. A failed learning rate choice that causes divergence after several hours of training is not merely an inconvenience — it is a substantial waste of computational resources. The paper's speech recognition experiments (Section 4.4) use 100–200 model replicas training on hundreds of hours of US English data, exactly the kind of setting where tuning cost is prohibitive.
Moreover, learning rate schedules — where the learning rate is manually decreased at specific epochs based on validation performance plateaus — add another layer of human judgment that doesn't scale. Different datasets, architectures, and tasks require different schedules, and the practitioner must monitor training and intervene. The paper's goal of a method that requires "no manual tuning of a learning rate" (Abstract) is therefore not a convenience feature; it is a prerequisite for scalable, automated machine learning pipelines.
Theoretical significance. Beyond the practical tuning burden, the learning rate problem exposes a deeper theoretical issue: the units are wrong. This is the paper's key conceptual insight, developed in Section 3.2. The gradient has units proportional to (assuming the cost function is unitless), so has units of — it does not match the parameter units. If the parameters represent, say, a weight in units of "pixels," the update should also be in units of "pixels." For SGD and momentum (Equation 4, Section 2.2.1):
This is a conceptual inconsistency: you are adding quantities with incompatible units. In physics-aware optimization, this would be a red flag. Second-order methods like Newton's method fix this because the Hessian provides the right scaling:
The inverse Hessian has units of (since the second derivative has units of ), so recovers units of . The paper's unit-correctness argument elevates the learning rate problem from a mere hyperparameter tuning annoyance to a structural flaw in first-order optimization — and positions ADADELTA's RMS ratio as a principled, unit-correct fix using only first-order information.
Prior Approaches and Their Specific Shortcomings
The paper dedicates Section 2 to surveying existing methods, identifying concrete failure modes for each major approach. The structure is a taxonomy: methods that anneal a global learning rate, methods that adapt per-dimension, and methods that use second-order information. Each category is found wanting in specific ways that ADADELTA is designed to address.
2.1. Learning Rate Annealing (Section 2.1): Manual, Coarse, Non-Adaptive
The simplest approach to improving SGD is to schedule the learning rate: start high for fast initial progress, then decrease it over time to converge smoothly. The paper identifies two variants:
Manual annealing based on validation plateau detection: the practitioner watches the validation curve and manually drops the learning rate when it flattens. This is labor-intensive and subjective — what constitutes a "plateau" varies across problems.
Automatic schedules (citing Robins and Monro, 1951 [1]): the learning rate follows a predetermined decay function, e.g., or (step decay).
The paper's criticism is twofold. First, these methods introduce additional hyperparameters — the initial rate , the decay rate or schedule, the decay frequency — that themselves require tuning. So the annealing approach doesn't eliminate the tuning problem; it relocates it. Second, annealing applies a single global scaling to all parameters simultaneously, ignoring that different dimensions of the parameter vector can relate to the cost function in completely different ways (more on this in Section 2.2). A weight in the first layer and a weight in the last layer of a deep network face fundamentally different gradient scales due to vanishing/exploding gradient phenomena, and annealing a single global learning rate cannot compensate for these per-dimension differences.
2.2. Per-Dimension First Order Methods (Section 2.2): Partial Solutions, Each with Distinct Weaknesses
The paper examines two methods and positions ADADELTA relative to each.
2.2.1. Momentum (Section 2.2.1). The momentum method (Rumelhart et al., 1986 [2]) is perhaps the oldest and most widely used SGD variant. Its update rule (Equation 4) maintains an exponentially decaying running average of past gradients:
where is a decay constant (typically 0.9). The paper explains momentum's intuitive benefits clearly: along dimensions where gradients consistently point in the same direction (e.g., along the bottom of a long, narrow valley in the loss surface), the term accumulates velocity, accelerating progress even when the instantaneous gradient is small. Along dimensions where the gradient sign oscillates (e.g., across the steep walls of the valley), the alternating signs cancel out in the momentum buffer, damping oscillations. This gives implicit per-dimension scaling — the effective step size varies per dimension based on gradient consistency — but the method still requires a manually tuned global learning rate . The paper's MNIST experiments (Table 1) show that momentum performance varies dramatically with : from 2.03% test error at to 89.68% at — a jump from excellent to essentially random performance over a single order of magnitude.
2.2.2. ADAGRAD (Section 2.2.2). ADAGRAD (Duchi et al., 2010 [3]) was the state-of-the-art adaptive method at the time. Its update rule (Equation 5) divides the gradient by the root-mean-square of all past gradients on a per-dimension basis:
The denominator accumulates squared gradients over the entire history of training, producing a per-dimension learning rate. Large-gradient dimensions get small effective step sizes; small-gradient dimensions get large effective step sizes. The paper acknowledges two attractive properties of this design:
-
Per-dimension scaling: Like second-order methods, it compensates for the fact that different layers in a deep network have gradients differing by orders of magnitude. This is especially valuable in deep networks where early layers need larger steps to overcome vanishing gradient effects.
-
Implicit annealing: As the denominator grows, the effective learning rate naturally decreases over time, providing a built-in annealing schedule without explicit programming.
However, the paper identifies two specific failure modes that motivate ADADELTA:
"Since the magnitudes of gradients are factored out in ADAGRAD, this method can be sensitive to initial conditions of the parameters and the corresponding gradients. If the initial gradients are large, the learning rates will be low for the remainder of training."
This is a subtle but devastating problem. In a random initialization, some parameters happen to receive large initial gradients through bad luck or architectural properties (e.g., the output layer often has large gradients early in training). ADAGRAD's denominator immediately penalizes these parameters with a small effective learning rate that persists throughout training — the parameters are "cursed" by their initial condition and can never recover. The only remedy is to increase the global learning rate , which brings back the tuning problem.
"Also, due to the continual accumulation of squared gradients in the denominator, the learning rate will continue to decrease throughout training, eventually decreasing to zero and stopping training completely."
The denominator is a monotonically increasing function of (since each term is non-negative). As , the denominator grows without bound, driving the effective learning rate to zero. After many iterations, the model simply stops learning — even if it hasn't converged. This is visible in the paper's MNIST results (Figure 1): ADAGRAD performs well for the first ~10 epochs but then plateaus as its denominator saturates, falling behind momentum and ADADELTA.
"We created our ADADELTA method to overcome the sensitivity to the hyperparameter selection as well as to avoid the continual decay of the learning rates."
This sentence (end of Section 2.2.2) is the paper's explicit mission statement for its method relative to ADAGRAD.
2.3. Second-Order Methods (Section 2.3): Correct But Computationally Expensive
The paper briefly surveys approaches that use curvature information to set learning rates properly:
Newton's method (Equation 3): , where is the full Hessian matrix. This gives optimal step sizes for quadratic problems and has correct units, but computing and inverting the Hessian for a model with millions or billions of parameters is completely infeasible.
Diagonal Hessian approximation (Becker and LeCun, 1988 [5]): If only the diagonal entries of the Hessian are used, the update becomes (Equation 6):
The diagonal Hessian can be computed with one additional forward-backward pass through the model — effectively doubling the computational cost per iteration over SGD — and provides curvature-aware per-dimension scaling. Still, this extra cost is substantial for large models, and the method hasn't seen widespread adoption.
Schaul et al. (2012) [6]: The most directly relevant competing method from the same period. Schaul et al. proposed combining diagonal Hessian information with ADAGRAD-style gradient statistics:
where and are expectations over a window of recent gradients. This method eliminates the manually tuned learning rate (one of ADADELTA's stated goals) and avoids ADAGRAD's infinite accumulation problem by using a finite window. However, it still requires computing the diagonal Hessian — the doubling of computation per iteration that ADADELTA specifically avoids. The paper notes in Section 3:
"After deriving our method we noticed several similarities to Schaul et al. [6], which will be compared to below."
The key difference is that ADADELTA approximates curvature information using only first-order statistics already available from gradient computation — the RMS of past updates — avoiding any second-order computation whatsoever.
How ADADELTA Positions Itself
The paper's positioning is clear from the relationship it draws to both ADAGRAD and Schaul et al.:
To ADAGRAD: ADADELTA adopts the core mechanism — dividing by the RMS of past gradients for per-dimension scaling — but replaces the infinite accumulation with a fixed-size exponential window (Section 3.1) to prevent the continual decay of learning rates. This uses the same decay constant as momentum, creating a locally adaptive estimate that responds to recent gradient magnitudes rather than being dominated by ancient history.
To Schaul et al.: ADADELTA achieves a similar effect (unit-correct, learning-rate-free updates) but replaces the expensive diagonal Hessian computation with a cheap first-order proxy: the RMS of past parameter updates themselves, using the insight from Equation 13 that . Under a locally smooth curvature assumption, the ratio of recent update magnitudes to recent gradient magnitudes approximates the inverse diagonal Hessian — and this ratio comes essentially for free.
To momentum: ADADELTA provides similar acceleration effects (the numerator accumulates past update magnitudes) but without a global learning rate.
To SGD: ADADELTA adds "trivial" computation — two exponential moving averages and a square root per dimension — and eliminates the learning rate hyperparameter entirely. The paper claims the overhead is "minimal computational overhead beyond vanilla stochastic gradient descent" (Abstract).
The paper's contribution is therefore not a single novel technique but rather a synthesis of existing ideas (exponential averaging from momentum, per-dimension scaling from ADAGRAD, unit correctness from second-order methods) into a unified update rule (Equation 14) that achieves four simultaneous goals:
- No learning rate to tune
- Per-dimension adaptive step sizes
- No continual decay (finite effective window)
- Minimal computational cost (first-order information only)
The experiments are designed to validate exactly these claims: insensitivity to hyperparameters ( and across orders of magnitude in Table 2), per-dimension adaptation (step size visualization in Figure 2 showing layer-dependent scaling), avoidance of ADAGRAD's slowdown (convergence beyond epoch 10 in Figure 1), and robust performance under noisy distributed gradients (200 replicas in Figure 4).
3. Technical Approach
3.1 Reader Orientation (Approachable Technical Breakdown)
ADADELTA is a per-dimension adaptive learning rate method — a drop-in replacement for the learning rate in stochastic gradient descent that computes a separate step size for each parameter automatically from training statistics, requiring no manual learning rate tuning. It solves the problem that SGD's global learning rate is simultaneously critical for convergence and extremely difficult to choose well, by constructing each parameter's step size as the ratio of two root-mean-square quantities: how much that parameter has been moving recently (numerator) divided by how large its gradients have been recently (denominator), which inherently produces correct physical units and naturally scales steps per dimension.
3.2 Big-Picture Architecture (Diagram in Words)
The ADADELTA system has exactly two moving parts layered on top of standard gradient computation:
-
Gradient accumulator — maintains an exponentially decaying running average of squared gradients for each parameter, producing
$\text{RMS}[g]_t$(root-mean-square of recent gradients). This serves as the denominator, providing per-dimension scaling that compensates for parameters whose gradients differ by orders of magnitude. -
Update accumulator — maintains an exponentially decaying running average of squared parameter updates for each parameter, producing
$\text{RMS}[\Delta x]_{t-1}$(root-mean-square of recent parameter changes, lagged by one iteration). This serves as the numerator, providing an approximation to the inverse diagonal Hessian curvature that gives the update correct units and acts as an acceleration term.
The flow is: compute gradient $g_t$ → update the gradient running average with $g_t^2$ → compute the step size as $\text{RMS}[\Delta x]_{t-1} / \text{RMS}[g]_t$ → multiply by the negative gradient to get the parameter update $\Delta x_t$ → apply the update to parameters → update the parameter-update running average with $\Delta x_t^2$ → repeat. Critically, the numerator uses the previous iteration's RMS of updates (lagged by one), which the paper notes provides robustness to sudden large gradients by keeping the numerator temporarily smaller than a new large denominator.
3.3 Roadmap for the Deep Dive
- First, Idea 1 — the finite window accumulator (Equation 8–10), because it addresses ADAGRAD's infinite accumulation problem and introduces the exponential decay mechanism that both ADADELTA accumulators share.
- Second, Idea 2 — the unit-correctness argument and Hessian approximation (Equations 11–14), because it is the paper's key conceptual innovation and explains why the numerator is chosen specifically as the RMS of past updates rather than any other statistic.
- Third, the complete ADADELTA algorithm (Algorithm 1), bringing both ideas together and showing the precise flow of computation, including initialization, the lagged numerator, and how the
$\epsilon$constant serves dual roles. - Fourth, design choices and alternatives rejected, explaining why a fixed window instead of infinite accumulation, why RMS of updates instead of another curvature proxy, why exponential decay instead of a hard window, and why the lagged numerator matters.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily a method derivation paper whose core idea is that a learning-rate-free, per-dimension adaptive update rule can be constructed by taking the ratio of two RMS quantities — one from past gradients (providing ADAGRAD-like scaling without infinite decay) and one from past parameter updates (providing correct units via a cheap diagonal Hessian approximation) — using only first-order information.
Idea 1: Accumulate Gradients Over a Finite Window (Fixing ADAGRAD's Continual Decay)
The paper derives ADADELTA by starting from ADAGRAD and fixing its two main problems: continual learning rate decay and sensitivity to the global learning rate. The first problem — continual decay — is addressed by replacing ADAGRAD's infinite accumulation window with a finite one.
The problem with ADAGRAD's denominator. In ADAGRAD (Equation 5), the denominator is $\sqrt{\sum_{\tau=1}^{t} g_\tau^2}$ — the sum of all squared gradients from the beginning of training to the current iteration $t$. Since each $g_\tau^2$ is non-negative, this sum grows monotonically with $t$. As $t \to \infty$, the denominator grows without bound (assuming gradients are not exactly zero forever, which they never are in practice), and the effective learning rate $\eta / \sqrt{\sum g_\tau^2}$ decays to zero. This means training eventually stalls regardless of whether the model has converged, a problem the paper identifies explicitly when describing ADAGRAD's behavior in Figure 1: "ADAGRAD performs well for the first 10 epochs of training, after which it slows down due to the accumulations in the denominator which continually increase."
The fix: exponential moving average over a window. Instead of summing over all time, the paper restricts the window of accumulated gradients to some fixed effective size $w$. The naive implementation — storing the last $w$ squared gradients and recomputing the sum — would be memory-inefficient (storing $w$ values per parameter). Instead, the paper implements this windowed accumulation using an exponentially decaying average, a standard technique familiar from momentum:
where $E[g^2]_t$ is the running exponentially decaying average of squared gradients at iteration $t$, $\rho \in [0, 1)$ is a decay constant that controls the effective window size, and $g^2_t$ is the element-wise square of the gradient at iteration $t$. The initialization is $E[g^2]_0 = 0$.
What it computes: this recurrence updates a running estimate of the expected squared gradient magnitude, giving more weight to recent gradients and exponentially less weight to older ones. A decay rate $\rho = 0.9$ means the contribution of a gradient decays by a factor of 0.9 per iteration, so gradients from more than $1/(1-\rho) = 10$ iterations ago contribute negligibly. The result at each iteration is a scalar per parameter that captures the recent "scale" or "energy" of gradients along that dimension.
Why this form: an exponential moving average is the standard way to maintain a local running statistic without storing individual samples. It requires only $O(1)$ memory per parameter (the previous average $E[g^2]_{t-1}$) and $O(1)$ computation per update (one multiply-add). A hard window of size $w$ would require storing $w$ previous squared gradients per parameter and summing them each iteration — $O(w)$ memory and $O(w)$ time, which is prohibitive for large models with millions of parameters. The exponential decay form approximates a soft window whose effective length is controlled by $\rho$; higher $\rho$ (e.g., 0.99) means longer memory, lower $\rho$ (e.g., 0.9) means shorter memory.
From running average to RMS. Since ADAGRAD uses the square root of accumulated squared gradients, the paper defines the RMS (root-mean-square) of the gradient running average:
where $\epsilon$ is a small constant added to improve numerical conditioning (preventing division by zero or near-zero when $E[g^2]_t$ is very small), following the practice of Becker and LeCun [5] who used a similar $\mu$ constant.
What it computes: $\text{RMS}[g]_t$ is the root-mean-square magnitude of recent gradients along each parameter dimension — a local, time-decaying estimate of the typical gradient scale. The $\epsilon$ constant prevents the denominator from being exactly zero at initialization (when $E[g^2]_0 = 0$) or when a parameter's gradients become extremely small late in training.
Why this form: taking the square root converts a variance-like quantity (expected squared gradient) back to the same scale as the gradient itself — a standard deviation-like measure. This matters for the units argument developed in Idea 2: the RMS has the same units as the gradient (units of $1/x$), which is needed for cancellation in the final update ratio. Using $E[g^2]_t$ directly (without the square root) would give units of $1/x^2$, which would not produce correct units in the final update.
The intermediate update rule (before Idea 2). With only this fixed-window modification, the update rule would be (Equation 10):
This solves the continual decay problem — $\text{RMS}[g]_t$ does not grow without bound because old gradients are exponentially forgotten — but it still requires a manually chosen global learning rate $\eta$. The RMS denominator provides per-dimension scaling (parameters with large gradients get small steps; parameters with small gradients get large steps) and implicit annealing (as gradients naturally shrink near convergence, the denominator shrinks, partially counteracting the decay), but the global $\eta$ remains to be tuned. This is the setup for Idea 2, which eliminates $\eta$ entirely.
Idea 2: Correct Units via a Hessian Approximation (Eliminating the Global Learning Rate)
The paper's second and more innovative idea addresses the global learning rate $\eta$ by observing that it exists to solve a unit mismatch problem. The gradient $g_t = \partial f / \partial x$ has units of $1 / \text{units of } x$ (assuming the cost function $f$ is unitless), so the SGD update $\Delta x_t = -\eta g_t$ has units that don't match the parameters — you're adding something with units of $1/x$ to something with units of $x$. The learning rate $\eta$ implicitly carries units of $x^2$ to patch this mismatch, but choosing it correctly requires knowing the curvature of the loss surface.
The unit mismatch problem formalized. The paper makes this explicit in Section 3.2. For SGD and momentum (Equation 11):
The parameter change $\Delta x$ has different units from the parameters $x$ themselves. ADAGRAD's update $\Delta x_t = -(\eta / \text{RMS}[g]_t) g_t$ also has mismatched units: $\text{RMS}[g]_t$ has the same units as $g_t$ (units of $1/x$), so the ratio $g_t / \text{RMS}[g]_t$ is unitless. The update is purely a dimensionless scaling of the negative gradient direction, and $\eta$ must supply the missing units of $x$.
Second-order methods are unit-correct. In contrast, Newton's method and other second-order approaches have the correct units (Equation 12):
Here $H^{-1}$ is the inverse Hessian. The second derivative $\partial^2 f / \partial x^2$ has units of $1 / (\text{units of } x)^2$, so $H^{-1}$ has units of $x^2$, and multiplying by $g$ (units of $1/x$) yields units of $x$ — matching the parameters. This is not merely a notational nicety; it means Newton's method automatically computes the right step size in the right units, provided the quadratic approximation is valid.
Deriving a first-order curvature proxy. The paper's key move is to approximate the inverse second derivative — which controls the optimal step size — using only first-order quantities that are already being computed. Starting from Newton's method with a diagonal Hessian assumption (so that each dimension's second derivative is independent), the paper rearranges the Newton update to isolate the inverse second derivative (Equation 13):
This equation says: the inverse second derivative equals the ratio of the parameter update to the gradient.
What this rearrangement means conceptually. If you've already taken a step, the step size you actually took divided by the gradient you followed tells you the effective inverse curvature along that direction. In a locally quadratic region where the Hessian is positive definite, this ratio is exactly the inverse diagonal Hessian element. In non-quadratic terrain, it is a local approximation that adapts as the loss surface changes.
Approximating the numerator with past updates. The current parameter update $\Delta x_t$ is not known when computing the step — it's what we're trying to compute. The paper assumes the curvature is locally smooth (doesn't change drastically from one iteration to the next) and approximates the numerator using the RMS of past parameter updates over a window, computed with the same exponential decay mechanism as the gradient RMS:
where $E[\Delta x^2]_t$ is the running exponentially decaying average of squared parameter updates, $\Delta x^2_t$ is the element-wise square of the parameter update at iteration $t$, and $\text{RMS}[\Delta x]_t$ is the root-mean-square of recent parameter update magnitudes. The $\epsilon$ constant is added for the same conditioning reason as in the gradient RMS.
The lagged numerator. Critically, the paper uses $\text{RMS}[\Delta x]_{t-1}$ — the RMS from the previous iteration — in the numerator (Equation 14):
This lagging is necessary because $\Delta x_t$ itself is computed from this equation, so it cannot be used to update $E[\Delta x^2]_t$ before being computed. The computation order is: compute $\Delta x_t$ using $\text{RMS}[\Delta x]_{t-1}$ → apply the update → compute $E[\Delta x^2]_t$ from $\Delta x_t^2$ for use in the next iteration.
Why this form — the ratio interpretation. The update can be understood as a two-part scaling:
The ratio $\text{RMS}[\Delta x]_{t-1} / \text{RMS}[g]_t$ is an adaptive, per-dimension, time-varying step size that replaces the global learning rate $\eta$. It automatically adjusts based on two signals:
- If gradients are large (
$\text{RMS}[g]_t$is high), the denominator makes the step size small — preventing overshooting in steep regions of the loss surface. This is the same intuition as ADAGRAD's per-dimension scaling. - If recent updates have been large (
$\text{RMS}[\Delta x]_{t-1}$is high), the numerator makes the step size large — providing momentum-like acceleration along dimensions where consistent progress is being made. This is analogous to how momentum accumulates velocity along consistent gradient directions.
The ratio has the correct units: $\text{RMS}[\Delta x]_{t-1}$ has units of $x$ (it's an RMS of parameter changes), $\text{RMS}[g]_t$ has units of $1/x$ (it's an RMS of gradients), so the ratio has units of $x^2$ — and multiplying by $g_t$ (units of $1/x$) yields $\Delta x_t$ with units of $x$, matching the parameters. The unit mismatch is resolved without any manually tuned learning rate.
Why this form — the Hessian approximation interpretation. The ratio $\text{RMS}[\Delta x]_{t-1} / \text{RMS}[g]_t$ can also be understood as an approximation to the inverse diagonal Hessian element $1 / (\partial^2 f / \partial x^2)$. From the rearrangement $1 / (\partial^2 f / \partial x^2) = \Delta x / (\partial f / \partial x)$, the ratio of RMS quantities approximates the ratio of instantaneous quantities, averaged over a recent window to reduce noise. The assumption of a diagonal Hessian means each parameter's curvature is treated independently — cross-parameter interactions are ignored, which is a standard simplifying assumption in adaptive methods (also used by Becker and LeCun [5] and ADAGRAD).
Why this form — the positivity guarantee. Since both RMS quantities are always non-negative (they're square roots of sums of squares), the ratio is always non-negative, and the update always follows the negative gradient direction $-g_t$. This is the same guarantee provided by Becker and LeCun's absolute-value diagonal Hessian (Equation 6), and it ensures the method is a descent method — each step moves in a direction that decreases the loss (to first order). This would not be guaranteed if the Hessian approximation could become negative (which can happen with exact diagonal Hessian elements in non-convex regions), potentially causing the method to ascend rather than descend.
The role of $\epsilon$ in the numerator. The constant $\epsilon$ is added to $E[\Delta x^2]_{t-1}$ inside the square root: $\text{RMS}[\Delta x]_{t-1} = \sqrt{E[\Delta x^2]_{t-1} + \epsilon}$. The paper notes that this serves two purposes: (1) at the very first iteration, $\Delta x_0 = 0$ so $E[\Delta x^2]_0 = 0$, and without $\epsilon$ the numerator would be zero, making the first update zero — training would never start; (2) later in training, if parameter updates become very small (near convergence), $\epsilon$ prevents the numerator from becoming zero, ensuring that some progress can still be made rather than stalling completely. In both cases, $\epsilon$ acts as a tiny "seed" that keeps the optimization alive when statistics are degenerate.
The Complete ADADELTA Algorithm
Algorithm 1 in the paper (reproduced at the end of Section 3) specifies the exact computation at each iteration. The procedure is:
Initialization (before any updates):
$E[g^2]_0 = 0$for all parameters (accumulator for squared gradients)$E[\Delta x^2]_0 = 0$for all parameters (accumulator for squared parameter updates)$x_1$is the initial parameter vector
At each iteration $t$ from 1 to $T$:
-
Compute the gradient
$g_t$for the current mini-batch. This is the standard backpropagation step — no modification. -
Update the gradient accumulator: This mixes the previous running average with the new squared gradient, weighted by
$\rho$. All operations are element-wise (per-dimension). -
Compute the parameter update: where
$\text{RMS}[\Delta x]_{t-1} = \sqrt{E[\Delta x^2]_{t-1} + \epsilon}$and$\text{RMS}[g]_t = \sqrt{E[g^2]_t + \epsilon}$. This is the core ADADELTA step: per-dimension adaptive step size times negative gradient direction. -
Update the parameter update accumulator: This mixes the previous running average of squared updates with the newly computed squared update, preparing the numerator for the next iteration.
-
Apply the update to parameters:
What the algorithm computes: at each iteration, for each parameter independently, it computes a step size as the ratio of how much that parameter has historically moved to how large its recent gradients have been, then takes a step in the negative gradient direction scaled by this ratio. The two exponential averages are updated to incorporate the new information, and the process repeats.
Computational overhead. The paper characterizes the cost as "minimal computational overhead beyond vanilla stochastic gradient descent" (Abstract) and "a trivial amount of extra computation per iteration over gradient descent" (Section 1). Concretely, per parameter per iteration, ADADELTA requires:
- 2 multiply-adds for the exponential averages (one for
$E[g^2]$, one for$E[\Delta x^2]$) - 2 square roots (one for each RMS)
- 1 division (the ratio)
- 1 multiplication by
$-g_t$
This is $O(d)$ for $d$ parameters and entirely negligible compared to the forward and backward passes through the network, which dominate the computational budget. The memory overhead is $2d$ (storing $E[g^2]$ and $E[\Delta x^2]$ per parameter), also negligible relative to the model parameters and optimizer states of modern methods.
Design Choices and Why Alternatives Were Rejected
Why exponential moving average instead of a hard window of size $w$?
The paper states that "storing $w$ previous squared gradients is inefficient." A hard window would require maintaining a circular buffer of $w$ values per parameter and recomputing the sum each iteration (or maintaining it with add-subtract operations). For a model with $10^7$ parameters and a window of $w = 100$, this would be $10^9$ stored values — gigabytes of memory purely for optimizer state. The exponential moving average uses $O(1)$ memory per parameter regardless of the effective window size. The decay rate $\rho$ maps to an effective window size of approximately $1/(1-\rho)$ iterations (the time constant of the exponential filter), so $\rho = 0.95$ gives an effective window of ~20 iterations, $\rho = 0.99$ gives ~100 iterations, etc.
Why the RMS of parameter updates specifically as the Hessian proxy?
The paper's derivation (Equation 13) shows that the inverse diagonal Hessian equals $\Delta x / (\partial f / \partial x)$. Any running statistic of $\Delta x$ could potentially serve as a numerator, but the RMS has several desirable properties:
- Always positive: ensures descent direction is always followed (unlike instantaneous
$\Delta x / g$which could be negative if the gradient and update had opposite signs due to noise). - Smooth: the exponential average reduces noise, providing stable step sizes that don't fluctuate wildly between iterations — critical for convergence in stochastic settings.
- Consistent with the denominator: using RMS for both numerator and denominator creates a symmetric structure where both quantities are on the same "scale" (standard-deviation-like measures), making the ratio well-behaved (typically order 1 after an initial transient).
- Unit-correct: RMS of
$\Delta x$has units of$x$, just as RMS of$g$has units of$1/x$, so the ratio inherits the correct units of$x^2$.
An alternative would be to use the $\ell_2$ norm of past $\Delta x$ directly (without squaring and square-rooting), but this would not produce the RMS form and would have different statistical properties (more sensitive to outliers, different noise averaging behavior). The paper does not discuss this alternative explicitly, but the choice of RMS is consistent with the denominator's form and the standard practice of using second-moment statistics in adaptive methods.
Why the decay rate $\rho$ is shared between the gradient and update accumulators?
The paper uses the same $\rho$ for both $E[g^2]$ and $E[\Delta x^2]$. This is not mathematically required — one could use different decay rates $\rho_g$ and $\rho_{\Delta x}$ — but sharing the parameter simplifies the method (one fewer hyperparameter to tune) and ensures the two running averages have the same effective memory length, which is reasonable if the relevant timescale for both gradient statistics and curvature changes is similar. The paper's hyperparameter sensitivity experiments (Table 2) only vary a single $\rho$, implicitly assuming the shared setting.
Why the diagonal Hessian approximation instead of a full or block-diagonal one?
The paper explicitly states "this derivation made the assumption of diagonal curvature so that the second derivatives could easily be rearranged." A diagonal Hessian treats each parameter's second derivative independently — ignoring off-diagonal elements that capture interactions between different parameters. This is a standard approximation in scalable optimization (used by ADAGRAD, RMSProp, Adam) because:
- Storage: a full Hessian requires
$O(d^2)$memory, infeasible for large$d$. - Computation: computing and inverting the full Hessian requires
$O(d^3)$or$O(d^2)$operations, prohibitive for large models. - Empirical adequacy: the diagonal approximation captures the dominant per-dimension scale variations (which can span orders of magnitude across layers) while ignoring cross-dimension correlations that are often less critical for step size selection.
Block-diagonal approximations (e.g., KFAC) are more accurate but substantially more expensive and were developed later than this paper.
Why $\text{RMS}[\Delta x]_{t-1}$ is specifically lagged rather than using a contemporaneous estimate?
The lagging is a logical necessity, not a design choice: $\Delta x_t$ depends on $\text{RMS}[\Delta x]_{t-1}$ through Equation 14, so the RMS cannot be computed from $\Delta x_t$ until after $\Delta x_t$ is computed. Using a contemporaneous $\text{RMS}[\Delta x]_t$ would create a circular dependency that would require solving a fixed-point equation. The paper notes an interesting side effect of this forced lagging: it provides robustness to sudden large gradients.
"An interesting side effect of this is that the system is robust to large sudden gradients which act to increase the denominator, reducing the effective learning rate at the current time step, before the numerator can react."
When an unusually large gradient occurs (e.g., from a noisy mini-batch or a sharp region of the loss surface), $\text{RMS}[g]_t$ increases immediately (since it's computed from the current gradient), making the step size small and preventing a destabilizing large update. The numerator $\text{RMS}[\Delta x]_{t-1}$ doesn't react until the next iteration (after $\Delta x_t$ has been incorporated into the update accumulator), providing a one-iteration damping buffer. This is a form of implicit gradient clipping that emerges naturally from the lagged structure.
Why $\epsilon$ is added to both numerator and denominator rather than just the denominator?
The $\epsilon$ in the denominator serves the standard purpose of preventing division by zero (as in Becker and LeCun [5] and ADAGRAD). The $\epsilon$ in the numerator serves a different purpose: ensuring the very first update is non-zero (since $E[\Delta x^2]_0 = 0$ at initialization) and preventing the update from stalling if parameter changes become extremely small late in training. Without $\epsilon$ in the numerator, $\text{RMS}[\Delta x]_0 = 0$ would make $\Delta x_1 = 0$ regardless of the gradient, and training would never start. The paper uses the same $\epsilon$ value for both, which is a simplification — one could use different values — but given that $\epsilon$ is shown to be insensitive across orders of magnitude (Table 2), the shared value is sufficient.
Why this approach over Schaul et al.'s method?
The paper acknowledges the similarity to Schaul et al. [6] (end of Section 3), which also achieves unit-correct, learning-rate-free updates. The key difference is the source of curvature information:
- Schaul et al.: requires computing the diagonal Hessian
$\text{diag}(H_t)$, which doubles the computational cost per iteration (one additional forward-backward pass). This is a substantial overhead for large models. - ADADELTA: approximates the inverse diagonal Hessian using
$\text{RMS}[\Delta x]_{t-1} / \text{RMS}[g]_t$, which is computed entirely from quantities already available — gradients ($g_t$) and the update that was just computed ($\Delta x_t$, used to update the accumulator for the next iteration). The computational overhead is "only one gradient computation per iteration" — exactly the same as SGD.
The tradeoff is accuracy of the curvature approximation versus computational cost. Schaul et al. use the exact (albeit noisy per-mini-batch) diagonal Hessian; ADADELTA uses a first-order approximation that assumes local smoothness. The paper's experiments implicitly test whether this cheaper approximation is adequate by comparing ADADELTA's convergence to other methods on realistic tasks.
4. Key Insights and Innovations
Innovation 1: Unit Correctness as a Design Principle for First-Order Optimizers
The most distinctive intellectual move in this paper is not any particular mathematical trick but rather the reframing of the learning rate problem as a unit mismatch problem. This is a genuinely novel diagnostic lens through which to evaluate and design optimization algorithms.
What the field did before. Prior to ADADELTA, the learning rate was universally treated as a hyperparameter to be tuned — a knob whose value mattered enormously but whose theoretical justification was thin. Methods like ADAGRAD (Duchi et al., 2010) provided per-dimension scaling but still required a global learning rate η, implicitly acknowledging that something needed to supply the missing units, without explicitly naming the problem as a units issue. Second-order methods like Newton's method and Becker and LeCun's diagonal Hessian approach [5] produced unit-correct updates, but the connection between unit correctness and the need for a learning rate was not articulated as a design principle — it was an incidental property of using curvature information.
The conceptual move. The paper makes explicit what was previously implicit: the gradient ∂f/∂x has units of 1/x (assuming a unitless cost function), so the SGD update Δx = −η g_t has units of 1/x — it doesn't match the parameters' units of x. The learning rate η exists to patch this mismatch by supplying units of x². This observation transforms the learning rate from a mysterious hyperparameter into a dimensional analysis problem: if you can construct a quantity with units of x² from first-order statistics, you can eliminate η entirely. This framing is the intellectual engine behind the entire method — the specific choice of RMS[Δx]_{t-1} / RMS[g]_t as that x²-valued quantity follows directly from the dimensional argument, not from empirical tuning.
Why this is fundamental, not incremental. This is a fundamental reframing rather than a refinement of existing methods. It changes the question from "how should we anneal the learning rate?" to "how do we construct an update with correct physical units using only first-order information?" The answer — approximate the inverse diagonal Hessian using the ratio of past parameter changes to past gradients — is both principled (derived from Newton's method, Equation 13) and practical (cheap to compute). Nothing in prior work on adaptive learning rates used dimensional analysis as an explicit design constraint; the paper elevates unit correctness from an afterthought to a first principle of optimizer design.
Evidence anchoring. The unit-correctness argument leads directly to the ADADELTA update rule (Equation 14), but its significance goes beyond this particular rule. It provides a framework for evaluating any optimizer: does it produce updates with the correct units? SGD and momentum do not; ADAGRAD does not; ADADELTA and second-order methods do. This diagnostic criterion is portable to future method development. The paper doesn't just propose a method; it gives the field a way to think about the learning rate problem that wasn't articulated before.
Innovation 2: First-Order Curvature Approximation via the Update-to-Gradient Ratio
The second conceptual innovation is the specific technique for approximating second-order curvature without computing second derivatives: use the ratio of past parameter updates to past gradients as a running estimate of the inverse diagonal Hessian.
What the field did before. Prior approaches to curvature-aware optimization fell into two camps: (1) compute or approximate the Hessian explicitly (Schaul et al., 2012 [6], which required an extra forward-backward pass per iteration to compute the diagonal Hessian; Becker and LeCun, 1988 [5], with similar cost), or (2) use per-dimension gradient statistics without explicit curvature modeling (ADAGRAD's denominator, which provides scaling but not a principled curvature estimate). The first camp produced accurate curvature information at prohibitive cost; the second camp was cheap but provided no theoretical connection to the Hessian.
The conceptual move. The paper's key insight — embodied in Equation 13 — is that if you rearrange Newton's method under a diagonal Hessian assumption, the inverse second derivative equals Δx / (∂f/∂x). That is, the curvature information you need is encoded in the relationship between the steps you've already taken and the gradients that drove those steps. If you've been taking large steps in response to small gradients along a particular dimension, that dimension has low curvature (flat terrain — safe to take big steps). If you've been taking small steps despite large gradients, that dimension has high curvature (steep terrain — need small steps). The ratio RMS[Δx]_{t-1} / RMS[g]_t is essentially a running, smoothed estimate of this curvature relationship.
Why this is more than just a trick. This insight bridges the gap between first-order and second-order methods in a way that's both computationally cheap and theoretically motivated. It's not a heuristic — it falls directly out of Newton's method under the assumption of locally smooth, diagonal curvature. The paper shows that you don't need to compute Hessian-vector products or maintain a Fisher information matrix estimate to get curvature-aware step sizes; you just need to keep track of what your optimizer has been doing. This is a fundamental insight about the information content of optimizer trajectories: the history of updates and gradients jointly contains a usable curvature signal.
The significance beyond ADADELTA. This idea — that the ratio of parameter changes to gradients approximates inverse curvature — generalizes beyond this specific method. It suggests that any optimizer maintaining state about past updates (momentum buffers, Adam's first and second moment estimates) implicitly encodes curvature information. The paper makes this encoding explicit and uses it as the primary mechanism for step size adaptation. This is a conceptual contribution that subsequent methods (including Adam, which combines momentum-like and RMS-like terms differently) implicitly build upon, even if they don't use the exact ratio form.
Evidence anchoring. The effectiveness of this approximation is demonstrated indirectly through the MNIST experiments (Table 2), where ADADELTA achieves competitive performance (1.83% test error at best settings) without any second-order computation, and more directly through the layer-dependent step size patterns in Figure 2, which show the method automatically assigning larger effective learning rates to lower layers (smaller gradients, need bigger steps) and smaller rates to upper layers — exactly the behavior a curvature-aware method should produce.
Innovation 3: A Unified Diagnostic Framework Revealing Why Prior Methods Fail
The paper's third contribution is less a single method and more a taxonomy of failure modes that explains why existing adaptive methods fall short, organized around two orthogonal problems: continual decay and unit mismatch. This diagnostic framework clarifies the landscape of first-order optimizers in a way that the paper's precursors did not.
What the field had before. Prior work presented a menu of methods — momentum, ADAGRAD, second-order approximations — each with their own update rules, but without a clear framework for understanding why some problems resist solution. ADAGRAD was known to slow down late in training, but the mechanism (infinite accumulation driving the denominator to infinity) was described as a property of the method rather than a symptom of a broader design flaw (the use of an infinite window). The unit mismatch problem in SGD and momentum was simply not discussed at all — everyone accepted that learning rates needed tuning without asking why a global scalar was structurally necessary.
The conceptual move. The paper decomposes the adaptive learning rate problem into two independent axes:
-
The accumulation window problem: How much gradient history should inform the current step size? ADAGRAD's infinite window causes continual decay; a finite window (implemented via exponential decay) prevents it. This is the focus of Section 3.1.
-
The unit correctness problem: How do we supply the missing units of x² without a manually tuned learning rate? ADAGRAD's denominator provides per-dimension scaling but is unitless (g_t / RMS[g]t is dimensionless), so η is still needed. Second-order methods solve this with Hessian information, but at high cost. The RMS[Δx]{t-1} numerator supplies the missing units cheaply. This is the focus of Section 3.2.
Why this framing matters. This decomposition turns what could be an ad-hoc method proposal into a principled synthesis. ADADELTA is not a random combination of exponential averaging and RMS ratios — it's the minimal modification to ADAGRAD that simultaneously fixes the window problem (exponential decay instead of infinite sum) and the unit problem (update RMS numerator instead of global η). Each piece of ADADELTA addresses a specific, named failure mode of a predecessor. The paper makes this explicit:
"We created our ADADELTA method to overcome the sensitivity to the hyperparameter selection as well as to avoid the continual decay of the learning rates." (Section 2.2.2)
Significance for method development. This diagnostic approach provides a template for future optimizer design: identify what structural property is missing (correct units? finite memory? per-dimension scaling? momentum?) and add a term that supplies it, rather than proposing black-box update rules and tuning them empirically. The paper's own Figure 1 can be read through this lens: ADAGRAD fails due to infinite accumulation (problem 1, visible as the post-epoch-10 plateau); SGD fails due to lack of per-dimension scaling and unit correctness (problem 2, visible as poor overall performance); momentum fails due to dependence on a tuned η (problem 2, visible in Table 1's extreme sensitivity). ADADELTA fixes both, and Figure 1 shows it matching ADAGRAD's early speed while continuing to improve — exactly the behavior predicted by the diagnostic framework.
Evidence anchoring. The failure mode analysis is validated by the controlled experiments in Table 1 vs. Table 2. Table 1 shows that SGD, momentum, and ADAGRAD all catastrophically fail under incorrect η (e.g., SGD at 89.68%, momentum at 89.68% with η = 1; ADAGRAD jumping from 1.79% to 43.76% as ϵ varies across orders of magnitude — though note ADAGRAD's sensitivity is to its own hyperparameter, not a learning rate directly). Table 2 shows ADADELTA varying only 0.6 percentage points (1.83% to 2.59%) across a 6-order-of-magnitude range of ϵ and a decade range of ρ — confirming that the structural fixes genuinely eliminate the sensitivity, not just mask it with a better default.
Innovation 4: Robustness to Sudden Large Gradients via Lagged Update Statistics
The paper identifies an emergent property of its lagged numerator design — robustness to gradient spikes — that was not an explicit design goal but emerges from the forced one-iteration lag between the denominator (computed from current gradients) and the numerator (computed from past updates). This is a subtle but practically important insight about how optimizer state can provide implicit gradient clipping.
What makes this distinctive. The paper notes:
"An interesting side effect of this is that the system is robust to large sudden gradients which act to increase the denominator, reducing the effective learning rate at the current time step, before the numerator can react."
This is not gradient clipping in the explicit sense (where gradients are rescaled to a maximum norm), nor is it the kind of damping provided by momentum (which smooths gradient directions over time). It's a one-iteration asymmetric damping: when an unusually large gradient appears, RMS[g]t spikes immediately (since it's updated with the current gradient), making the denominator large and the step size small for this iteration. The numerator RMS[Δx]{t-1} is based on previous updates, which were computed from smaller gradients, so it hasn't yet grown to match the new gradient scale. The result is that the current step is automatically downsized relative to what it would be if both numerator and denominator responded simultaneously. Only in the next iteration, after the current (now-small) update has been incorporated into E[Δx²], does the numerator adjust — but by then the gradient spike may have passed.
Why this matters beyond the specific mechanism. This is an early example of what would later become a more explicit design principle in optimizers: using temporal asymmetry in state updates to provide robustness. Modern methods like AdamW and LAMB incorporate various forms of debiasing and normalization, but the core idea that lagged statistics can provide implicit stabilization without additional hyperparameters (like a gradient clipping threshold) originates here as an observed emergent property. The paper doesn't make a big theoretical deal of this — it's presented as an "interesting side effect" — but it's a genuine insight about the dynamics of adaptive methods that wasn't articulated in prior work on momentum or ADAGRAD.
Evidence anchoring. The paper doesn't provide a dedicated ablation showing the effect of the lag (e.g., comparing to a version where numerator and denominator are both based on t − 1), so this insight is more conceptual than empirically validated. However, the speech recognition experiments with 200 distributed replicas (Figure 4) demonstrate ADADELTA's robustness under conditions of high gradient noise — exactly the scenario where gradient spikes are most common — without any explicit gradient clipping. The method "performs well, quickly converging to the same frame accuracy as the other methods" despite the "significants amount of noise" from 200 asynchronous replicas, consistent with the implicit spike-damping mechanism.
Innovation 5: Demonstration That Hyperparameter Insensitivity Is Achievable in First-Order Methods
The paper's claim that ADADELTA is "insensitive to hyperparameters" (Abstract) and "robust to ... selection of hyperparameters" (Abstract) is backed by an experiment (Table 2) that constitutes an empirical finding with significant practical implications, even if the theoretical basis for the insensitivity follows from the method's construction.
What the field had before. Prior adaptive methods — particularly ADAGRAD — were known to be less sensitive than SGD to learning rate choice, but still required careful tuning of their own hyperparameters (the global η in ADAGRAD; the decay constant and ϵ in various methods). The idea that an optimizer could work well with any reasonable setting of its hyperparameters, without tuning, was aspirational but not demonstrated. Schaul et al.'s method [6] aimed for this but required Hessian computation; no prior first-order method had shown flat performance across orders-of-magnitude hyperparameter variation.
What the paper shows. Table 2 reports MNIST test error after 6 epochs for ADADELTA across 12 hyperparameter combinations: ρ ∈ {0.9, 0.95, 0.99} and ϵ ∈ {1e−2, 1e−4, 1e−6, 1e−8}. The range of ϵ spans 6 orders of magnitude; ρ varies across most of its valid range [0, 1). The test error varies from 1.83% (ρ = 0.95, ϵ = 1e−6) to 2.59% (ρ = 0.9, ϵ = 1e−2) — a range of only 0.76 percentage points. This is in stark contrast to Table 1, where SGD test error varies from 2.26% to 58.10%, momentum from 2.03% to 89.68%, and ADAGRAD from 1.79% to 43.76% across similar ranges of their respective hyperparameters.
Why this is a genuine innovation, not just a good result. The insensitivity is not accidental — it follows from the method's construction. Because the update is a ratio of two RMS quantities that respond to the same decay dynamics and both contain the same ϵ, the method is approximately invariant to the scale of ϵ as long as ϵ is small relative to the typical values of E[g²] and E[Δx²] during active training. When gradients and updates are large, ϵ is negligible in both numerator and denominator, and they cancel. When gradients and updates become very small near convergence, ϵ dominates in both, and the ratio RMS[Δx] / RMS[g] converges to 1 — an effective learning rate that allows continued, smooth progress without the explosion that would occur in other methods. The paper notes this convergence in Section 4.3's discussion of Figure 2:
"Near the end of training these step sizes converge to 1. This is typically a high learning rate that would lead to divergence in most methods, however this convergence towards 1 only occurs near the end of training when the gradients and parameter updates are small."
The step size of 1 is safe precisely because the gradient is tiny — the actual parameter change Δx = −1 · g_t is proportionally tiny. This automatic annealing toward a stable final learning rate is a property that emerges from the ratio structure and the shared ϵ, not something that was manually scheduled or tuned. It's a form of implicit learning rate decay that requires no schedule, no monitoring, and no additional hyperparameters — and it's the reason the hyperparameter insensitivity holds.
Significance for practice. This finding — that a first-order method can be effectively hyperparameter-free — was important for the practical adoption of adaptive methods in deep learning. While ADADELTA itself was eventually superseded by Adam (which reintroduced a learning rate but combined momentum-like and RMS-like terms differently), the demonstration that hyperparameter insensitivity is achievable set a standard that subsequent methods could be measured against. It also directly motivated the paper's large-scale speech experiments (Section 4.4), where the authors explicitly note: "the hyperparameters did not need to be tuned, showing that ADADELTA is a robust learning rate method that can be applied in a variety of situations" — from MNIST with 500 hidden units to speech models with 2560 hidden units and 100–200 distributed replicas, all with the same ρ = 0.95, ϵ = 1e−6.
Evidence anchoring. The claim is directly supported by Table 2 (MNIST hyperparameter sweep), Figure 3 (speech with 100 replicas, using the same hyperparameters as MNIST without tuning), and Figure 4 (speech with 200 replicas and rectified linear units, again with the same hyperparameters). The robustness across model sizes (500/300 hidden units vs. 4 layers of 2560 units), nonlinearities (tanh, logistic, ReLU), data modalities (images, speech), and distributed settings (1 to 200 replicas) with zero hyperparameter adjustment is the empirical backbone of the insensitivity claim.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. The MNIST handwritten digit classification dataset, consisting of 60,000 training images and 10,000 test images of 28×28 grayscale digits. The speech dataset comprises "several hundred hours of US English data collected using Voice Search, Voice IME, and read data," with inputs being 26 frames of 40 log-energy filter bank outputs and outputs being 8,000 senone labels from a GMM-HMM forced alignment system.
-
Base model(s). For MNIST: a feedforward neural network with two hidden layers — 500 units in the first layer and 300 in the second — followed by a softmax output layer. Both tanh and rectified linear (ReLU) activation functions are tested. For speech: a neural network with 4 hidden layers of 2560 units each, trained with either logistic or ReLU nonlinearities, using the distributed training system of Dean et al. (2012) [4] with 100 or 200 model replicas. The paper justifies the model choice by noting that ReLU units "work better in practice than tanh, and their non-saturating nature further tests each of the methods at coping with large variations of activations and gradients" (Section 4.1).
-
Metrics. MNIST: test set error rate (percentage of misclassified digits) after a specified number of epochs. Speech: frame classification accuracy on the test set (percentage of frames correctly assigned to their senone label). Both are standard supervised classification metrics computed by comparing network outputs to ground truth labels. The cross-entropy objective is used for training in all experiments.
-
Baselines. Three methods are compared throughout: (1) SGD — vanilla stochastic gradient descent with fixed global learning rate η (Robbins and Monro, 1951 [1]); (2) Momentum — SGD with a momentum term controlled by decay constant ρ and global learning rate η (Rumelhart et al., 1986 [2]); (3) ADAGRAD — per-dimension adaptive learning rates via accumulation of squared gradients with a global learning rate η (Duchi et al., 2010 [3]). The paper also references the method of Schaul et al. (2012) [6] as a comparison point for MNIST test error at 6 epochs (2.10% reported), though no head-to-head training run is performed — only the published number is cited.
-
Generation budget / compute accounting. The paper does not use FLOPs or parameter-update counts as a primary comparison metric. Instead, all comparisons are made at the same number of training epochs or wall-clock time. For MNIST, results are reported at 6 epochs (for hyperparameter sensitivity) and 50 epochs (for convergence behavior, Figure 1). For speech, training progress is measured in hours of wall-clock time (Figures 3 and 4), reflecting the distributed training environment where computational throughput is the practical constraint. The paper argues ADADELTA has "minimal computational overhead beyond vanilla stochastic gradient descent" and "a trivial amount of extra computation per iteration over gradient descent" (Abstract, Section 1), but does not provide wall-clock timing comparisons per epoch.
-
Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. MNIST uses the standard train/test split. For the speech experiments, there is no mention of a held-out validation set used for hyperparameter selection — the paper states that "the same settings of ϵ = 1e−6 and ρ = 0.95 from the MNIST experiments were used for this setup" (Section 4.4), meaning hyperparameters were transferred directly with zero tuning on the speech data. This is intentional: it demonstrates the claimed insensitivity by showing that defaults from a completely different task work without adjustment.
Main Quantitative Results
MNIST: Fast Initial Convergence and Competitive Final Performance
Headline result (6 epochs, tanh activations). ADADELTA achieves 2.00% test error after 6 epochs of training with tanh nonlinearities, 500-300 hidden units, and mini-batches of 100 images. This compares favorably to the 2.10% test error reported by Schaul et al. [6] (Section 4.1). Hyperparameters are ϵ = 1e−6, ρ = 0.95.
Headline result (6 epochs, ReLU activations, hyperparameter sweep). Across 12 hyperparameter combinations (ρ ∈ {0.9, 0.95, 0.99}; ϵ ∈ {1e−2, 1e−4, 1e−6, 1e−8}), ADADELTA achieves a minimum test error of 1.83% (ρ = 0.95, ϵ = 1e−6) and a maximum of 2.59% (ρ = 0.9, ϵ = 1e−2) — a range of only 0.76 percentage points (Table 2). All 12 combinations produce errors between 1.83% and 2.59%.
Convergence behavior over 50 epochs (Figure 1). Training the same network with ReLU activations for 50 epochs reveals distinct temporal dynamics for each method:
- SGD (optimal η from Table 1): performs worst throughout, with test error remaining higher than all other methods at every epoch.
- Momentum (optimal η from Table 1): shows the strongest final performance, converging to the lowest test error by epoch 50. The curve steadily decreases throughout training.
- ADAGRAD (optimal η from Table 1): matches the fastest initial convergence, outperforming momentum in the first ~10 epochs, but then plateaus — its error curve flattens while momentum and ADADELTA continue to improve. The paper attributes this to "the accumulations in the denominator which continually increase" (Section 4.1).
- ADADELTA (ρ = 0.95, ϵ = 1e−6): "matches the fast initial convergence of ADAGRAD while continuing to reduce the test error, converging near the best performance which occurs with momentum" (Section 4.1). Unlike ADAGRAD, ADADELTA does not plateau; unlike momentum, it requires no learning rate tuning.
The paper acknowledges that momentum ultimately converges to a slightly better final solution, and hypothesizes why in the discussion of Figure 2: "having no explicit annealing schedule imposed on the learning rate could be why momentum with the proper hyperparameters outperforms ADADELTA later in training" (Section 4.3). With momentum, oscillations near minima are damped; with ADADELTA, oscillations "can accumulate in the numerator," potentially preventing the finest convergence. The paper suggests adding an explicit annealing schedule to ADADELTA as future work.
Hyperparameter Sensitivity: ADADELTA vs. Baselines (Tables 1 and 2)
SGD sensitivity (Table 1, first column). With η = 1e0: 2.26% error. With η = 1e−4: 58.10% error (essentially random — a 10-class problem has 90% expected error for random guessing, but 58% likely reflects the network settling on a consistent incorrect strategy). The acceptable range spans roughly 2–3 orders of magnitude, with sharp degradation outside this range.
Momentum sensitivity (Table 1, second column). With η = 1e−1: 2.03% error. With η = 1e0: 89.68% error (divergence — the network fails to learn almost entirely). With η = 1e−2: 2.68%. The acceptable range is extremely narrow — performance collapses catastrophically when η increases by a single order of magnitude from 1e−1 to 1e0.
ADAGRAD sensitivity (Table 1, third column). The paper varies ϵ (ADAGRAD's conditioning constant) rather than η, reporting: ϵ = 1e−4 gives 1.79% error; ϵ = 1e0 gives 43.76% error. Note that ADAGRAD also has a global learning rate η which was presumably tuned separately — the table only sweeps ϵ. The sensitivity to ϵ (nearly 42 percentage points of variation) is still substantial.
ADADELTA sensitivity (Table 2). The method has two hyperparameters: ρ (decay rate) and ϵ (conditioning constant). The paper sweeps both jointly:
- ρ = 0.9: errors from 1.90% (ϵ = 1e−6) to 2.59% (ϵ = 1e−2)
- ρ = 0.95: errors from 1.83% (ϵ = 1e−6) to 2.58% (ϵ = 1e−2)
- ρ = 0.99: errors from 2.00% (ϵ = 1e−8) to 2.32% (ϵ = 1e−2)
The worst result (2.59%) is still competitive — better than any SGD result except the optimal one, and better than many of the ADAGRAD and momentum results. The paper's language is careful: "the two hyperparameters do not significantly alter performance" (Section 4.2). This is a qualitative claim supported by the stark contrast between the 0.76-point range in Table 2 and the 40–87 point ranges in Table 1.
Effective Learning Rate Dynamics (Figure 2)
Headline observation. Figure 2 visualizes step sizes (the ratio RMS[Δx]_{t-1} / RMS[g]_t) and parameter updates for 10 randomly selected dimensions in each of the 3 weight matrices during 25 epochs of MNIST training with tanh nonlinearities. The left column shows step sizes; the right column shows the corresponding Δx values. Data is plotted every 60 batches.
Layer-dependent adaptation. The paper identifies three properties from this visualization:
-
Compensating for vanishing gradients: "the step sizes, or effective learning rates ... are larger for the lower layers of the network and much smaller for the top layer at the beginning of training" (Section 4.3). This is exactly the behavior that gradient-based scaling should produce: lower layers receive smaller gradients during backpropagation (due to gradient attenuation through the network), and ADADELTA compensates by assigning them larger effective learning rates. The top layer, which receives the largest gradients, gets the smallest step sizes.
-
Convergence of step sizes to 1: "near the end of training these step sizes converge to 1" (Section 4.3). When gradients and parameter updates become very small (approaching convergence), both RMS[g]t and RMS[Δx]{t-1} shrink, and the ϵ constants in the numerator and denominator begin to dominate. Since the same ϵ is used in both, the ratio approaches ϵ/ϵ = 1. The paper notes: "This is typically a high learning rate that would lead to divergence in most methods, however this convergence towards 1 only occurs near the end of training when the gradients and parameter updates are small." The effective step is Δx = −1 · g_t, and since g_t itself is tiny, the actual parameter change is proportionally tiny — this is a form of implicit, smooth annealing.
-
Parameter updates smoothly tend toward zero: "when the step sizes become 1, the parameter updates (shown on the right of Fig. 2) tend towards zero. This occurs smoothly for each of the weight matrices effectively operating as if an annealing schedule was present" (Section 4.3). The simultaneous behavior of RMS[g]t → ϵ and RMS[Δx]{t-1} → ϵ means Δx = −(ϵ/ϵ) · g_t = −g_t, and since g_t → 0 near convergence, Δx → 0 naturally without any explicit schedule.
Large-Scale Speech Recognition (Figures 3 and 4)
Headline result (100 replicas, logistic nonlinearities, Figure 3). ADADELTA (ρ = 0.95, ϵ = 1e−6) compared against ADAGRAD on a speech recognition task with 100 distributed model replicas, logistic hidden units, and 4 hidden layers of 2560 units each. The paper reports: "Notice our method initially converges faster and outperforms ADAGRAD throughout training in terms of frame classification accuracy on the test set" (Section 4.4). The frame accuracy curves show ADADELTA maintaining a consistent advantage over ADAGRAD from approximately 20 hours onward, with the gap widening as training progresses. At 100 hours, ADADELTA achieves roughly 36–37% frame accuracy vs. approximately 34–35% for ADAGRAD.
Headline result (200 replicas, ReLU nonlinearities, Figure 4). ADADELTA compared against ADAGRAD and Momentum with 200 distributed replicas and rectified linear hidden units. Hyperparameters are again the MNIST defaults (ρ = 0.95, ϵ = 1e−6). The paper states: "Despite having 200 replicates which inherently introduces significants amount of noise to the gradient accumulations, the ADADELTA method performs well, quickly converging to the same frame accuracy as the other methods" (Section 4.4). All three methods — ADADELTA, ADAGRAD, and Momentum — converge to roughly 35–36% frame accuracy by 160 hours, with ADADELTA showing competitive or slightly faster initial convergence.
Practical significance. The method's hyperparameters from MNIST (a 500-300 unit network on 28×28 images) transfer without modification to a 4×2560-unit network on 40-dimensional filterbank features trained across a distributed cluster. This is the paper's strongest evidence for the claim of robustness "to various data modalities and selection of hyperparameters" (Abstract) — the same ϵ = 1e−6, ρ = 0.95 work on both tasks without tuning.
Ablation Studies and Robustness Checks
The paper does not present formal ablation studies in the modern sense (e.g., systematically removing components of ADADELTA and measuring degradation). However, several experimental choices serve as implicit robustness checks and comparative ablations:
Hyperparameter sweep as implicit sensitivity ablation (Table 2). While presented as a hyperparameter sensitivity analysis for ADADELTA, the comparison with Table 1 serves as a de facto ablation: what happens when you replace a manually tuned global learning rate with ADADELTA's adaptive ratio? The answer is that performance becomes effectively flat across hyperparameter choices that would catastrophically degrade competing methods. This is the central empirical claim of the paper.
Activation function robustness (MNIST: tanh vs. ReLU; Speech: logistic vs. ReLU). ADADELTA is tested with tanh (MNIST 6-epoch experiment), ReLU (MNIST 50-epoch experiment, Figure 1, Tables 1–2), logistic (speech with 100 replicas, Figure 3), and ReLU (speech with 200 replicas, Figure 4). The same hyperparameters (ρ = 0.95, ϵ = 1e−6) are used across all activation functions with no degradation requiring tuning. This implicitly ablates the concern that the method's dynamics depend on particular activation properties (e.g., saturating vs. non-saturating, bounded vs. unbounded). The paper specifically notes that ReLU's "non-saturating nature further tests each of the methods at coping with large variations of activations and gradients" (Section 4.1), making this a stress test.
Distributed noise robustness (100 vs. 200 replicas). Increasing the number of asynchronous model replicas from 100 to 200 increases the staleness and noise of gradient updates (since replicas compute gradients on slightly out-of-date parameters). ADADELTA's performance does not degrade — it "quickly converges to the same frame accuracy as the other methods" (Section 4.4) — implicitly demonstrating robustness to the increased gradient variance that distributed training introduces.
Missing ablation: the lagged numerator. The paper does not compare the standard ADADELTA (lagged numerator RMS[Δx]_{t-1}) against a version using a contemporaneous numerator (RMS[Δx]_t, which would require two iterations or a fixed-point solve) or a version without the numerator entirely (reverting to the intermediate update of Equation 10 with a fixed η). The robustness to sudden large gradients attributed to the lag (Section 3.2, final paragraph) is therefore a conceptual claim rather than an empirically verified property — there is no experiment showing that performance degrades when the lag is removed.
Missing ablation: finite window vs. infinite accumulation. The paper argues that replacing ADAGRAD's infinite sum with an exponential window prevents continual decay, but there is no direct comparison between ADAGRAD and a version of ADADELTA that uses ADAGRAD's infinite accumulation (with the numerator still present). The evidence is comparative across methods (ADADELTA vs. ADAGRAD in Figure 1), not an ablation of the window mechanism within ADADELTA itself.
Missing ablation: the ratio form vs. additive combination. ADADELTA's update uses a ratio (RMS[Δx]_{t-1} / RMS[g]_t). An alternative design would be an additive combination of numerator and denominator effects, as later used in Adam (which combines momentum-like and RMS-like terms as m_t / (√v_t + ϵ)). The paper does not test an additive variant.
Negative result: ADADELTA does not surpass well-tuned momentum at full convergence (Figure 1). This is an honest negative result that the paper does not hide. Despite its adaptive advantages and hyperparameter insensitivity, ADADELTA converges near — not beyond — the best performance of momentum with a properly tuned learning rate. The paper attributes this to the lack of explicit annealing, and this limitation is discussed in Section 4.3. This is a meaningful negative finding because it establishes a performance ceiling for the ratio-based approach: hyperparameter elimination and unit correctness come at a small cost in asymptotic convergence quality relative to a method that has the luxury of manual learning rate scheduling.
Critical Assessment
Claim 1: ADADELTA requires no manual tuning of a learning rate.
What was tested. The paper tests ADADELTA across 12 hyperparameter combinations (Table 2) spanning ρ ∈ {0.9, 0.95, 0.99} and ϵ ∈ {1e−2, 1e−4, 1e−6, 1e−8}, and shows that MNIST test error varies only from 1.83% to 2.59%. The same hyperparameters (ρ = 0.95, ϵ = 1e−6) are then used without modification on speech tasks with different architectures, activation functions, and distributed settings (Section 4.4, Figures 3–4), where they perform competitively.
What was not tested. The paper does not sweep ρ over a wider range approaching 0 or 1 (e.g., ρ = 0.5, ρ = 0.999). It does not test whether the same defaults work well on fundamentally different problem types (e.g., sequence modeling, computer vision beyond MNIST, generative modeling). It does not test sensitivity to the choice of initial accumulator values E[g²]₀ and E[Δx²]₀ — these are fixed to zero, and varying them might matter in some settings. The claim of "no manual tuning" is well-supported for the tested domains but its generality across deep learning workloads is assumed, not demonstrated.
Conditional assessment. The claim holds on the tested tasks provided the user accepts the paper's default hyperparameters (effectively tuning by paper recommendation rather than by search). For a practitioner deploying ADADELTA on a new problem, the paper offers ρ = 0.95 and ϵ = 1e−6 as a default that worked on MNIST and speech — this is a recommended setting drawn from the paper's experiments, not evidence that any setting of ρ and ϵ will work equally well on any problem. The 2.59% worst-case error in Table 2 shows some variation exists, even if small.
Claim 2: ADADELTA is insensitive to hyperparameters.
What was tested. Table 2 sweeps two hyperparameters jointly and shows a 0.76-percentage-point range on MNIST. Table 1 provides the contrast: SGD, momentum, and ADAGRAD vary by 40–87 percentage points across similar sweeps. The paper concludes that ADADELTA's hyperparameters "do not significantly alter performance."
What was not tested. The sweeps in Tables 1 and 2 are not directly comparable in meaning. Table 1 sweeps η (the learning rate) for SGD and momentum, and ϵ for ADAGRAD. These are fundamentally different hyperparameters with different roles. ADAGRAD's sensitivity to ϵ cannot be directly compared to ADADELTA's sensitivity to ϵ because ADADELTA places ϵ in both numerator and denominator (where it partially cancels), while ADAGRAD places it only in the denominator. A fairer comparison would report ADAGRAD's sensitivity to η, its learning rate, which the paper does not sweep. Similarly, ADADELTA's sensitivity to ρ is not directly comparable to momentum's sensitivity to η — they are different parameters. The comparison is illustrative rather than rigorous.
Conditional assessment. The claim is supported in a practical sense: ADADELTA is demonstrably more robust than the tested baselines under the tested variations. However, the "insensitivity" is relative, not absolute. A variation from 1.83% to 2.59% is small but could matter in settings where 0.76% test error separates state-of-the-art from runner-up (e.g., later MNIST results would reach errors well below 1%). The paper's characterization as "do not significantly alter performance" is reasonable for the scale of the task and era.
Claim 3: ADADELTA overcomes ADAGRAD's continual learning rate decay.
What was tested. Figure 1 shows ADAGRAD's test error plateauing after ~10 epochs while ADADELTA's continues to decrease. This is direct visual evidence of the infinite accumulation problem and ADADELTA's fix.
What was not tested. The paper does not test whether this advantage persists at much longer training horizons. If trained for 500 epochs instead of 50, would ADADELTA eventually stall as well (perhaps from the denominator becoming dominated by ϵ, causing step sizes to converge to 1)? The convergence of step sizes to 1 documented in Figure 2 suggests a different asymptotic regime than ADAGRAD's decay to zero, but the long-term consequences are not explored. There is also no test of whether a simple modification to ADAGRAD — e.g., using the same exponential moving average in its denominator but keeping the global η — would recover ADADELTA's continued progress without the update RMS numerator. This would isolate whether the finite window alone fixes ADAGRAD's problem or whether the full ADADELTA ratio is necessary.
Conditional assessment. The qualitative behavior is clearly demonstrated: ADAGRAD plateaus, ADADELTA does not. The causal attribution to the exponential window (Idea 1) is theoretically sound but not empirically isolated — the improvement could partially stem from the update RMS numerator (Idea 2) rather than the window alone.
Claim 4: ADADELTA provides per-dimension adaptive learning rates that compensate for layer-dependent gradient scales.
What was tested. Figure 2 visualizes step sizes for 10 dimensions in each of 3 weight matrices, showing systematically larger step sizes in lower layers and smaller step sizes in the top layer. This is exactly the expected behavior from the RMS[g]_t denominator: lower layers have smaller gradients and thus get larger effective learning rates; upper layers have larger gradients and get smaller rates.
What was not tested. The paper does not verify that this per-dimension adaptation actually improves optimization relative to a method that applies the same per-dimension scaling but without the update RMS numerator (i.e., Equation 10 with a tuned η). The ablation that would isolate the benefit of the per-dimension scaling is missing — there is no comparison where per-dimension adaptation is removed while keeping the rest of the method intact. The visualization is suggestive but does not prove causality.
Conditional assessment. The qualitative behavior is convincingly visualized and matches the mechanistic design. The causal link to improved optimization is supported only by the aggregate performance comparisons (Figures 1, 3, 4), which confound multiple design choices (finite window, unit correctness, per-dimension scaling). Isolating each mechanism's contribution would require targeted ablation experiments that are not present.
General assessment of experimental strength
Strengths. The paper's experiments are well-chosen to validate its four stated benefits (Section 1): (1) no manual learning rate — demonstrated by hyperparameter sweep (Tables 1–2) and cross-task transfer of defaults; (2) insensitive to hyperparameters — shown by the flat Table 2; (3) separate dynamic learning rate per dimension — visualized in Figure 2; (4) minimal computation over gradient descent — claimed but not empirically benchmarked; (5) robust to large gradients, noise, and architecture choice — tested via ReLU, distributed replicas, and activation function variation; (6) applicable in local or distributed environments — tested via single-machine MNIST and 100–200-replica speech. The paper's honesty about ADADELTA not surpassing tuned momentum at convergence (Figure 1) lends credibility.
Weaknesses. The experimental scope is narrow: one vision dataset (MNIST) and one internal speech dataset, both classification tasks. There are no experiments on recurrent networks, convolutional networks, generative models, or tasks with different loss landscapes (e.g., regression, ranking, reinforcement learning). The hyperparameter sweep (Table 2) tests only 12 combinations; a grid with finer granularity or random search might reveal failure regions. The computational overhead claim ("minimal") is stated but never measured — there are no wall-clock timing comparisons, FLOPs counts, or memory usage measurements against SGD or momentum. The speech results use internal Google data and a proprietary distributed system, making them unreproducible. There are no error bars, confidence intervals, or multiple random seeds reported for any experiment, making it impossible to assess whether the differences between methods in Figures 1, 3, and 4 are statistically significant.
Experiments that would have strengthened the paper. (1) A controlled ablation comparing ADADELTA to Equation 10 (the intermediate update rule with exponential window but a fixed global η), to isolate the benefit of the update RMS numerator. (2) A comparison to Schaul et al. [6] on the same hardware and dataset rather than citing their published number. (3) Wall-clock timing of one epoch for SGD vs. ADADELTA vs. momentum on the same model to quantify the "minimal" overhead claim. (4) Experiments on at least one non-classification task to test generality of the unit-correctness argument. (5) Training for more than 50 epochs on MNIST to characterize the asymptotic gap between ADADELTA and momentum observed in Figure 1. (6) A version of ADAGRAD modified to use exponential averaging (fixing the infinite accumulation problem) but still requiring a global η, to test whether ADADELTA's main advantage comes from the window or from eliminating η.
Overall. The experiments support the paper's headline claims at the level of "proof of concept" for a novel optimization method, but they do so in a narrow domain and without the rigorous ablation, statistical, and computational-cost analysis that would elevate the findings from promising to definitive. This is characteristic of a technical report introducing a new method — the goal is to establish feasibility and demonstrate advantages on representative tasks, leaving thorough characterization to subsequent work (which, for ADADELTA, largely occurred through the adoption and analysis of its ideas in RMSProp and Adam).
6. Limitations and Trade-offs
6.1 The Method Does Not Surpass Well-Tuned Momentum at Convergence
The assumption or constraint. ADADELTA eliminates the learning rate hyperparameter by deriving an adaptive step size from the ratio of two RMS quantities. The paper implicitly assumes that this adaptation can match or exceed the asymptotic convergence quality of methods that benefit from explicit learning rate scheduling. However, the experiments reveal a gap between this assumption and reality.
The consequence. On the MNIST classification task trained for 50 epochs, ADADELTA "converges near the best performance which occurs with momentum" but does not surpass or even match it — momentum with a properly tuned learning rate reaches a lower test error by epoch 50 (Figure 1, Section 4.1). Both methods continue improving throughout training, but momentum maintains a consistent advantage in the later epochs. This means ADADELTA's hyperparameter insensitivity comes at a cost: the method sacrifices some asymptotic optimization quality relative to what is achievable when learning rate tuning is performed well.
The paper diagnoses the likely cause in Section 4.3:
"having no explicit annealing schedule imposed on the learning rate could be why momentum with the proper hyperparameters outperforms ADADELTA later in training... With momentum, oscillations that can occur near a minima are smoothed out, whereas with ADADELTA these can accumulate in the numerator."
The accumulated oscillations in the numerator RMS[Δx]_{t-1} effectively create noise in the step size calculation near convergence, preventing the fine-grained descent that an annealing schedule enables. Momentum's explicit decay of velocity smooths these oscillations; ADADELTA has no mechanism to distinguish between genuine progress and noisy back-and-forth movement.
What evidence exists in the paper. Figure 1 shows the gap directly: after epoch 50, momentum's test error is visibly lower than ADADELTA's, with both curves still decreasing. Figure 2 provides mechanistic evidence: the step sizes converge to 1 near the end of training (left column), and the parameter updates smoothly tend toward zero (right column), but this convergence is driven by gradient shrinkage rather than by an explicit damping of oscillatory behavior. The paper does not present MNIST results beyond 50 epochs, so it is unknown whether ADADELTA would eventually close the gap or whether the gap would persist or widen.
Mitigation status. The paper does not attempt to fix this issue in the presented method. It explicitly flags it as a direction for future work:
"An annealing schedule could possibly be added to the ADADELTA method to counteract this in future work."
This is a genuine, self-acknowledged limitation. The authors are transparent that their hyperparameter-free method leaves some asymptotic performance on the table relative to a method that benefits from careful human-designed scheduling.
6.2 The Empirical Scope Is Limited to Feedforward Classification on Two Datasets
The assumption or constraint. The paper evaluates ADADELTA on exactly two tasks: MNIST handwritten digit classification (a feedforward network with 500 and 300 hidden units) and a large-scale speech recognition task (a feedforward network with 4 hidden layers of 2560 units). The method is proposed as a general-purpose drop-in replacement for SGD's learning rate, applicable to "any parameters for which a derivative can be obtained" (Section 1). Yet the experiments cover only supervised classification with feedforward architectures.
The consequence. The paper provides no evidence about ADADELTA's behavior on problem classes with fundamentally different loss landscapes or architectural properties. Specifically untested are:
- Recurrent neural networks, where gradient scales can vary dramatically across time steps due to vanishing and exploding gradient dynamics — a setting where per-dimension adaptation is arguably most needed and also most challenging.
- Convolutional neural networks, which were already state-of-the-art for vision tasks in 2012 (Krizhevsky et al., 2012) but are absent from the evaluation.
- Tasks with non-classification loss functions, such as regression (MSE loss), structured prediction, ranking losses, or generative modeling objectives — all of which have different gradient statistics and curvature properties that could interact differently with the RMS ratio.
- Very deep networks (the paper tests at most 4 hidden layers; modern networks were already going to 8+ layers in 2012), where the per-dimension scaling behavior documented in Figure 2 would be stressed across many more layers with proportionally more extreme gradient scale variations.
The speech recognition task demonstrates scaling to a larger model (4 × 2560 = 10,240 hidden units total) and distributed training, but the architecture remains feedforward and the task remains classification. The architectural diversity tested — feedforward with tanh, ReLU, or logistic activations — is important but narrow relative to the claim of broad applicability.
What evidence exists in the paper. All experimental evidence comes from Section 4: MNIST (Section 4.1–4.3) and speech (Section 4.4). There are no results for recurrent, convolutional, or generative architectures. There is no discussion of how the method's dynamics might differ on problems where gradients have different temporal or spatial correlation structures.
Mitigation status. None. The paper does not acknowledge the narrowness of its empirical scope as a limitation. It claims ADADELTA is "robust to... different model architecture choices" (Abstract) and "a robust learning rate method that can be applied in a variety of situations" (Section 5), but the "variety" of architectures tested is exclusively feedforward networks. The robustness to architecture choice is asserted rather than demonstrated across architecture families.
6.3 The Speech Experiments Are Unreproducible and Lack Standardized Benchmarks
The assumption or constraint. The paper's most impressive scaling demonstration — training on hundreds of hours of speech data with 100–200 distributed replicas — relies on internal Google infrastructure (the distributed system of Dean et al., 2012 [4]) and a proprietary dataset ("several hundred hours of US English data collected using Voice Search, Voice IME, and read data," Section 4.4). The exact data, training setup, and hyperparameters for the baseline methods on speech are not fully specified.
The consequence. The speech results cannot be independently reproduced or verified. A practitioner cannot run ADADELTA on the same speech task to confirm the reported performance or to benchmark their own implementation. More importantly, the results cannot be compared to subsequent work or standardized benchmarks, which makes it impossible to assess ADADELTA's performance relative to methods developed later (e.g., Adam, which would be introduced in 2014 and become the dominant adaptive optimizer). The internal nature of the speech data also means that the difficulty of the task — and thus the meaningfulness of the reported ~36% frame accuracy — is opaque to the reader.
Additionally, the paper provides no detail on how the baseline methods (ADAGRAD and Momentum) were tuned on the speech task. For the MNIST experiments, Table 1 reports sensitivity to hyperparameters for each baseline, making it clear that their performance depends heavily on tuning. For the speech experiments, the paper states only that ADADELTA used the same ρ = 0.95 and ϵ = 1e−6 from MNIST (Section 4.4). It is unclear whether the baselines were carefully tuned on the speech data or whether their hyperparameters were also transferred from MNIST — and if so, whether the transferred values were optimal. If ADAGRAD and Momentum were not tuned for the speech task while ADADELTA enjoyed cross-task hyperparameter robustness, the comparison in Figures 3 and 4 is systematically biased in ADADELTA's favor.
What evidence exists in the paper. Figures 3 and 4 present frame accuracy over wall-clock time for the speech experiments. Figure 3 compares ADADELTA vs. ADAGRAD; Figure 4 compares ADADELTA vs. ADAGRAD and Momentum. No hyperparameter sweep or sensitivity analysis is reported for any method on the speech data. No details are given about the baseline hyperparameters or their tuning procedure.
Mitigation status. None. The paper acknowledges the distributed system and data source in the text but treats the speech experiments as validation of the method's scalability rather than as reproducible benchmarks. There is no discussion of the reproducibility limitation. This was more accepted in the 2012 era — large-scale experiments at Google were understood to be infrastructure demonstrations — but it limits the enduring value of the empirical evidence for the method's claims.
6.4 The Computational Overhead Claim Is Never Quantified
The assumption or constraint. Throughout the paper, ADADELTA's computational cost is characterized qualitatively as "minimal" and "trivial." The Abstract claims "minimal computational overhead beyond vanilla stochastic gradient descent"; Section 1 claims "a trivial amount of extra computation per iteration over gradient descent"; and Section 5 claims "trivial computational overhead compared to SGD." These are strong claims about the method's practical deployability, but they are never supported with actual measurements.
The consequence. The reader cannot assess the true cost of adopting ADADELTA. The method requires per-parameter: two exponential moving average updates (each a multiply-add), two square root operations, one division, and one multiplication — plus storing two additional floating-point values per parameter (E[g²] and E[Δx²]). While these operations are indeed O(d) for d parameters and likely small relative to the forward and backward passes through a neural network, the actual wall-clock overhead is architecture-dependent and training-framework-dependent. For small models (like the 500-300 unit MNIST network), the optimizer overhead could be a non-trivial fraction of the per-iteration time. For models where gradient computation is cheap (e.g., fully-connected layers with small batch sizes), the optimizer cost might be noticeable.
The memory overhead — two additional floating-point values per parameter, tripling the optimizer state relative to SGD (which stores zero or one value) — is also not discussed. For a model with 100 million parameters, ADADELTA requires approximately 800 MB of additional memory (two 32-bit floats per parameter), which is small relative to activations and the model itself but not negligible.
What evidence exists in the paper. None. There are no wall-clock timing comparisons between ADADELTA and SGD or momentum for a fixed number of epochs. There are no FLOPs counts for the optimizer update relative to the forward-backward pass. There are no memory usage measurements. The speech experiments measure performance in wall-clock hours (Figures 3–4), but the x-axis is total training time, which conflates optimizer overhead with forward-backward computation and communication costs — it does not isolate the optimizer's contribution to runtime.
Mitigation status. None. The paper asserts minimal overhead without evidence. This was not uncommon for optimization papers of the era, which typically focused on convergence in terms of iterations or epochs rather than wall-clock time. However, for a method that explicitly positions itself as a practical, low-overhead alternative to SGD ("minimal computational overhead," "trivial amount of extra computation"), the absence of timing data is a gap between the claim and its support.
6.5 The Hyperparameter Insensitivity Claim Is Demonstrated on Only 12 Combinations Across a Narrow Range
The assumption or constraint. Table 2 reports ADADELTA's MNIST test error for 12 combinations of ρ ∈ {0.9, 0.95, 0.99} and ϵ ∈ {1e−2, 1e−4, 1e−6, 1e−8}. Based on this grid, the paper concludes that "the two hyperparameters do not significantly alter performance" (Section 4.2) and that the method is "insensitive to hyperparameters" (Abstract).
The consequence. The conclusion of hyperparameter insensitivity is drawn from a sparse grid over a limited range. Several potential failure regions are not tested:
- ρ approaching 0 (e.g., ρ = 0.5, ρ = 0.1): The exponential window becomes very short, making the RMS estimates highly noisy. This could cause step sizes to fluctuate wildly between iterations, potentially destabilizing training.
- ρ approaching 1 (e.g., ρ = 0.999): The effective window becomes very long, approaching ADAGRAD's infinite accumulation behavior. This could reintroduce the continual learning rate decay that ADADELTA was designed to prevent. At ρ = 0.99 (tested), the effective window is approximately 1/(1 − 0.99) = 100 iterations; at ρ = 0.999, it would be ~1000 iterations, and the dynamics might qualitatively change.
- ϵ much larger than the tested range (e.g., ϵ = 1e−1, ϵ = 1e0): When ϵ is comparable to or larger than typical values of
E[g²]andE[Δx²], the ratioRMS[Δx]_{t-1} / RMS[g]_tis dominated by ϵ/ϵ = 1 everywhere, reducing ADADELTA to vanilla SGD with a fixed learning rate of 1. This could cause divergence in early training when gradients are large, since a learning rate of 1 is typically far too high. - ϵ extremely small (e.g., ϵ = 1e−16 or 0): Numerical underflow or division by actual zero could occur when gradients truly vanish, particularly in low-precision arithmetic.
The tested grid also does not include the interaction between ρ and the model architecture — ρ = 0.95 might work well for the tested network sizes, but a different decay rate might be needed for networks with very different gradient timescales (e.g., RNNs with long temporal dependencies).
What evidence exists in the paper. Table 2 provides the only systematic evidence for hyperparameter insensitivity. The speech experiments (Figures 3–4) use a single setting (ρ = 0.95, ϵ = 1e−6) and do not test alternatives, so they demonstrate cross-task transfer of a single default, not insensitivity to variation on the speech task itself. There is no ablation showing that alternative ρ or ϵ values would produce similar speech results.
Mitigation status. The paper does not acknowledge the limited range of the hyperparameter sweep as a limitation. The insensitivity claim is stated without qualification. A practitioner adopting ADADELTA on a new task with very different gradient dynamics (e.g., a deep transformer with highly non-stationary gradient statistics) has no guidance on whether the default values remain safe or whether the claimed insensitivity generalizes beyond the tested grid and tasks.
6.6 The Method Has No Mechanism for Incorporating Momentum-Style Acceleration Into the Gradient Direction
The assumption or constraint. ADADELTA's update follows the raw negative gradient direction −g_t, with the adaptive step size RMS[Δx]_{t-1} / RMS[g]_t scaling the magnitude but not modifying the direction. The paper's derivation from Newton's method (Equation 13–14) assumes a diagonal Hessian, which implies that the optimal update direction is aligned with the negative gradient — curvature only rescales the step size, not the direction.
The consequence. This is a fundamental architectural limitation: ADADELTA cannot perform the kind of gradient-direction smoothing that makes momentum effective in ill-conditioned loss landscapes. In a long, narrow valley (a classic difficult optimization geometry), the negative gradient points almost perpendicular to the valley floor, zigzagging across the steep walls rather than progressing along the valley bottom. Momentum accumulates direction along the valley (where gradients consistently point the same way) while canceling oscillations across the valley (where gradient signs alternate), effectively rotating the update direction toward the valley axis. ADADELTA's step size adaptation reduces the magnitude of steps across the steep walls (since gradients are large there, RMS[g]_t is large, and the step is small) but does not alter the direction — the update still points across the valley, just with a shorter stride.
This means ADADELTA lacks one of the two mechanisms that make momentum successful: direction smoothing via temporal averaging of gradients. The numerator RMS[Δx]_{t-1} provides acceleration (large past updates lead to large future steps) but this acceleration is isotropic — it scales the gradient uniformly, unlike momentum which modifies the effective gradient direction per-dimension. ADADELTA and momentum address orthogonal aspects of the optimization problem (step size vs. direction), and the paper's final result — that momentum with a tuned learning rate ultimately surpasses ADADELTA at convergence (Figure 1) — is consistent with this architectural limitation.
What evidence exists in the paper. The paper does not directly analyze this tradeoff, but the evidence is implicit in two places. First, Equation 14 clearly shows Δx_t is proportional to −g_t — there is no momentum buffer or direction-modifying term. Second, Figure 1 shows that momentum outperforms ADADELTA at convergence, which the paper attributes to oscillation handling (Section 4.3) without connecting it to the absence of direction smoothing. The speech results (Figure 4) show ADADELTA converging to roughly the same accuracy as momentum, which suggests the direction-smoothing advantage may be less critical in some architectures or loss landscapes — but the paper does not investigate this.
Mitigation status. The paper does not present the lack of direction smoothing as a design choice or limitation. It is simply not part of the method. The paper's suggestion to add annealing to ADADELTA (Section 4.3) addresses the symptom (oscillations near convergence) without addressing the structural cause (raw gradient direction). A natural extension — combining ADADELTA's per-dimension step size adaptation with momentum-style gradient smoothing — would later be realized in Adam (Kingma and Ba, 2014), which uses separate exponential moving averages for the gradient (direction, like momentum) and squared gradient (scale, like ADADELTA/RMSProp). ADADELTA's design precludes this combination within its current formulation because the numerator RMS[Δx]_{t-1} provides acceleration but not direction modification.
7. Implications and Future Directions
How This Work Changes the Landscape
ADADELTA does not inaugurate a new paradigm — it operates firmly within the gradient descent framework that has dominated optimization since Robbins and Monro (1951). Nor does it reframe the fundamental problem of optimizing neural networks. What it accomplishes is more targeted but still significant: it provides a principled answer to the question "why do learning rates need to exist?" and, having answered that question, constructs a method that eliminates them.
The paper's lasting conceptual contribution is the unit-correctness diagnostic. Before ADADELTA, the learning rate was a mysterious knob — everyone knew it mattered enormously, but no one articulated why a scalar multiplier was structurally necessary in the gradient descent update. The paper's dimensional analysis (Equations 11–13) reveals the answer: the gradient has the wrong units (1/x rather than x), and the learning rate exists to supply the missing units of x². This is not just a clever observation — it is a diagnostic lens that can be applied to any optimizer. Does the method produce updates with correct physical units? If not, it implicitly requires a tuned hyperparameter to patch the mismatch. If so, it has the potential to be hyperparameter-free, provided its curvature approximation is adequate. Future optimizer designers — whether they adopt ADADELTA's specific ratio or not — now have this lens available to them. The question shifts from "how should we tune the learning rate?" to "how do we construct an update with units of x from first-order statistics?"
This diagnostic also reconciles a tension in the prior literature between the theoretical appeal of second-order methods and the practical dominance of first-order methods with manual tuning. Second-order methods (Newton, Becker and LeCun [5]) were known to produce unit-correct updates with principled curvature scaling, but their computational cost made them impractical. First-order methods (SGD, momentum, ADAGRAD) were practical but required fragile tuning. The paper shows that this tradeoff is not fundamental — you can achieve unit-correct, curvature-aware updates at first-order cost, provided you're willing to approximate the diagonal Hessian using the history of parameter updates and gradients. The "impossible trinity" of correct units, low computational cost, and hyperparameter freedom turns out to be achievable after all.
Which research directions become more attractive. The paper makes the search for hyperparameter-free, curvature-aware optimizers feel like a solvable problem rather than a pipe dream. It validates the strategy of deriving optimizer structure from first principles (dimensional analysis, Newton's method rearrangement) rather than from empirical tuning, encouraging more theoretically-grounded optimizer design. It also makes explicit what had been implicit in adaptive methods: that optimizer state (the accumulators for E[g²] and E[Δx²]) is not just a smoothing mechanism — it is encoding curvature information that can replace explicit second-order computation. This insight would directly motivate subsequent work on methods that extract richer curvature signals from first-order statistics, including Adam (Kingma and Ba, 2014), which uses separate momentum and RMS accumulators, and KFAC (Martens and Grosse, 2015), which generalizes the diagonal approximation to block-diagonal form while still operating at manageable cost.
Which research directions become less attractive. The paper makes the idea of manually tuned global learning rate schedules feel increasingly archaic. If a first-order method with "trivial" overhead can eliminate the learning rate entirely, the burden of proof shifts to methods that still require one: what additional benefit (better convergence, stronger theoretical guarantees, different capabilities) justifies the tuning cost? The paper also implicitly argues against the approach of Schaul et al. [6] — computing explicit second-order information (diagonal Hessian) when a first-order approximation may suffice. The modest performance improvement from exact diagonal Hessian computation (Schaul et al. reported 2.10% error vs. ADADELTA's 2.00% on the same MNIST architecture at 6 epochs) does not obviously justify the 2× computational cost per iteration. This shifts the burden of proof onto methods that require explicit curvature computation: they must demonstrate substantially better performance or faster convergence to justify their overhead relative to first-order curvature approximations.
Follow-Up Research This Work Enables
Characterizing the asymptotic gap between ratio-based and momentum-based methods. The paper's Figure 1 shows momentum with a tuned learning rate converging to a lower MNIST test error than ADADELTA after 50 epochs, and the authors hypothesize this is due to ADADELTA's lack of explicit annealing. A targeted follow-up would train both methods to full convergence (e.g., 200–500 epochs on MNIST, or on CIFAR-10 with a ConvNet) and measure: (a) does ADADELTA eventually close the gap, or does it saturate at a higher error floor? (b) does adding an annealing schedule to ADADELTA (e.g., decaying the effective step size by multiplying the update by a schedule factor, or by increasing ρ over time) allow it to match momentum's asymptotic performance while retaining hyperparameter insensitivity? (c) is the gap architecture-dependent — does it widen for deeper networks or specific activation functions where oscillations near minima are more pronounced? This would establish whether ADADELTA's asymptotic limitation is a fundamental cost of its ratio-based design or a fixable consequence of lacking explicit decay.
Isolating the contribution of the update RMS numerator from the gradient RMS denominator. The paper presents ADADELTA as simultaneously fixing two problems with ADAGRAD: infinite accumulation (via exponential moving average) and unit mismatch (via the update RMS numerator). These two fixes are confounded in all experiments. A controlled ablation would compare: (1) ADAGRAD with its infinite sum replaced by an exponential moving average (identical to ADADELTA's denominator but still requiring a global η), (2) pure ADADELTA with the update RMS numerator, and (3) ADAGRAD with both modifications (i.e., ADADELTA). This would answer: how much of ADADELTA's improvement over ADAGRAD comes from the finite window alone, and how much from eliminating η? If the finite-window ADAGRAD already matches ADADELTA's performance, then the unit-correctness argument, while elegant, is not the primary source of empirical gains. If the update RMS numerator provides additional benefit, the unit-correctness argument is empirically validated as well as theoretically principled. Training on MNIST for 50 epochs would suffice to distinguish these hypotheses.
Stress-testing hyperparameter insensitivity at extreme ρ and ϵ values. Table 2 sweeps ρ ∈ {0.9, 0.95, 0.99} and ϵ ∈ {1e−2, 1e−4, 1e−6, 1e−8} and finds consistent performance. The natural extension is to test the boundaries: ρ = 0.5 (very short memory, noisy step sizes — does training become unstable?), ρ = 0.999 (approaching ADAGRAD-like memory — does continual decay re-emerge?), ϵ = 1e0 (ϵ dominates the RMS ratio, ADADELTA approximates SGD with η = 1 — does this cause divergence in early training?), ϵ = 0 (does removing the conditioning constant cause division-by-zero failures or is the method surprisingly robust?). Training a small MNIST network for a few epochs would reveal failure modes at the extremes. This would produce a practical "safe operating range" for practitioners — the paper currently demonstrates that the defaults work, but not where the boundaries of failure lie.
Testing the method on recurrent neural networks for sequence modeling. ADADELTA's per-dimension adaptation is motivated partly by the need to compensate for vanishing gradients in deep networks (Section 4.3, Figure 2). RNNs exhibit even more extreme gradient scale variation — not just across layers, but across time steps within the same parameter (the recurrent weight matrix receives gradients that vary dramatically depending on sequence position). This is the stress test for ADADELTA's per-dimension adaptation: can the method handle gradient scales that vary not just by layer but dynamically within the same parameter across mini-batches? A natural experiment is character-level language modeling or polyphonic music prediction (both standard RNN benchmarks in 2012) with a single-layer or stacked LSTM, comparing ADADELTA against SGD with gradient clipping, momentum, and ADAGRAD. The hypothesis is that ADADELTA's exponential window (which forgets old gradient statistics) should handle RNNs' non-stationary gradient distributions better than ADAGRAD's infinite accumulation. Gradient clipping's interaction with ADADELTA's implicit spike-damping mechanism (Section 3.2) would also be instructive: does ADADELTA reduce or eliminate the need for explicit gradient clipping in RNNs?
Combining ADADELTA's step size adaptation with momentum-style gradient direction smoothing. ADADELTA scales the raw negative gradient by a per-dimension step size but does not modify the update direction — it follows −g_t exactly (Equation 14). Momentum, in contrast, modifies the direction by accumulating a velocity buffer but does not adapt the per-dimension step size beyond what the momentum decay implicitly provides. These mechanisms address orthogonal aspects of the optimization problem: step size (how far to move) and direction (which way to move). The natural combination — smoothing the gradient direction with an exponential moving average before scaling by ADADELTA's RMS ratio — would produce an update like Δx_t = −(RMS[Δx]_{t-1} / RMS[m]t) · m_t, where m_t = β m{t-1} + (1 − β)g_t is a momentum buffer. This is essentially what Adam (Kingma and Ba, 2014) would later do, but with the key difference that Adam uses the RMS of gradients as the denominator (like RMSProp) rather than ADADELTA's ratio of update RMS to gradient RMS. A head-to-head comparison of ADADELTA, Adam, and ADADELTA+momentum on MNIST and CIFAR-10 would clarify whether the update RMS numerator adds value over the simpler gradient RMS denominator when gradient-direction smoothing is already present.
Extending the unit-correctness argument to optimizers that handle non-diagonal curvature. ADADELTA's derivation assumes a diagonal Hessian — each parameter's curvature is treated independently (Section 3.2). This is computationally cheap but ignores correlations between parameters, which can be significant in neural networks (e.g., weights in the same layer often exhibit structured covariance). The paper's rearrangement (Equation 13) — 1/(∂²f/∂x²) = Δx / (∂f/∂x) — generalizes to the full matrix case: H^{-1} = Δx (∂f/∂x)⁻¹ under a quadratic approximation (though care is needed with matrix inverses). A natural extension would maintain a running estimate of the outer product of parameter updates and gradients, E[Δx · gᵀ], approximating the inverse Hessian as a full or block-diagonal matrix. The computational and memory costs would be higher (O(n²) or O(nb) for block size b), but the method would capture parameter interactions that the diagonal approximation misses. A targeted experiment on a small network where the Hessian can be computed exactly (e.g., a 2–3 layer MNIST classifier) could compare the true inverse Hessian, ADADELTA's diagonal approximation, and a block-diagonal or low-rank extension in terms of their ability to predict optimal step sizes, establishing the marginal value of richer curvature modeling.
Practical Applications and Downstream Use Cases
Default optimizer for rapid prototyping and hyperparameter-free baselines. The most immediate practical use case is as a drop-in replacement for SGD when tuning a learning rate is impractical — during exploratory model development, when training many model variants for architecture search, or when setting up baselines on new datasets. ADADELTA with ρ = 0.95 and ϵ = 1e−6 provides competitive performance on both the small-scale MNIST task (1.83% error after 6 epochs) and the large-scale speech task (Figure 3) without any task-specific tuning. A practitioner can start training immediately without a learning rate grid search, iterate on architecture and data pipeline, and switch to a carefully tuned optimizer (e.g., momentum with a schedule) only for final performance polishing. The paper's demonstration that the same hyperparameters transfer from MNIST (500/300 hidden units, images) to speech (4×2560 hidden units, audio) suggests the defaults generalize reasonably well, though this should be verified on the specific task at hand.
Distributed training environments where tuning is prohibitively expensive. The speech recognition experiments (Section 4.4, Figures 3–4) train on a distributed cluster with 100–200 model replicas, where each training run consumes hundreds of machine-hours. In such settings, a failed learning rate choice that causes divergence is not just inconvenient — it represents a substantial waste of computational resources that may require days or weeks to reschedule. ADADELTA's hyperparameter insensitivity is a form of risk reduction: starting a large distributed training job with ADADELTA's defaults is far less likely to produce a catastrophic failure than starting with a manually chosen SGD learning rate that could be an order of magnitude too high. Even if ADADELTA converges slightly slower or to a slightly worse final solution than a perfectly-tuned momentum schedule (as suggested by Figure 1), the expected cost — accounting for the probability of failed tuning runs — may favor ADADELTA. The paper's demonstration that ADADELTA handles 200-replica gradient noise "well" (Figure 4) is directly relevant to practitioners using asynchronous distributed training.
Edge deployment where optimizer simplicity reduces binary size and memory. ADADELTA's state consists of exactly two scalars per parameter (E[g²] and E[Δx²]), compared to one for momentum and zero for SGD. This 2× increase over SGD in optimizer memory is modest in absolute terms — for a model with 1 million parameters, it adds approximately 8 MB (two 32-bit floats per parameter). In resource-constrained environments (mobile devices, embedded systems, on-device fine-tuning), this small memory cost may be acceptable in exchange for eliminating the learning rate tuning process entirely. ADADELTA requires no learning rate schedule logic, no validation-based early stopping for schedule decisions, and no hyperparameter storage — the optimizer is self-contained and stateless beyond the two accumulators. This reduces code complexity for deployment pipelines where simplicity and reliability are prioritized over asymptotic convergence quality.
When to Prefer This Method
The paper positions ADADELTA primarily against two alternatives: ADAGRAD (which it aims to fix) and manually-tuned momentum/SGD (which it aims to replace). The tradeoffs are clearest in three scenarios:
-
Prefer ADADELTA over ADAGRAD when training for many iterations or epochs. ADAGRAD's denominator accumulates squared gradients from the beginning of training, causing the effective learning rate to decay monotonically toward zero. ADADELTA's exponential window prevents this decay; the paper's Figure 1 shows ADAGRAD plateauing after ~10 epochs on MNIST while ADADELTA continues to improve. Any task requiring sustained learning over tens or hundreds of epochs — which includes most deep learning applications — benefits from this fix. The speech experiments (Figure 3) confirm the advantage at scale: ADADELTA maintains a consistent accuracy lead over ADAGRAD throughout ~100 hours of distributed training.
-
Prefer ADADELTA over SGD/momentum when learning rate tuning is impractical or too costly. The paper demonstrates in Table 1 that SGD, momentum, and ADAGRAD all require their learning rate (or conditioning constant) to be set within roughly an order of magnitude to achieve reasonable performance — deviations of 1–2 orders of magnitude cause catastrophic degradation (58%+ error on MNIST for SGD and momentum). ADADELTA, in contrast, varies only 0.76 percentage points across a 6-order-of-magnitude epsilon sweep and a decade range of ρ (Table 2). For rapid prototyping, architecture search, hyperparameter sweeps over other dimensions, or large distributed runs where failed attempts are expensive, ADADELTA's insensitivity eliminates a major source of experimental variance and wasted compute.
-
Prefer momentum with a tuned schedule over ADADELTA when asymptotic convergence quality is paramount and tuning is feasible. The paper acknowledges that momentum "with the proper hyperparameters outperforms ADADELTA later in training" on MNIST (Figure 1, Section 4.3). The hypothesized mechanism is that momentum damps oscillations near minima whereas ADADELTA's numerator accumulates them, preventing the finest convergence. In a production setting where a model will be trained once and deployed widely, the investment in careful learning rate tuning (potentially including learning rate schedules, warmup, and decay) is justified, and momentum or a more sophisticated adaptive method (Adam, AdamW) with tuning is likely preferable to ADADELTA's default settings. The paper recommends ADADELTA for speed and convenience, not for squeezing out the last fraction of a percent of accuracy.
