Gaussian Error Linear Units (GELUs)

🎯 Pitch

Neurons don't have to choose between being stochastic regularizers and activation functions—GELU merges them into a single, simple equation that beats ReLU and ELU across vision, language, and speech. It works by weighting inputs by their Gaussian cumulative probability, effectively soft-dropping low values instead of harshly zeroing them out. The result is a non-convex, non-monotonic activation that acts like a built-in regularizer and consistently improves test accuracy.

1. Executive Summary

This paper introduces the Gaussian Error Linear Unit (GELU), a novel neural network activation function defined as $x\Phi(x)$ , where $\Phi(x)$ is the standard Gaussian cumulative distribution function. Unlike the ReLU, which gates inputs by their sign (multiplying by zero or one deterministically), the GELU weights inputs by their magnitude relative to other inputs — it is the expected transformation of a stochastic regularizer that probabilistically multiplies inputs by zero or one based on how large they are. Across computer vision (MNIST, CIFAR-10/100), natural language processing (Twitter POS tagging), and speech tasks (TIMIT), the GELU consistently outperforms both ReLU and ELU activations — achieving, for instance, a CIFAR-100 median error of 20.74% versus 21.77% for ReLU and 22.98% for ELU — establishing that a non-convex, non-monotonic activation with a probabilistic interpretation of stochastic regularization can serve as a drop-in replacement that improves accuracy with no additional hyperparameters.

2. Context and Motivation

The Core Problem: Activation Functions Are Architectural Choices Without a Probabilistic Basis

The fundamental question this paper addresses is deceptively simple: what mathematical function should a neuron apply to its input to enable effective deep learning? This matters because activation functions are not merely architectural details — they are the sole source of nonlinearity in otherwise purely linear deep networks. Without them, a stack of linear transformations collapses into a single linear transformation, making depth useless. The choice of activation function therefore determines what kinds of functions a network can represent and how easily it can learn them through gradient-based optimization.

This gap is significant for several reasons the paper establishes implicitly:

Representational power: Different nonlinearities enable different function classes. A network's capacity to approximate complex decision boundaries depends directly on the curvature and non-monotonicity of its activation function. An activation function that is too simple (e.g., piecewise linear) may require more depth or width to achieve the same representational power as one with richer curvature.
Training dynamics: The activation function directly shapes the loss landscape that gradient descent navigates. The ReLU's hard zeroing creates "dead neurons" that never recover, while sigmoids suffer from vanishing gradients in their saturation regimes. The interaction between activation function, optimizer, and initialization determines whether training converges quickly, slowly, or not at all.
Regularization interaction: Modern neural networks rely heavily on stochastic regularizers like dropout (Srivastava et al., 2014). Yet, as the paper observes, "nonlinearities and dropout thus determine a neuron's output together, yet the two innovations have remained distinct. More, neither subsumed the other because popular stochastic regularizers act irrespectively of the input and nonlinearities are aided by such regularizers." This separation means practitioners must tune two independent components that influence the same quantity — the neuron's output — without a unified framework for understanding their interaction.

The Historical Trajectory: From Probabilistic to Engineering-Driven

The paper traces a revealing historical arc that motivates its contribution (Section 1):

First-generation activations: binary threshold units. McCulloch & Pitts (1943) and Hopfield (1982) used hard binary decisions — a neuron fires or it doesn't, with no gradation. These had a clean interpretation (mimicking biological all-or-nothing action potentials) but were incompatible with backpropagation because the threshold function's derivative is zero almost everywhere.

Second-generation: sigmoidal smoothing. The sigmoid function smoothed the binary threshold into a continuous, differentiable curve, enabling gradient-based training while preserving a "firing rate" interpretation — the output could be viewed as a probability or a normalized firing frequency. But as the paper notes, "as networks became deeper, training with sigmoid activations proved less effective than the non-smooth, less-probabilistic ReLU." The sigmoid's saturation at extreme values causes vanishing gradients: when the neuron is "certain" (output near 0 or 1), the gradient is near zero, and no learning occurs. In deep networks, this problem compounds across layers, effectively halting training for early layers.

Third-generation: the ReLU. Nair & Hinton (2010) introduced the Rectified Linear Unit: $\text{ReLU}(x) = \max(0, x) = x \cdot \mathbf{1}_{x > 0}$ . This is "non-smooth" and "less-probabilistic" — it makes a hard, deterministic gating decision based purely on the input's sign. Despite having "less of a statistical motivation," the ReLU "remains a competitive engineering solution which often enables faster and better convergence than sigmoids." Why? Because its gradient is either 0 (for negative inputs) or 1 (for positive inputs), avoiding the vanishing gradient problem in the active regime. The paper acknowledges this triumph of engineering pragmatism over statistical elegance.

A refinement: the ELU. Clevert et al. (2016) proposed the Exponential Linear Unit, which modifies the ReLU by allowing negative outputs for negative inputs: $\text{ELU}(x) = x$ for $x > 0$ and $\alpha(e^x - 1)$ for $x \leq 0$ . The motivation is that negative outputs can push the mean activation closer to zero, which "sometimes increases training speed" by reducing internal covariate shift. However, the ELU introduces a hyperparameter $\alpha$ and uses an exponential computation, adding cost and complexity. Moreover, as the paper later notes, "some have noted that ELUs have an exploding gradient with residual networks" (Shah et al., 2016), requiring architectural workarounds like batch normalization after residual blocks.

The Unresolved Gap: Where Existing Approaches Fall Short

Despite this progression, the paper identifies a fundamental schism that no prior activation function bridges:

ReLU and its variants are deterministic, sign-based gates with no probabilistic interpretation. The ReLU asks: "Is the input positive? If so, pass it through unchanged. If not, output zero." This is a hard decision rule with no notion of uncertainty. Yet the paper observes that modern neural network training is fundamentally probabilistic — dropout randomly masks neurons, zoneout randomly preserves hidden states, and stochastic depth randomly drops layers. These regularizers work because they inject noise that prevents co-adaptation and simulates training an ensemble of sub-networks (a "pseudoensemble," Bachman et al., 2014). But they act independently of the input: dropout flips a coin regardless of whether the neuron is strongly activated or barely firing. The activation function and the regularizer, despite both influencing the same neuron's output, live in separate conceptual worlds with no shared framework.

Sigmoid and threshold units are probabilistic but poorly behaved. The sigmoid has a clean probabilistic interpretation (it is the CDF of a logistic distribution, and $\sigma(x)$ can be read as $P(\text{neuron fires})$ ), but its training properties are inferior — vanishing gradients, non-zero-centered outputs, and computationally expensive exponentials. The early probabilistic activations were abandoned not because the probabilistic perspective was wrong, but because the specific function (sigmoid) had poor optimization characteristics.

No activation function is derived as the expectation of a stochastic regularizer. This is the paper's key framing insight. Existing activations are either deterministic engineering choices (ReLU, ELU) or smooth approximations to deterministic thresholds (sigmoid). None arise from asking: "What if we designed a stochastic regularizer that multiplies the input by zero or one, where the probability of keeping the input depends on the input's magnitude, and then took the expected value of this stochastic transformation as our deterministic activation?" This construction would unify the activation function and the regularizer — the activation is the expected behavior of a dropout-like process, making the deterministic training pass a smooth approximation to a stochastic inference process.

The Mechanism: Merging Dropout, Zoneout, and ReLU into a Single Probabilistic Framework

The paper constructs its motivation by decomposing existing techniques into a common algebraic form (Section 2):

A ReLU can be written as $x \cdot \mathbf{1}_{x > 0}$ — the input multiplied by an indicator (0 or 1) that depends deterministically on the input's sign.

Dropout (Srivastava et al., 2014) can be written as $x \cdot m$ where $m \sim \text{Bernoulli}(p)$ — the input multiplied by a random mask that depends on nothing but a fixed probability $p$ . Dropout is input-independent.

Zoneout (Krueger et al., 2016), a regularizer for RNNs, can be written as $x \cdot m$ where $m \sim \text{Bernoulli}(p_{\text{zoneout}})$ — again, the mask probability is fixed and input-independent.

Adaptive Dropout (Ba & Frey, 2013) comes closest to the paper's idea: it multiplies the input by a random mask where the mask probability depends on the input — specifically, using a logistic function of the input. But as the paper notes, adaptive dropout was designed as a regularizer "used in tandem with nonlinearities" rather than as a nonlinearity itself, and "uses a logistic not standard normal distribution."

The paper's synthetic insight is: what if we combine these by making the mask probability $\Phi(x)$ — the standard Gaussian CDF — and then take the expectation to produce a deterministic activation?

Concretely: let $m \sim \text{Bernoulli}(\Phi(x))$ , where $\Phi(x) = P(X \leq x)$ for $X \sim \mathcal{N}(0,1)$ . The stochastic transformation is $x \cdot m$ , which is $x$ with probability $\Phi(x)$ and $0$ with probability $1 - \Phi(x)$ . The expected value is:

$\mathbb{E}[x \cdot m] = x \cdot \Phi(x) + 0 \cdot (1 - \Phi(x)) = x\Phi(x)$

This is the GELU.

Why the Gaussian Distribution Specifically?

The choice of the standard normal CDF is not arbitrary. The paper justifies it with a critical empirical observation:

"We choose this distribution since neuron inputs tend to follow a normal distribution, especially with Batch Normalization."

Batch Normalization (Ioffe & Szegedy, 2015) explicitly standardizes layer inputs to have zero mean and unit variance, making them approximately $\mathcal{N}(0,1)$ -distributed. This means $\Phi(x)$ has a direct probabilistic interpretation: it is the probability that a randomly sampled neuron input (from the same layer's distribution) is less than or equal to the current input $x$ . In other words, $\Phi(x)$ tells you how large $x$ is relative to its peers. An input at the mean ( $x = 0$ ) has $\Phi(0) = 0.5$ — it is "dropped" half the time. An input two standard deviations above the mean ( $x = 2$ ) has $\Phi(2) \approx 0.977$ — it is almost always kept. An input two standard deviations below the mean ( $x = -2$ ) has $\Phi(-2) \approx 0.023$ — it is almost always dropped.

This gives the GELU a probabilistic interpretation of input-dependent stochastic regularization: the neuron's output is the input scaled by how typical or extreme it is relative to the layer's activation distribution. This is fundamentally different from the ReLU, which asks "is the input positive?" without any notion of relative magnitude.

How This Paper Positions Itself

The paper positions the GELU as the deterministic expectation of a stochastic regularizer, bridging the conceptual gap between activation functions and dropout-like regularization. The key claims are:

Unification: The GELU subsumes aspects of dropout, zoneout, and adaptive dropout into the activation function itself. Rather than applying an activation (e.g., ReLU) and then separately applying a stochastic regularizer (e.g., dropout), the GELU's deterministic form is already the expected output under a dropout-like process. This is not merely a theoretical curiosity — the paper demonstrates empirically that "it is possible to train competitive MNIST and TIMIT networks solely with this stochastic regularizer, all without using any nonlinearity" (Section 2, emphasis added), showing that the stochastic process underlying the GELU can substitute for both the activation and the regularizer simultaneously.
Probabilistic interpretation: Unlike the ReLU, which "gates the input depending upon its sign," the GELU "weights its input depending upon how much greater it is than other inputs" (Section 4). This is a fundamentally different inductive bias: the neuron's output is not just "on or off" but reflects a continuous measure of how strongly the input stands out from the background distribution of activations.
Empirical superiority: The GELU is not merely conceptually elegant — it "matches or exceeds models with ReLUs or ELUs across tasks from computer vision, natural language processing, and automatic speech recognition" (Section 1). This is critical because conceptual elegance alone does not drive adoption; the GELU must work better in practice.
No new hyperparameters: By fixing $\mu = 0, \sigma = 1$ , the GELU introduces no additional hyperparameters compared to the ReLU, making it a true drop-in replacement. This is a deliberate contrast to alternatives like the ELU (which has $\alpha$ ) or PReLU (which has a learnable slope parameter), and to the paper's own discussion of using "different CDFs" — the paper explicitly chooses not to make $\mu$ and $\sigma$ learnable, instead relying on Batch Normalization to justify the standard normal assumption.

The paper's position is thus: the GELU is not just "another activation function" but a reconceptualization of what an activation function can be — a deterministic expectation of an input-dependent stochastic regularizer that naturally incorporates the behavior of dropout into the activation itself, outperforms existing activations, and requires no tuning. The extensive experiments across vision, language, and speech (Sections 3.1–3.5) are designed to demonstrate that this probabilistic derivation translates into practical, consistent accuracy gains.

A Note on the SiLU and Naming Priority

The paper also introduces the Sigmoid Linear Unit (SiLU), defined as $x\sigma(x)$ , as a special case that uses the logistic CDF instead of the Gaussian CDF. The authors note that the SiLU "performs worse than GELUs but usually better than ReLUs and ELUs," making it "also a reasonable nonlinearity choice" (Section 4). Appendix B details a subsequent priority dispute: other researchers later proposed $x\sigma(x)$ under the name "swish" (Ramachandran et al., 2017) without initially citing this work. The GELU paper was on arXiv in June 2016, predating the "swish" paper by over a year, and the SiLU name has since been adopted by major frameworks (PyTorch, TensorFlow). This history is significant not just for credit assignment but because it illustrates that the core idea — using a CDF to weight inputs probabilistically — originated here, with the Gaussian variant (GELU) proving empirically superior to the logistic variant (SiLU/swish) for the large-scale Transformer architectures that would later dominate the field (BERT, GPT).

3. Technical Approach

3.1 Reader Orientation

This paper proposes a new activation function — a drop-in replacement for the ReLU — that computes its output as the input multiplied by the standard Gaussian cumulative distribution function evaluated at that input. The problem it solves is that existing activation functions (ReLU, ELU, sigmoid) are either deterministic engineering choices with no probabilistic interpretation or have good probabilistic interpretations but poor optimization behavior, and the GELU bridges this gap by being the deterministic expectation of an input-dependent stochastic regularizer, unifying the activation function with dropout-like behavior in a single, mathematically principled function that outperforms alternatives across vision, language, and speech tasks.

3.2 Big-Picture Architecture (Diagram in Words)

The system is not a complex multi-component architecture but rather a single mathematical function that replaces the activation function in any neural network. The "architecture" has two logical components:

The stochastic process (the conceptual foundation): a random mask $m \sim \text{Bernoulli}(\Phi(x))$ multiplies the neuron input $x$ by zero or one, where the probability of keeping the input is $\Phi(x) = P(X \leq x)$ for $X \sim \mathcal{N}(0,1)$ — the input's percentile under the standard normal distribution. Larger inputs are more likely to be preserved; smaller inputs are more likely to be dropped.
The deterministic activation (what is actually used): the expected value of this stochastic process, $\text{GELU}(x) = x\Phi(x)$ , which is the limit of averaging infinitely many stochastic forward passes. This deterministic function is the actual activation used during both training and inference.

Information flows through a GELU neuron as follows: a pre-activation input $x$ (typically the output of a linear transformation, often batch-normalized) enters the activation function → the Gaussian CDF $\Phi(x)$ is computed (or approximated) → the input is multiplied by this CDF value → the result $x\Phi(x)$ becomes the neuron's output and feeds into the next layer's linear transformation. The entire process is a pointwise transformation with no state, no learned parameters, and no interaction across neurons.

The key architectural insight is that the activation function subsumes what would traditionally be a separate stochastic regularizer: rather than applying a ReLU and then separately applying dropout, the GELU's deterministic form already represents the expected behavior under an input-dependent dropout process. This means the activation and the regularizer are no longer "distinct innovations" — they are unified in a single function.

3.3 Roadmap for the Deep Dive

First, the stochastic process that motivates the GELU — the input-dependent Bernoulli mask, its relationship to Adaptive Dropout, zoneout, and standard dropout, and why the Gaussian distribution is chosen — because this is the conceptual foundation from which the deterministic form is derived.
Second, the derivation of the deterministic GELU as an expectation, the closed-form expression in terms of the error function, and the two practical approximations (the tanh and sigmoid forms) — because these are what practitioners actually implement.
Third, the relationship to prior activations (ReLU as a special case, ELU as an asymptotic cousin, SiLU as the logistic-CDF variant) — to establish exactly what is new and how the GELU connects to the existing landscape.
Fourth, the practical recommendations (using momentum, choosing an approximation, the role of Batch Normalization in justifying the standard normal assumption) — because these determine whether the GELU works in practice.
Fifth, a brief note on the SiLU and the design choice not to make $\mu$ and $\sigma$ learnable — to clarify what the paper deliberately chose not to do and why.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper whose core idea is that an activation function can be derived as the expected value of an input-dependent stochastic regularizer, yielding a non-convex, non-monotonic function that outperforms ReLUs and ELUs across diverse tasks without introducing new hyperparameters.

The Stochastic Process: Input-Dependent Bernoulli Masking

The paper begins not with the GELU itself but with a stochastic regularizer that motivates it. Consider a neuron receiving input $x$ . Instead of applying a deterministic nonlinearity like the ReLU, imagine multiplying $x$ by a random binary mask $m$ :

$m \sim \text{Bernoulli}(\Phi(x))$

where $\Phi(x)$ is the cumulative distribution function (CDF) of the standard normal distribution, defined as:

$\Phi(x) = P(X \leq x), \quad X \sim \mathcal{N}(0, 1)$

This means:

With probability $\Phi(x)$ , the mask $m = 1$ , and the neuron outputs $x$ (the input passes through unchanged).
With probability $1 - \Phi(x)$ , the mask $m = 0$ , and the neuron outputs $0$ (the input is "dropped").

What this process computes: a stochastic transformation of the neuron input where the probability of keeping the input depends on the input's own value. An input at the mean ( $x = 0$ ) has $\Phi(0) = 0.5$ , so it is dropped half the time. An input two standard deviations above the mean ( $x = 2$ ) has $\Phi(2) \approx 0.977$ , so it is kept 97.7% of the time. An input two standard deviations below the mean ( $x = -2$ ) has $\Phi(-2) \approx 0.023$ , so it is kept only 2.3% of the time.

Why this form: this merges the behaviors of three existing techniques into a single mechanism. A ReLU can be written as $x \cdot \mathbf{1}_{x > 0}$ — the input multiplied by a zero-one gate that depends deterministically on the input's sign. Standard dropout multiplies the input by a zero-one mask that is stochastically determined but independent of the input — $m \sim \text{Bernoulli}(p)$ for a fixed dropout keep probability $p$ . Zoneout (Krueger et al., 2016) does the same for RNN hidden states. The paper's proposal makes the mask probability depend on the input while remaining stochastic, combining the input-dependence of ReLU gating with the non-determinism of dropout. Adaptive Dropout (Ba & Frey, 2013) also uses input-dependent mask probabilities but (a) uses a logistic distribution rather than a Gaussian, and (b) is designed as a regularizer applied on top of a nonlinearity rather than as the nonlinearity itself.

The paper reports a striking empirical finding that validates this construction:

"We found that it is possible to train competitive MNIST and TIMIT networks solely with this stochastic regularizer, all without using any nonlinearity."

This is significant because it demonstrates that the stochastic process underlying the GELU is not merely a theoretical curiosity — it can replace the activation function entirely when used in its stochastic form. The nonlinear behavior emerges from the input-dependent masking rather than from an explicit nonlinear function.

The choice of the Gaussian CDF specifically is justified by a critical property of modern networks:

"We choose this distribution since neuron inputs tend to follow a normal distribution, especially with Batch Normalization."

Batch Normalization (Ioffe & Szegedy, 2015) standardizes each layer's pre-activations to have zero mean and unit variance. This means that after Batch Normalization, the pre-activation inputs to the activation function are approximately $\mathcal{N}(0,1)$ -distributed by construction. Under this distribution, $\Phi(x)$ has a direct probabilistic interpretation: it is the probability that a randomly sampled pre-activation from the same layer is less than or equal to $x$ . In other words, $\Phi(x)$ is $x$ 's percentile rank within its layer's activation distribution. This interpretation — that the GELU scales the input by how much it stands out from its peers — is fundamentally different from the ReLU's interpretation of gating by sign.

If the neuron inputs were not approximately normal, the standard Gaussian CDF would be a poor match — the percentile interpretation would break because $\Phi(x)$ would no longer correspond to the actual empirical CDF of the layer's activations. Batch Normalization is therefore not just a training convenience for the GELU; it is what makes the probabilistic interpretation coherent. The paper does not explore what happens without Batch Normalization, but the theoretical motivation depends on it.

The Deterministic GELU: From Expectation to Closed Form

Stochastic regularizers are useful during training (they prevent co-adaptation and simulate ensembles), but at inference time, neural networks typically use deterministic computations. The standard approach with dropout is to scale activations by the keep probability at test time (or equivalently, use inverted dropout during training), which approximates the expected value of the stochastic network. The paper follows the same logic: take the expectation of the stochastic process to produce a deterministic activation function.

The stochastic transformation is:

$\text{output} = x \cdot m, \quad m \sim \text{Bernoulli}(\Phi(x))$

Taking the expectation over the mask:

$\mathbb{E}[x \cdot m] = x \cdot \mathbb{E}[m] = x \cdot \Phi(x)$

since the expected value of a Bernoulli random variable is its success probability. The $x$ factor is constant with respect to the mask distribution because $\Phi(x)$ depends on $x$ but the mask draw $m$ is independent of $x$ given $\Phi(x)$ — $\Phi(x)$ determines the Bernoulli parameter, and then the coin is flipped.

This yields the fundamental definition:

$\text{GELU}(x) = x\Phi(x)$

What this computes: the input multiplied by the probability that a standard normal random variable is less than or equal to that input. For $x$ far above zero, $\Phi(x) \approx 1$ , so $\text{GELU}(x) \approx x$ — the function is approximately linear in the positive tail. For $x$ far below zero, $\Phi(x) \approx 0$ , so $\text{GELU}(x) \approx 0$ — the function squashes negative inputs to near zero. For $x$ near zero, $\Phi(x) \approx 0.5$ , so $\text{GELU}(x) \approx 0.5x$ — the function attenuates but does not eliminate small inputs. Unlike the ReLU, which has a hard zero for negative inputs, the GELU has a smooth, curved transition through the origin.

Why this form: taking the expectation converts the stochastic regularizer into a deterministic function that can be used at both training and inference time without sampling. This is the same principle as dropout's test-time scaling, but built into the activation function itself. The resulting function is non-convex, non-monotonic (it curves upward for negative inputs near zero before being squashed), and smooth (infinitely differentiable), which the paper argues allows it to "more easily approximate complicated functions than can ReLUs or ELUs" (Section 4). The curvature everywhere — even in the positive domain where ReLUs and ELUs are purely linear — means the GELU provides nonlinearity across the entire input range, not just at a single threshold.

The standard Gaussian CDF $\Phi(x)$ is not an elementary function — it cannot be expressed in closed form using basic arithmetic, exponentials, and trigonometric functions. However, it is universally computed in terms of the error function (erf), which is available in all major numerical libraries. The relationship is:

$\Phi(x) = \frac{1}{2}\left[1 + \text{erf}\left(\frac{x}{\sqrt{2}}\right)\right]$

Therefore, the exact GELU can be written as:

$\text{GELU}(x) = x \cdot \frac{1}{2}\left[1 + \text{erf}\left(\frac{x}{\sqrt{2}}\right)\right]$

What this computes: the same quantity as $x\Phi(x)$ , but expressed in terms of functions available in standard math libraries. The error function $\text{erf}(z) = \frac{2}{\sqrt{\pi}} \int_0^z e^{-t^2} dt$ is the integral of the Gaussian from $0$ to $z$ , scaled so that $\text{erf}(\infty) = 1$ . The transformation $\text{erf}(x/\sqrt{2})$ maps the standard normal CDF to the error function: the factor $1/\sqrt{2}$ adjusts for the fact that the standard normal has variance 1 (so its density is $e^{-x^2/2}/\sqrt{2\pi}$ ) while the error function's integrand is $e^{-t^2}$ (corresponding to variance 1/2).

Why this form: this is the exact computation, but the error function itself requires numerical integration or series expansion, which is relatively slow. For large-scale neural network training where the activation function is called billions of times, even a small per-call overhead can accumulate significantly. The paper therefore provides approximations.

The Two Approximations: Trading Exactness for Speed

The paper provides two approximations to the GELU that avoid the computational cost of the error function while maintaining very close numerical agreement. The choice of which to use depends on whether "greater feedforward speed is worth the cost of exactness."

Approximation 1: The tanh form (used in all paper experiments)

$\text{GELU}(x) \approx 0.5x\left(1 + \tanh\left[\sqrt{\frac{2}{\pi}}\left(x + 0.044715x^3\right)\right]\right)$

What this computes: an approximation to $x\Phi(x)$ using the hyperbolic tangent, which is a fast, hardware-accelerated function on GPUs. The inner expression $\sqrt{2/\pi}(x + 0.044715x^3)$ is a polynomial approximation to the inverse probit function (the quantile function of the normal distribution composed with the CDF, remapped to the tanh domain). The outer scaling $0.5(1 + \tanh(\cdot))$ maps the $(-1, 1)$ range of tanh to $(0, 1)$ , approximating the CDF.

Why this form: the tanh function is highly optimized in GPU hardware (it is implemented in CUDA intrinsics), making this approximation significantly faster than computing the error function while maintaining what the paper describes as a "close approximation" to the true CDF. The specific coefficients (0.5, $\sqrt{2/\pi}$ , 0.044715) come from Choudhury (2014) and are optimized to minimize the maximum absolute error in approximating $\Phi(x)$ over the real line. The cubic term $0.044715x^3$ corrects for the fact that a linear argument to tanh would poorly match the tails of the Gaussian CDF.

The paper uses this approximation in every experiment. This is a critical practical detail: all the empirical results demonstrating the GELU's superiority are actually evaluating this tanh approximation, not the exact error-function formulation. The distinction matters for reproducibility — a naive reimplementation that computes $\text{erf}$ exactly might be slower but should be functionally identical up to the approximation error.

Approximation 2: The sigmoid form

$\text{GELU}(x) \approx x\sigma(1.702x)$

where $\sigma(z) = 1/(1 + e^{-z})$ is the logistic sigmoid function.

What this computes: the input multiplied by a scaled and shifted sigmoid. The factor $1.702$ stretches the sigmoid horizontally so that its shape matches the Gaussian CDF as closely as possible (specifically, it minimizes the integrated squared difference between $\sigma(cx)$ and $\Phi(x)$ ).

Why this form: the sigmoid is even faster to compute than tanh on some hardware and is even more widely implemented. However, the sigmoid decays as $e^{-x}$ in the left tail and approaches 1 as $1 - e^{-x}$ in the right tail, while the Gaussian CDF decays as $e^{-x^2/2}$ in both tails. This means the sigmoid approximation is less accurate in the tails — it is heavier-tailed than the true Gaussian CDF, meaning extreme negative inputs will be less aggressively squashed and extreme positive inputs will approach the identity line more slowly. The paper acknowledges this inferiority: "we found that a Sigmoid Linear Unit (SiLU) $x\sigma(x)$ performs worse than GELUs but usually better than ReLUs and ELUs" (Section 4). The $1.702$ scaling factor mitigates but does not eliminate this mismatch.

The paper mentions the sigmoid approximation as an alternative for speed-critical applications but uses the tanh form exclusively in experiments, suggesting that the tanh approximation represents the preferred practical tradeoff between accuracy and speed.

Relationship to Prior Activation Functions: What the GELU Inherits and What Is New

The GELU does not exist in a vacuum. The paper explicitly connects it to ReLUs, ELUs, and sigmoid-based activations to clarify what is genuinely novel.

GELU as a smoothed ReLU. Consider the ReLU: $\text{ReLU}(x) = \max(0, x) = x \cdot \mathbf{1}_{x > 0}$ (where $\mathbf{1}$ is the indicator function, returning 1 when the condition is true and 0 otherwise). This is a "hard gating" decision: the input is passed through exactly when positive and zeroed exactly when negative. Now consider the GELU with $\mu = 0$ and variable $\sigma$ :

$\text{GELU}_{\sigma}(x) = x\Phi_{\sigma}(x) = xP(X \leq x), \quad X \sim \mathcal{N}(0, \sigma^2)$

As $\sigma \to 0$ , the normal distribution becomes a delta function at zero. The CDF $\Phi_{\sigma}(x)$ becomes a step function: 0 for $x < 0$ , 1 for $x > 0$ . Therefore:

$\lim_{\sigma \to 0} \text{GELU}_{\sigma}(x) = x \cdot \mathbf{1}_{x > 0} = \text{ReLU}(x)$

The GELU with $\sigma = 1$ is a smoothed version of the ReLU where the hard step at zero is replaced by the smooth Gaussian CDF. The paper notes this explicitly: "the GELU can be viewed as a way to smooth a ReLU. To see this, recall that $\text{ReLU} = \max(x, 0) = x\mathbf{1}(x > 0)$ ... while the GELU is $x\Phi(x)$ if $\mu = 0, \sigma = 1$ . Then the CDF is a smooth approximation to the binary function the ReLU uses, like how the sigmoid smoothed binary threshold activations" (Section 4).

This ReLU-as-limiting-case property means that a GELU network can approximate a ReLU network if that turns out to be optimal, but it has additional flexibility from the smooth transition region. The paper does not argue that the GELU necessarily outperforms the ReLU because of this extra flexibility — the empirical evidence is the argument — but the connection establishes that the GELU is strictly more general.

GELU asymptotics match ReLU. For large positive $x$ , $\Phi(x) \to 1$ , so $\text{GELU}(x) \to x$ . For large negative $x$ , $\Phi(x) \to 0$ , so $\text{GELU}(x) \to 0$ . These limits match the ReLU exactly: "the ReLU and GELU are equal asymptotically" (Section 4). The difference is only in the transition region near zero.

GELU vs. ELU: the Cauchy connection. The ELU (Clevert et al., 2016) is defined as:

$\text{ELU}(x) = \begin{cases} x & \text{if } x > 0 \\ \alpha(e^x - 1) & \text{if } x \leq 0 \end{cases}$

The paper draws a non-obvious connection: "if we used the cumulative distribution function of the standard Cauchy distribution, then the ELU (when $\alpha = 1/\pi$ ) is asymptotically equal to $xP(C \leq x)$ , $C \sim \text{Cauchy}(0, 1)$ for negative values and for positive values is $xP(C \leq x)$ if we shift the line down by $1/\pi$ " (Section 4). This establishes that the ELU can be viewed as another CDF-based activation — just using the heavy-tailed Cauchy distribution rather than the Gaussian, with a shift for positive values. The fact that the ELU needs a shift (the $-1/\pi$ term for positive inputs) to match the CDF form highlights a difference: the GELU uses the CDF directly without any shift, making the positive-domain behavior naturally asymptotic to identity.

The paper does not delve deeply into this connection — it is mentioned as a "fundamental relation" — but it provides theoretical unity: both the GELU and the ELU can be seen as CDF-weighted activations, with different choices of distribution and treatment of the positive domain. The GELU applies the CDF weight everywhere, while the ELU is identity in the positive domain and uses an exponential (related to the Cauchy CDF's negative tail) in the negative domain.

The SiLU: same form, different distribution. If instead of the Gaussian CDF $\Phi(x)$ , one uses the logistic CDF — which is simply the sigmoid function $\sigma(x) = 1/(1 + e^{-x})$ — the resulting activation is:

$\text{SiLU}(x) = x\sigma(x)$

The paper introduces this as the Sigmoid Linear Unit and notes that it "performs worse than GELUs but usually better than ReLUs and ELUs" (Section 4). The logistic distribution has heavier tails than the Gaussian (the logistic density decays as $e^{-|x|}$ while the Gaussian decays as $e^{-x^2/2}$ ), meaning the SiLU is less aggressive about squashing extreme negative inputs and slower to approach the identity line for extreme positive inputs. In the central region, $\sigma(x)$ and $\Phi(x)$ are very similar in shape (the logistic and Gaussian CDFs differ by at most about 0.02 in the Kolmogorov-Smirnov sense), but the tail behavior matters for the activation's curvature properties.

The SiLU demonstrates that the specific choice of CDF matters — not just the general form $x \cdot \text{CDF}(x)$ . The Gaussian's lighter tails produce empirically better performance, which the paper attributes to the fact that neuron inputs are approximately normally distributed under Batch Normalization. Using a logistic CDF when the actual input distribution is Gaussian means the SiLU's probabilistic interpretation is less well-matched to the data, even though the functional form is identical. This is an important design lesson: the distribution matters for the interpretation, and matching the CDF to the empirical input distribution (made normal by Batch Normalization) yields better performance than a generic heavy-tailed alternative.

Design Choices: What the Paper Deliberately Does NOT Do

Several "natural" extensions are available but deliberately not pursued, and understanding these omissions clarifies the paper's design philosophy.

Fixed $\mu = 0$ , $\sigma = 1$ : no learnable parameters. The paper explicitly states:

"We could use the CDF of $\mathcal{N}(\mu, \sigma^2)$ and have $\mu$ and $\sigma$ be learnable hyperparameters, but throughout this work we simply let $\mu = 0$ and $\sigma = 1$ . Consequently, we do not introduce any new hyperparameters in the following experiments."

This is a deliberate design decision that distinguishes the GELU from parameterized activations like PReLU (He et al., 2015), which learns a slope parameter for negative inputs, or the "swish" paper's later addition of a learnable $\beta$ parameter to $x\sigma(\beta x)$ . The paper argues implicitly that Batch Normalization already standardizes the inputs to $\mathcal{N}(0,1)$ , so learnable $\mu$ and $\sigma$ would be redundant — they would just undo the normalization that Batch Normalization performs. By keeping the GELU parameter-free, the paper ensures it is a true drop-in replacement for the ReLU, requiring zero additional tuning and introducing zero additional optimization variables.

Different CDFs are possible but the Gaussian is preferred. The paper explicitly notes that "we could use different CDFs" and provides the SiLU as the logistic alternative. But the Gaussian is chosen for the probabilistic match with Batch Normalization's output distribution, and the paper's experiments show that the Gaussian version (GELU) outperforms the logistic version (SiLU). The paper does not explore other distributions (Laplace, Student's t, etc.), leaving open the question of whether some other CDF might outperform the Gaussian. The empirical argument is not that the Gaussian is uniquely optimal but that it works well and has a clean justification.

The stochastic form is a conceptual device, not a practical training method. Although the paper notes that it is possible to train networks with the stochastic version (the Bernoulli masking process) without any explicit nonlinearity, all experiments use the deterministic GELU. The stochastic version serves as the conceptual bridge between dropout and activation functions, but the practical method is to use the expectation. This is analogous to how dropout is motivated by stochastic masking but deployed with deterministic scaling at test time.

Practical Recommendations for Using the GELU

The paper includes two practical tips (Section 4) that are essential for successful use:

Use an optimizer with momentum. The paper states:

"First we advise using an optimizer with momentum when training with a GELU, as is standard for deep neural networks."

This is not GELU-specific — all the experiments use Adam (Kingma & Ba, 2015), which includes momentum via its exponential moving average of gradients — but the paper emphasizes it because the GELU's non-convexity and non-monotonicity might interact poorly with purely stochastic gradient descent without momentum. The smooth curvature of the GELU means the loss landscape is different from the piecewise-linear landscape of the ReLU, and momentum helps navigate this curved landscape. The recommendation does not arise from ablation studies — the paper does not compare SGD-with-momentum to SGD-without-momentum for the GELU — but from the theoretical properties of the function and the standard practices of deep learning at the time.

Use a close approximation to the true Gaussian CDF. The paper warns:

"Second, using a close approximation to the cumulative distribution function of a Gaussian distribution is important."

The supporting evidence is the underperformance of the SiLU ( $x\sigma(x)$ ), which uses the logistic CDF as an approximation to the Gaussian CDF. The paper is effectively arguing that the specific shape of the Gaussian CDF — with its lighter tails and specific curvature — matters for performance, and that approximate forms that deviate from this shape (like the sigmoid) will underperform. The tanh approximation, by contrast, is specifically designed to closely match the Gaussian CDF, and the paper uses it in all experiments. This recommendation implies that implementers should not casually substitute the sigmoid for the tanh approximation or use a rough polynomial fit — the quality of the CDF approximation directly affects the activation's behavior and the network's performance.

4. Key Insights and Innovations

Innovation 1: The Activation Function as the Expected Value of an Input-Dependent Stochastic Regularizer

Before this paper, activation functions and stochastic regularizers occupied entirely separate conceptual categories. An activation function (ReLU, sigmoid, tanh) was a deterministic nonlinearity — a function you applied to a neuron's pre-activation to introduce nonlinearity into the network. A stochastic regularizer (dropout, zoneout, adaptive dropout) was a random perturbation — noise you injected during training to prevent co-adaptation and simulate an ensemble. They influenced the same quantity (the neuron's output) but were designed independently, tuned independently, and understood through entirely separate theoretical lenses. The paper's observation that "nonlinearities and dropout thus determine a neuron's output together, yet the two innovations have remained distinct" (Section 1) diagnoses a genuine conceptual schism in how the field thought about neural network components.

The GELU's foundational contribution is to dissolve this distinction entirely. By deriving the activation function as $\mathbb{E}[x \cdot m]$ where $m \sim \text{Bernoulli}(\Phi(x))$ , the paper demonstrates that a single mathematical object — an input-dependent stochastic regularizer — gives rise to both the regularizer (when used stochastically) and the activation function (when the expectation is taken). The deterministic GELU is the expected behavior of the stochastic regularizer; the stochastic regularizer is a randomized version of the deterministic GELU. They are not two components that happen to work well together; they are two views of the same underlying process.

This is a conceptual advance, not merely a new function. Prior work had explored input-dependent dropout (Adaptive Dropout; Ba & Frey, 2013) but always as an add-on regularizer applied after a separate nonlinearity. The paper's insight is that if you make the dropout probability depend on the input in a specific way — using the Gaussian CDF evaluated at that input — then the regularizer alone, without any explicit nonlinearity, is sufficient to train competitive networks. The paper states this explicitly: "it is possible to train competitive MNIST and TIMIT networks solely with this stochastic regularizer, all without using any nonlinearity" (Section 2). This is a striking empirical demonstration that the nonlinear behavior traditionally provided by a separate activation function can emerge entirely from input-dependent stochastic masking — the "activation" and the "regularization" are the same mechanism viewed at different noise levels.

The significance of this reframing extends beyond the specific GELU function. It opens a design space: any CDF can be used to define a stochastic regularizer, and taking its expectation yields a corresponding deterministic activation function. The paper explicitly notes this — "We could use different CDFs" (Section 2) — and provides the SiLU ( $x\sigma(x)$ , using the logistic CDF) as a concrete alternative. This establishes a principled framework for generating activation functions from desired stochastic regularization behaviors, or vice versa. A practitioner who wants a specific dropout profile (e.g., heavy-tailed, light-tailed, asymmetric) can choose the corresponding CDF and obtain both the stochastic regularizer and the deterministic activation as two sides of the same coin.

This unification is fundamental, not incremental. It does not refine the ReLU or add a learnable parameter to an existing function; it changes what an activation function is from an arbitrary engineering choice to the deterministic limit of a principled stochastic process. The fact that the resulting function outperforms ReLUs and ELUs across diverse tasks (Figures 2–7) validates that this conceptual shift translates to practical gains, but the idea itself — the dissolution of the activation/regularizer boundary — is the primary intellectual contribution.

Innovation 2: Weighting by Magnitude Rather Than Gating by Sign

The ReLU and its variants (LeakyReLU, PReLU, ELU) all share a fundamental inductive bias: the treatment of an input depends primarily on its sign. The ReLU asks "is $x > 0$ ?" and applies a hard binary decision — pass the input through unchanged or zero it completely. The ELU asks the same question but softens the negative regime with an exponential decay. LeakyReLU uses a learned or fixed small slope for negative inputs. In every case, the decision boundary is at zero, and the qualitative behavior changes abruptly at that point. This sign-based gating is not derived from any principle; it is an engineering choice that happened to work well — as the paper notes, the ReLU "remains a competitive engineering solution which often enables faster and better convergence than sigmoids" despite "having less of a statistical motivation" (Section 1).

The GELU replaces sign-based gating with magnitude-based weighting. The function $x\Phi(x)$ does not ask "is the input positive?" but rather "how large is this input relative to other inputs in the same layer?" The paper makes this distinction explicit in Section 4:

"ReLU gates the input depending upon its sign, while the GELU weights its input depending upon how much greater it is than other inputs."

This is a fundamentally different inductive bias. Under the ReLU, an input of $+0.001$ is treated identically to an input of $+100$ — both pass through unchanged (gradient of 1). Under the GELU, $\Phi(0.001) \approx 0.5004$ while $\Phi(100) \approx 1.0$ , so the small input is attenuated by roughly half while the large input passes through at full strength. Similarly, an input of $-0.001$ is zeroed by the ReLU but attenuated to roughly $-0.0005$ by the GELU (since $\Phi(-0.001) \approx 0.4996$ ), preserving some signal from slightly-negative inputs that the ReLU completely discards.

Why this matters: the GELU's behavior is calibrated to the expected distribution of activations in a batch-normalized network. Because Batch Normalization makes pre-activations approximately $\mathcal{N}(0, 1)$ , $\Phi(x)$ is interpretable as the input's percentile rank within the layer's activation distribution. An input at the mean ( $x = 0$ ) has $\Phi(0) = 0.5$ — it is "average," so the GELU passes it through at half strength, reflecting uncertainty about whether this moderate activation is meaningful. An input at $x = 2$ has $\Phi(2) \approx 0.977$ — it is in the 98th percentile, a clear outlier, so the GELU passes it through almost unchanged. An input at $x = -2$ has $\Phi(-2) \approx 0.023$ — it is in the 2nd percentile, strongly below average, so the GELU nearly zeros it out. The transition is smooth: there is no arbitrary threshold at zero, just a continuous scaling based on how typical or extreme the input is.

This probabilistic interpretation of relative magnitude is a conceptual advance over the ReLU's arbitrary zero threshold. The ReLU's decision boundary at zero is justified by nothing deeper than "it works." The GELU's smooth weighting is justified by the empirical distribution of activations: inputs that are typical (near the mean of the layer's activation distribution) are uncertain and should be attenuated; inputs that are extreme outliers are likely signal and should be preserved. This connects the activation function to the statistical properties of the data flowing through the network, rather than treating it as a fixed transformation independent of the network's internal representations.

The practical consequence is that the GELU provides nonlinearity everywhere, not just at a single threshold. The ReLU is piecewise linear: linear with slope 1 for $x > 0$ , linear with slope 0 for $x < 0$ . Its only curvature is the non-differentiable kink at zero. The ELU adds curvature in the negative regime (the exponential decay) but remains linear in the positive regime. The GELU, by contrast, "is not linear in the positive domain and exhibits curvature at all points" (Section 4). This means the GELU can represent richer function classes with fewer neurons, because each neuron provides nonlinear transformation across its entire input range rather than switching between two linear regimes at a single point. The paper hypothesizes that "increased curvature and non-monotonicity may allow GELUs to more easily approximate complicated functions than can ReLUs or ELUs" (Section 4) — a representational capacity argument that complements the probabilistic motivation.

This is a fundamental shift in how to think about activation design, not an incremental tweak. It replaces the question "at what threshold should the activation switch behavior?" (the ReLU/ELU/PReLU family's design question) with "how should the activation scale the input based on its typicality?" — a question that has a principled answer given the layer's activation distribution.

Innovation 3: The Activation Function as a Pseudoensemble Mechanism in a Single Forward Pass

The pseudoensemble interpretation of dropout — that training with stochastic masking approximates training an ensemble of exponentially many sub-networks that share parameters (Bachman et al., 2014) — was well-established before this paper. Dropout's effectiveness is attributed to this implicit ensembling: each training step updates a different random subset of the network, and at test time the full network approximates the averaged prediction of all those sub-networks.

The GELU takes this idea and embeds it directly into the deterministic activation function. Because $\text{GELU}(x) = \mathbb{E}_{m \sim \text{Bernoulli}(\Phi(x))}[x \cdot m]$ , a single deterministic forward pass through a GELU network computes the expected output of an ensemble of stochastic networks, where each ensemble member corresponds to a different draw of the input-dependent Bernoulli masks. Unlike dropout, which requires multiple stochastic forward passes (or weight scaling) to approximate the ensemble prediction, the GELU computes the ensemble expectation exactly in one deterministic pass.

This is a qualitatively different kind of pseudoensemble than dropout provides. In dropout, the mask probability is fixed (e.g., $p = 0.5$ ) and independent of the input — every neuron has the same chance of being dropped regardless of whether it is strongly or weakly activated. The resulting pseudoensemble averages over networks that differ in which neurons are present, but with no regard for whether those neurons were contributing meaningful signal. In the GELU's stochastic process, the mask probability depends on the input's magnitude — strongly positive neurons are almost always kept, strongly negative neurons are almost always dropped, and moderate neurons are uncertain. The pseudoensemble therefore averages over networks that adaptively preserve or drop neurons based on how much signal they carry for the current input.

This is significant because it means the GELU's deterministic forward pass inherently performs a form of input-dependent model averaging that would require multiple stochastic forward passes to achieve with standard dropout. The paper does not frame this in terms of Bayesian model averaging or uncertainty quantification — those connections would come later — but the groundwork is here: the GELU's output for a given input is the expectation under a posterior-like distribution over sub-network structures, where the "posterior" probability of keeping a neuron depends on how strongly that neuron responds to the input. A neuron that fires strongly (large positive pre-activation) is "confident" and included with high probability; a neuron that barely fires (near-zero pre-activation) is "uncertain" and included roughly half the time; a neuron that is strongly inhibited (large negative pre-activation) is "confidently irrelevant" and nearly always excluded.

The empirical result that competitive networks can be trained solely with the stochastic regularizer, without any nonlinearity (Section 2) is the strongest evidence for this interpretation. It demonstrates that the pseudoensemble effect alone — the averaging over input-dependent subnetworks — provides sufficient representational power to solve MNIST and TIMIT without any explicit nonlinear transformation. The fact that taking the expectation (the deterministic GELU) works even better suggests that the pseudoensemble averaging is the core mechanism, and the deterministic form simply computes it more efficiently.

This insight is fundamental, not incremental. It reveals that a well-designed activation function can serve as an implicit ensemble mechanism, providing the benefits of stochastic regularization (robustness, reduced co-adaptation) without the computational overhead of actually sampling multiple masks. The field would later build on this idea implicitly — the dominance of GELU in Transformers (BERT, GPT) may be partly attributable to this built-in regularization effect, which is particularly valuable in overparameterized architectures — but the paper itself frames it as a unification of two previously separate design choices.

Innovation 4: Non-Monotonicity as a Deliberate Design Feature, Not a Bug

Activation functions before the GELU were predominantly monotonic. The sigmoid is monotonically increasing. The tanh is monotonically increasing. The ReLU is monotonically increasing (it is non-decreasing: flat for $x < 0$ , strictly increasing for $x > 0$ ). The ELU is monotonically increasing. LeakyReLU and PReLU are monotonically increasing. Monotonicity was so ingrained in activation function design that it was essentially an unstated assumption — of course an activation function should be monotonic; why would a larger input ever produce a smaller output?

The GELU violates this assumption. For negative inputs near zero, the GELU is non-monotonic: it dips slightly negative, reaches a minimum, and then increases toward zero as $x$ becomes more negative and $\Phi(x)$ approaches zero. Specifically, $\text{GELU}(x) = x\Phi(x)$ is negative for $x < 0$ (since $x < 0$ and $\Phi(x) > 0$ ), but the magnitude $|x\Phi(x)|$ is not monotonic in $x$ — it first increases as $x$ goes from $0$ to roughly $-0.5$ , then decreases toward zero as $\Phi(x)$ decays faster than $|x|$ grows for $x \ll 0$ . This creates a small "bump" in the negative region where the function's output becomes more negative before relaxing back toward zero.

The paper does not emphasize this non-monotonicity heavily — it is mentioned as a property in Section 4 ("this non-convex, non-monotonic function") — but it represents a significant departure from prior design philosophy. The non-monotonicity is not an accident or an artifact; it is a direct consequence of the probabilistic derivation. The GELU asks "how likely is this input to be preserved under input-dependent dropout?" and for slightly negative inputs, the answer is "moderately likely" (since $\Phi(-0.5) \approx 0.31$ ), so the activation outputs a moderately negative value. For very negative inputs, the preservation probability drops faster than the magnitude grows, so the output approaches zero. This non-monotonic behavior has no analog in ReLU or ELU families.

Why this matters: non-monotonic activations can represent functions that monotonic activations cannot without additional layers. A monotonic activation combined with linear transformations can only produce monotonic functions of the input (since composition of monotonic functions is monotonic). To represent a non-monotonic relationship — say, a function that increases, then decreases, then increases again — a monotonic network needs at least two layers (to create a "bump" through subtraction of shifted monotonic functions). A non-monotonic activation can potentially represent such relationships in fewer layers because each neuron can itself encode non-monotonic behavior. The paper's hypothesis that the GELU's curvature and non-monotonicity "may allow GELUs to more easily approximate complicated functions" (Section 4) points toward this representational efficiency argument.

Subsequent empirical evidence from the paper's own experiments provides indirect support: the GELU consistently outperforms ReLUs and ELUs at the same network depth and width (Figures 2–7), suggesting that GELU networks can represent the target functions more effectively with the same architectural budget. The WideResNet CIFAR-100 result is particularly informative: a 40-layer network with GELUs achieves 20.74% error versus 21.77% for ReLU and 22.98% for ELU (Figure 7), a gap that cannot be attributed to differences in optimization dynamics alone at this depth, and likely reflects the GELU's richer representational capacity per layer.

This insight is fundamental in the sense that it challenges an unstated axiom of activation function design (monotonicity) and shows that violating it can be beneficial — but it is not fully developed in the paper. The paper does not provide ablation studies isolating the effect of non-monotonicity from other GELU properties (e.g., smoothness, probabilistic interpretation), nor does it compare against other non-monotonic activations. The contribution here is therefore more of an opening of a design space than a complete exploration: the paper demonstrates that non-monotonic activations can work and work well, without fully characterizing why or when. The field would later explore this direction more systematically (e.g., Swish/SiLU, Mish, and other non-monotonic activations), but the GELU paper is the first to show that non-monotonicity — when derived from a principled stochastic process rather than arbitrarily introduced — is a feature, not a flaw.

5. Experimental Analysis

Evaluation Methodology

Dataset. Six datasets spanning computer vision, natural language processing, and speech: MNIST classification (grayscale images, 10 classes, 60k training / 10k test), MNIST autoencoding (same data, self-supervised reconstruction), Twitter POS tagging (1000 training / 327 validation / 500 test tweets with 25 tags), TIMIT frame classification (3696 training / 1152 validation / 192 test audio sentences with 39 phone labels), CIFAR-10 (color images, 10 classes, 50k training / 10k test), and CIFAR-100 (color images, 100 classes, 50k training / 10k test). These were chosen to test the GELU across diverse modalities, dataset sizes, and task types (supervised classification, self-supervised reconstruction, sequence labeling).
Base model(s). Task-specific architectures, not a single model family. For MNIST classification: fully connected 8-layer networks, 128 neurons wide. For MNIST autoencoding: a deep autoencoder with layers of width 1000, 500, 250, 30, 250, 500, 1000. For Twitter POS tagging: a two-layer network with 256 neurons per layer using pretrained word vectors from a 56-million-tweet corpus. For TIMIT: a five-layer, 2048-neuron wide classifier with 11-frame input context and 26 MFCC+energy+derivative features per frame. For CIFAR-10: a 9-layer convolutional network following Salimans & Kingma (2016) with batch normalization (architecture detailed in Appendix A). For CIFAR-100: a 40-layer Wide Residual Network with widening factor 4 (Zagoruyko & Komodakis, 2016). These architectures represent a range of depths (2 to 40 layers), widths (128 to 2048 neurons), and structural types (fully connected, convolutional, residual), demonstrating that GELU gains are not architecture-specific.
Metrics. All supervised tasks report test set error rate (%) or log loss at the epoch of lowest validation error (for TIMIT, "median test error chosen at the lowest validation error"). MNIST autoencoding reports reconstruction error (mean squared loss). For MNIST robustness experiments (Figure 3), test set accuracy and log loss are measured under increasing uniform noise added to inputs. For MNIST classification with dropout (Figure 2), both training and validation log loss curves are shown. The primary metric throughout is which activation achieves the lowest error/log loss at convergence, with median statistics over multiple runs to account for training stochasticity.
Baselines. Two activation functions: ReLU (Nair & Hinton, 2010), defined as $\max(0,x)$ , and ELU (Clevert et al., 2016), defined as $x$ for $x > 0$ and $\alpha(e^x - 1)$ for $x \leq 0$ with $\alpha = 1$ . The paper explicitly states it does "not evaluate nonlinearities like the LReLU because of its similarity to ReLUs" (Section 3). The SiLU ( $x\sigma(x)$ ) is mentioned in Section 4 as performing "worse than GELUs but usually better than ReLUs and ELUs," but is not included as a full baseline in the main experiments — it serves as an ablation of the CDF choice rather than a primary comparator.
Generation budget / compute accounting. Since this paper evaluates activation functions rather than generation strategies, "compute" is measured indirectly through training convergence speed (epochs to reach a given loss) and final performance at convergence — there is no explicit FLOP counting or per-sample budget. Fair comparison is ensured by using identical architectures, optimizers, weight initializations, batch sizes, and learning rate tuning protocols across all three activations for each task. The only variable changed is the activation function itself, making differences in convergence and final accuracy attributable to the activation. The paper does note that "we do not introduce any new hyperparameters" (Section 2), meaning the GELU uses no additional tunable parameters that could give it an unfair advantage through extra tuning.
Cross-validation / statistical protocol. The paper reports medians of multiple independent runs (five runs for MNIST classification and Twitter POS tagging; three runs for MNIST autoencoding, TIMIT, CIFAR-10, and CIFAR-100). Learning rates are tuned over {10⁻³, 10⁻⁴, 10⁻⁵} for most tasks, with 5k validation examples held out from the training set for MNIST and CIFAR-10 to select the best rate. For the CIFAR-10 experiments specifically, the paper tunes learning rates "with 5k validation examples then train[s] on the whole training set again based upon the learning rate from cross validation." Weights are initialized with "unit norm rows" (following Hendrycks & Gimpel, 2016; Mishkin & Matas, 2016; Saxe et al., 2014) because "this has positive impact on each nonlinearity's performance." The optimizer is Adam (Kingma & Ba, 2015) for all experiments except CIFAR-100, which uses Nesterov momentum with a cosine annealing schedule (Loshchilov & Hutter, 2016; T₀ = 50, η = 0.1). No formal statistical significance tests (confidence intervals, p-values) are reported — the evaluation relies on median-based comparisons and consistency across multiple tasks.

Main Quantitative Results

MNIST Classification (Section 3.1, Figures 2 and 3)

Convergence without dropout (Figure 2, left): The GELU demonstrates the lowest median training log loss across 50 epochs among the three activations. The validation log loss curves (fainter, upper lines in Figure 2) show the GELU maintaining a consistent advantage over both ReLU and ELU, with the ReLU performing worst. All three activations converge without divergence, indicating the GELU does not introduce training instability.

Convergence with dropout at keep rate 0.5 (Figure 2, right): Adding dropout substantially increases the log loss for all activations (note the y-axis scale change: the validation loss range shifts from ~0.00–0.14 without dropout to ~0.1–0.5 with dropout). The GELU again achieves the lowest median training log loss, though the gap narrows compared to the no-dropout setting. The ELU and ReLU curves track more closely together under dropout than without, but the GELU retains a visible advantage, particularly in training loss. The paper notes this as evidence that "although the GELU is inspired by a different stochastic process, it comports well with dropout" (Section 3.1) — the GELU's built-in input-dependent stochastic regularization does not conflict with or obviate the benefits of explicit dropout; the two can be combined.

Robustness to input noise (Figure 3): Using classifiers trained without dropout, the paper evaluates test set accuracy and log loss as uniform noise Unif[-a, a] is added to test examples at increasing strengths a (where a = 3 represents the maximum noise). For test set accuracy (Figure 3, left), the GELU matches or exceeds both ReLU and ELU across all noise levels. At zero noise (a = 0), the GELU achieves approximately 0.98 accuracy versus roughly 0.97 for both ELU and ReLU. As noise increases to a = 3, the GELU maintains approximately 0.68 accuracy versus roughly 0.64 for ELU and 0.62 for ReLU. For test set log loss (Figure 3, right), the GELU tracks the ELU closely and both outperform the ReLU by a substantial margin at all noise levels above a = 0.5. At maximum noise (a = 3), the GELU achieves approximately 7.5 test log loss versus roughly 8 for ELU and 18 for ReLU. The paper summarizes that "GELUs display robustness matching or exceeding ELUs and ReLUs."

What these results establish: The GELU learns faster (lower training loss at each epoch) and generalizes better (lower validation loss) than ReLUs and ELUs on a standard fully-connected MNIST benchmark, both with and without dropout. The robustness results further show that GELU-trained networks are not brittle — they handle distribution shift (additive input noise) at least as well as the best alternative (ELU) and substantially better than ReLUs, particularly in log loss.

MNIST Autoencoding (Section 3.2, Figure 4)

Headline result: Across two learning rates (10⁻³ and 10⁻⁴), the GELU achieves substantially lower reconstruction error than both ReLU and ELU throughout training, and this advantage persists at convergence.

Learning rate 10⁻³ (Figure 4, left): The GELU converges to a final test reconstruction error of approximately 0.0040–0.0045 at 250 epochs, compared to roughly 0.0055–0.0060 for the ReLU and roughly 0.0065–0.0070 for the ELU. The GELU's training curve (dark, lower) is consistently below the other activations from approximately epoch 20 onward. Notably, the ELU performs worst at this learning rate despite being the best performer on some other tasks — the paper does not explain this reversal but notes that at a learning rate of 0.01 "ELUs diverged, and GELUs and ReLUs converged poorly," indicating that the ELU is more sensitive to learning rate in this autoencoding setting.

Learning rate 10⁻⁴ (Figure 4, right): The ranking is preserved but the gaps narrow. The GELU converges to roughly 0.0040 test reconstruction error, the ReLU to roughly 0.0045–0.0050, and the ELU to roughly 0.0055–0.0060. The convergence is slower overall at this lower learning rate — all activations take approximately twice as many epochs to reach the same loss as at 10⁻³ — but the GELU maintains a clear advantage.

What these results establish: The GELU's superiority extends to self-supervised learning (autoencoding), not just supervised classification. More importantly, the GELU "accommodates different learning rates" — it performs well at both 10⁻³ and 10⁻⁴ without divergence or degradation, while the ELU diverges at 0.01 and underperforms at 10⁻³. This learning rate robustness is a practical advantage: practitioners can use a wider range of learning rates without activation-specific failures.

The autoencoder architecture is notably deep and narrow at the bottleneck (1000 → 500 → 250 → 30 → 250 → 500 → 1000), meaning the GELU must propagate gradients effectively through many layers of varying width. The fact that the GELU outperforms at both the high learning rate (where optimization is aggressive) and the low learning rate (where optimization is conservative) suggests that its gradient properties — smooth, non-vanishing for positive inputs, non-zero for moderate negative inputs — contribute to robust optimization across learning rate regimes.

Twitter POS Tagging (Section 3.3)

Headline result: On a small-data NLP task (1000 training tweets, 25 output tags), the GELU achieves a median test set error of 12.57%, compared to 12.67% for ReLU and 12.91% for ELU.

The network architecture is modest (two layers, 256 neurons each, dropout keep probability 0.8) and uses pretrained word vectors from a 56-million-tweet corpus (Owoputi et al., 2013). The input to the tagger is the concatenation of the target word's vector and its left and right neighbors' vectors, making the input dimension 3 × embedding_dim. The paper trains each network five times per learning rate (tuned over {10⁻³, 10⁻⁴, 10⁻⁵}) and reports the median test set error.

The absolute differences are small: GELU outperforms ReLU by 0.10 percentage points and ELU by 0.34 percentage points. The paper does not report whether these differences are statistically significant, and with only 500 test tweets and 25 output tags (roughly 20 expected examples per tag), the test set is small enough that a handful of correctly classified examples could swing the error rate. The paper frames this as the GELU demonstrating better generalization from limited data, consistent with the pattern across other tasks, but the margin is narrow enough that this experiment alone would not be convincing. Its value lies in the consistency of GELU > ReLU > ELU across yet another modality and task type.

TIMIT Frame Classification (Section 3.4, Figure 5)

Headline result: On phone recognition from audio frames, the GELU achieves a median test error of 29.3%, compared to 29.5% for ReLU and 29.6% for ELU.

The architecture is a five-layer, 2048-neuron wide classifier with 39 output phone labels and a dropout rate of 0.5, following Mohamed et al. (2012) and Srivastava (2013). The input is 11 frames of audio features (26 MFCC, energy, and derivative features per frame, concatenated), and the network must predict the center frame's phone. The training/validation/test split follows the standard TIMIT protocol (3696/1152/192 sentences).

Figure 5 shows training and validation log loss curves (median of five runs). The GELU achieves the lowest training log loss throughout training, with the ELU and ReLU tracking closely together at slightly higher loss. The validation curves (fainter, upper lines) show the GELU consistently below the other activations, though all three are tightly clustered — the final validation log loss for GELU is approximately 1.32 at epoch 30, compared to roughly 1.34 for both ReLU and ELU. The median test error is chosen at the epoch with the lowest validation error.

As with Twitter POS tagging, the absolute gaps are small (0.2–0.3 percentage points). However, TIMIT phone recognition is a well-established benchmark where gains are typically hard-fought — the state of the art at the time using deep belief networks (Mohamed et al., 2012) achieved comparable error rates, and the paper notes that the GELU edges out the baselines despite using an identical architecture and training procedure. The paper emphasizes that the GELU matches or exceeds the baselines, not that it dramatically outperforms them — the claim is one of consistent, if modest, superiority across tasks.

CIFAR-10 Classification (Section 3.5, Figure 6)

Headline result: On a 9-layer convolutional network without data augmentation, the GELU achieves a median test error of 7.89%, compared to 8.16% for ReLU and 8.41% for ELU.

The architecture is from Salimans & Kingma (2016) and is detailed in Appendix A: ZCA whitening, Gaussian noise (σ = 0.15), then three blocks of 3×3 convolutions (96 → 96 → 96 channels) followed by 2×2 max pooling and dropout (p = 0.5), then three more convolutions (192 → 192 → 192 channels), max pooling, dropout, then 3×3 → 1×1 → 1×1 convolutions, global average pooling, and softmax. Batch normalization is used throughout "to speed up training." No data augmentation is applied.

Figure 6 shows median classification error (three runs) over 200 epochs, with the learning rate decaying linearly to zero from epoch 100 to 200. The training curves (darker, lower) show all three activations converging to near-zero training error — the networks essentially memorize the training set — but the test curves (lighter, upper) show clear separation. The GELU's test error drops below 8% around epoch 120 and continues to decline slowly, while the ReLU and ELU plateau earlier at higher error rates. The final median test errors (7.89% GELU, 8.16% ReLU, 8.41% ELU) represent a 0.27 percentage point improvement over ReLU and a 0.52 percentage point improvement over ELU.

At the time of writing, the architecture from Salimans & Kingma (2016) "recently obtained state of the art on CIFAR-10 without data augmentation" (Section 3.5). The GELU therefore improves upon a state-of-the-art result purely through activation function substitution, without any architectural changes or additional regularization. The gap of 0.27 percentage points over ReLU is small in absolute terms but non-trivial at this performance level — it represents roughly a 3.3% relative reduction in error (from 8.16% to 7.89%).

CIFAR-100 Wide Residual Network (Section 3.5, Figure 7)

Headline result: On a 40-layer Wide Residual Network (widening factor 4) with CIFAR-100, the GELU achieves a median test error of 20.74%, compared to 21.77% for ReLU and 22.98% for ELU.

This is the largest-scale experiment in the paper and the result that carries the most weight. The WideResNet (Zagoruyko & Komodakis, 2016) is a modern, high-performing architecture, and CIFAR-100's 100 fine-grained classes make it substantially more challenging than CIFAR-10. The training uses a cosine annealing schedule (Loshchilov & Hutter, 2016) with T₀ = 50 and η = 0.1, Nesterov momentum, and a dropout keep probability of 0.7.

An important architectural detail: the paper notes that "some have noted that ELUs have an exploding gradient with residual networks (Shah et al., 2016), and this is alleviated with batch normalization at the end of a residual block. Consequently, we use a Conv-Activation-Conv-Activation-BatchNorm block architecture to be charitable to ELUs" (Section 3.5). This is a critical fairness consideration: the block ordering is deliberately chosen to prevent the ELU from failing due to a known incompatibility with residual connections, rather than from any inherent inferiority of the activation itself. The fact that the ELU still underperforms despite this accommodating architecture strengthens the case for the GELU.

Figure 7 shows training log loss (with dropout on) and test log loss (with dropout off) over 50 epochs. All three activations converge, but the GELU achieves the lowest test log loss by a visible margin. The final test error of 20.74% for GELU is 1.03 percentage points better than ReLU and 2.24 percentage points better than ELU — substantially larger absolute gaps than in the smaller-scale experiments. The paper also notes that "without our changes described above, the original 40-4 WideResNet with a ReLU obtains 22.89%" (citing Zagoruyko & Komodakis, 2016), meaning the paper's ReLU baseline (21.77%) already benefits from the architectural adjustment made for ELU compatibility, and the GELU improves further still.

This result is particularly significant because it demonstrates that the GELU's advantages scale to deep, modern architectures — not just shallow fully-connected networks or small CNNs. The 1 percentage point gap on a 100-class problem with a 40-layer network is a meaningful practical improvement that cannot be attributed to optimization noise or hyperparameter luck (three runs, median reported, consistent with the paper's pattern of GELU > ReLU > ELU across all tasks).

Summary of Aggregate Performance Across All Tasks

The paper reports six tasks (MNIST classification, MNIST autoencoding, Twitter POS, TIMIT, CIFAR-10, CIFAR-100), and the GELU achieves the lowest error or log loss on all six. The ReLU ranks second on four tasks and third on two; the ELU ranks third on four tasks and second on two. The margins range from narrow (0.10–0.30 percentage points on Twitter POS and TIMIT) to substantial (1.03–2.24 percentage points on CIFAR-100). No task shows the GELU underperforming either baseline.

Ablation Studies and Robustness Checks

Dropout compatibility (Figure 2, right vs. left): The GELU's performance advantage over ReLU and ELU persists when dropout (keep rate 0.5) is added to the 8-layer MNIST network. In the no-dropout setting, the GELU validation log loss curve is visibly separated from the other two activations throughout training. With dropout, all three log loss curves shift upward (higher loss) and the gaps narrow, but the GELU maintains the lowest training loss. The paper interprets this as the GELU "comport[ing] well with dropout" — the GELU's built-in input-dependent stochastic regularization does not render explicit dropout redundant, nor does dropout erase the GELU's advantage. The two regularization mechanisms appear to be complementary rather than competing.

Input noise robustness (Figure 3): On MNIST classifiers trained without dropout, the GELU's accuracy under increasing uniform input noise matches the ELU (both outperform ReLU substantially), while its log loss tracks closer to the ELU and far below the ReLU. At the highest noise level (a = 3), the GELU test set accuracy is approximately 0.68 vs. ~0.64 for ELU and ~0.62 for ReLU; GELU log loss is approximately 7.5 vs. ~8 for ELU and ~18 for ReLU. This demonstrates that GELU-trained networks are not merely overfitting to clean test data — they are as robust or more robust than ELU-trained networks to distribution shift, and far more robust than ReLU-trained networks in terms of log loss (calibrated uncertainty).

Learning rate sensitivity (Figure 4 and various): The MNIST autoencoding experiment explicitly sweeps two learning rates (10⁻³ and 10⁻⁴). The GELU outperforms at both, and the paper notes that at 0.01, "ELUs diverged, and GELUs and ReLUs converged poorly" (Section 3.2), indicating that the GELU shares the ReLU's tolerance for moderate learning rates while the ELU is more fragile. Across the other tasks, learning rates are tuned over {10⁻³, 10⁻⁴, 10⁻⁵} for each activation independently, and the GELU is never reported to diverge or require special treatment. The implicit conclusion is that the GELU is at least as learning-rate-robust as the ReLU.

CIFAR-100 architectural accommodation for ELU: The paper explicitly modifies the residual block ordering to Conv-Activation-Conv-Activation-BatchNorm "to be charitable to ELUs" because ELUs are known to have exploding gradients with standard residual blocks (Shah et al., 2016). This is an informal ablation: it shows that the ELU's underperformance (22.98% vs. GELU's 20.74%) is not due to a known incompatibility that could be fixed with better architecture design — the architecture was already adjusted to help the ELU, and it still lost. The paper also reports that the original WideResNet with ReLU achieved 22.89% error (Zagoruyko & Komodakis, 2016), while the paper's ReLU baseline with the ELU-friendly block ordering achieves 21.77%, confirming that the architectural change helped the ReLU as well (by 1.12 percentage points) but the GELU still wins.

The stochastic form as an existence proof (Section 2): The paper reports that "it is possible to train competitive MNIST and TIMIT networks solely with this stochastic regularizer, all without using any nonlinearity." This is not a formal ablation with tables and curves, but it serves as a critical conceptual validation: the stochastic process underlying the GELU (input-dependent Bernoulli masking with Gaussian CDF probabilities) is sufficient on its own to act as both regularizer and nonlinearity. The deterministic GELU is the expectation of this process; the fact that the stochastic version works confirms that the deterministic form is not just a mathematically convenient smoothing of the ReLU but genuinely encodes the expected behavior of a viable training procedure. No specific error rates are provided for these stochastic-only networks, so the claim remains qualitative.

CDF choice: GELU vs. SiLU (Section 4): The paper states that "a Sigmoid Linear Unit (SiLU) $x\sigma(x)$ performs worse than GELUs but usually better than ReLUs and ELUs." No specific numbers are provided — this is an informal ablation discussed in Section 4 rather than a full experiment. However, it demonstrates that the choice of CDF matters: the Gaussian CDF's lighter tails produce better performance than the logistic CDF's heavier tails, even though the functional form $x \cdot \text{CDF}(x)$ is the same. Combined with the theoretical justification that "neuron inputs tend to follow a normal distribution, especially with Batch Normalization," this supports the claim that matching the CDF to the empirical input distribution is important.

Approximation quality (Section 4): The paper provides two GELU approximations (tanh and sigmoid) and notes that "using a close approximation to the cumulative distribution function of a Gaussian distribution is important." The tanh approximation (0.5x(1 + tanh[√(2/π)(x + 0.044715x³)])) is used in all experiments and is a close match to the true Φ(x). The sigmoid approximation (xσ(1.702x)) is offered as a faster alternative but is implicitly warned against for best performance — the SiLU underperforms the GELU, and the sigmoid approximation to the GELU would share the logistic CDF's heavier tails. No experiment directly compares the tanh-approximated GELU against the exact erf-based GELU, so the approximation error's effect on performance is not quantified. All reported GELU results are for the tanh approximation, not the exact function.

Critical Assessment

Claim from the executive summary: "The GELU consistently outperforms both ReLU and ELU activations across all considered computer vision, natural language processing, and speech tasks." The experiments genuinely support this claim: on six tasks spanning three modalities, the GELU achieves the lowest error or log loss in every case. However, the claim should be qualified in several ways that the paper's framing sometimes glosses over:

First, the margins are often small and no statistical significance is established. On Twitter POS tagging (12.57% vs. 12.67% vs. 12.91%) and TIMIT (29.3% vs. 29.5% vs. 29.6%), the gaps are tenths of a percentage point. With 500 test tweets and 192 test TIMIT sentences, and with only three to five runs reported as medians, random variation could plausibly produce these rankings even if all three activations were equally effective. The paper never reports standard deviations, confidence intervals, or any formal test of whether the observed differences are statistically significant. The consistency of the ranking (GELU > ReLU > ELU across all six tasks) is suggestive but does not substitute for statistical rigor — six independent experiments, each with a non-negligible probability of ranking reversal under the null hypothesis of no difference, do not constitute a formal rejection of the null.

Second, the largest and most convincing margin (CIFAR-100: 20.74% vs. 21.77% vs. 22.98%) comes with a caveat about architectural accommodation for ELU. The paper adjusted the residual block ordering specifically to help the ELU, which also helped the ReLU (reducing its error from the original 22.89% to 21.77%). This means the comparison is both fair (the ELU is not handicapped by a known incompatibility) and unfair in a subtle way: the paper does not explore whether the GELU might benefit from a similarly accommodating architecture, nor whether the standard block ordering (which produced 22.89% ReLU error) would change the GELU/ReLU gap. The reported 1.03 percentage point GELU-over-ReLU advantage is measured in an architecture that was not optimized for either — it was optimized to prevent the ELU from exploding. This cuts both ways: it means the GELU's advantage is robust to suboptimal architecture choices, but it also means we don't know what the gap would be in a GELU-optimized architecture.

Third, the paper trains only three to five runs per setting and reports medians. For CIFAR-100 (three runs), a single outlier run could shift the median by a substantial amount. The paper does not report the spread of results (min, max, or standard deviation), making it impossible to assess whether the three GELU runs were tightly clustered around 20.74% or whether one lucky run pulled the median down. Modern best practices would report error bars or at minimum the range across runs, particularly for the headline CIFAR-100 and CIFAR-10 results.

Fourth, the claim of "consistency" across tasks masks substantial heterogeneity in margin size. The GELU's advantage is large and practically meaningful on CIFAR-100 (1.03 points over ReLU) and MNIST autoencoding (visible separation in loss curves), but it is tiny on TIMIT and Twitter POS tagging (0.2–0.3 points). A practitioner choosing an activation function for a small-data NLP task might reasonably conclude that the GELU offers negligible benefit over the ReLU, while a practitioner training deep CNNs on large-scale image classification would see a more compelling advantage. The paper's narrative of uniform superiority is supported by the ranking data but the practical import varies substantially by task scale and type.

Claim: "The GELU weights inputs by their magnitude, rather than gates inputs by their sign as in ReLUs." The experiments do not directly test this mechanistic claim — they demonstrate that the GELU outperforms the ReLU, from which the paper infers that magnitude-based weighting is superior to sign-based gating. No experiment isolates the effect of the weighting mechanism from other GELU properties (smoothness, non-monotonicity, probabilistic interpretation). The CIFAR-100 result could be due to the GELU's curvature everywhere (providing richer representations), or its smooth gradients (improving optimization), or its non-monotonicity (enabling more efficient function approximation), rather than specifically the magnitude-weighting-vs-sign-gating distinction. The paper's ablation of the SiLU ( $x\sigma(x)$ , which also weights by magnitude but uses a heavier-tailed CDF) shows that the CDF choice matters, but does not test whether any magnitude-weighting function outperforms any sign-gating function — it compares one magnitude-weighting function to two sign-gating functions and finds it superior, which is evidence but not a controlled test of the hypothesized mechanism.

A missing experiment: exact GELU vs. tanh-approximated GELU. The paper uses the tanh approximation in all experiments but never compares it against the exact erf-based computation. If the approximation introduces meaningful error in the tails or near the origin, the reported "GELU" results might not reflect what an exact implementation would achieve. Conversely, if the exact GELU were meaningfully better than the approximation, the paper's practical recommendation to use the approximation might be suboptimal. This omission is understandable — the tanh form is faster and the approximation is designed to be close — but it means we cannot be certain that the reported gains over ReLU and ELU are from the GELU's mathematical form rather than from serendipitous properties of the specific approximation used.

A missing experiment: learnable μ and σ. The paper explicitly notes that μ and σ could be made learnable but deliberately chooses not to, keeping the GELU parameter-free. This is a reasonable design choice for a paper arguing the GELU is a drop-in ReLU replacement, but it leaves open the question of whether learnable distribution parameters would further improve performance. Given that Batch Normalization already standardizes activations, the benefit might be small, but the paper provides no evidence either way. The later "swish" paper (Ramachandran et al., 2017) would add a learnable β parameter to $x\sigma(\beta x)$ , and if such a parameter improves the SiLU, it might also improve the GELU. The paper's decision to forgo learnable parameters is presented as a feature ("we do not introduce any new hyperparameters"), but it may also mean the GELU is not operating at its full potential.

A missing experiment: GELU without Batch Normalization. The entire probabilistic motivation for the GELU depends on the assumption that "neuron inputs tend to follow a normal distribution, especially with Batch Normalization." If Batch Normalization is what makes inputs approximately normal, then the GELU's theoretical justification is contingent on using Batch Normalization. But the paper never tests the GELU without Batch Normalization — all experiments either explicitly use it (CIFAR-10/100) or were conducted in an era when it was standard. Would the GELU still outperform ReLUs in a network without Batch Normalization, where pre-activations might not be normally distributed? The paper provides no evidence. This is not a fatal omission — Batch Normalization was ubiquitous and the GELU was designed for Batch-Normalized networks — but it means the GELU's probabilistic interpretation is a conditional justification, not a universal one.

A missing experiment: deeper/wider scaling to saturation. The paper tests a range of architectures (2 to 40 layers, 128 to 2048 neurons wide) but does not systematically study how the GELU advantage scales with network capacity. Does the GELU's advantage grow, shrink, or plateau as networks become very deep? Does it help more in overparameterized regimes (where regularization matters more) or underparameterized regimes (where representational capacity matters more)? The WideResNet experiment (40 layers) shows the largest GELU advantage, hinting that the benefit may increase with depth, but this is a single data point, not a scaling study. The paper's later impact — GELU becoming the default in very deep Transformers like BERT and GPT — retrospectively supports the hypothesis that GELU scales well to extreme depths, but the paper itself provides no systematic evidence for this.

The stochastic-regularizer-only training claim is qualitative. The paper states that competitive MNIST and TIMIT networks can be trained with only the stochastic regularizer and no nonlinearity, but provides no error rates, training curves, or comparison to baselines for this claim. This is a conceptual proof-of-concept rather than a rigorous experimental result. It would have been informative to see how the stochastic-only networks compared to ReLU networks (with and without dropout) — if the stochastic regularizer alone matches or exceeds ReLU+dropout, that would be a powerful validation of the unified activation-regularization framework. The paper leaves this as a qualitative assertion in the GELU formulation section (Section 2) rather than elevating it to a full experiment.

Robustness to input noise is tested only on MNIST. Figure 3 demonstrates GELU robustness on MNIST with uniform noise, but this is a single dataset with a single noise type. Would the GELU's robustness advantage hold for CIFAR-10/100 under common corruptions (blur, contrast, weather)? Would it hold for adversarial perturbations? The paper makes no claims about adversarial robustness, and input noise on MNIST is a weak test of generalization under distribution shift — MNIST digits remain recognizable even under extreme uniform noise. The robustness result is therefore suggestive but narrow.

Overall assessment: The experiments demonstrate a consistent empirical pattern — GELU ≥ ReLU ≥ ELU across six tasks — that, in aggregate, makes a compelling case that the GELU is a viable and often superior alternative to existing activations. The paper's central claim of consistent outperformance is supported by the directional consistency of the results. However, the practical significance of the advantage varies dramatically by task (from negligible on TIMIT to meaningful on CIFAR-100), and the paper does not provide the statistical tools (error bars, significance tests) or mechanistic ablations that would allow a reader to understand why the GELU outperforms or to predict when the advantage will be large versus small. The omission of certain experiments — exact-vs-approximate GELU, GELU without Batch Normalization, learnable parameters, deeper scaling — means the paper establishes the GELU as a strong empirical finding without fully characterizing its operating envelope or the mechanisms underlying its advantage. The experiments are sufficient to motivate adoption and further study but insufficient to close the book on when and why the GELU works best.

6. Limitations and Trade-offs

Limitation 1: The GELU's Probabilistic Motivation Depends on Batch Normalization, Yet This Dependency Is Never Tested in Isolation

The entire conceptual foundation of the GELU — that xΦ(x) computes the input multiplied by the probability that a standard normal random variable is less than or equal to that input — rests on a specific empirical claim about the distribution of neuron pre-activations. The paper states:

"We choose this distribution since neuron inputs tend to follow a normal distribution, especially with Batch Normalization."

This is a conditional justification: if neuron inputs are approximately N(0,1), then Φ(x) is interpretable as x's percentile rank within the layer's activation distribution, and the GELU's magnitude-based weighting inherits a clean probabilistic meaning. The paper does not claim that neuron inputs are normally distributed in general — it claims they are approximately normal because of Batch Normalization. The mechanism that justifies the GELU (Batch Normalization) is logically prior to and independent of the GELU itself.

The consequence: Without Batch Normalization, the GELU's probabilistic interpretation collapses. If pre-activations are not normally distributed — for instance, in networks using Layer Normalization, weight normalization, or no normalization at all — then Φ(x) is no longer the empirical CDF of the layer's inputs. A pre-activation of x = 1.0 might correspond to the 84th percentile under N(0,1), but if the actual activation distribution is skewed, heavy-tailed, or has a different variance, Φ(1.0) could be wildly miscalibrated. The GELU would still output some value — it is a deterministic function regardless of its inputs' distribution — but the intended behavior (weighting inputs by how typical they are relative to their peers) would no longer hold. The function would become, in effect, an arbitrary smooth nonlinearity whose shape was chosen for a distribution that does not match the data flowing through it.

This is not merely a theoretical concern. The paper never reports a single experiment without Batch Normalization. Every architecture in the paper either explicitly uses Batch Normalization (CIFAR-10/100: "using batch normalization to speed up training") or was designed in an era when it was standard for the architectures used (the 8-layer MNIST fully-connected network, the 5-layer TIMIT classifier). The GELU's empirical superiority is therefore conditional on the presence of Batch Normalization, but the strength of this dependency is completely unmeasured. A practitioner deploying the GELU in a Transformer with Layer Normalization (which became standard in architectures like the original BERT and GPT — both of which eventually adopted GELU) cannot know from this paper whether the activation's advantage transfers or whether the distributional mismatch degrades performance relative to activations with less distribution-dependent justifications.

What evidence exists in the paper: Zero. The paper contains no ablation comparing GELU performance with and without Batch Normalization. It contains no measurement of how normal the pre-activation distributions actually are under the tested architectures. It contains no comparison of GELU against ReLU or ELU in a network that uses an alternative normalization scheme (Layer Normalization, weight normalization) or no normalization at all. The claim that neuron inputs are normally distributed "especially with Batch Normalization" is presented as motivation, not as a tested hypothesis.

Mitigation status: The paper does not acknowledge this as a limitation, does not measure it, and does not suggest future work on the GELU's dependence on input distribution. The entire probabilistic motivation is treated as a derivation of the functional form rather than as an empirical claim requiring validation. In fairness, Batch Normalization was so universal at the time of writing (2016) that testing without it may have seemed pointless — any serious deep network used Batch Normalization. But the subsequent adoption of GELU in Transformer architectures (which overwhelmingly use Layer Normalization, not Batch Normalization) retrospectively makes this omission significant. The fact that GELU does work well in Transformers suggests the distributional assumption may be less critical than the paper implies, but the paper provides no evidence or analysis to help a practitioner understand why or when the assumption matters.

Limitation 2: No Statistical Rigor — All Results Are Medians of 3–5 Runs Without Any Measure of Variance

Every quantitative claim in the paper rests on comparisons of median performance across a small number of training runs — typically three or five — without any reported measure of dispersion. The paper states:

"Each 8-layer, 128 neuron wide neural network is trained for 50 epochs... We tune over the learning rates... and take the median results for five runs." (MNIST, Section 3.1)

"Over three runs we obtain the median convergence curves in Figure 7." (CIFAR-100, Section 3.5)

The paper never reports standard deviations, min/max ranges, confidence intervals, or the results of any statistical test (t-test, bootstrap, Mann-Whitney) comparing activation functions. The reader is asked to compare point estimates — "12.57% for the GELU, 12.67% for the ReLU, and 12.91% for the ELU" (Twitter POS) — without knowing whether these medians are stable or whether the differences could easily reverse under additional runs.

The consequence: The paper's headline claim — that the GELU "consistently" outperforms ReLU and ELU — is supported by the direction of the ranking across six tasks, but the magnitude of the advantage is uninterpretable without variance estimates. On Twitter POS tagging (500 test examples, 25 output tags), the GELU's 0.10 percentage point advantage over ReLU (12.57% vs. 12.67%) could reflect a single correctly classified example out of 500. With only five runs, the median being 0.10 lower for GELU does not rule out the possibility that the true expected error rates are equal and the observed difference is sampling noise. On TIMIT (192 test sentences), the 0.2–0.3 percentage point gap similarly corresponds to a tiny number of test examples.

The problem is most acute for CIFAR-100, where the paper reports only three runs to produce the headline 20.74% (GELU) vs. 21.77% (ReLU) vs. 22.98% (ELU) medians. With three runs, the median is simply the middle value — a single outlier run (in either direction) can shift it substantially. If one of the three GELU runs happened to be particularly lucky, the reported 20.74% could be unrepresentative. Without knowing the spread — Did the three GELU runs produce 20.5%, 20.7%, 21.0%, or 19.0%, 20.7%, 28.0%? — the practitioner cannot assess whether the 1.03 percentage point advantage over ReLU is reliable or fragile.

This is not merely a reporting omission; it undermines the paper's ability to make relative claims about which results are most significant. The paper implicitly treats all six results as equally supportive of the GELU's superiority, but if we had error bars, the CIFAR-100 advantage (large absolute gap, but only three runs) might look less statistically robust than the MNIST autoencoding advantage (clear separation in loss curves across 250 epochs), while the Twitter POS and TIMIT advantages might vanish entirely into overlapping confidence intervals. The paper's narrative of uniform, cross-task superiority would either be strengthened (if the variances are small) or weakened (if the variances are large enough to make the small-margin results statistically insignificant).

What evidence exists in the paper: The paper reports the number of runs for each experiment (five for MNIST classification and Twitter POS tagging; three for MNIST autoencoding, TIMIT, CIFAR-10, and CIFAR-100) but never reports any measure of variance. The learning curves in Figures 2 and 4–7 show "median" curves without shaded regions indicating run-to-run variability. The bar plots in Figure 1 (from the paper's later positioning) aggregate results but similarly lack error bars. The paper contains no discussion of statistical methodology beyond specifying medians and run counts.

Mitigation status: The paper does not acknowledge this as a limitation. The choice to report medians rather than means is a reasonable robustness measure against outlier runs, but reporting medians without any measure of dispersion provides only half the statistical picture. The paper's consistency argument — GELU ranks first on all six tasks — is a form of informal meta-analysis (a sign test would reject the null hypothesis of equal performance at p ≈ 0.016 if all six rankings were independent and equally likely under the null), but this is never stated or tested. A practitioner cannot determine from the paper whether the GELU's advantage on their specific task of interest is likely to be real or attributable to run-to-run variance. Modern best practices (post-2020) would require at minimum standard deviations or bootstrapped confidence intervals; the paper predates these norms but the limitation remains for anyone evaluating the strength of its empirical claims.

Limitation 3: All Experiments Use a Single Approximate Implementation of the GELU — the True Function Is Never Evaluated

The paper provides the exact definition of the GELU in terms of the error function:

$\text{GELU}(x) = x \cdot \frac{1}{2}\left[1 + \text{erf}\left(\frac{x}{\sqrt{2}}\right)\right]$

However, every experiment in the paper uses a tanh-based approximation, not the exact erf-based computation:

$\text{GELU}(x) \approx 0.5x\left(1 + \tanh\left[\sqrt{\frac{2}{\pi}}\left(x + 0.044715x^3\right)\right]\right)$

The paper is explicit about this: "we used the former in every experiment in this paper" (Section 4, referring to the tanh approximation). The motivation is clear — the tanh function is fast and GPU-accelerated, while the error function requires more expensive numerical computation — but the consequence is that all reported "GELU" results are actually results for a specific polynomial approximation to the GELU. The paper never measures how closely this approximation matches the true function, never compares the approximation against the exact GELU on any task, and never quantifies whether any performance difference between "GELU" and ReLU/ELU could be partially attributable to the approximation's specific numerical properties rather than to the GELU's mathematical form.

The consequence: The tanh approximation is not the GELU; it is an approximation to the GELU. The approximation was designed to be close — the cubic term 0.044715x³ is optimized to match the Gaussian CDF's shape — but the paper provides no maximum error bound, no comparison of the approximation's derivatives against the true GELU's derivatives, and no experiment demonstrating that the exact GELU would produce identical or even similar results. If the approximation introduces systematic errors — for instance, if it slightly overestimates or underestimates Φ(x) in specific regions — those errors are baked into every empirical result in the paper.

This matters for two reasons. First, it creates a reproducibility ambiguity: a practitioner who implements the GELU using the exact erf formulation (because they read the mathematical definition and want to use the "true" function) might obtain different results than those reported in the paper. The paper's empirical claims are not about xΦ(x) in general; they are about 0.5x(1 + tanh[√(2/π)(x + 0.044715x³)]) in particular. Second, the paper's theoretical framework — the probabilistic interpretation, the connection to input-dependent dropout, the CDF-as-percentile-rank argument — all refer to the exact xΦ(x), not to the tanh approximation. If the approximation's slight deviations from the true CDF happen to improve optimization (e.g., by providing slightly larger gradients in regions where the true CDF is very flat), then the paper's theoretical justification is not the cause of the empirical gains — the gains are partly a happy accident of the specific approximation chosen. Conversely, if the approximation slightly degrades the true GELU's performance, then the paper understates the GELU's potential.

The paper also provides a second approximation — xσ(1.702x) — but explicitly notes that the SiLU (xσ(x), without the 1.702 scaling) "performs worse than GELUs." The 1.702 scaling factor is chosen to make the sigmoid shape match the Gaussian CDF, but the paper never reports whether xσ(1.702x) was tested as an alternative GELU approximation or whether it was only considered as a conceptual bridge to the SiLU. The existence of multiple approximations with different accuracy-speed tradeoffs, and the paper's exclusive use of one without comparison to the others or to the exact function, leaves the practitioner without guidance on which implementation to use.

What evidence exists in the paper: None. There is no experiment comparing the tanh-approximated GELU against the exact erf-based GELU. There is no measurement of the approximation error (maximum absolute error, mean squared error, error in derivatives). There is no ablation demonstrating that the specific cubic coefficient (0.044715) is better than alternative values or that the tanh form is superior to the sigmoid approximation. The paper simply uses the tanh approximation throughout and reports results as "GELU."

Mitigation status: The paper acknowledges the existence of the approximation tradeoff — "if greater feedforward speed is worth the cost of exactness" — but frames this as a practitioner choice between the exact function (slow, exact) and the approximations (fast, approximate). It does not acknowledge that the reported results are for the approximation, not the exact function, and it does not characterize the approximation's fidelity. The paper does not suggest that future work should compare exact vs. approximate GELU performance or characterize the approximation's impact. For a paper introducing a new activation function, the fact that every empirical demonstration of that function's superiority actually uses a different (if closely related) function is a significant evidentiary gap — the function being advocated for (xΦ(x)) is not the function being tested.

Limitation 4: The GELU's Advantage on Small-Dataset Tasks Is Too Small to Be Practically Meaningful — and Possibly Statistically Insignificant

Three of the paper's six experiments — Twitter POS tagging (1000 training examples, 500 test), TIMIT frame classification (3696 training, 192 test), and arguably MNIST classification (60k training but results reported as log loss curves rather than test error) — show GELU advantages measured in tenths of a percentage point of test error. On Twitter POS tagging, the ranking is 12.57% (GELU), 12.67% (ReLU), 12.91% (ELU) — a spread of 0.34 percentage points covering all three activations. On TIMIT, the ranking is 29.3% (GELU), 29.5% (ReLU), 29.6% (ELU) — a spread of 0.3 percentage points.

The consequence: For a practitioner choosing an activation function, a 0.1–0.3 percentage point improvement on a small test set is not a meaningful basis for decision-making. On TIMIT's 192 test sentences, a 0.2 percentage point difference corresponds to less than half a sentence correctly classified that would have been misclassified under the alternative activation. This difference is smaller than what could be caused by different random seeds, different train/validation splits, or minor variations in hyperparameter tuning that the paper's protocol (tuning learning rate over three values with 5k validation examples) might not fully control.

More importantly, the paper does not establish that these differences are generalizable rather than dataset-specific. The Twitter POS tagger uses pretrained word vectors from a specific 56-million-tweet corpus; the TIMIT system uses a specific 11-frame MFCC feature representation. Would the GELU maintain its 0.1–0.3 point advantage on a different POS tagging dataset? On a different phone recognition corpus? On the same datasets with different preprocessing or feature extraction? The paper provides no evidence because each task is tested on a single dataset with a single architecture. The small margins on these tasks mean that even minor changes to the experimental setup (different word vectors, different acoustic features, different train/validation split) could plausibly reverse the ranking.

This limitation interacts with the absence of variance estimates (Limitation 2): if the run-to-run standard deviation on TIMIT is, say, 0.3 percentage points, then the observed ranking (29.3% vs. 29.5% vs. 29.6%) is entirely consistent with all three activations having identical expected performance. The paper's narrative treats every ranking as evidence for the GELU's superiority, but on these small-data tasks, the evidence is too weak to support practical action.

What evidence exists in the paper: The raw numbers themselves demonstrate the small margins. The paper reports the Twitter POS and TIMIT results without comment on their magnitude, treating them as additional data points in the consistent pattern of GELU > ReLU > ELU. The learning curves in Figure 5 (TIMIT) show the three activations' validation losses nearly overlapping throughout training, with the GELU only visibly separating in the final few epochs. The paper does not report whether the differences are statistically significant or whether they hold across alternative train/test splits.

Mitigation status: The paper does not mitigate this limitation — it does not acknowledge that some of its results are too close to be practically meaningful, does not provide statistical tests, and does not test the small-data tasks on multiple datasets to demonstrate generalization. The limitation is partially addressed by the larger-scale experiments (CIFAR-10 and especially CIFAR-100), where the margins are larger (0.27 and 1.03 percentage points respectively) and more likely to be practically meaningful. However, the paper's claim of cross-task consistency implicitly gives equal weight to the small-margin and large-margin experiments, which overstates the strength of the evidence from the small-data tasks. A practitioner evaluating the paper should discount the Twitter POS and TIMIT results as suggestive but inconclusive, and focus on the CIFAR and autoencoding experiments as the primary empirical support for the GELU's superiority.

Limitation 5: The Paper Provides No Mechanistic Understanding of Why the GELU Outperforms — Only Correlational Evidence That It Does

The paper offers several hypotheses for why the GELU might outperform ReLUs and ELUs:

Magnitude-based weighting ("the GELU weights its input depending upon how much greater it is than other inputs") rather than sign-based gating.
Non-monotonicity and curvature everywhere ("increased curvature and non-monotonicity may allow GELUs to more easily approximate complicated functions").
The probabilistic interpretation as a built-in pseudoensemble mechanism (the expected output under input-dependent stochastic regularization).
Smooth gradients (the GELU is infinitely differentiable, unlike the ReLU's non-differentiable kink at zero).

However, none of these hypotheses are tested in isolation. Every experiment in the paper compares the GELU against the ReLU and ELU, where all three activations differ along multiple dimensions simultaneously — deterministic vs. probabilistic motivation, sign-gating vs. magnitude-weighting, convex vs. non-convex, monotonic vs. non-monotonic, piecewise-linear vs. fully curved, zero-output for negatives vs. attenuated negative outputs. When the GELU wins, the paper cannot attribute the win to any specific property; when it wins by different margins on different tasks, the paper cannot explain why the margin varies.

The consequence: A practitioner cannot predict from this paper when the GELU will provide large gains versus small gains. The CIFAR-100 advantage (1.03 percentage points) is substantially larger than the TIMIT advantage (0.2 points), but the paper offers no explanation. Is the GELU especially helpful for very deep networks (40 layers on CIFAR-100 vs. 5 layers on TIMIT)? For networks with residual connections? For tasks with many output classes (100 vs. 39)? For larger datasets (50k vs. 3.7k training examples)? For convolutional architectures vs. fully-connected? Any of these hypotheses could be tested by the paper's existing data — the experiments span a range of depths, widths, dataset sizes, and architectures — but the paper makes no attempt to correlate GELU advantage with any architectural or data property. The results are presented as a flat list of six victories without analysis of patterns in the victory margins.

This also means the paper provides no guidance for future activation function design. If the GELU works because of non-monotonicity, then future work should explore other non-monotonic functions. If it works because of smooth gradients, then any smooth activation should match or exceed it. If it works because of the probabilistic interpretation, then other CDF-based activations might do even better. But because the paper does not isolate these mechanisms, it provides no basis for choosing among these hypotheses. The single CDF ablation — the SiLU (xσ(x)) performs "worse than GELUs but usually better than ReLUs and ELUs" — is reported qualitatively in Section 4 without specific numbers, and even this comparison confounds distribution shape (Gaussian vs. logistic CDF) with tail behavior, curvature, and approximation properties. The SiLU could underperform the GELU because the logistic distribution is a poor match for Batch-Normalized inputs, or because lighter tails are beneficial, or because of unrelated optimization dynamics — the comparison does not isolate the CDF-matching mechanism.

What evidence exists in the paper: The paper demonstrates that the GELU outperforms across six tasks, but this is evidence that it works, not why it works. The SiLU comparison in Section 4 is the closest thing to a mechanistic ablation, but it is qualitative ("performs worse") without experimental details and confounds multiple variables. The paper contains no experiments that vary only one GELU property while holding others constant — for example, comparing the GELU against a monotonic version with matched curvature, or against a sign-gating function with matched smoothness, or against the stochastic regularizer at different noise levels to isolate the pseudoensemble effect.

Mitigation status: The paper does not claim to provide mechanistic understanding — it is framed as an empirical contribution introducing a new function and demonstrating its effectiveness. The Section 4 discussion offers hypotheses ("increased curvature and non-monotonicity may allow...") but does not test them. The paper does not suggest future mechanistic work. However, the absence of mechanistic understanding is a practical limitation for deployment: a practitioner who tries the GELU on a new task and finds no improvement over ReLU has no guidance from the paper about why — whether the task lacks the properties that make GELU helpful, or whether they implemented it incorrectly, or whether they need to adjust other hyperparameters (learning rate, initialization, normalization) to realize the gains. The paper's empirical approach demonstrates existence but not operating conditions.

Limitation 6: The Experiments Use Architectures and Tuning Protocols Optimized for ReLU-Based Training, Potentially Understating GELU Gains

Every architecture and training protocol in the paper was originally designed and tuned for networks using ReLU activations — the 8-layer MNIST network, the Salimans & Kingma (2016) CIFAR-10 architecture, the WideResNet (Zagoruyko & Komodakis, 2016), the TIMIT classifier from Mohamed et al. (2012). The paper's experimental methodology is to swap the activation function while keeping everything else — depth, width, learning rate tuning range {10⁻³, 10⁻⁴, 10⁻⁵}, optimizer choice (Adam or Nesterov momentum), weight initialization (unit norm rows), batch size, and regularization (dropout rates, noise levels) — identical across activations. This makes the comparison fair in the sense that the GELU is not given an unfair advantage through extra tuning, but it also means the GELU is evaluated in architectures and hyperparameter regimes that were not optimized for its specific properties.

The consequence: The GELU changes the optimization landscape — it has different gradient properties (smooth everywhere, non-zero gradient for negative inputs), different saturation behavior (soft squashing rather than hard zeroing), and different interactions with weight initialization (since its output distribution differs from the ReLU's). An architecture or training protocol optimized for the ReLU might be suboptimal for the GELU. For instance, the paper uses a specific weight initialization (unit norm rows) because "this has positive impact on each nonlinearity's performance" (Section 3.1), but this initialization was developed for ReLU-like activations. A GELU-specific initialization might further improve performance. Similarly, the learning rate tuning range {10⁻³, 10⁻⁴, 10⁻⁵} was chosen based on ReLU conventions — the GELU's smoother gradients might allow or benefit from larger learning rates that would cause ReLU divergence, or might prefer a different learning rate schedule. The paper does not explore GELU-specific learning rates outside this ReLU-derived range.

The CIFAR-100 experiment reveals this limitation concretely. The paper modifies the residual block ordering from the standard Conv-BatchNorm-ReLU-Conv-BatchNorm-ReLU to Conv-Activation-Conv-Activation-BatchNorm specifically "to be charitable to ELUs" because ELUs are known to have exploding gradient issues with the standard ordering (Shah et al., 2016). This architectural change also happens to improve the ReLU baseline (from the original 22.89% error to 21.77%), demonstrating that the ReLU's performance is sensitive to block ordering. The paper never explores whether a different block ordering — one optimized for the GELU — might further improve the GELU's performance. The GELU might benefit from different normalization placement, different residual connection patterns, or different width/depth tradeoffs than the architectures inherited from ReLU-centric design.

This is not a flaw in the experimental design per se — controlling for architecture and hyperparameters is the standard method for isolating the effect of the activation function — but it means the reported GELU advantages are lower bounds. The GELU might achieve even larger gains if architectures and training protocols were co-designed with it rather than borrowed from ReLU-optimized baselines. A practitioner who adopts the GELU and also tunes other hyperparameters (wider learning rate search, GELU-aware initialization, different normalization placement) might see larger improvements than the paper reports. Conversely, a practitioner who swaps in the GELU without any other changes — exactly as the paper does — can expect gains matching the paper's reported magnitudes.

What evidence exists in the paper: The paper's methodology is explicitly to keep everything except the activation function constant. The CIFAR-100 architectural change is the only exception, and it was made for the ELU's benefit, not the GELU's. The paper never reports experiments with GELU-specific tuning, nor does it discuss whether the fixed hyperparameter ranges might disadvantage or advantage the GELU relative to baselines.

Mitigation status: The paper does not acknowledge this as a limitation — it presents the fixed-hyperparameter comparison as a feature (fairness) rather than a potential understatement of GELU performance. In practice, this limitation is partially self-correcting: if the GELU becomes widely adopted, the community will naturally co-evolve architectures and training recipes around it, as indeed happened with Transformers (where GELU became the default and architectures were subsequently designed with GELU in mind). But within the scope of this paper, the limitation means the reported numbers should be interpreted as "GELU advantage under ReLU-optimized conditions," which may understate the achievable advantage under GELU-optimized conditions. The paper does not suggest future work on GELU-specific architecture design or hyperparameter optimization.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper causes a conceptual reframing of what an activation function is, rather than an incremental refinement of existing functions. Before the GELU, the field's mental model of an activation function was a deterministic nonlinearity — a mathematical transformation applied to a neuron's pre-activation to introduce nonlinearity into an otherwise linear stack of layers. The ReLU, sigmoid, tanh, and ELU all fit this mold. They were designed as functions first, justified by their optimization properties (vanishing gradients, dead neurons, zero-centered outputs), and any connection to stochastic regularization existed only in the separate design of techniques like dropout — which acted on top of the activation, not as the activation.

The GELU shifts this framing by demonstrating that an activation function can be the deterministic expectation of an input-dependent stochastic regularizer. The activation function is no longer an arbitrary engineering choice; it is the limit of averaging infinitely many stochastic forward passes where each neuron's output is probabilistically multiplied by zero or one, with the probability depending on how large that neuron's input is relative to the layer's activation distribution. The consequence is that the boundary between "activation function" and "stochastic regularizer" dissolves: the same mathematical object — an input-dependent Bernoulli mask with Gaussian CDF probabilities — serves as both a trainable stochastic regularizer (when sampled) and a deterministic activation (when the expectation is taken). The paper makes this explicit by showing that competitive MNIST and TIMIT networks can be trained "solely with this stochastic regularizer, all without using any nonlinearity" (Section 2), demonstrating that the nonlinear behavior traditionally provided by a separate activation function can emerge entirely from input-dependent masking.

This is not a paradigm shift in the Kuhnian sense — it does not overturn a dominant theoretical framework, because the field had no deep theoretical framework for activation design to begin with. Rather, it is a new design principle: activation functions can be derived from desired stochastic regularization behaviors, and vice versa. The paper explicitly opens this design space by noting that "we could use different CDFs" (Section 2) and providing the SiLU (xσ(x)) as a logistic-CDF alternative. Any CDF defines a stochastic regularizer (multiply by zero or one with input-dependent probability) and its expectation defines a corresponding deterministic activation. This principled framework for generating activation functions — choose a distribution that matches your layer's expected input distribution, use its CDF to weight inputs, and take the expectation — replaces ad-hoc functional form search with a theoretically motivated procedure.

The work also resolves a latent tension in the relationship between activation functions and dropout. Before the GELU, dropout was understood as an ensemble method that approximately averages over exponentially many subnetworks (the "pseudoensemble" interpretation, Bachman et al., 2014). But dropout was always external to the activation — applied after the nonlinearity — and its probability was fixed and input-independent. The GELU embeds a better version of this pseudoensemble mechanism directly into the activation: each neuron is probabilistically included or excluded based on whether it carries signal for the current input, not based on an arbitrary fixed coin flip. The resulting deterministic forward pass computes the expectation of an input-adaptive ensemble in a single pass, without the need for Monte Carlo sampling or weight scaling. This reframes the activation function as an implicit ensemble mechanism, not merely a source of nonlinearity.

Research directions that become more attractive after this work include:

CDF-based activation design: The GELU demonstrates that matching the CDF to the expected input distribution matters — the Gaussian CDF works well because Batch Normalization makes inputs approximately normal, while the logistic CDF (SiLU) works less well despite the identical functional form x · CDF(x). This opens the door to activation functions using other distributions (Laplace, Student's-t, asymmetric distributions) chosen to match the empirical input distributions of specific architectures or normalization schemes.
Unified regularization-activation design: If an activation function is the expectation of a stochastic regularizer, then practitioners can design both simultaneously — choose a stochastic masking profile that provides desired regularization properties, and the corresponding deterministic activation follows automatically. This eliminates the need to tune dropout rates and activation functions as separate hyperparameters.
Input-dependent regularization beyond dropout: The GELU's mechanism of input-dependent Bernoulli masking can be generalized to other stochastic regularizers — input-dependent Gaussian noise injection, input-dependent zoneout for RNNs, or input-dependent stochastic depth for residual networks — with corresponding deterministic activations derived by taking expectations.

Research directions that become less attractive include:

Monotonic activation function search: The GELU is non-monotonic (it dips slightly negative before approaching zero for negative inputs) and the paper explicitly notes this as a potential advantage (Section 4). This suggests that the long-standing implicit assumption that activations should be monotonic is unnecessarily restrictive, and future work should explicitly explore non-monotonic functions rather than constraining the search space.
Piecewise-linear activation design: The ReLU family (ReLU, LeakyReLU, PReLU, ELU's positive domain) is piecewise linear — linear with slope 1 for positives, linear with some slope for negatives. The GELU is "not linear in the positive domain and exhibits curvature at all points" (Section 4), and it consistently outperforms piecewise-linear alternatives. This suggests that curvature everywhere — not just at a threshold or in the negative regime — is beneficial for representational power, making piecewise-linear designs less attractive as a starting point for new activation functions.

Follow-Up Research This Work Enables

Characterizing GELU advantage as a function of network depth and width. The paper's experiments span depths from 2 to 40 layers and widths from 128 to 2048 neurons, but the GELU advantage is not systematically analyzed as a function of architectural scale. The largest advantage appears on the deepest network (40-layer WideResNet on CIFAR-100: 1.03 percentage points over ReLU), while the smallest advantages appear on shallower, narrower networks (TIMIT 5-layer: 0.2 points; Twitter POS 2-layer: 0.1 points). A systematic scaling study — training identically-structured networks at depths {2, 5, 10, 20, 40, 80} and widths {64, 128, 256, 512, 1024, 2048} with GELU vs. ReLU on CIFAR-10/100 or ImageNet — would determine whether the GELU's advantage monotonically increases with depth (supporting a representational capacity hypothesis), plateaus (supporting an optimization advantage that matters most at moderate depth), or shows a U-shaped pattern. Such a study would provide practitioners with a predictive model: given your architecture's depth and width, how large an improvement can you expect from switching to GELU?

Ablation of GELU properties through controlled synthetic activations. The GELU differs from the ReLU along multiple dimensions simultaneously — smoothness, non-monotonicity, curvature in the positive domain, probabilistic interpretation, negative-value allowance. The paper's experiments compare GELU against ReLU and ELU, where all these properties change at once, so it is impossible to attribute the performance gain to any specific mechanism. A controlled ablation would design a family of synthetic activation functions that vary only one property at a time: (a) a "smoothed ReLU" that uses a softplus-like transition at zero but remains linear in the positive domain and monotonic; (b) a "curved ReLU" that uses xΦ(x) for positives and zero for negatives, isolating curvature in the positive domain; (c) a "monotonic GELU" that removes the negative-region dip while preserving the CDF-weighting everywhere else; (d) a "sign-gating GELU" that uses the GELU's smoothness but with hard zeroing at zero. Training these synthetic activations against GELU and ReLU on CIFAR-10/100 would identify which properties contribute how much to the performance gap. A strong negative result — e.g., the smoothed ReLU matching the GELU — would imply the GELU's advantage is purely about smooth gradients and its CDF-based probabilistic motivation is incidental.

GELU with and without Batch Normalization. The paper's probabilistic motivation for the GELU — that Φ(x) is interpretable as x's percentile rank within the layer's activation distribution — depends entirely on the assumption that "neuron inputs tend to follow a normal distribution, especially with Batch Normalization" (Section 2). However, the paper never tests the GELU without Batch Normalization. A direct experiment would train identical networks (e.g., the 8-layer MNIST architecture and the 9-layer CIFAR-10 architecture) with GELU vs. ReLU vs. ELU under three normalization conditions: Batch Normalization, Layer Normalization, and no normalization (using careful weight initialization to enable training). If the GELU's advantage persists without Batch Normalization but diminishes (or vanishes), it would validate the CDF-matching hypothesis — the Gaussian CDF is well-matched to Batch-Normalized inputs but mismatched to other distributions — and suggest that practitioners using alternative normalization schemes should consider CDFs matched to their specific input distributions (e.g., a logistic CDF for Layer-Normalized inputs if they follow a heavier-tailed distribution). If the GELU's advantage is equally large across all normalization schemes, the probabilistic motivation is a red herring and the GELU works for other reasons (smoothness, curvature) that are independent of input distribution.

Exact GELU vs. tanh-approximated GELU vs. sigmoid-approximated GELU. The paper uses the tanh approximation 0.5x(1 + tanh[√(2/π)(x + 0.044715x³)]) in all experiments but never compares it against the exact erf-based GELU x · 0.5(1 + erf(x/√2)). The paper also provides a sigmoid-based approximation xσ(1.702x) as a faster alternative but never evaluates it experimentally. A careful comparison on CIFAR-10/100 and MNIST autoencoding would measure: (a) whether the tanh approximation's slight deviations from the true Gaussian CDF meaningfully affect accuracy or convergence — if the exact GELU outperforms the tanh approximation, the paper's reported numbers understate the GELU's potential; (b) whether the sigmoid approximation's heavier tails (logistic vs. Gaussian decay) degrade performance enough to warrant the tanh approximation's computational cost; (c) the actual wall-clock speed difference between the three implementations on modern GPU hardware. The result would give practitioners an evidence-based recommendation: use the exact erf form if accuracy matters and erf is hardware-accelerated, use the tanh form for the best speed-accuracy tradeoff, or use the sigmoid form only if the speed gain outweighs a quantified accuracy loss.

GELU in recurrent architectures and sequence models. The paper evaluates the GELU on feedforward (MNIST), convolutional (CIFAR-10/100), and frame-based (TIMIT) architectures, but not on recurrent neural networks (RNNs, LSTMs, GRUs) or the Transformer architectures that would later make the GELU famous. The GELU's probabilistic interpretation — input-dependent stochastic masking — connects directly to zoneout (Krueger et al., 2016), an RNN regularizer that stochastically preserves hidden states. A natural experiment would replace the tanh activation in an LSTM's input, forget, output, and cell update gates with GELU, and compare against standard LSTM and zoneout-regularized LSTM on language modeling (Penn Treebank, WikiText-2) or sequence classification. If GELU-LSTMs match or exceed zoneout-LSTMs without requiring stochastic masking at training time, it would demonstrate that the GELU's expectation-based pseudoensemble mechanism generalizes from feedforward dropout to recurrent zoneout, providing a unified activation choice across architecture families. A negative result — GELU underperforming tanh in LSTMs — would reveal a boundary condition: the GELU's CDF-weighting may be poorly matched to the saturating dynamics that make tanh effective in gated recurrent architectures.

The stochastic GELU as a training algorithm. The paper mentions in passing that competitive networks can be trained with the stochastic regularizer alone (input-dependent Bernoulli masking, no deterministic activation), but provides no numbers or training curves. A systematic study would train the 8-layer MNIST network and the CIFAR-10 network using the stochastic GELU during training (sampling a fresh Bernoulli mask at each forward pass) and compare against: (a) deterministic GELU, (b) ReLU + dropout at various rates, (c) deterministic GELU + dropout. The key metrics would be final test accuracy and robustness to input noise (as in Figure 3). If stochastic-GELU training matches or exceeds deterministic GELU, it would validate the paper's unifying framework — the stochastic process is not just a conceptual motivation but a viable training algorithm in its own right — and potentially enable new training schemes that anneal from stochastic to deterministic behavior over the course of training. If stochastic-GELU underperforms, it would suggest that taking the expectation loses something that the stochastic process gains from explicit noise injection, complicating the paper's narrative of the deterministic GELU as the "expected" behavior.

Practical Applications and Downstream Use Cases

Drop-in replacement for ReLU in existing architectures. The most direct application implied by the paper is replacing ReLU activations with the GELU in any neural network that currently uses ReLU or ELU. The paper demonstrates this across six tasks with no hyperparameter changes — the same learning rate tuning range {10⁻³, 10⁻⁴, 10⁻⁵}, the same batch sizes, the same weight initialization, and the same optimizer settings work for GELU as for ReLU. A practitioner training a convolutional network for image classification can swap relu for gelu (using the tanh approximation for speed, as in the paper) and expect, based on the CIFAR-10 and CIFAR-100 results, a 0.3–1.0 percentage point improvement in test error with no additional tuning budget. The CIFAR-100 result — 20.74% GELU vs. 21.77% ReLU on a 40-layer WideResNet — is the most relevant for modern deep architectures, suggesting the largest gains come in deeper networks where the GELU's smooth curvature and non-vanishing gradients for negative inputs may compound across layers.

Improved robustness to input perturbations in production classifiers. Figure 3 demonstrates that GELU-trained MNIST classifiers are as robust as ELU-trained classifiers and substantially more robust than ReLU-trained classifiers under increasing uniform input noise, measured in both accuracy and log loss. At the maximum noise level (a = 3), the GELU achieves ~0.68 accuracy vs. ~0.62 for ReLU, and ~7.5 log loss vs. ~18 for ReLU — the log loss gap is particularly striking, indicating that the GELU's predictions remain better calibrated under distribution shift. For a production image classifier deployed on user-uploaded photos (which may have compression artifacts, unusual lighting, or sensor noise), switching from ReLU to GELU could improve robustness to these real-world perturbations without requiring adversarial training, data augmentation, or additional regularization. The log loss improvement suggests that GELU classifiers would produce less confidently wrong predictions on out-of-distribution inputs, which is valuable for safety-critical applications where detecting uncertainty matters.

Autoencoder pretraining and representation learning. The MNIST autoencoding experiment (Figure 4) shows the GELU achieving substantially lower reconstruction error than ReLU and ELU across two learning rates (10⁻³ and 10⁻⁴). At learning rate 10⁻³, the GELU's final reconstruction error is approximately 0.0040–0.0045 vs. 0.0055–0.0060 for ReLU — a relative improvement of roughly 25–30%. For self-supervised representation learning pipelines that use autoencoding as a pretraining objective (e.g., denoising autoencoders, masked autoencoders), switching the encoder and decoder activations from ReLU to GELU could yield better reconstructions and, by extension, better representations for downstream tasks. The GELU's learning rate robustness (it accommodates both 10⁻³ and 10⁻⁴ well, while ELU diverges at 0.01) also makes it a safer choice for practitioners who may not have the budget to extensively tune learning rates for their autoencoding architecture.

Small-data NLP and speech tasks where every fraction of a point matters. The Twitter POS tagging and TIMIT experiments demonstrate GELU advantages of 0.1–0.3 percentage points on tasks with limited training data (1000 tweets, 3696 TIMIT sentences). While the paper's statistical rigor is limited (no confidence intervals), the directional consistency across six tasks suggests the advantage is real, if small. For production NLP systems — part-of-speech taggers, named entity recognizers, dependency parsers — trained on modest annotated datasets, replacing ReLU with GELU in the classification head or feature extractor could yield a small but practically valuable accuracy improvement at zero implementation cost (no new hyperparameters, no architectural changes). The benefit is most relevant when the system is part of a pipeline where errors compound — a 0.1 point improvement in POS tagging accuracy could reduce cascading errors in downstream parsing or information extraction. The GELU's tangent-hyperbolic approximation is trivially implementable in any deep learning framework, making this a low-risk, low-effort change for practitioners seeking marginal gains on small-data tasks.

🎯 Pitch

1. Executive Summary

2. Context and Motivation

The Core Problem: Activation Functions Are Architectural Choices Without a Probabilistic Basis

The Historical Trajectory: From Probabilistic to Engineering-Driven

The Unresolved Gap: Where Existing Approaches Fall Short

The Mechanism: Merging Dropout, Zoneout, and ReLU into a Single Probabilistic Framework

Why the Gaussian Distribution Specifically?

How This Paper Positions Itself

A Note on the SiLU and Naming Priority

3. Technical Approach

3.1 Reader Orientation

3.2 Big-Picture Architecture (Diagram in Words)

3.3 Roadmap for the Deep Dive

3.4 Detailed, Sentence-Based Technical Breakdown

The Stochastic Process: Input-Dependent Bernoulli Masking

The Deterministic GELU: From Expectation to Closed Form

The Two Approximations: Trading Exactness for Speed

Relationship to Prior Activation Functions: What the GELU Inherits and What Is New

Design Choices: What the Paper Deliberately Does NOT Do

Practical Recommendations for Using the GELU

4. Key Insights and Innovations

Innovation 1: The Activation Function as the Expected Value of an Input-Dependent Stochastic Regularizer

Innovation 2: Weighting by Magnitude Rather Than Gating by Sign

Innovation 3: The Activation Function as a Pseudoensemble Mechanism in a Single Forward Pass

Innovation 4: Non-Monotonicity as a Deliberate Design Feature, Not a Bug

5. Experimental Analysis

Evaluation Methodology

Main Quantitative Results

MNIST Classification (Section 3.1, Figures 2 and 3)

MNIST Autoencoding (Section 3.2, Figure 4)

Twitter POS Tagging (Section 3.3)

TIMIT Frame Classification (Section 3.4, Figure 5)

CIFAR-10 Classification (Section 3.5, Figure 6)

CIFAR-100 Wide Residual Network (Section 3.5, Figure 7)

Summary of Aggregate Performance Across All Tasks

Ablation Studies and Robustness Checks

Critical Assessment

6. Limitations and Trade-offs

Limitation 1: The GELU's Probabilistic Motivation Depends on Batch Normalization, Yet This Dependency Is Never Tested in Isolation

Limitation 2: No Statistical Rigor — All Results Are Medians of 3–5 Runs Without Any Measure of Variance

Limitation 3: All Experiments Use a Single Approximate Implementation of the GELU — the True Function Is Never Evaluated

Limitation 4: The GELU's Advantage on Small-Dataset Tasks Is Too Small to Be Practically Meaningful — and Possibly Statistically Insignificant

Limitation 5: The Paper Provides No Mechanistic Understanding of Why the GELU Outperforms — Only Correlational Evidence That It Does

Limitation 6: The Experiments Use Architectures and Tuning Protocols Optimized for ReLU-Based Training, Potentially Understating GELU Gains

7. Implications and Future Directions

How This Work Changes the Landscape

Follow-Up Research This Work Enables

Practical Applications and Downstream Use Cases

Adjacent in domain

Instance Normalization: The Missing Ingredient for Fast Stylization

TensorFlow: A system for large-scale machine learning