Stronger Normalization-Free Transformers

ArXiv: 2512.10938

🎯 Pitch

A simple learned error function called Derf can completely replace normalization layers in Transformers and outperform them across vision, speech, DNA, and language tasks. Rather than fitting better, Derf works through improved generalization—showing higher training loss but better downstream accuracy everywhere it’s tested.

---

1. Executive Summary

This paper studies how to design point-wise functions that can replace and surpass normalization layers in Transformer architectures, systematically analyzing the function properties—zero-centeredness, boundedness, center sensitivity, and monotonicity—that enable stable training, and then conducting a large-scale empirical search across candidate functions on Vision Transformers (ViT-Base) and Diffusion Transformers (DiT) trained on ImageNet-1K. From this search, the authors introduce Derf (Derf(x) = γ * erf(αx + s) + β), a learnable dynamic error function that constrains extreme activations via the Gaussian CDF shape while applying independent scalar mappings to each element, as an alternative to both standard normalization and the prior Dynamic Tanh (DyT). Derf consistently outperforms LayerNorm, RMSNorm, and DyT across vision (ViT-B: 82.8% vs. 82.3% LN; DiT-L/4 FID: 43.94 vs. 45.91 LN), speech (wav2vec 2.0), DNA modeling (Caduceus: 87.3% vs. 86.9% LN), and language (GPT-2: matching LN at 2.94 validation loss), establishing that a well-designed point-wise function can surpass normalization layers entirely—with the performance gains stemming from stronger generalization rather than improved fitting capacity, as evidenced by Derf's higher training loss yet superior downstream accuracy across all tested domains.

2. Context and Motivation

The Core Problem: Normalization Layers Are Pervasive but Problematic

The fundamental issue this paper addresses is the ubiquitous yet costly dependence of modern deep learning architectures on normalization layers. Since Batch Normalization (Ioffe and Szegedy, 2015) revolutionized deep network training by stabilizing activation distributions, normalization has become an architectural requirement rather than an option in virtually all Transformer models (Vaswani et al., 2017; Dosovitskiy, 2021). Every attention block, every feed-forward block, and often the final output projection is wrapped with a LayerNorm or RMSNorm. These components are treated as indispensable—the paper cites them as having "long been viewed as indispensable components of deep learning architectures" (Abstract).

This matters for several practical and theoretical reasons that make normalization layers a genuine deployment bottleneck rather than a mere architectural inconvenience:

Memory access and synchronization overhead. Normalization layers compute activation statistics (mean and variance) across groups of activations at every forward pass. For LayerNorm specifically, each token's representation $x \in \mathbb{R}^C$ requires computing:

$\mu = \frac{1}{C} \sum_{k=1}^C x_k, \quad \sigma^2 = \frac{1}{C} \sum_{k=1}^C (x_k - \mu)^2$

These reduction operations require gathering statistics across the channel dimension, which on modern hardware (GPUs, TPUs) translates to additional memory accesses, synchronization barriers, and non-trivial latency. The paper cites Zhang and Sennrich (2019), Chen et al. (2020), and Yang et al. (2022) as documenting this overhead. In large-scale training or inference settings—particularly with large batch sizes where computation is already heavily optimized—this statistical computation becomes a measurable fraction of total runtime. The issue is architectural, not merely quantitative: normalization fundamentally requires cross-element communication in a way that point-wise operations do not.

Batch size sensitivity. Several normalization methods are highly sensitive to batch size, with inappropriate settings leading to unstable training. The paper references Wu and He (2018), Lian and Liu (2019), and Singh and Krishnan (2020) as documenting these failure modes. While LayerNorm avoids the explicit batch-dependence of BatchNorm by computing statistics per-token rather than per-batch, the issue reveals a deeper fragility: normalization methods couple the transformation applied to each activation with the statistical properties of other activations, making them vulnerable to distribution shifts in deployment and complicating scenarios like streaming inference or small-batch evaluation.

Theoretical opacity. Despite extensive study into how normalization helps—stabilizing gradient flow (Balduzzi et al., 2017; Daneshmand et al., 2020), smoothing the loss landscape (Santurkar et al., 2018; Bjorck et al., 2018), reducing sensitivity to initialization (Zhang et al., 2019; De and Smith, 2020)—there remains no single agreed-upon explanation for why normalization is necessary. The multiplicity of mechanisms (gradient stabilization, implicit learning rate tuning, sharpness reduction) suggests that normalization layers are solving several problems simultaneously, without a clear specification of which problem is primary. This theoretical opacity makes it difficult to design targeted replacements: without knowing exactly what normalization does, how do you know what a replacement must do?

The Gap: Normalization-Free Methods That Can Match, But Not Surpass, Normalization

The recognition that normalization layers impose real costs has motivated a growing body of work on normalization-free training. However, prior to this paper, all such methods shared a common ceiling: they aimed to match the performance of normalization-based architectures, with none demonstrating the ability to consistently outperform them.

The prior state of the art: Dynamic Tanh (DyT). The most directly relevant predecessor is Dynamic Tanh (Zhu et al., 2025), which introduced the idea that a simple point-wise S-shaped function can replace normalization layers. DyT applies the transformation:

$\text{DyT}(x) = \gamma \cdot \tanh(\alpha x) + \beta$

where $\alpha$ is a learnable scalar that controls the saturation point, and $\gamma, \beta$ are per-channel affine parameters (identical in form to those in LayerNorm). The key insight from Zhu et al. was empirical: they observed that LayerNorm's input-output mapping often resembles an S-shaped saturation curve in practice, with extreme activations being compressed toward zero and moderate activations passing through roughly linearly. The $\tanh$ function naturally produces this shape, and adding a learnable $\alpha$ parameter allows the model to adapt the sharpness of the saturation to each layer's needs.

DyT achieved results that were comparable to LayerNorm across multiple architectures—it did not degrade performance, which was itself a significant finding. It demonstrated that normalization is not strictly necessary for training Transformers; a suitably shaped point-wise function can maintain stability and achieve equivalent accuracy.

But DyT left a critical question unanswered. The paper frames its motivation directly against this gap:

"This work has established the foundation for point-wise functions that match the performance of normalization layers, yet functions that can surpass them remain unexplored. In this work, we aim to discover point-wise functions that outperform normalization layers to push toward stronger Transformer architectures." (Section 1)

DyT proved that matching is possible, but it did not investigate whether matching is the ceiling. The design space of point-wise functions is vast—even restricting to S-shaped, zero-centered functions, there are infinitely many candidates (the CDF of any symmetric distribution, rational approximations, trigonometric variants, etc.). DyT selected $\tanh$ based on conceptual similarity to LayerNorm's observed behavior, not through systematic exploration of alternatives. The paper explicitly identifies this as the unexplored territory:

"a comprehensive analysis of the design space for these statistics-free operators remains missing" (Section 2, Point-wise functions).

Where prior normalization-free methods fall short. The paper situates itself within a broader landscape of normalization-free approaches, each with distinct limitations:

• Parameter and optimization-level methods use tailored initialization schemes (Zhang et al., 2019; De and Smith, 2020; Bachlechner et al., 2021), self-normalizing activations (Klambauer et al., 2017), weight normalization (Salimans and Kingma, 2016; Brock et al., 2021a), or adaptive gradient clipping (Brock et al., 2021b) to maintain stable gradients without normalization. These methods modify the optimization procedure rather than the architecture, meaning they may require different hyperparameters for each model or task and do not provide a drop-in replacement for the normalization layer itself.

• Architectural modification methods restructure the model through simplifications (He and Hofmann, 2024), Softmax-only formulations (Jha and Reagen, 2024), or bounded convolutional operators (Liu et al., 2017, 2018). These approaches require redesigning the architecture rather than substituting a single component, limiting their applicability to existing model families.

• Point-wise function methods, most notably DyT (Zhu et al., 2025), provide true drop-in replacements—you remove the LayerNorm and insert the point-wise function—but prior work had demonstrated only performance parity, not superiority. Stollenwerk (2025) provided theoretical analysis revealing mathematical similarities between normalization operations and dynamic activation functions, but this work was analytical rather than empirical and did not propose new function designs.

The unifying limitation across all these approaches is that they are defensive: they ask "how can we remove normalization without hurting performance?" rather than "can removing normalization improve performance?" This paper is the first to systematically pursue the latter question.

How This Paper Positions Itself

The paper positions itself not as proposing yet another normalization-free method, but as systematically exploring the design space of point-wise functions to find those that can actually outperform normalization. This is a qualitative shift in ambition—from replacement to improvement.

The approach proceeds in two phases, each addressing a distinct gap in prior work:

Phase 1: Understanding why point-wise functions work (Section 3). While DyT demonstrated that a specific point-wise function ($\tanh$) could replace normalization, it did not analyze which properties of that function were responsible for its success. Was the S-shape essential? The boundedness? The zero-centeredness? Without understanding the functional requirements for a normalization replacement, designing better alternatives is guesswork. The paper's property analysis (zero-centeredness, boundedness, center sensitivity, monotonicity) is the first systematic study of which function properties are necessary or beneficial for training Transformers without normalization. This analysis serves both as diagnostic (explaining why DyT works) and as prescriptive (defining the feasible design space for new candidates).

Phase 2: Searching within the feasible design space for superior functions (Section 4). Armed with the property constraints, the paper constructs a large candidate set of functions that satisfy all identified requirements and empirically evaluates them across multiple architectures (ViT, DiT). This search is not random—it is guided by the property analysis—and it produces a clear winner: $\text{erf}(x)$, the error function (rescaled Gaussian CDF), augmented with learnable parameters as Derf. The search demonstrates that seemingly similar S-shaped functions (Table 7) can produce meaningfully different performance, and that the choice of function matters beyond simply satisfying the basic property constraints.

The key conceptual leap. The paper's deepest insight is that the performance gains from Derf over normalization come from better generalization, not better fitting (Section 6.1). This is demonstrated through a clever experimental design: measuring training loss in evaluation mode (with stochastic regularization disabled) and showing that Derf exhibits higher training loss than LayerNorm while achieving lower validation error. This finding reframes the normalization-free question completely:

• Prior work implicitly assumed that normalization layers are optimal for optimization (they minimize training loss) and that matching them requires matching their optimization benefits.
• This paper shows that point-wise functions may be worse optimizers (higher training loss) but better regularizers (lower validation error). The reduced adaptability of point-wise functions—they apply the same transformation regardless of activation statistics after training—acts as an implicit regularizer that prevents overfitting to training statistics.

This regularization hypothesis (stated explicitly in Section 6.1 Discussion) provides a positive rationale for removing normalization, not just a cost-based one. It suggests that normalization layers may be overfitting to training-time activation statistics in ways that hurt generalization, and that replacing them with fixed-function transformations can improve test-time performance by reducing this form of overfitting.

Comparison to the broader normalization literature. The paper's empirical scope is intentionally broad, spanning classification (ViT), generation (DiT), self-supervised representation learning (wav2vec 2.0), and sequence modeling (HyenaDNA, Caduceus, GPT-2). This multi-domain evaluation distinguishes the work from prior normalization-free methods that typically evaluated on one or two tasks. The consistent gains across vision, speech, DNA, and language—each with fundamentally different data modalities and training paradigms—strengthen the claim that the benefit of Derf is architectural rather than task-specific.

What the paper is not claiming. It is important to note what this work does not assert: it does not claim that Derf is universally optimal (the search is empirical and limited to the candidate functions considered), it does not provide theoretical guarantees about why $\text{erf}$ outperforms $\tanh$ (the approximation experiment in Section 7.2 shows that simply scaling $\tanh$ to match $\text{erf}$'s shape is insufficient, but the underlying mechanism remains empirical), and it does not address the cost of difficulty estimation or adaptive allocation (the focus is purely on architectural substitution). The contribution is a systematic empirical demonstration that the choice of point-wise function matters, that properties can be identified to constrain the search, and that a specific choice ($\text{erf}$) consistently outperforms both normalization and the prior point-wise state-of-the-art.

Connecting to the present. The paper's motivation is particularly timely given the dominance of Transformer architectures in foundation models across modalities. As models scale to hundreds of billions of parameters, the computational overhead of normalization—however small per-layer—accumulates across thousands of layers and millions of training steps. A drop-in replacement that is both simpler (no cross-element communication, trivial implementation) and more performant offers immediate practical value for large-scale training pipelines, while the regularization insight suggests that removing normalization may have benefits that compound at larger scales where overfitting to training statistics becomes a greater concern.

3. Technical Approach

3.1 Reader Orientation

This is a systematic design-space exploration paper that studies how to replace the statistical normalization layers found in every modern Transformer block with a much simpler operation — a fixed mathematical function applied independently to each activation value — and, crucially, discovers that the right function can not only replace normalization but outperform it. The system is not a trained model or a complex pipeline, but a search methodology for finding the optimal point-wise function shape, culminating in the introduction of Derf (Dynamic erf), a learnable transformation based on the Gaussian error function that achieves superior performance across vision, generation, speech, DNA modeling, and language tasks.

The problem this solves is that normalization layers — specifically LayerNorm and RMSNorm — impose computational overhead (gathering activation statistics across channels requires cross-element communication and synchronization) and may overfit to training-time activation statistics, hurting generalization. The solution's shape is deceptively simple: replace every normalization layer with a single parametric function applied independently to each activation value, where the function is chosen from a carefully constrained design space and validated through exhaustive empirical search across architectures and domains.

3.2 Big-Picture Architecture (Diagram in Words)

The technical contribution has three major components that build on each other:

1. Property Analysis Framework (Section 3) — A controlled experimental methodology for isolating how individual function properties (zero-centeredness, boundedness, center sensitivity, monotonicity) affect training stability and final performance. This component takes candidate point-wise functions as input, systematically modifies them along each property dimension, trains ViT-Base models to convergence, and outputs a set of necessary conditions for viable normalization replacements.

2. Candidate Function Construction and Search (Section 4) — A systematic pipeline for generating point-wise function candidates that satisfy all identified properties, including natural functions (erf, tanh, arctan), transformed basic functions (sin(x) clipped to [−1,1], exponential variants, rational forms), clipped unbounded functions (arcsinh clipped to [−1,1]), and canonical ratio functions. Each candidate is instantiated in the unified parametric form y = γ * f(αx + s) + β, evaluated on ViT-Base (classification accuracy) and DiT (FID), and ranked by performance.

3. Derf Instantiation and Multi-Domain Validation (Sections 5–6) — The winning candidate erf(αx + s) is formalized as a drop-in layer, integrated into Transformers across five domains (ViT, DiT, wav2vec 2.0, HyenaDNA/Caduceus, GPT-2), and compared against LayerNorm, RMSNorm, and DyT baselines. A separate fitting-vs-generalization analysis computes evaluation-mode training loss to determine whether performance gains come from better optimization or better regularization.

Information flows linearly: property analysis → candidate construction → search → winner instantiation → multi-domain validation → generalization analysis. Each phase constrains the next: properties define the feasible design space, the search finds the optimal design within that space, validation confirms domain independence, and the generalization experiment explains why the optimal design works.

3.3 Roadmap for the Deep Dive

• First, the unified parametric form for point-wise normalization replacements: the equation y = γ * f(αx + s) + β that all candidate functions share, including what each parameter does and why this form subsumes both DyT and traditional normalization.

• Second, the property analysis methodology: how each of the four properties (zero-centeredness, boundedness, center sensitivity, monotonicity) is independently tested through controlled function modifications, the specific experimental setups for each property (ViT-Base on ImageNet-1K), and the convergence/failure patterns observed.

• Third, the search space construction: how candidate functions are generated from four function families (natural, transformed basic, clipped unbounded, canonical ratio), the transformations applied to satisfy property constraints, and the evaluation protocol (ViT top-1 accuracy, DiT FID).

• Fourth, the Derf formulation and initialization: the specific mathematical form Derf(x) = γ * erf(αx + s) + β, parameter initialization values, and ablation of the shift parameter s.

• Fifth, the multi-domain evaluation protocol: the architecture-specific integration details for ViT, DiT, wav2vec 2.0, HyenaDNA/Caduceus, and GPT-2, including any domain-specific adjustments (e.g., DiT's zero-initialization removal, GPT-2's per-layer α initialization).

• Sixth, the fitting-vs-generalization analysis: the evaluation-mode training loss measurement procedure, and what the pattern Norm < Derf < DyT in training loss implies about the mechanisms driving Derf's performance advantage.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systematic design-space exploration paper whose core idea is that the choice of point-wise function matters materially for performance — seemingly similar S-shaped curves produce measurably different results — and that by constraining the design space with property requirements and exhaustively searching within it, one can find a function (erf) that consistently outperforms both normalization layers and the prior state-of-the-art point-wise alternative (tanh / DyT).

---

The Unified Parametric Form for Point-Wise Normalization Replacements

All point-wise functions considered in this paper are instantiated through a common parametric template, described in Section 3:

$y = \gamma \cdot f(\alpha x + s) + \beta$

where $f(\cdot)$ is a chosen base function (e.g., tanh, erf, arctan, etc.), $\alpha$ is a learnable input scaling parameter that controls how sharply the function transitions through its active region, $s$ is a learnable shift parameter that offsets the input horizontally, $\gamma$ is a learnable per-channel output scaling vector (analogous to the scaling weight in LayerNorm), and $\beta$ is a learnable per-channel output bias vector (analogous to the bias term in LayerNorm). The input $x$ is a single activation value from the tensor; the transformation is applied independently to each element, meaning there is no cross-element communication whatsoever.

What this form computes: Given an input activation $x$ (a scalar from anywhere in the network), the transformation first shifts and scales it to $\alpha x + s$, then passes it through the chosen base function $f$ (which is fixed, non-learnable, and shared across all positions and channels), and finally applies a learned per-channel affine transformation $\gamma$ (multiplicative) and $\beta$ (additive). The output is a scalar of the same shape. Operationally, if the input is a tensor of shape [batch, tokens, channels], each of the batch × tokens × channels scalar elements passes through exactly the same point-wise transformation sequence, with $\alpha$ and $s$ shared across all elements and $\gamma$, $\beta$ varying only across the channel axis.

Why this form: The design makes explicit what is flexible and what is fixed. The base function $f$ defines the qualitative shape (S-shaped saturating, linear near zero, bounded extremes) that determines the function's stabilization properties during training — this is the object of the property analysis and the search. The learnable parameters $\alpha$ and $s$ allow each layer to adapt the operating range of this fixed shape — how wide the linear region is, whether it is centered on zero, how quickly saturation kicks in — without changing the fundamental functional form. The affine parameters $\gamma$ and $\beta$ exactly mirror the output affine transformation in LayerNorm, providing the model the same per-channel rescaling and bias capability it already expects. This separation of concerns — fixed shape for stability, learnable scalar parameters for per-layer adaptation, learned vectors for per-channel representation — allows the core search question to focus purely on $f$. Alternative designs that baked adaptation into $f$ itself (e.g., making $f$ a neural network) would conflate shape selection with parameter learning and would not serve as a drop-in replacement for existing normalization layers.

The paper introduces the shift parameter $s$ as a novel addition beyond the DyT formulation (which used $\gamma \cdot \tanh(\alpha x) + \beta$ without $s$). An ablation in Section 7.1 (Table 14) demonstrates that $s$ provides consistent but function-dependent improvements — erf benefits more from the shift than tanh does — and that the gains from Derf over DyT are not solely attributable to this additional parameter. Table 15 further shows that making $s$ a per-channel vector rather than a scalar yields negligible performance difference, so the scalar form is adopted for efficiency.

---

Property Analysis Methodology: How Each Function Property Affects Training

The property analysis in Section 3 is the diagnostic cornerstone of the paper. Rather than speculating about what makes a point-wise function work, the authors design controlled experiments where a single property is varied while all others are held constant, training ViT-Base models on ImageNet-1K to convergence and measuring top-1 accuracy. The goal is to identify the necessary conditions for a function to serve as a viable normalization replacement, which then constrains the search space in Section 4.

Experimental framework. All property experiments share the same backbone: a ViT-Base architecture where every LayerNorm is replaced identically with the parametric form $y = \gamma \cdot f(\alpha x) + \beta$ (the shift parameter $s$ is not used in the property analysis; it is introduced later). The base function $f$ is modified to systematically vary the property under study. Three base functions are used across most experiments — tanh(x), erf(x), and arctan(x) — to ensure observed patterns are not idiosyncratic to a particular functional form. Training follows the standard ViT configuration from Table 17: AdamW optimizer, base learning rate 4e-3, weight decay 0.05, cosine decay schedule with 20 warmup epochs over 300 total epochs, effective batch size 4096, with standard ImageNet augmentations (rand-m9-mstd0.5-inc1, mixup 0.8, cutmix 1.0, random erase 0.25, label smoothing 0.1). Results are reported as top-1 accuracy on the ImageNet-1K validation set.

---

Zero-Centeredness

Definition. A function is zero-centered if its output values are balanced around zero, such that positive and negative activations of similar magnitude produce outputs of similar magnitude and opposite sign. Formally, a function $f$ is zero-centered if $f(-x) = -f(x)$ (odd symmetry), though the experiments test both strict and approximate centering through parameterized perturbations.

Why it matters for training. Normalization layers inherently recenter activations — LayerNorm subtracts the mean, producing outputs with expected value approximately zero. This centering stabilizes gradient flow by preventing systematic biases in activation distributions from accumulating across layers. If a point-wise function is not centered, it introduces a constant bias that shifts the mean activation at each layer, which can compound multiplicatively through deep networks and potentially cause training instability (large gradients, saturation, or divergence).

Experimental setup (Section 3.1, Table 1). For each base function (erf, tanh, arctan), two types of shifts are applied:

Horizontal shift: $f_{\text{horiz}}(x) = f(x + \lambda_{\text{horiz}})$, where $\lambda_{\text{horiz}}$ controls how far the function is shifted along the input axis. A positive $\lambda_{\text{horiz}}$ means the function's center moves to the left (inputs need to be more negative to hit the active region), while a negative shift moves it to the right. This tests whether the function's operating region must be centered on input zero.

Vertical shift: $f_{\text{vert}}(x) = f(x) + \lambda_{\text{vert}}$, where $\lambda_{\text{vert}}$ adds a constant offset to all outputs. A positive $\lambda_{\text{vert}}$ biases outputs upward, breaking symmetry even if the base function is odd. This tests whether the function's output range must be symmetric around zero.

For both types, $\lambda$ is varied over the values {±0.5, ±1, ±2} across all three base functions, with $\lambda = 0$ (the unmodified zero-centered function) serving as baseline.

Results and interpretation (Table 1). The patterns are consistent across all three base functions:

• Horizontal shifts are moderately tolerated: with |$\lambda_{\text{horiz}}$| ≤ 0.5, performance remains close to baseline (erf: 82.6% at $\lambda$=0 vs. 82.5% at $\lambda$=±0.5; tanh: 82.5% vs. 82.4–82.5%). At |$\lambda_{\text{horiz}}$| = 1.0, a small degradation appears (erf: 82.1% at +1, 82.0% at -1), and at |$\lambda_{\text{horiz}}$| ≥ 2, training diverges entirely for all functions. The takeaway: the operating region must be approximately centered on input zero, but small misalignments (up to ~0.5) are forgivable.

• Vertical shifts are more damaging: even at |$\lambda_{\text{vert}}$| = 0.5, performance drops visibly (erf: 82.6% → 82.3%), and the degradation worsens monotonically with increasing |$\lambda_{\text{vert}}$|. At |$\lambda_{\text{vert}}$| ≥ 2, training fails for all functions. The takeaway: output symmetry matters — a constant bias in activation space is more harmful than a shifted operating range, likely because it systematically biases downstream layer inputs.

A subtle observation: horizontal shifts show asymmetry effects — the impact is not always perfectly symmetric around λ=0. For example, erf at $\lambda_{\text{horiz}}$ = -1 achieves 82.5% while at +1 it achieves 82.1%, suggesting that the specific direction of shift interacts with the function's shape and the optimizer's trajectory. Despite these asymmetries, the overall message is clear: zero-centeredness is a requirement for stable convergence. Candidate functions that violate this property (e.g., shifted sigmoids, functions with nonzero asymptotic values on both tails) are excluded from the search space.

---

Boundedness

Definition. A function is bounded if its output is constrained within a finite range for all possible inputs. Formally, there exist constants $a, b \in \mathbb{R}$ such that $a \leq f(x) \leq b$ for all $x$ in the domain. For the paper's purposes, functions are typically scaled to the range [-1, 1]. An unbounded function, by contrast, can produce arbitrarily large outputs as inputs grow.

Why it matters for training. The Deep Learning training lore is rich with stories of exploding activations — unbounded transformation functions (like ReLU without normalization) allow activation variances to grow without limit as signals propagate through successive layers, leading to exploding gradients and numerical overflow. Normalization layers directly counteract this: by dividing by the standard deviation, they keep activation scales bounded regardless of how large pre-normalization values become. A point-wise function that is bounded similarly prevents unbounded growth: no matter how large or small the input, the output stays within [a, b], providing a hard ceiling on per-layer activation magnitude.

Experimental setup — Method 1: Clamping unbounded functions (Section 3.2, Table 2). Three inherently unbounded S-shaped functions are selected: arcsinh(x) (inverse hyperbolic sine, which grows like ln(x) asymptotically), logsign(x) = sign(x) * ln(|x| + 1), and linear(x) = x. Each is evaluated in its original unbounded form and in a clipped version:

$y = \text{clip}(f_u(x), -\lambda_u, \lambda_u)$

where $\lambda_u$ is the clipping threshold, varied over {0.5, 0.8, 1.0, 2.0, 3.0, 5.0}. The clipping operation simply truncates outputs to [−λ_u, λ_u], creating an enforced bound.

Experimental setup — Method 2: Linearly interpolating toward unboundedness (Section 3.2, Table 3). For functions that are naturally bounded, the authors create a continuous family that gradually becomes unbounded:

$y = (1 - \lambda_b) f_b(x) + \lambda_b x, \quad \lambda_b \in (0, 1)$

When $\lambda_b = 0$, the function is purely bounded; when $\lambda_b = 1$, it becomes the identity (linear, unbounded). Intermediate values blend the two behaviors. The interpolation parameter $\lambda_b$ is varied over {0.01, 0.1, 0.5} for four bounded base functions: erf, tanh, arctan, and isru (the integral of the sigmoid, another bounded S-shape).

Results and interpretation. For Method 1 (Table 2):

• Of the three unbounded functions, only arcsinh(x) and logsign(x) converge effectively (82.2% each); linear(x) fails entirely. This establishes that some unbounded functions can train — boundedness is not an absolute requirement — but the linear case shows that unboundedness without any saturation mechanism is fatal.
• For functions that converge, clipping consistently improves performance across all λ_u values. For arcsinh(x), the clipped versions achieve 82.3–82.4% versus 82.2% for the unbounded case. For logsign(x), clipped versions achieve 82.3–82.4% versus 82.2% unbounded. The improvement is small but consistent, and intriguingly, the optimal clipping threshold is not at the extremes — moderate values (~1.0–3.0) perform best, while very tight clipping (0.5) or very loose clipping (5.0) yields slightly lower accuracy.

For Method 2 (Table 3):

• As $\lambda_b$ increases (more linear, less bounded), performance degrades monotonically for all four functions. At $\lambda_b = 0.01$ (0.1% linear), performance already drops slightly (erf: 82.6% → 82.4%; tanh: 82.5% → 82.4%). At $\lambda_b = 0.1$ (10% linear), the drop is clearer (erf: 82.3%; tanh: 82.3%). At $\lambda_b = 0.5$ (50% linear), training diverges for all functions. This shows that even a small unbounded component degrades performance, and a majority-linear function catastrophically fails.

Growth rate limitation (Section 3.2, "Limitation of growth rate," Table 4, Figure 3). The paper makes a further, more nuanced observation: among unbounded functions that do converge, there is an upper limit on their growth rate — how quickly outputs grow with input magnitude. The authors evaluate five unbounded functions with increasing growth rates on the positive half-axis:

• logsign(x) — slowest growth (logarithmic), converges at 82.2%
• arcsinh(x) — moderate growth (logarithmic-like), converges at 82.2%
• logquad(x) = sign(x) * ln(x² + 1) — faster growth, converges at 82.1%
• power23(x) = sign(x) * x^{2/3} — faster still, fails to converge
• linear(x) — linear growth, fails to converge

The fastest function that still allows training is logquad(x), which grows as ln(x² + 1) ~ 2ln(|x|) asymptotically. Any function with polynomial or faster growth (sublinear power x^{2/3}, linear) causes optimization divergence early in training. The paper attributes this to variance amplification: rapidly growing functions fail to suppress the variance of large activations, producing large gradient norms that destabilize the optimizer.

Design implication. Boundedness is not strictly necessary — some unbounded functions work — but it is practically essential because it provides a margin of safety. Functions that are naturally bounded remove the need to worry about growth rates and clipping thresholds, simplifying the search space. All candidate functions in Section 4 are bounded (usually to [-1, 1]), either naturally or through explicit clipping.

---

Center Sensitivity

Definition. Center sensitivity characterizes how responsive a point-wise function is to small input variations near zero. A function with high center sensitivity has a steep slope near the origin — small changes in input produce clear changes in output. A function with low center sensitivity has a flat region around zero (or near-zero), where inputs must be large to produce any meaningful output. The paper operationalizes this property by introducing a controllable flat region around the origin.

Why it matters for training. Most activations in a well-trained network concentrate near zero, particularly after normalization. If the point-wise function has a flat region around zero, it essentially blocks signal propagation for the majority of activations — small perturbations in the input produce negligible output changes, making gradient flow through that layer effectively zero. This is analogous to the "saturation" problem in sigmoid and tanh activations, but here the flatness is at the functional design level rather than an emergent property of saturated inputs.

Experimental setup (Section 3.3, Table 5). The authors modify each base function to include a symmetric flat region around the origin of width $\lambda$, where $\lambda$ is the center sensitivity scale:

• For inputs $x \in [-\lambda, \lambda]$, the function output is set to zero.
• For $|x| > \lambda$, the positive and negative parts of the function are shifted outward to maintain continuity at the boundaries $x = \pm\lambda$.

This creates a function with a "dead zone" near zero. A smaller $\lambda$ means a narrower dead zone (higher center sensitivity — the function becomes responsive sooner), while a larger $\lambda$ means a wider dead zone (lower center sensitivity — inputs must be larger to produce any output). The parameter $\lambda$ is varied over {0, 0.1, 0.5, 1.0, 2.0, 3.0} across three base functions: erf, tanh, and arctan. $\lambda = 0$ corresponds to the original function with no dead zone (maximal center sensitivity).

Results and interpretation (Table 5).

• The best performance is always achieved at $\lambda = 0$ (no dead zone). For erf: 82.6% at λ=0 vs. 82.5% at λ=0.1, 82.1% at λ=1.0, 81.3% at λ=2.0.
• The degradation is nonlinear: for $\lambda \leq 0.5$, the drop is small (82.6% → 82.5% for erf), suggesting the optimizer can partially compensate for a small dead zone by learning larger $\alpha$ values that effectively stretch the active region. But once $\lambda \geq 1.0$, the degradation accelerates sharply, and at $\lambda \geq 3.0$, training diverges.
• This pattern is consistent across all three base functions, with erf showing the most robustness (smallest relative drop at moderate λ) and arctan the most sensitivity.

The training loss curves in Appendix A.2 (Figure 7) confirm this interpretation: as $\lambda$ increases, the training loss (in evaluation mode) rises monotonically, indicating that the flat zone directly limits the model's fitting capacity — it is not just a regularization effect but a genuine constraint on what the model can represent.

Design implication. Center sensitivity is a strict requirement: the function must have nonzero slope near the origin to propagate signals and gradients for the majority of activations. Functions that are locally flat at zero (e.g., 1 - cos(x) or any function with a zero derivative at the origin) are excluded from the candidate set. This property is closely related to the "S-shaped" requirement implicit in prior work (the S-shape ensures steepness near zero and saturation at extremes), but the paper formalizes it as an independently testable property.

---

Monotonicity

Definition. A function is monotonic if it preserves the relative ordering of inputs — larger inputs always produce larger outputs (monotonically increasing) or smaller outputs (monotonically decreasing). A non-monotonic function has at least one region where the derivative changes sign, meaning the output can increase and then decrease (or vice versa) over the input range.

Why it matters for training. A non-monotonic function distorts the relative ordering of activations: two inputs with $x_1 < x_2$ could produce outputs $f(x_1) > f(x_2)$, inverting their ranking. Moreover, since a non-monotonic function necessarily has regions where its derivative changes sign, the backpropagated gradient signal can flip direction — an input that should produce a positive gradient instead gets a negative one (or vice versa), creating conflicting optimization signals. This is particularly problematic for attention mechanisms, where relative activation magnitudes encode importance weights.

Experimental setup (Section 3.4, Table 6, Figure 4). The authors test four categories of monotonicity behavior:

1. Monotonically increasing functions: the standard forms of erf, tanh, and arctan, which all preserve input ordering.
2. Monotonically decreasing functions: the negated versions, $f_{\text{neg}}(x) = -f(x)$, which flip the ordering but preserve it consistently.
3. Hump-shaped functions: functions that rise to a peak and then fall, specifically dampx(x) = 2x/(1+x²) (like a derivative of a sigmoid) and dampexp(x) = 2.72x * exp(-|x|) (a product of linear and exponential decay), both scaled to fit [-1, 1].
4. Oscillatory functions: periodic functions, specifically sin(x), which cyclically increase and decrease.

To isolate the effect of monotonicity, all functions are rescaled so that their output range matches the monotonic baselines ([-1, 1]), and they share the property of being zero-centered, bounded, and center-sensitive. This makes monotonicity the only varying property.

Results and interpretation (Table 6).

• Monotonically decreasing functions perform almost identically to increasing ones. For erf: 82.6% increasing vs. 82.5% decreasing; for tanh: 82.5% vs. 82.5%; for arctan: 82.3% vs. 82.2%. The negligible gap shows that the direction of monotonicity is irrelevant — what matters is that the ordering is consistent across the entire input range.
• Non-monotonic functions consistently underperform. The sinusoidal function achieves 81.6% — a significant drop from the monotonic baselines. The hump-shaped functions fare worse: dampx(x) at 80.7% and dampexp(x) at 81.2%.
• Training loss curves (Appendix A.3, Figure 8) show that non-monotonic functions exhibit higher training loss throughout optimization, indicating reduced fitting capacity — not just poorer generalization.

The paper notes an interesting negative result: the monotonically decreasing versions perform so similarly to the increasing ones that it confirms the key requirement is consistent ordering, not the sign of the ordering. This means candidate functions can include mirrored versions without penalty, expanding the design space.

Design implication. Monotonicity (either increasing or decreasing) is a requirement. Functions with humps, oscillations, or any non-monotonic behavior are excluded from the candidate set. Combined with the previous three properties, the design space is now constrained to functions that are approximately zero-centered, bounded, center-sensitive (nonzero slope near zero), and monotonic — essentially, smooth S-shaped curves that saturate at extremes.

---

Candidate Function Construction and Search

The design space defined by the four properties. After establishing that effective normalization replacements must be zero-centered, bounded, center-sensitive, and monotonic, the paper constructs a candidate set of functions that satisfy these constraints (Section 4, Table 7, Figure 5). The functions are drawn from four families:

1. Natural functions (Figure 9): Functions that innately satisfy all four properties without modification. These include erf(x) (the rescaled Gaussian CDF), tanh(x) (the hyperbolic tangent, rescaled sigmoid), and arctan(x) (scaled so the output range matches [-1, 1]). All three are S-shaped, odd-symmetric, and bounded. They differ primarily in their tails — how sharply they approach the asymptotic bounds — and their center slopes: erf has a slightly broader linear region near zero than tanh, while arctan has the gentlest approach to saturation.

2. Transformed basic functions (Figure 10): Functions constructed by taking simple primitives — power functions, exponentials, polynomials — and applying transformations (translation, scaling, rotation, mirroring) to satisfy all four properties. Examples include:

• satursin(x): The standard sin(x) function clipped to the range [−π/2, π/2] before applying sin, producing an S-shape that saturates at ±1.
• expsign(x) = -sign(x) * (exp(-|x|) - 1): Built from exponential decay, mirrored to be odd, producing outputs that approach ±1 as inputs grow.
• isru(x) = x/(√(x² + 1)): A rational approximation to the sigmoid shape, derived from the integrated sigmoid, which smoothly transitions from linear near zero to saturating at ±1.
• cubsign(x) = x³/(|x|³ + 1): A cubic rational that produces an S-shape.

The transformations applied include sign-flipping to ensure odd symmetry, scaling to bound the range to [-1, 1], and clipping to enforce hard saturation boundaries.

3. Clipped unbounded functions (Figure 11): Functions that innately satisfy zero-centeredness, center sensitivity, and monotonicity, but are unbounded in their original form. To satisfy the boundedness requirement, their outputs are explicitly clipped to [-1, 1]. Examples include:

• arcsinhclip(x) = clip(arcsinh(x), -1, 1): The inverse hyperbolic sine, clipped.
• logsignclip(x) = clip(sign(x) * ln(|x| + 1), -1, 1): A logarithmic growth function, clipped.
• logquadclip(x) = clip(sign(x) * ln(x² + 1), -1, 1): The fastest-growing function that still converges (from the growth-rate analysis), clipped.
• power23clip(x) = clip(sign(x) * x^{2/3}, -1, 1): A sublinear power function, clipped. (Note: this function actually failed to converge in the property analysis, but is included here in clipped form to test if boundedness rescues it.)
• linearclip(x) = clip(x, -1, 1): The identity function, clipped to [-1, 1] — the simplest possible bounded, monotonic, zero-centered, center-sensitive function.

For logsign, logquad, and power23, the negative branch is constructed by mirroring the positive branch around the origin to ensure odd symmetry (since the original forms are defined only for positive inputs or have non-monotonic behavior).

4. Canonical ratio functions (Figure 12): Functions constructed using the canonical ratio form f(x)/(|f(x)| + 1), which naturally enforces boundedness and monotonicity for any monotonically increasing base f(x). By choosing f(x) to be an odd, zero-centered base function, the resulting ratio automatically satisfies all four properties. Examples include:

• smoothsign(x) = x/(1+|x|): Uses f(x) = x, producing an S-shape similar to a sigmoid but with fatter tails.
• saturlog(x) = sign(x) * ln(|x|+1) / (ln(|x|+1) + 1): Uses f(x) = sign(x) * ln(|x|+1), wrapped in the ratio form.

Quantitative evaluation protocol (Section 4). Each candidate function is instantiated in the unified form:

$y = \gamma \cdot f(\alpha x + s) + \beta$

and evaluated on two architectures:

1. ViT-Base trained on ImageNet-1K with the standard 300-epoch training recipe from Table 17. Performance is measured as top-1 validation accuracy.
2. DiT-B/4 and DiT-L/4 trained on ImageNet-1K for class-conditional image generation with the configuration from Table 18. The authors found that the default DiT learning rate was suboptimal, so they sweep three values {1e-4, 2e-4, 4e-4} for all methods (including LayerNorm baselines) and report the best result. Additionally, they observe that DiT's zero initialization negatively affects point-wise function models, so they remove it for those variants while retaining it for LayerNorm. Performance is measured as FID (Fréchet Inception Distance) using the standard ImageNet reference batch, where lower is better.

Search results (Table 7). Even though the candidate functions appear highly similar in shape (all are S-shaped, bounded to [-1, 1], zero-centered, monotonic — see Figure 5), their empirical performance shows meaningful differences:

• Best performer: erf(x) achieves 82.8% top-1 on ViT-B and 63.23 FID on DiT-B/4 — the best numbers in both columns.
• tanh(x) achieves 82.6% top-1 and 63.71 FID — close but consistently below erf.
• satursin(x) ties tanh on ViT (82.6%) and achieves 63.90 FID on DiT — a strong showing for a non-standard function.
• isru(x) achieves only 82.3% top-1 and 65.72 FID — notably worse despite similar shape.
• cubsign(x) is the worst performer at 81.4% top-1 and 70.22 FID.
• The clipped linear function (linearclip) achieves 82.3% top-1 — remarkably good for such a simple function — but 66.08 FID, far from the best.
• There is a clear rank correlation between ViT accuracy and DiT FID, suggesting the function choice matters for both discriminative and generative tasks in similar ways.

The spread from best to worst (~1.4% top-1, ~7 FID points) demonstrates that the specific function shape matters beyond mere property satisfaction. All these functions satisfy the four necessary properties, yet performance varies substantially. This justifies the search approach and identifies erf as the standout candidate.

---

Derf Formulation and Parameter Initialization

Formulation (Section 5). From the search, erf(x) emerges as the single best-performing base function. The full Derf layer is defined as:

$\text{Derf}(x) = \gamma \cdot \text{erf}(\alpha x + s) + \beta$

where $\text{erf}(x) = \frac{2}{\sqrt{\pi}} \int_0^x e^{-t^2} dt$ is the error function — the rescaled cumulative distribution function (CDF) of a standard Gaussian distribution, mapping real numbers to $(-1, 1)$. The learnable parameters are: $\alpha$ (scalar) controlling the steepness of the transition through the active region, $s$ (scalar) controlling horizontal offset, $\gamma$ (per-channel vector) controlling output scaling, and $\beta$ (per-channel vector) controlling output bias.

What it computes. Given an input tensor $x$ of arbitrary shape, each element $x_i$ is first shifted and scaled to $\alpha x_i + s$, then passed through the erf function which compresses it into the range $(-1, 1)$ with a shape that is approximately linear near zero (where most activations concentrate) and saturating at extremes. The per-channel affine parameters then rescale and shift this compressed value. The result is an element-wise transformation that constrains extreme activations (via erf's saturation) while preserving linear behavior for moderate values, exactly mimicking the functional role of normalization's re-centering and re-scaling but without computing any statistics.

Why this form outperforms tanh. The paper does not offer a theoretical explanation for why erf beats tanh, but the approximation experiment in Section 7.2 provides an empirical hint. The authors fit a scaled tanh to match erf's shape by minimizing the L1 norm:

$\min_{\varepsilon} \int_{-\infty}^{+\infty} |\tanh(\varepsilon x) - \text{erf}(x)| dx$

The optimal scaling factor is $\varepsilon \approx 1.205$. When this optimized tanh(1.205x) is evaluated (Table 16), it achieves comparable or slightly improved performance over the standard tanh(x) but still falls short of erf(x). For example, on ViT-B: tanh(x) gets 82.6%, tanh(1.205x) gets 82.7%, erf(x) gets 82.8%. On DiT-L: tanh(x) gets 45.48 FID, tanh(1.205x) gets 45.13 FID, erf(x) gets 43.94 FID. The gap persists even with scaling, indicating that the difference is not just about the slope or width of the function — erf has a qualitative shape difference that scaling alone cannot replicate. The paper leaves open the question of what specific geometric property causes the advantage.

Parameter initialization (Section 5). The initialization scheme is designed to make Derf behave like LayerNorm at the start of training:

• $\gamma$ is initialized to an all-one vector (matching LayerNorm's default scaling).
• $\beta$ is initialized to an all-zero vector (matching LayerNorm's default bias).
• $\alpha$ is initialized to 0.5 (not specified why this value; it likely provides a moderate slope that avoids both extreme saturation and near-linearity at initialization).
• $s$ is initialized to 0 (no horizontal shift at initialization — the function is initially centered).

These initialization values are used across all experiments unless otherwise specified (Section 5, "Parameter initialization"). For the GPT-2 language model specifically, the paper found that the $\alpha$ initialization needed per-position tuning: they tried different $\alpha$ initialization values for the point-wise layer following the attention block versus other layers, sweeping combinations and reporting the best (Table 21).

Structural integration. To use Derf in any Transformer architecture, each normalization layer is replaced one-to-one with a Derf layer — the pre-attention normalization, the pre-FFN normalization, and any final output normalization are all substituted identically. No other architectural changes are required. This drop-in compatibility is a key practical advantage: existing model implementations need only swap nn.LayerNorm(...) for Derf(...).

---

Multi-Domain Evaluation Protocol

The paper evaluates Derf across five domains, each with its own architecture, training paradigm, and evaluation metric. The consistent theme is that Derf replaces the default normalization layer with no other changes to the training recipe, and in some cases with minor adjustments to learning rates or initialization.

Vision Transformers (Section 6, Table 8). ViT-Base and ViT-Large are trained on ImageNet-1K using the standard supervised classification pipeline from Table 17 (AdamW, lr=4e-3, weight decay=0.05, 300 epochs, cosine decay, 20 warmup epochs, batch size 4096, standard augmentations). Each model is trained three times: with LayerNorm (the baseline), with DyT, and with Derf. Top-1 accuracy on the validation set is reported. ViT-L uses a slightly different AdamW momentum setting (β₁=0.9, β₂=0.95 instead of the default β₂=0.999) and higher stochastic depth (0.5 vs. 0.15). The results show Derf achieving 82.8% (B) and 83.8% (L) versus LayerNorm at 82.3% and 83.1%, for ΔLN of +0.5% and +0.7% respectively, and versus DyT at 82.5% and 83.6%, for ΔDyT of +0.3% and +0.2% (Table 8). Additional baselines in Table 22 (Appendix D) include RMSNorm and GroupNorm, neither of which surpass Derf.

Diffusion Transformers (Section 6, Table 9). Three DiT model sizes are trained (B/4, L/4, XL/2) on ImageNet-1K for class-conditional image generation, following the default DiT training configuration from Table 18 but with learning rate sweeping ({1e-4, 2e-4, 4e-4}) for all methods to ensure fair comparison. Two important modifications for point-wise functions: the zero initialization (present in the original DiT LayerNorm) is removed for DyT and Derf because it negatively affected their performance, and the affine parameters $\gamma, \beta$ are retained for class conditioning across all methods (including LayerNorm) to maintain architectural consistency. FID is measured using the standard ImageNet reference batch. Results show Derf achieving 63.23 (B/4), 43.94 (L/4), and 18.92 (XL/2) versus LayerNorm at 64.93, 45.91, and 19.94, with ΔLN of −1.70, −1.97, and −1.02 FID respectively (Table 9). Versus DyT, the improvements are −0.71, −1.72, and −1.91 FID. Notably, DyT actually underperforms LayerNorm on XL/2 (20.83 vs. 19.94), while Derf improves over both. RMSNorm is also evaluated in Table 23 and performs worse than LayerNorm on the smaller models but comparably on XL/2.

Speech models (Section 6, Table 10). Two wav2vec 2.0 Transformer models (Base and Large) are trained on the LibriSpeech dataset for self-supervised speech representation learning, using the official implementation and hyperparameters from Table 19. A key architectural note: the first GroupNorm layer and the LayerNorm after the convolutional feature extractor are retained even in Derf models because they serve as "data normalization to handle the unnormalized input data" — they normalize the raw audio features before they enter the Transformer, which is a different role from the internal normalization within the Transformer blocks. All models are run in fp32 precision instead of the default bf16 (the authors found this necessary, though the reason is not specified). Validation loss is reported. Derf achieves 1.93 (Base) and 1.90 (Large) versus LayerNorm at 1.95 and 1.92, for ΔLN of −0.02 and −0.02 (Table 10). DyT achieves 1.95 and 1.91. RMSNorm is also evaluated in Table 24 and matches or slightly underperforms LayerNorm.

DNA models (Section 6, Table 11). Two architectures are evaluated: HyenaDNA (using LayerNorm by default) and Caduceus (using RMSNorm by default). Both are pretrained on the human reference genome (GRCh38) and evaluated on the GenomicBenchmarks dataset, which contains multiple subtasks for genomic sequence classification. Training follows the official implementations and hyperparameters from Table 20 with no modifications. Results are reported as average accuracy over all subtasks. Derf achieves 85.7% (Hyena, ΔNorm +0.5%) and 87.3% (Caduceus, ΔNorm +0.4%), with identical gains over DyT at +0.5% and +0.4% respectively (Table 11). Both LayerNorm and RMSNorm are evaluated for each architecture in Table 25, showing that Derf outperforms regardless of which normalization was originally used.

Language models (Section 6, Table 12). A GPT-2 (124M parameters) is pretrained on the OpenWebText dataset using the configuration from Table 21. For Derf and DyT specifically, the paper found that $\alpha$ initialization needed per-position tuning: they sweep initial $\alpha$ values separately for the point-wise layers following attention blocks ({0.5, 1.0, 2.0, 4.0}) and other point-wise layers ({0.1, 0.3, 0.5, 1.0}), reporting the best combination. Validation loss is reported. Derf achieves 2.94, matching LayerNorm exactly (ΔLN = 0.00) and outperforming DyT at 2.97 (ΔDyT = −0.03). RMSNorm achieves 2.95 in the extended comparison (Table 26). This is the one domain where Derf does not clearly surpass normalization — it matches it — but the authors note this still represents an improvement over DyT, which previously showed a performance gap versus LayerNorm.

Summary of domain-specific adjustments. The key pattern across all five domains is that Derf requires minimal hyperparameter tuning — in most cases, no changes at all beyond removing zero initialization for DiT. The only case where per-layer initialization tuning was needed was GPT-2, and even there the final result matches LayerNorm without degradation. This "drop-in" property is a significant practical advantage over prior normalization-free methods that required careful redesign of initialization schemes or optimizer settings.

---

Fitting-vs-Generalization Analysis

Motivation (Section 6.1). Given that Derf consistently outperforms normalization layers despite being structurally simpler, the paper asks a crucial diagnostic question: where do the gains come from? Specifically, does Derf enable the model to fit the training data better (stronger optimization), or does it regularize the model to generalize better (stronger generalization)? This distinction is critical because it has opposite implications: if Derf improves optimization, normalization layers were suboptimal for training; if Derf improves generalization, normalization layers may be overfitting to training statistics.

Experimental design: evaluation-mode training loss. A direct comparison of training loss during optimization is confounded by stochastic regularization (drop-path, data augmentation) and train-time preprocessing that differ from evaluation. To isolate pure fitting capacity, the authors measure training loss after training, in evaluation mode, with all stochastic regularization disabled:

1. For ViT: Switch to evaluation mode, disable drop-path, mixup, cutmix, label smoothing, and all data augmentations. Apply only test-time preprocessing (center crop, normalize). Compute cross-entropy loss on the full training set.
2. For DiT: Switch to evaluation mode. Apply test-time preprocessing. Compute diffusion MSE loss over the first 100 training batches. (DiT does not use drop-path, so no stochastic regularization needs disabling.)
3. For wav2vec 2.0, HyenaDNA, Caduceus, and GPT-2: Switch to evaluation mode, disable dropout/drop-path where present, apply test-time preprocessing, compute loss on the full training set.

The resulting "evaluation-mode training loss" measures how well each model fits the training data under deterministic conditions, separated from the effects of stochastic regularization.

Results (Table 13). The pattern is consistent across all nine model configurations:

• Normalization layers (Norm) achieve the lowest training loss in all cases. For ViT-B: 0.2623 (Norm) vs. 0.2681 (Derf) vs. 0.2714 (DyT). For ViT-L: 0.2034 vs. 0.2066 vs. 0.2083. For wav2vec 2.0 Base: 1.8509 vs. 1.8821 vs. 1.8946. For GPT-2: 2.9478 vs. 2.9702 vs. 2.9822. The ordering is always: Norm < Derf < DyT.
• Derf achieves the second-lowest training loss, consistently below DyT but above normalization.
• The gap between Norm and Derf is small but consistent (e.g., 0.2623 vs. 0.2681 for ViT-B — a difference of 0.0058, or roughly 2.2% relative).

Interpretation — Part 1: Derf does not improve fitting capacity. The consistently higher training loss of Derf (and DyT) compared to normalization layers indicates that point-wise functions are worse optimizers — they result in a higher training loss, meaning the model cannot fit the training data as tightly. This directly refutes the hypothesis that Derf's performance gains come from better optimization. Normalization layers, by adapting to activation statistics, allow the model to minimize training loss more effectively.

Interpretation — Part 2: Derf improves generalization through implicit regularization. Despite higher training loss, Derf achieves lower validation error (higher top-1 accuracy, lower FID, lower validation loss) than normalization layers. This means Derf must provide stronger generalization — the gap between training and validation performance is smaller, indicating less overfitting. The paper hypothesizes that this generalization benefit arises from the limited adaptability of point-wise functions:

"Normalization layers adapt their transformation based on training statistics, allowing them to dynamically fit activation distributions throughout training. In contrast, point-wise functions are controlled by only a small set of learnable scalar parameters (e.g., α for DyT and α, s for Derf) that do not adapt to activation statistics after training. They apply the same transformation regardless of activation distribution. This limited adaptability constrains overfitting and effectively serves as an implicit regularizer, leading to improved generalization." (Section 6.1, Discussion)

In operational terms: during training, LayerNorm computes μ and σ from each batch's activations and applies a different transformation at each training step (since statistics vary across batches). These batch-specific transformations may create a form of overfitting to batch-level activation statistics — the model learns to rely on normalization adjustments that are present during training but not representative of the true data distribution. At test time, with larger batches or different activation distributions, this reliance can hurt. Point-wise functions, by applying the same fixed transformation regardless of batch statistics, cannot overfit in this way — they must learn a transformation that works universally, which acts as a regularizer.

Interpretation — Part 3: Derf strikes a better fitting-generalization balance than DyT. Within the point-wise function family, Derf achieves lower training loss than DyT while presumably maintaining similar regularization properties. This means Derf has stronger fitting capacity than DyT (it can fit the training data better) while retaining the implicit regularization benefit of being a point-wise function. Derf therefore sits at a sweeter spot in the bias-variance tradeoff: better fitting than DyT, better generalization than normalization.

The paper does not explain why erf enables better fitting than tanh — it is an empirical observation. But the combination — (1) normalization layers have the best fitting capacity but weaker generalization, (2) point-wise functions have weaker fitting capacity but stronger generalization, and (3) among point-wise functions, erf has better fitting capacity than tanh — produces the observed performance hierarchy: Derf > {LayerNorm, DyT} across all tested domains.

This analysis is the paper's most theoretically significant contribution: it reframes normalization-free design from an optimization problem ("how do we match normalization's training stability without statistics?") to a regularization problem ("can we improve generalization by replacing adaptive statistics with fixed transformations?"). The answer, demonstrated empirically, is yes — and erf is the function that best executes this tradeoff among the candidates tested.

4. Key Insights and Innovations

Innovation 1: Reframing Normalization-Free Design as a Regularization Problem Rather Than an Optimization Problem

The dominant assumption throughout the normalization literature—from BatchNorm (Ioffe and Szegedy, 2015) through LayerNorm (Ba et al., 2016) to normalization-free methods like DyT (Zhu et al., 2025)—has been that normalization layers are fundamentally optimization tools. The framing, explicit or implicit, was: normalization stabilizes training by controlling activation statistics, smoothing the loss landscape, and enabling faster convergence. Under this view, removing normalization creates an optimization deficit—training becomes unstable or slower—and the challenge for normalization-free methods is to recover that optimization capability through other means (careful initialization, adaptive gradient clipping, bounded nonlinearities).

This paper fundamentally challenges that framing. The evaluation-mode training loss experiment in Section 6.1 (Table 13) reveals a pattern that inverts the conventional wisdom: normalization layers actually achieve lower training loss than point-wise functions, not higher. Across all nine model configurations spanning vision, generation, speech, DNA, and language, the ordering is consistently Norm < Derf < DyT in training loss. Normalization-based models fit the training data better—they are stronger optimizers. Point-wise functions, by contrast, show higher training loss yet deliver superior downstream performance. The implication is that normalization layers are not suboptimal in the optimization sense; they are suboptimal in the generalization sense.

This is a conceptual inversion with substantial implications. Prior work evaluated normalization-free methods by asking "does this train as well as LayerNorm?"—checking training curves, convergence speed, and final training loss. This paper shows that metric points in the wrong direction: the best normalization-free method (Derf) actually trains worse than LayerNorm in terms of fitting capacity. The correct metric is generalization gap—the difference between training and validation performance—and by this measure, point-wise functions are superior because their limited adaptability acts as an implicit regularizer.

The regularization mechanism is subtly different from standard techniques like weight decay or dropout. Those methods add noise or constraints during training to prevent overfitting. Point-wise functions achieve regularization through an architectural constraint: they apply the same fixed transformation regardless of activation statistics, whereas normalization layers adapt their transformation to each batch's statistics. This adaptation—precisely the feature that makes normalization powerful for optimization—may cause overfitting to batch-level activation distributions that don't generalize to test conditions. The paper doesn't prove this mechanism directly, but the consistent pattern of higher training loss with lower validation error across architectures and modalities makes it the most parsimonious explanation.

The significance of this reframing extends beyond this paper. It suggests that the entire line of normalization-free research has been optimizing the wrong objective. Methods that carefully engineer initialization schemes or gradient clipping to match normalization's training dynamics (Zhang et al., 2019; Brock et al., 2021b) may be inadvertently reducing the very property—limited adaptability—that gives point-wise functions their generalization advantage. The goal shouldn't be to make point-wise functions train more like normalization; it should be to find the function shape that optimally balances fitting capacity against the implicit regularization benefit. This paper's search procedure (Section 4) and the discovery that erf outperforms tanh on this tradeoff—better fitting than tanh, better generalization than normalization—is the first demonstration of optimizing for this new objective.

Innovation 2: A Systematic Property Analysis That Makes the Design Space for Normalization Replacements Legible

Prior to this work, the design of point-wise functions for normalization replacement was guided largely by conceptual analogy rather than empirical constraint. DyT (Zhu et al., 2025) chose tanh because LayerNorm's input-output mapping "often resembles an S-shaped saturation curve in practice"—a qualitative observation, not a principled design rule. Other work explored bounded activations or self-normalizing functions based on theoretical properties of gradient propagation (Klambauer et al., 2017), but there was no systematic study of which functional properties are necessary, which are sufficient, and which are irrelevant for training Transformers without normalization.

The paper's property analysis (Section 3) provides exactly this. By isolating four functional properties—zero-centeredness, boundedness, center sensitivity, and monotonicity—and testing each through controlled perturbations on three base functions (erf, tanh, arctan), the authors convert qualitative intuition into quantitative constraint. The findings are not merely confirmatory ("yes, these properties matter"); they contain non-obvious nuance:

• Zero-centeredness is a requirement, but asymmetric. Vertical shifts (output bias) are more damaging than horizontal shifts (input offset) of the same magnitude (Table 1). This isn't predicted by any existing theory—it's an empirical regularity that constrains the design space in a specific way.

• Boundedness is practically essential but not theoretically mandatory. Some unbounded functions (arcsinh, logsign) can train (Table 2), but their growth rate must be slower than polynomial (Table 4, Figure 3). The paper identifies logquad(x) as the fastest-growing function that still converges—a concrete boundary in the design space that didn't exist before.

• Center sensitivity is a strict requirement with a sharp failure threshold. Small dead zones around zero (λ ≤ 0.5) are tolerable, but at λ ≥ 1.0 degradation accelerates and at λ ≥ 3.0 training fails entirely (Table 5). This quantifies precisely how much "flatness" the function can have before signal propagation fails.

• Monotonicity matters, but direction doesn't. Monotonically decreasing functions perform identically to increasing ones (Table 6a), while non-monotonic functions universally underperform (Table 6b). This is the cleanest result in the analysis—consistent ordering is essential, the sign of ordering is irrelevant—and it simplifies the design space by making mirrored functions equally valid.

What makes this analysis a genuine contribution rather than an obvious catalog is that it produces a design space with boundaries. Before this work, the space of point-wise functions was infinite and unstructured—any transformation from ℝ to ℝ was a candidate. After the analysis, the viable space is constrained to functions that are approximately zero-centered, bounded (naturally or through clipping), responsive near zero (nonzero derivative at the origin), and monotonic. Functions violating any of these properties are known to fail (or degrade) with quantified thresholds. This transforms the search problem from "try everything" to "search within a well-defined region," which is what enables the systematic evaluation in Section 4.

The analysis also serves as a diagnostic tool for future work. Any newly proposed normalization replacement can be checked against these four properties: does it satisfy them? If not, what specific failure mode does the property analysis predict? This makes the design space legible—new entrants can be evaluated against established constraints rather than requiring full training runs to assess viability.

Innovation 3: Demonstrating That the Specific Function Shape Matters Beyond Property Satisfaction, With erf as the Empirical Optimum

The most straightforward reading of the property analysis would be: "any S-shaped function that's zero-centered, bounded, center-sensitive, and monotonic should work about equally well." The search results in Table 7 refute this. Despite all 16 candidate functions satisfying the four identified properties—and despite their visual similarity when plotted (Figure 5)—performance varies substantially: from 82.8% down to 81.4% top-1 on ViT-Base, and from 63.23 up to 70.22 FID on DiT-B/4. The 1.4-point accuracy spread and 7-point FID spread are large enough to matter in practical deployment, and they emerge purely from the choice of base function shape, with all other architectural and training details held constant.

This finding establishes that the property constraints are necessary but not sufficient. They define the feasible region of function space, but within that region, there are meaningful performance differences that cannot be explained by property satisfaction alone. Something about erf's specific curvature—its tail behavior, the rate at which it approaches saturation, the exact shape of its transition through the linear regime—provides an advantage over equally well-qualified alternatives like tanh, satursin, and arctan. The approximation experiment in Section 7.2 (Table 16) deepens the mystery: scaling tanh to minimize its L1 distance from erf (producing tanh(1.205x)) improves performance over standard tanh but still falls short of erf itself. This means the performance gap is not simply a matter of matching erf's slope or width—there is a qualitative shape difference that simple scaling cannot capture.

The paper does not resolve why erf works best. It doesn't propose a theory linking erf's curvature to gradient flow, loss landscape geometry, or representation learning dynamics. But the empirical demonstration itself is a significant contribution because it establishes that the choice of point-wise function is a design degree of freedom worth optimizing. Before this paper, the field's state of knowledge was: (1) normalization works, (2) tanh can match it. After this paper, the state of knowledge is: (1) normalization works but may overfit, (2) point-wise functions can outperform normalization, (3) among point-wise functions, the specific shape matters materially, and (4) erf is the best shape found so far. Each step represents a genuine advance in understanding, even though the mechanism behind step (4) remains empirical rather than theoretical.

The practical consequence is that Derf (erf with learnable parameters) should be the default choice for anyone replacing normalization layers with point-wise functions. The paper demonstrates this across five domains with consistent improvements—not just on the search architectures (ViT and DiT) but on previously unseen tasks (speech, DNA, language) that were not part of the search procedure. This cross-domain transfer is important: it suggests that erf's advantage is not specific to vision or to the architectures on which it was selected, but reflects a more general property of the function that benefits Transformer training broadly.

Innovation 4: Establishing Point-Wise Functions as a Domain-General Architectural Primitive

Normalization layers have historically been domain-specific in their design. BatchNorm was developed for CNNs on image classification; LayerNorm for RNNs and Transformers on sequence data; InstanceNorm for style transfer; GroupNorm as a compromise between batch-dependent and instance-dependent normalization. Each variant reflects assumptions about which activation groups share meaningful statistics—batch dimension, channel dimension, spatial dimension—and the choice is often tied to the data modality and architecture.

This paper, together with DyT (Zhu et al., 2025) which it builds on, establishes a countervailing principle: a single point-wise function, applied identically to every activation regardless of modality or architecture, can replace all of these domain-specific normalization variants and outperform them. Derf is evaluated on image classification (ViT), image generation (DiT), speech representation learning (wav2vec 2.0), DNA sequence modeling (HyenaDNA, Caduceus), and language modeling (GPT-2)—five domains with fundamentally different data characteristics, architectural requirements, and training paradigms. In every case, Derf matches or exceeds the domain-standard normalization, and in most cases exceeds it by a clear margin.

What makes this distinctive is not the multi-domain evaluation itself, but what it implies about the relationship between normalization and architecture. The traditional view is that normalization is coupled to architecture: the choice of normalization method depends on what makes sense for the data and the model structure. The point-wise function view is that normalization can be decoupled from architecture: a single element-wise transformation, with no cross-element communication and no domain-specific assumptions, works universally. This is a stronger claim than "Derf works on these five tasks." It suggests that the function of normalization—stabilizing training, controlling activation scales—does not actually require the statistical aggregation that has defined normalization layers since their inception. A fixed function shape, with per-layer learnable parameters for slope and shift, is sufficient.

The theoretical implications are significant. If statistical aggregation is unnecessary, then the benefits of normalization must arise from the shape of the transformation rather than from its ability to adapt to activation statistics. This aligns with the regularization interpretation (Innovation 1): normalization works because it imposes a specific functional form (centering, scaling, saturating extremes) that is beneficial for gradient flow and representation learning, not because the specific values of μ and σ computed from each batch are meaningful. The fact that a fixed function like erf can outperform an adaptive one like LayerNorm suggests that the adaptivity may actually be harmful in some regimes—a claim that would be difficult to make without the multi-domain consistency this paper demonstrates.

Practically, this universality is valuable because it simplifies model design. Rather than choosing between LayerNorm, RMSNorm, GroupNorm, or BatchNorm based on the task and architecture, practitioners can default to Derf across the board. The paper shows this works: Derf outperforms RMSNorm on architectures where RMSNorm is the default (Caduceus), outperforms LayerNorm where it is the default (ViT, DiT, wav2vec 2.0), and matches both on GPT-2 (Tables 22–26 in Appendix D). The only architecture-specific adjustment needed was α initialization tuning for GPT-2, and even there the final result matched LayerNorm without degradation.

This universality also positions point-wise functions as a candidate for the long-sought goal of a unified Transformer block—a single architectural template that works across modalities without modality-specific normalization choices. While this paper doesn't explicitly pursue that agenda, the evidence that Derf works across vision, speech, DNA, and language without domain-specific modifications provides strong empirical support for such unification.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use ImageNet-1K (Deng et al., 2009), consisting of ~1.28 million training images and 50,000 validation images across 1,000 classes, with the standard training/validation split. For domain-specific evaluations, additional datasets are used: LibriSpeech (Panayotov et al., 2015) for speech (960 hours of read English speech), the human reference genome GRCh38 (2013) for DNA pretraining, GenomicBenchmarks (Grešová et al., 2023) for DNA downstream evaluation, and OpenWebText for language modeling. The paper does not itself introduce new datasets; it evaluates on established benchmarks to ensure comparability.

Base model(s). The primary architectures are ViT-Base and ViT-Large (Dosovitskiy, 2021) for image classification, DiT-B/4, DiT-L/4, and DiT-XL/2 (Peebles and Xie, 2023) for image generation, wav2vec 2.0 Base and Large (Baevski et al., 2020) for speech representation learning, HyenaDNA (Nguyen et al., 2023) and Caduceus (Schiff et al., 2024) for DNA sequence modeling, and GPT-2 (124M parameters) (Radford et al., 2019) for language modeling. The property analysis and function search (Sections 3–4) use ViT-Base and DiT as the evaluation platforms, chosen because they are established, well-tested Transformer architectures with standard training recipes that provide reliable comparative baselines. The multi-domain evaluation (Section 6) extends to all listed architectures to test domain generality.

Metrics. For ViT classification, the metric is top-1 accuracy (%) on the ImageNet-1K validation set — the fraction of images for which the model's highest-probability class matches the ground truth. For DiT generation, the metric is Fréchet Inception Distance (FID), measured using the standard ImageNet "reference batch" evaluation protocol, with lower FID indicating higher-quality generated images (closer to real image statistics). For wav2vec 2.0, the metric is final validation loss on the LibriSpeech validation set — a standard proxy for representation quality in self-supervised speech models. For DNA models, the metric is average classification accuracy (%) across all subtasks in the GenomicBenchmarks dataset. For GPT-2, the metric is validation loss on the OpenWebText validation set. The paper also introduces an evaluation-mode training loss (Section 6.1) to measure fitting capacity: training loss computed after optimization in evaluation mode with all stochastic regularization disabled, using test-time preprocessing.

Baselines. Three primary baselines are used throughout:

• LayerNorm (LN) (Ba et al., 2016): the default normalization in ViT, DiT, wav2vec 2.0, and HyenaDNA. Standard formulation: y = γ * (x − μ)/√(σ² + ε) + β with per-token statistics computed across the channel dimension.
• RMSNorm (Zhang and Sennrich, 2019): the default normalization in Caduceus and common in LLMs (LLaMA, Qwen, DeepSeek). Similar to LayerNorm but without mean subtraction: y = γ * x/√(mean(x²) + ε) + β.
• Dynamic Tanh (DyT) (Zhu et al., 2025): the prior state-of-the-art point-wise normalization replacement. Formulation: DyT(x) = γ * tanh(αx) + β where α is a learnable scalar.

Additional baselines evaluated in Appendix D include GroupNorm (GN) (Wu and He, 2018) for ViT, tested because some vision architectures (ConvNeXt, DETR, Swin Transformer) use it as their default normalization. All baselines are trained under identical configurations to the corresponding Derf models, with only the normalization layer replacement varying.

Generation budget / compute accounting. Since all methods being compared are architectural substitutions that replace one layer type with another, there is no "generation budget" in the sense of test-time scaling papers. The relevant comparison framework is computational cost per layer, measured via wall-clock runtime of forward and backward passes through a single normalization-equivalent layer. The paper implements DyT, Derf, and LayerNorm using custom Triton kernels and benchmarks them under a unified setup with identical batch sizes, sequence lengths, and hidden dimensions (Appendix F, Figure 13, Table 27). This ensures that any reported performance gains are not achieved through increased computation. The efficiency analysis shows that Derf and DyT have nearly identical runtime in both forward and backward passes across all practical hidden dimensions; in the forward pass, Derf and LayerNorm are comparable; in the backward pass, Derf is slightly slower at small dimensions but becomes faster as hidden dimension grows (since normalization's reduction and synchronization overhead increases with dimension).

Cross-validation / statistical protocol. The paper does not employ cross-validation for its main results, as the architectures are large models trained on fixed datasets with standard train/validation splits. The function search in Section 4 evaluates all candidates on the same ViT-Base and DiT training runs using the standard ImageNet-1K training and validation sets — there is no held-out search validation set because the search space is the function choice itself, not hyperparameters that could overfit to a validation set. The learning rate sweep for DiT ({1e-4, 2e-4, 4e-4} for all methods, including baselines) ensures fair comparison by giving each method its best learning rate rather than using a single default. For the GPT-2 results, the paper sweeps α initialization values ({0.5, 1.0, 2.0, 4.0} for attention layers, {0.1, 0.3, 0.5, 1.0} for other layers) and reports the best combination — this is effectively a hyperparameter search on the validation set, though the paper does not frame it as cross-validation. Since the paper's core claim is about a fixed architectural substitution rather than a learned policy, the absence of cross-validation is reasonable: there are no hyperparameters to select per-question, and the function choice is validated across five independent domains that were not used for the search (speech, DNA, language) or on the search architectures themselves (ViT, DiT) where the function was selected.

Main Quantitative Results

Property Analysis: Identifying Necessary Conditions for Viable Normalization Replacements

Headline finding. The property analysis (Section 3) establishes four properties as necessary for a point-wise function to serve as an effective normalization replacement, with quantified thresholds for acceptable deviation from each property. All experiments use ViT-Base trained on ImageNet-1K for 300 epochs under the configuration in Table 17.

Zero-centeredness (Table 1). Across three base functions (erf, tanh, arctan), horizontal shifts (input offset) of magnitude |λ| ≤ 0.5 are well-tolerated — erf at λ=0 achieves 82.6% while λ=±0.5 achieves 82.5% — but performance degrades steadily beyond this, and training fails entirely when |λ| ≥ 2 for all functions. Vertical shifts (output bias) are more damaging: even |λ| = 0.5 produces a visible drop (erf: 82.6% → 82.3%), degradation is monotonic with |λ|, and |λ| ≥ 2 causes training divergence for all functions. The key quantification: the acceptable deviation from zero-centeredness is asymmetric, with vertical shifts being roughly twice as damaging as horizontal shifts of the same magnitude.

Boundedness (Tables 2–4). For naturally unbounded functions, the clipped versions consistently outperform their unclipped counterparts: arcsinh(x) improves from 82.2% (unclipped) to 82.3–82.4% (clipped at various thresholds); logsign(x) similarly improves from 82.2% to 82.3–82.4%. The linear interpolation experiment (Table 3) shows that performance degrades monotonically as functions are made less bounded: at λ_b = 0.01 (1% linear component), erf drops from 82.6% to 82.4%; at λ_b = 0.1, to 82.3%; at λ_b = 0.5, training fails for all four tested functions. The growth rate analysis (Table 4) identifies logquad(x) at 82.1% as the fastest-growing unbounded function that still converges; power23(x) and linear(x) — which grow as x^{2/3} and x respectively — cause immediate training divergence.

Center sensitivity (Table 5). Performance is optimal with no dead zone around zero (λ=0). For erf: 82.6% at λ=0, 82.5% at λ=0.1, 82.1% at λ=1.0, 81.3% at λ=2.0. For λ ≥ 3.0, training fails across all functions. Training loss curves (Appendix A.2, Figure 7) show monotonic loss increase with λ, confirming that the flat zone directly limits fitting capacity rather than merely changing regularization. The acceptable tolerance is λ ≤ 0.5 — beyond this, degradation becomes substantial.

Monotonicity (Table 6). Monotonically decreasing functions perform near-identically to their increasing counterparts: erf at 82.6% (increasing) vs. 82.5% (decreasing); tanh at 82.5% vs. 82.5%; arctan at 82.3% vs. 82.2%. Non-monotonic functions consistently underperform: oscillatory sin(x) achieves 81.6%; hump-shaped dampx(x) achieves 80.7%; dampexp(x) achieves 81.2%. Training loss curves (Appendix A.3, Figure 8) confirm that non-monotonic functions exhibit higher loss throughout optimization. The finding that direction of monotonicity is irrelevant — only consistency matters — simplifies the design space by making mirrored versions equally valid candidates.

Function Search: erf Emerges as the Optimal Design

Headline finding (Table 7). Among 16 candidate point-wise functions evaluated on ViT-Base (top-1 accuracy) and DiT-B/4 and DiT-L/4 (FID), erf(x) achieves the best performance on both architectures: 82.8% top-1 on ViT (+0.5% over LayerNorm at 82.3%, +0.3% over DyT at 82.5%) and 63.23 FID on DiT-B/4 (−1.70 over LayerNorm at 64.93, −0.71 over DyT at 63.94). On DiT-L/4, Derf achieves 43.94 FID (−1.97 over LayerNorm at 45.91, −1.72 over DyT at 45.66).

The performance spread across candidates is substantial despite all functions satisfying the four property constraints. On ViT-Base: the best function (erf, 82.8%) exceeds the worst (cubsign, 81.4%) by 1.4 percentage points. On DiT-B/4: the best (erf, 63.23 FID) beats the worst (cubsign, 70.22 FID) by nearly 7 FID points. The second-best candidates — tanh (82.6% top-1, 63.71 FID) and satursin (82.6% top-1, 63.90 FID) — are consistently close to but below erf. Functions like isru(x) (82.3%, 65.72 FID) and arctan(x) (82.4%, 67.07 FID) show notably worse performance despite similar visual shapes (Figure 5), demonstrating that subtle curvature differences have measurable impact.

A notable finding: linearclip(x) — the identity function clipped to [−1, 1] — achieves a respectable 82.3% top-1 on ViT, tying isru(x) and outperforming several more complex candidates on classification, but its DiT FID of 66.08 places it near the bottom of the generation ranking. This task-dependent pattern suggests that the function's suitability may vary between discriminative and generative objectives, making erf's consistent dominance across both tasks particularly meaningful.

Multi-Domain Evaluation: Derf Consistently Outperforms Normalization

Headline finding (Tables 8–12, Section 6). Derf outperforms LayerNorm and DyT across vision classification, image generation, speech representation, DNA sequence modeling, and language modeling, with gains ranging from modest (matching LN on GPT-2) to substantial (−1.97 FID improvement on DiT-L/4, +0.7% top-1 on ViT-L).

Vision Transformers (Table 8, Table 22). On ViT-Base, Derf achieves 82.8% vs. LayerNorm at 82.3% (Δ = +0.5%) and DyT at 82.5% (Δ = +0.3%). On ViT-Large, Derf achieves 83.8% vs. LayerNorm at 83.1% (Δ = +0.7%) and DyT at 83.6% (Δ = +0.2%). The extended comparison in Table 22 adds RMSNorm (ViT-B: 82.4%, ViT-L: 83.0%) and GroupNorm (ViT-B: 82.5%, ViT-L: 83.1%), neither of which reaches Derf's performance.

Diffusion Transformers (Table 9, Table 23). On DiT-B/4, Derf achieves 63.23 FID vs. LayerNorm at 64.93 (Δ = −1.70) and DyT at 63.94 (Δ = −0.71). On DiT-L/4: 43.94 FID vs. 45.91 (Δ = −1.97) and 45.66 (Δ = −1.72). On DiT-XL/2: 18.92 FID vs. 19.94 (Δ = −1.02) and 20.83 (Δ = −1.91). Notably, DyT actually underperforms LayerNorm on DiT-XL/2 (20.83 vs. 19.94), while Derf improves over both — indicating that at larger scales, the function choice becomes more impactful. RMSNorm (Table 23) performs worse than LayerNorm on smaller DiT models (65.08 on -B/4, 45.02 on -L/4) but comparably on -XL/2 (20.76), still below Derf.

Speech models (Table 10, Table 24). On wav2vec 2.0 Base, Derf achieves validation loss 1.93 vs. LayerNorm at 1.95 (Δ = −0.02) and DyT at 1.95 (Δ = −0.02). On wav2vec 2.0 Large: 1.90 vs. 1.92 (Δ = −0.02) and 1.91 (Δ = −0.01). RMSNorm (Table 24) achieves 1.95 on Base and 1.93 on Large — matching or slightly worse than LayerNorm. The consistent −0.02 improvement on both model sizes, while small in absolute terms, is reliable across the architecture scale.

DNA models (Table 11, Table 25). On HyenaDNA, Derf achieves 85.7% average accuracy vs. LayerNorm at 85.2% (Δ = +0.5%) and DyT at 85.2% (Δ = +0.5%). On Caduceus: 87.3% vs. RMSNorm at 86.9% (Δ = +0.4%) and DyT at 86.9% (Δ = +0.4%). The extended comparison (Table 25) evaluates both LN and RMSNorm for each architecture: on HyenaDNA, RMSNorm achieves 85.2% (matching LN and DyT); on Caduceus, LN achieves 87.0% (slightly above its native RMSNorm at 86.9%, but below Derf). Derf is the only method to exceed 87% on Caduceus and 85.5% on HyenaDNA.

Language models (Table 12, Table 26). On GPT-2 (124M), Derf achieves validation loss 2.94, matching LayerNorm exactly (Δ = 0.00) and outperforming DyT at 2.97 (Δ = −0.03). RMSNorm (Table 26) achieves 2.95 — slightly worse than Derf and LN. This is the only domain where Derf does not clearly surpass normalization, though it does close the gap that DyT had left (DyT was previously worse than LN on language modeling; Derf eliminates that deficit). The paper notes that α initialization required per-position tuning for GPT-2 (sweeping across attention-layer and other-layer α values separately), making this the only architecture where Derf needed any hyperparameter adjustment beyond the defaults used in other domains.

Fitting-vs-Generalization Analysis: Derf's Gains Come from Better Regularization, Not Better Optimization

Headline finding (Table 13). Across all nine model configurations spanning five architectures and multiple scales, evaluation-mode training loss follows the consistent ordering: Norm < Derf < DyT. Normalization layers achieve the lowest training loss in every case — they are the strongest optimizers. Derf achieves the second-lowest, and DyT the highest. Since Derf delivers superior validation performance despite higher training loss than normalization, its gains must come from better generalization (smaller gap between training and validation performance) rather than better fitting.

Specific numbers: on ViT-B, training losses are 0.2623 (Norm), 0.2681 (Derf), 0.2714 (DyT). On ViT-L: 0.2034, 0.2066, 0.2083. On DiT-B: 0.1531, 0.1533, 0.1535 (gap narrows on this task). On DiT-L: 0.1501, 0.1510, 0.1518. On DiT-XL: 0.1432, 0.1436, 0.1440. On wav2vec 2.0 Base: 1.8509, 1.8821, 1.8946. On wav2vec 2.0 Large: 1.8241, 1.8563, 1.8641. On HyenaDNA: 1.1297, 1.1526, 1.1631. On Caduceus: 0.8917, 0.9129, 0.9203. On GPT-2: 2.9478, 2.9702, 2.9822.

The gap between Norm and Derf varies by task: it is larger on discriminative/classification tasks (ViT-B: 0.0058 difference, ViT-L: 0.0032) and on self-supervised tasks (wav2vec 2.0 Base: 0.0312 difference, Large: 0.0322) than on generative tasks (DiT-B: 0.0002, DiT-L: 0.0009, DiT-XL: 0.0004) where the fitting gap nearly vanishes. Despite these task-dependent variations, the ranking Norm < Derf < DyT holds universally, establishing that Derf improves over DyT by recovering some of the fitting capacity that DyT lost relative to normalization, while retaining the implicit regularization benefit that all point-wise functions share.

Ablation Studies and Robustness Checks

Effect of the learnable shift parameter s (Table 14, Section 7.1): Removing s from the formulation degrades performance consistently across functions and metrics. On ViT-Base: erf drops from 82.8% to 82.6% (−0.2%); tanh from 82.6% to 82.5% (−0.1%); satursin from 82.6% to 82.4% (−0.2%); isru from 82.3% to 82.2% (−0.1%); arctan from 82.4% to 82.3% (−0.1%); arcsinhclip from 82.5% to 82.4% (−0.1%). On DiT-B/4 FID: erf worsens from 63.23 to 63.39 (+0.16); tanh from 63.71 to 63.94 (+0.23); satursin from 63.90 to 65.28 (+1.38 — the largest sensitivity). The degree of improvement from s varies across functions: satursin benefits substantially more than erf or tanh, and functions with weaker baseline performance (isru, arctan) show smaller improvements. Critically, erf without s (82.6%) still outperforms tanh with s (82.6% is a tie, but with 63.39 vs. 63.71 FID on DiT, erf without s beats tanh with s), establishing that Derf's advantage over DyT is not solely attributable to the additional shift parameter — erf's base shape intrinsically outperforms tanh's.

Scalar vs. vector shift parameter s (Table 15, Section 7.1): Using a per-channel vector s instead of a scalar s yields nearly identical performance. On ViT-Base: erf achieves 82.8% with both vector and scalar; arctan achieves 82.5% (vector) vs. 82.4% (scalar); arcsinhclip achieves 82.5% with both. The negligible difference justifies using the scalar form for parameter efficiency — a per-channel vector adds C parameters per layer instead of 1, with no corresponding accuracy benefit.

Approximating erf with scaled tanh (Table 16, Section 7.2): Fitting tanh(εx) to match erf(x) by minimizing the L1 distance yields an optimal scaling factor ε ≈ 1.205. On ViT-Base, tanh(1.205x) achieves 82.7% — improving over standard tanh at 82.6% but still below erf at 82.8%. On ViT-L: 83.7% vs. 83.6% (tanh) and 83.8% (erf). On DiT-B: 63.88 FID vs. 63.71 (tanh) and 63.23 (erf). On DiT-L: 45.13 FID vs. 45.48 (tanh) and 43.94 (erf). The scaled tanh bridges part of the gap between DyT and Derf but does not close it, indicating that the performance difference is not solely about erf's slope or width — there is a qualitative shape difference that simple input scaling cannot capture. This is a notable negative result: the most obvious hypothesis for why erf outperforms tanh (different slope) is tested and rejected.

Robustness across normalization variants (Appendix D, Tables 22–26): Derf is compared against not only each model's default normalization but also against alternative normalization methods. On ViT (Table 22), Derf outperforms RMSNorm (ViT-B: 82.4%, ViT-L: 83.0%) and GroupNorm (82.5%, 83.1%). On DiT (Table 23), Derf outperforms RMSNorm across all model sizes (DiT-B: 65.08, -L: 45.02, -XL: 20.76). On wav2vec 2.0 (Table 24), Derf outperforms RMSNorm (Base: 1.95, Large: 1.93). On DNA models (Table 25), Derf is the only method to top both LN and RMSNorm on both HyenaDNA (LN: 85.2%, RMSNorm: 85.2%, Derf: 85.7%) and Caduceus (LN: 87.0%, RMSNorm: 86.9%, Derf: 87.3%). On GPT-2 (Table 26), Derf matches LN (2.94) and beats RMSNorm (2.95). This establishes that Derf's advantage is not specific to the normalization variant it replaces — it is consistently superior or equal regardless of whether the baseline is LN, RMSNorm, or GN.

Robustness across model scales: The improvements hold across model sizes within each architecture family — ViT-Base (+0.5%) and ViT-Large (+0.7%), DiT-B/4 (−1.70 FID), -L/4 (−1.97), -XL/2 (−1.02), wav2vec 2.0 Base (−0.02 loss) and Large (−0.02), HyenaDNA and Caduceus (both +0.4–0.5% accuracy). The gain does not diminish with scale — in fact, it slightly increases for ViT and is largest for DiT-L/4 among the DiT variants. On DiT-XL/2, the gap between DyT and LayerNorm actually reverses (DyT underperforms LN by −0.89 FID, while Derf outperforms LN by −1.02), showing that at larger scales the function choice becomes more consequential, not less.

Domain transfer from search to evaluation: The function search was conducted on ViT-Base and DiT (vision models). Derf (erf) was selected based on its performance on these architectures. The subsequent evaluations on speech (wav2vec 2.0), DNA (HyenaDNA, Caduceus), and language (GPT-2) all show Derf outperforming or matching the baselines — these domains were never used for function selection. This establishes that erf's advantage transfers across modalities and is not an artifact of overfitting the function choice to the search architectures.

Computational cost comparison (Appendix F, Figure 13, Table 27): Derf and DyT exhibit nearly identical wall-clock runtime in both forward and backward passes across all tested hidden dimensions (5K to 15K). In the forward pass, Derf and LayerNorm are comparable (both ~0.42 ms at 15K hidden dimension). In the backward pass, Derf can be slightly slower than LayerNorm at small dimensions (~5K) but becomes faster as hidden dimension increases, since normalization's cross-element reduction and synchronization overhead grows with dimension while point-wise operations scale independently. At 15K hidden dimension, backward times are: Derf 0.73 ms, DyT 0.73 ms, LN 0.74 ms — essentially identical. This confirms that Derf's performance improvements are not achieved through higher computational cost; the method is at least as efficient as LayerNorm.

Critical Assessment

Claim 1: "Well-designed point-wise functions can surpass normalization layers."

This claim is well-supported for the specific definition of "surpass" as "achieve better downstream task performance." The evidence spans five domains, multiple model scales, and alternative normalization baselines (LN, RMSNorm, GN). On vision classification (ViT), image generation (DiT), speech (wav2vec 2.0), and DNA (HyenaDNA, Caduceus), Derf achieves statistically meaningful improvements over the best normalization baseline. The gains are consistent in direction across all tested configurations: Derf > Norm in 14 of 15 comparisons (the one exception being GPT-2 where Derf matches LN at 2.94 and beats DyT at 2.97).

However, what "surpass" means operationally is worth scrutinizing. The improvements are real but modest in absolute terms: +0.5% top-1 on ViT-B, −1.97 FID on DiT-L/4, −0.02 validation loss on wav2vec 2.0, +0.5% accuracy on DNA. These are not transformative gains — they represent incremental but reliable improvements over already well-tuned baselines. The paper does not demonstrate that Derf enables qualitatively different capabilities or that the gains compound at larger scales (GPT-2 being a relevant counterexample where the gain is zero). The claim "surpass" is technically accurate but the magnitude should be contextualized: Derf provides a consistent small improvement, not a breakthrough.

A more significant qualification: the paper does not establish that Derf is optimal in a theoretical sense or that erf is uniquely suited to replace normalization. The function search evaluated 16 candidates; there are infinitely many functions satisfying the four properties. The claim that Derf is "the strongest choice" (Figure 1 caption) is true within the evaluated set, but the paper cannot rule out that some other function — perhaps a rational approximation to erf, or a learned parametric shape — would perform even better. The scaled-tanh approximation experiment (Section 7.2) shows that simple modifications to tanh partially close the gap, suggesting that the performance landscape may have multiple local optima, not a single sharp peak at erf.

Claim 2: "The performance gains of Derf largely stem from its improved generalization rather than stronger fitting capacity."

This claim is strongly supported by the evaluation-mode training loss experiment (Table 13). The pattern Norm < Derf < DyT in training loss, combined with Derf's superior validation performance, requires that Derf generalizes better than normalization — there is no other way to achieve lower validation error from higher training error. The mechanism proposed (point-wise functions serve as implicit regularizers by not adapting to batch-level statistics) is plausible and consistent with the data, but the paper does not directly test it. Alternative explanations — e.g., that erf's specific curvature imposes a beneficial inductive bias on the learned representations, or that it interacts favorably with attention mechanisms — are not ruled out. The claim about why generalization improves is therefore a reasonable hypothesis, not a demonstrated fact.

A minor tension: the paper states that "Derf exhibits stronger fitting power than DyT" because it achieves lower training loss (Section 6.1, Discussion). But both Derf and DyT have higher training loss than normalization — so Derf's "stronger fitting" is only relative to DyT, not in absolute terms. The paper's framing sometimes conflates "Derf generalizes better than normalization" with "Derf fits better than DyT" — these are distinct comparisons along different axes, and the language could be clearer about which comparison is being made.

Claim 3: "Derf consistently outperforms LayerNorm, RMSNorm, and DyT across a wide range of domains."

Supported with the caveat that GPT-2 is an exception where Derf matches but does not outperform LayerNorm. The paper honestly reports this (ΔLN = 0.00 in Table 12) without trying to spin it as a win. The cross-domain transfer from search architectures (vision) to evaluation architectures (speech, DNA, language) is a genuine strength of the experimental design — it provides evidence that the function choice generalizes rather than overfitting to the search task. One might argue that the speech and DNA evaluations are on relatively small-scale models compared to state-of-the-art, and that testing on a GPT-2 of only 124M parameters leaves open the question of whether Derf scales to large language models (GPT-3 scale, Llama scale). The paper acknowledges no such limitation, but it is a reasonable concern: if the improvement on GPT-2 is zero, would it remain zero or become negative at larger scales? The DiT scaling pattern (where DyT actually degrades vs. LN at XL/2 scale while Derf improves) suggests that function choice can interact with model scale in non-monotonic ways, making extrapolation risky.

Missing experiments and baselines:

• No comparison to learned normalization functions. The paper compares Derf to fixed-form normalization (LN, RMSNorm, GN) and to another fixed point-wise function (DyT). It does not compare to methods that learn the normalization transformation, such as adaptive normalization schemes or learned activation functions. This is a defensible choice — the paper's scope is point-wise functions as architectural primitives — but it means the claim "best normalization replacement" is limited to the classes of replacements considered.

• No combination of Derf with normalization. Would applying Derf after LayerNorm help? Would Derf in some layers and LayerNorm in others be optimal? The paper treats this as an either-or substitution but does not explore hybrid architectures, which might capture the optimization benefits of normalization and the regularization benefits of point-wise functions simultaneously.

• No large-scale language model evaluation. GPT-2 (124M) is the only language model tested. Modern LLMs are 1,000× larger, and the paper's findings cannot be assumed to transfer. The DiT results (Section 6, Table 9) show that scale can affect the relative performance of normalization methods — DyT vs. LN flips at XL/2 — so the absence of larger LLM experiments is a genuine gap.

• Limited training budget comparison. The evaluation-mode training loss analysis (Table 13) shows that Derf has higher training loss than LayerNorm at the end of training. This raises the question: does Derf simply need more training epochs to match LayerNorm's fitting capacity, after which it would retain its generalization advantage and widen the performance gap? Or does the fitting gap persist regardless of training duration? The paper does not investigate training-time scaling — all experiments use fixed, architecture-standard training durations — which leaves open whether the gains are robust to different training budgets.

• No statistical significance reporting. For results like ViT-B +0.5% and wav2vec 2.0 −0.02 validation loss, the paper provides no confidence intervals, standard deviations from multiple seeds, or statistical tests. Given the relatively small absolute improvements, it is not obvious that all reported gains are statistically significant. The consistency of the pattern across architectures and tasks provides informal confidence, but formal statistical rigor is absent.

Conditional nature of the claims: The paper's central contributions hold under the conditions tested but come with implicit scope limitations that should be made explicit:

• The claims hold for Transformer architectures — the paper does not test Derf on CNNs, RNNs, or MLP-only architectures. The function search was conducted on ViT and DiT (both Transformers), and the multi-domain evaluation sticks to Transformer variants (wav2vec 2.0 is a Transformer-based speech model; HyenaDNA and Caduceus use structured state-space layers but still employ Transformer-like normalization patterns; GPT-2 is a pure Transformer). The property analysis and design principles may not generalize to architectures where normalization plays a fundamentally different role.

• The claims hold for the specific training recipes used — each model follows a standard, well-tuned training configuration. The paper does not investigate whether Derf is robust to different optimizers, learning rate schedules, batch sizes, or data augmentation strategies. If Derf's generalization advantage stems from implicit regularization, it might interact with other regularizers (weight decay, dropout, data augmentation) in ways not explored here.

• The claims are about final performance, not training dynamics — the paper does not report training curves comparing Derf to LayerNorm. A practitioner switching to Derf might discover that their model converges differently (faster, slower, with different stability properties) even if final performance improves. The property analysis in Section 3 briefly discusses early-training divergence for functions violating the four properties, but no systematic training-dynamics comparison is provided for Derf vs. LayerNorm.

Overall, the experimental evidence strongly supports the paper's primary claims within their scope: point-wise functions can replace normalization, the specific function choice matters materially, erf is the best among those tested, and the gains come from generalization rather than optimization. The paper's transparency about limitations (the GPT-2 parity result, the scaled-tanh negative result, the absence of theoretical explanation for erf's advantage) strengthens credibility. The main weakness is the relatively narrow architectural scope (Transformer-only) and model scale range, which leaves the most impactful potential application — large language model training — unvalidated.

6. Limitations and Trade-offs

6.1 The Function Search Is Conducted on Only Two Vision Architectures at a Single Scale

The assumption or constraint. The entire function search that identified erf as the optimal point-wise function was conducted exclusively on ViT-Base (classification) and DiT-B/4 and DiT-L/4 (image generation), all trained on ImageNet-1K. The paper states this explicitly in Section 4: "We conduct an empirical search on two representative vision architectures: Vision Transformer (ViT-Base) and Diffusion Transformer (DiT-B/4 and DiT-L/4)." The remaining domains — speech (wav2vec 2.0), DNA (HyenaDNA, Caduceus), and language (GPT-2) — were used only for evaluation of the already-selected Derf, not for the search itself. This creates a fundamental selection bias: erf was chosen because it performed best on vision tasks specifically, and its subsequent success on non-vision domains is presented as validation, but this does not establish that erf is the optimal function for those other domains.

The consequence. The search procedure cannot guarantee that erf is the best point-wise function for speech, DNA, or language modeling — it only guarantees that among the 16 candidates tested, erf performs best on ViT and DiT, and also performs well on the other domains when evaluated. A different function might outperform erf on language modeling if the search had been conducted on GPT-2 training runs, or on speech if conducted on wav2vec 2.0. The paper provides no evidence about whether the relative ranking of candidate functions is consistent across modalities. In fact, Table 7 provides indirect evidence that rankings can shift: linearclip(x) achieves a respectable 82.3% top-1 on ViT-B (matching isru(x) and outperforming several more complex candidates), but its DiT-B/4 FID of 66.08 places it near the bottom of the generation ranking. This task-dependence of function performance suggests that the optimal function for classification may not be optimal for generation, and by extension, may not be optimal for speech or DNA modeling. The search's restriction to vision means the paper has found a function that is broadly good across domains, but not necessarily domain-optimal for any domain other than vision.

What evidence exists in the paper. Table 7 (the search results) and Tables 8–12 (the multi-domain evaluation) contain all relevant evidence. The search space itself — 16 candidate functions evaluated on two vision architectures — is documented in Section 4. The paper makes no claim that the search was conducted on non-vision domains, and the multi-domain results simply report Derf's performance without comparing against other candidate functions on those domains (i.e., we don't know how satursin(x) or isru(x) would perform on GPT-2 or wav2vec 2.0). The paper presents the domain transfer as evidence of robustness, which it is, but does not discuss the selection bias inherent in choosing the function from vision tasks.

Mitigation status. The paper does not address this limitation explicitly. The authors frame the multi-domain results as validation that Derf "consistently achieves stronger performance" (Section 6), which is true, but they do not discuss whether a different function might achieve even stronger performance on non-vision domains. Future work would need to replicate the search procedure on each target domain — or, more efficiently, develop a theoretical understanding of why erf works that could predict domain transfer without exhaustive search. The paper's scaled-tanh approximation experiment (Section 7.2) gestures toward this direction but does not provide sufficient theory to predict cross-domain optimality.

---

6.2 No Large-Scale Language Model Evaluation; Scaling Behavior Is Unknown for LLMs

The assumption or constraint. The only language model evaluated is GPT-2 at 124M parameters, trained on OpenWebText. This is the smallest commonly-studied GPT-2 variant, roughly three orders of magnitude smaller than modern production language models (GPT-3 at 175B, Llama 3 at 8B–405B). The paper does not evaluate Derf on any larger language model, nor does it provide evidence about how Derf's performance relative to LayerNorm scales with model size in the language domain. The GPT-2 result itself is the weakest in the paper: Derf achieves validation loss 2.94, matching LayerNorm exactly (ΔLN = 0.00), compared to clear improvements on vision, speech, and DNA.

The consequence. The behavior of Derf at LLM scale is entirely uncharacterized. Several scenarios are plausible and the paper provides no evidence to distinguish among them: (1) Derf continues to match LayerNorm, as on GPT-2, providing a cost-equivalent drop-in replacement with no accuracy penalty; (2) Derf falls behind LayerNorm at larger scales, similar to how DyT underperformed LayerNorm on DiT-XL/2 (20.83 FID vs. 19.94, Table 9) despite matching or outperforming on smaller DiT variants; (3) Derf begins to outperform LayerNorm at larger scales, similar to how the ViT improvement grew from +0.5% on ViT-B to +0.7% on ViT-L (Table 8). The DiT scaling pattern — where function choice interacts with model scale in non-monotonic ways — is particularly concerning: DyT matched LayerNorm on DiT-B/4 (63.94 vs. 64.93) and DiT-L/4 (45.66 vs. 45.91) but fell behind by ~0.9 FID on DiT-XL/2 (20.83 vs. 19.94), while Derf maintained its advantage across all scales. This demonstrates that point-wise function performance is not scale-invariant and that extrapolating from small models to large ones is unreliable. Since GPT-2 at 124M is closer to DiT-B/4 in parameter count than to any production LLM, the GPT-2 results cannot be assumed to predict behavior at billion-parameter scale.

What evidence exists in the paper. Table 12 reports the GPT-2 (124M) validation loss. Table 21 specifies the GPT-2 training configuration, noting that Derf required per-position α initialization tuning ({0.5, 1.0, 2.0, 4.0} for attention layers, {0.1, 0.3, 0.5, 1.0} for other layers) — the only architecture where such tuning was needed, suggesting that language modeling may be more sensitive to point-wise function initialization than other domains. The DiT scaling results across B/4, L/4, and XL/2 in Table 9 provide indirect evidence about scale interactions, but these are for image generation, not language modeling. The paper provides no language model scaling curve, no experiments with larger GPT-2 variants (355M, 774M, 1.5B), and no evaluation on any modern LLM architecture (Llama, Qwen, etc.). The absence of evidence is itself the limitation.

Mitigation status. The paper does not acknowledge this as a limitation. The GPT-2 result is presented neutrally — "Derf achieves comparable performance to LN, while clearly outperforming DyT" (Section 6) — without noting that "comparable" means "no improvement" in the one domain most relevant to the largest current ML training budgets. The paper does not suggest future work on LLM-scale evaluation, despite this being the most impactful potential application. A practitioner considering Derf for LLM pretraining would need to run their own scaling experiments to determine whether the vision and DNA improvements translate to their setting, with the GPT-2 parity result providing only weak evidence either way.

---

6.3 The Mechanism Behind erf's Advantage Over tanh Remains Unexplained; No Theoretical Guidance for Future Design

The assumption or constraint. The paper establishes empirically that erf outperforms tanh (and 14 other functions) on the search architectures, and validates that this advantage persists across domains. However, Section 7.2 explicitly demonstrates that the most obvious hypothesis — that erf simply has a different slope or width than tanh — is incorrect: fitting tanh(εx) to minimize L1 distance from erf(x) (optimal ε ≈ 1.205) improves performance over standard tanh but does not close the gap to erf (Table 16). The paper states this as a finding but does not propose an alternative mechanism:

"This indicates that simply scaling tanh(x) is insufficient to match the behavior or performance of erf(x)." (Section 7.2)

No further analysis of why erf works better is provided — no gradient flow analysis, no loss landscape visualization, no study of how erf's specific curvature interacts with attention or FFN layers, no comparison of the functions' Taylor expansions or their behavior under typical activation distributions.

The consequence. The function search in Section 4 is purely empirical: it identifies erf as the best among 16 candidates without providing a framework for predicting whether other, untested functions might be better still. The design space of point-wise functions satisfying the four property constraints is infinite — smooth S-shaped curves that are zero-centered, bounded, center-sensitive, and monotonic form a continuous family that cannot be exhaustively searched. Without understanding which geometric property of erf's shape matters, future work cannot efficiently search larger candidate spaces or design novel functions with guaranteed improvements. The scaled-tanh experiment rules out one hypothesis (slope/width) but replaces it with nothing. This leaves derf as a point solution — a specific function that works well for reasons unknown — rather than a design principle that could generate further improvements.

This is not merely an academic concern. The gap between the best candidate (erf, 82.8%) and the second-best (tanh, 82.6%) on ViT-B is only 0.2 percentage points — small enough that a slightly different candidate, not included in the 16, might close or reverse the gap on some tasks. The gap is larger on DiT (63.23 vs. 63.71 FID on -B/4), suggesting that erf's advantage is real but possibly domain-dependent. Without a mechanistic explanation, practitioners cannot assess whether erf's advantage would hold on their specific task, architecture, and scale, or whether a different S-shaped function might be better suited.

What evidence exists in the paper. Table 7 provides the raw search results. Table 16 shows the scaled-tanh approximation experiment. Figure 5 visualizes all candidate functions — their similarity is visually striking, underscoring how small geometric differences produce measurable performance differences. Appendix B provides function definitions but no analysis of their mathematical relationships. The paper's entire theoretical contribution consists of the four property constraints (Section 3), which are necessary conditions for viability but not sufficient to discriminate among viable candidates. The mechanism by which erf improves generalization over tanh — the paper's central finding from Section 6.1 — is explained only at the level of "point-wise functions have limited adaptability which acts as a regularizer," which applies equally to erf and tanh and does not explain the difference between them.

Mitigation status. The paper does not claim to explain erf's advantage and is transparent about the negative result from the tanh-approximation experiment. The limitation is in what the paper does not provide rather than in what it claims falsely. However, the paper also does not flag this explanatory gap as a limitation or suggest future theoretical work to close it. The Discussion in Section 6.1 attributes Derf's overall advantage over normalization to "improved generalization rather than stronger fitting capacity" but does not attempt to decompose how much of the generalization improvement comes from being a point-wise function (shared with DyT) versus being specifically erf (unique to Derf). The fitting-capacity comparison shows Norm < Derf < DyT in training loss, which only explains why Derf beats DyT (better fitting); it does not explain why Derf beats normalization (better generalization). The implicit regularization hypothesis explains the latter uniformly for all point-wise functions but cannot account for Derf's advantage over DyT — that requires an additional, unexplained mechanism.

---

6.4 The Training Recipe Sensitivity of Point-Wise Functions Is Not Systematically Characterized

The assumption or constraint. Throughout the paper, Derf (and DyT) is evaluated using the default training recipes of each architecture — the standard hyperparameters, optimizers, learning rate schedules, and data augmentation strategies that were designed for and tuned with normalization layers. The paper makes no systematic study of whether Derf is robust to changes in these recipes, or whether the reported improvements would hold under different optimization choices. The paper in fact encountered recipe sensitivity in several places but treated these as isolated fixes rather than systematic phenomena:

• For DiT, the paper found that "the default learning rate is suboptimal for the models in this work" and swept {1e-4, 2e-4, 4e-4} for all methods including LayerNorm (Section 4, "Quantitative evaluation" paragraph). This means the DiT results come from a modified training recipe, not the original default.
• For DiT, the paper also found that "the zero initialization negatively affects the performance of Derf models and other point-wise function models," so it removed zero initialization for those variants while retaining it for LayerNorm (Section 4). This is an asymmetric change that improves point-wise functions but not LayerNorm.
• For GPT-2, the paper found that α initialization required per-position tuning — sweeping different α values for attention-layer point-wise functions versus other point-wise functions — and "report the best validation loss" (Table 21). This is a hyperparameter search on the validation set that was not applied to the LayerNorm baseline.

The consequence. The reported improvements of Derf over LayerNorm may partially reflect the fact that Derf received more hyperparameter attention than the baselines. The DiT learning rate sweep was applied to all methods (fair), but the removal of zero initialization was asymmetric (only point-wise functions benefit), and the GPT-2 α sweep was not mirrored by a corresponding sweep of LayerNorm-specific parameters (e.g., ε or initialization). This asymmetry is not necessarily disqualifying — the paper's goal is to demonstrate that point-wise functions can work, and showing what recipe changes are needed is part of that demonstration. However, it complicates the claim that Derf is a simple "drop-in replacement." In practice, a practitioner switching from LayerNorm to Derf may need to re-tune learning rates, remove zero initializations, and potentially adjust per-layer α initializations to recover the reported gains.

More broadly, the paper provides no evidence about whether Derf's regularization advantage (Section 6.1) interacts with other regularizers. If Derf functions as an implicit regularizer by limiting adaptability to batch statistics, then combining Derf with other regularizers (weight decay, dropout, data augmentation, stochastic depth) might produce different interactions than combining those same regularizers with LayerNorm. The paper trains all models with their architecture-standard regularization (the ViT recipe in Table 17 includes mixup 0.8, cutmix 1.0, random erase 0.25, label smoothing 0.1, drop path 0.15–0.5, and EMA 0.9999), but does not ablate these to see whether Derf's improvement persists under different regularization strengths. A plausible failure mode: if Derf already provides strong implicit regularization, adding explicit regularizers on top might over-regularize and hurt performance, while LayerNorm — which provides less implicit regularization — might benefit more from the same explicit regularizers. The paper provides no data to assess this.

What evidence exists in the paper. Table 17, 18, 19, 20, 21 document the training configurations and note the DiT zero-initialization removal and the GPT-2 α sweep. The evaluation-mode training loss in Table 13 provides indirect evidence about the regularization hypothesis but does not test interactions with explicit regularizers. No ablation of weight decay, data augmentation strength, or stochastic depth is provided for Derf vs. LayerNorm. The paper does not report training curves, making it impossible to assess whether Derf converges faster, slower, or with different stability properties than LayerNorm.

Mitigation status. The paper does not frame recipe sensitivity as a limitation. The DiT changes are described as necessary to achieve fair comparison ("we go through three learning rates... for all models... and report the best result"), which is a reasonable fairness argument. The GPT-2 α sweep is acknowledged ("We try multiple combinations of these initialization settings and report the best validation loss," Table 21 caption) but not discussed as a potential confound. The paper does not suggest systematic study of recipe robustness as future work. For a practitioner, the practical consequence is that adopting Derf may require non-trivial hyperparameter re-tuning — the paper demonstrates that Derf can work well, but not that it works well out of the box with existing training configurations.

---

6.5 The Four Property Constraints May Not Be Complete; The Design Space May Exclude Functions That Could Outperform Derf

The assumption or constraint. The entire function search (Section 4) is constrained to functions satisfying four properties: zero-centeredness, boundedness (natural or through clipping), center sensitivity (nonzero derivative near zero), and monotonicity. These properties were identified through controlled experiments in Section 3 that showed functions violating them either underperform or fail to converge entirely. The paper treats these as necessary conditions for viability, which constrains the search space to S-shaped, saturating functions. However, the property analysis only tests necessity — it shows that violating any property tends to hurt performance — but does not test whether the conjunction of these four properties is sufficient or whether other, equally important properties exist that were not tested. The space of point-wise functions is infinite, and the property constraints reduce it to a continuous family of S-shaped curves. By restricting to this family, the search may exclude functions that violate one or more properties but could be made to work through architectural modifications (e.g., different initialization, different learning rates, different placement within the block).

The consequence. The paper cannot claim that Derf is the best possible point-wise function — only that it is the best among S-shaped functions satisfying the four properties. More subtly, the four properties may interact with each other in ways not captured by the independent-property experiments in Section 3. For example, the boundedness analysis (Section 3.2) showed that some unbounded functions can converge, just with slightly worse performance than their clipped versions (arcsinh: 82.2% unclipped vs. 82.4% clipped; logsign: 82.2% vs. 82.4%). This raises the possibility that a carefully designed unbounded function with the right growth rate might combine the benefits of unboundedness (no hard saturation, enabling the model to represent very large activations when needed) with the stability benefits of the other three properties, potentially outperforming bounded functions. The paper explores growth rate limitations (Table 4) but does not search over unbounded functions with deliberately chosen growth rates as candidates — all candidates in Table 7 are bounded to [-1, 1], either naturally or through clipping.

Similarly, the center sensitivity analysis (Section 3.3) showed that small dead zones (λ ≤ 0.5) are well-tolerated (erf drops only from 82.6% to 82.5%). A function with a very slight flat region near zero might provide additional robustness to noise or small activation variations while maintaining trainability — the paper does not explore this regime. The search is constrained to functions with maximal center sensitivity (no dead zone), which may be locally optimal but not necessarily globally optimal in the space of all possible sensitivity profiles.

What evidence exists in the paper. The property analysis (Section 3, Tables 1–6) establishes the four properties as necessary, with quantitative thresholds for acceptable deviation. The function search (Section 4, Table 7) evaluates 16 functions satisfying these constraints. The boundedness analysis documents that clipping unbounded functions improves them but does not explore whether the clipping threshold itself is optimal — the functions are simply clipped to [-1, 1] without sweep over alternative ranges. The paper provides no analysis of whether the four identified properties are the only ones that matter, or whether the conjunction of four properties is sufficient to guarantee viability. No experiment tests whether functions violating one property but compensating with another (e.g., unbounded but with very strong center sensitivity to control variance) could succeed.

Mitigation status. The paper does not claim the four properties are sufficient or complete. The systematic approach — identify constraints, search within them — is methodologically sound and represents genuine progress over prior ad-hoc function selection (e.g., DyT choosing tanh based on qualitative similarity to LayerNorm). However, the paper also does not discuss the possibility that these constraints might be overly restrictive, or that the "optimal" function might lie outside them. Future work could investigate: (1) whether any of the four properties can be relaxed without performance degradation through compensatory mechanisms (e.g., careful initialization, adaptive learning rates); (2) whether additional properties exist that further constrain the space and might identify functions better than erf; (3) whether a continuous parameterization of the S-shaped function family could enable gradient-based optimization of the function shape itself, rather than discrete search over candidates.

---

6.6 No Combined Evaluation of Derf with Partial or Interleaved Normalization

The assumption or constraint. The paper evaluates Derf as a complete replacement for all normalization layers in each architecture: "To integrate Derf into a transformer-based architecture, we replace each normalization layer with a corresponding Derf layer" (Section 5). This all-or-nothing substitution is compared against all-LayerNorm, all-RMSNorm, and all-DyT baselines. The paper never evaluates hybrid architectures where some layers use normalization and others use Derf, or where Derf is applied in sequence with normalization (e.g., Derf after LayerNorm, or LayerNorm after Derf). The implicit assumption is that point-wise functions and normalization are substitutes, not complements.

The consequence. This is a missed opportunity to combine the complementary strengths identified in Section 6.1: normalization layers provide stronger fitting capacity (lower training loss), while point-wise functions provide stronger generalization (lower validation error despite higher training loss). A hybrid architecture that uses LayerNorm in some layers (perhaps early layers where optimization stability is critical) and Derf in others (perhaps later layers where overfitting to training statistics is more harmful) might achieve better overall performance than either pure approach. Similarly, applying Derf in addition to LayerNorm (rather than instead of it) might retain normalization's optimization benefits while adding Derf's regularization benefit, potentially achieving the best of both worlds.

The paper's own analysis demonstrates that Derf and normalization have different, potentially complementary, error characteristics. Table 13 shows the training loss ordering Norm < Derf < DyT consistently, but the gap between Norm and Derf varies dramatically by task: it's large on classification (ViT-B: 0.2623 vs. 0.2681, a 2.2% relative increase) and near-zero on generation (DiT-B: 0.1531 vs. 0.1533, a 0.13% increase). This task-dependence suggests that the optimal allocation of normalization vs. point-wise functions might vary across layers within a single model — some layers might benefit more from normalization's fitting capacity, others from Derf's regularization. The paper provides no framework for making such per-layer allocation decisions.

What evidence exists in the paper. The evaluation-mode training loss experiment (Table 13) provides the raw data showing that Norm and Derf have complementary strengths. The paper's discussion in Section 6.1 explicitly frames the tradeoff: normalization layers "allow dynamic fitting of activation distributions" (better optimization) while point-wise functions "limit adaptability" (better generalization). But this framing leads to a conclusion about choosing between them rather than combining them. No experiment tests any hybrid configuration. The architecture descriptions (Section 6, Appendix C) confirm that all normalization layers are replaced identically.

Mitigation status. The paper does not acknowledge this as a limitation or suggest hybrid architectures as future work. The "drop-in replacement" framing — Derf as a direct substitute for LayerNorm — is clean and practically useful, but it may have prevented exploration of more nuanced architectures that could outperform the pure approaches. Given the paper's careful analysis of the fitting vs. generalization tradeoff, the absence of any hybrid experiments is a notable gap: the analysis provides the diagnostic tools to understand why a hybrid might work but never tests whether it actually does. This is likely the most actionable direction for immediate follow-up work, since it requires no new function search or theoretical advances — only architectural experimentation combining the components already characterized in this paper.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around normalization in deep learning from a defensive posture—"how can we remove normalization without hurting performance?"—to an offensive one: "can we improve performance by replacing normalization with something simpler?" This is a qualitative change in ambition that reframes normalization not as a regrettable but necessary architectural tax, but as a potentially suboptimal design choice that can be improved upon.

The magnitude of this shift should be characterized precisely. This is not a paradigm shift in the Kuhnian sense—the paper does not overthrow a theoretical framework or introduce a new class of models. Rather, it is a reframing with practical teeth. Prior work treated normalization-free methods as cost-reduction plays: remove LayerNorm to save the memory accesses and synchronization overhead of computing per-token statistics, and accept a small accuracy penalty (or at best, break even). DyT (Zhu et al., 2025) established that breaking even is possible, which was itself a significant finding. This paper demonstrates that you can do better than break even—consistently, across modalities, with a drop-in replacement that requires no cross-element communication whatsoever.

The most landscape-changing empirical finding is the fitting-vs-generalization decomposition in Section 6.1 (Table 13). The consistent ordering Norm < Derf < DyT in training loss, combined with Derf's superior validation performance, establishes that normalization layers are stronger optimizers but weaker regularizers than well-designed point-wise functions. This is not an incremental refinement of prior knowledge—it inverts the conventional diagnosis. The normalization literature has overwhelmingly studied normalization as an optimization aid (stabilizing gradients, smoothing the loss landscape, enabling higher learning rates). This paper provides evidence that normalization may actually hurt generalization by overfitting to batch-level activation statistics, and that removing this adaptivity—far from being a cost to be mitigated—can be a source of improved test-time performance.

The paper also resolves a latent contradiction in the normalization-free literature. Prior work split into two camps: architectural methods (He and Hofmann, 2024; Jha and Reagen, 2024) that redesigned blocks from scratch to operate without normalization, and point-wise methods (Zhu et al., 2025) that provided drop-in replacements. The architectural camp achieved strong results but required per-architecture redesign; the point-wise camp achieved simplicity but only parity. This paper demonstrates that the point-wise approach can achieve both simplicity and superiority, unifying the two desiderata and making the architectural redesign approach less attractive for most use cases. The fact that Derf matches or exceeds LayerNorm, RMSNorm, and GroupNorm across vision, generation, speech, DNA, and language (Tables 8–12, 22–26) without any per-domain architectural changes establishes point-wise functions as a domain-general primitive, which architectural methods have not achieved.

A subtle but important shift: the paper makes "what is the optimal S-shaped function shape?" a legitimate research question with measurable consequences. Before this work, function shape selection was either ad-hoc (DyT chose tanh based on qualitative similarity to LayerNorm's input-output mapping) or motivated by theoretical properties of gradient propagation (e.g., SELU's self-normalizing property). This paper shows that among functionally similar S-shaped curves—all zero-centered, bounded, monotonic, and center-sensitive—performance varies by 1.4 percentage points on ViT classification and 7 FID points on DiT generation (Table 7). This legitimizes fine-grained function design as an axis of architectural innovation, analogous to how activation function design became a research subfield after the demonstration that ReLU, LeakyReLU, GELU, and Swish produce meaningfully different results despite their apparent similarity.

The paper also redirects research attention in a specific way: it makes verifier/model robustness less central (since there is no verifier—the "regularization" comes from the architectural constraint, not from an external scoring function) and instead focuses attention on the geometric properties of element-wise transformations as a design degree of freedom. This is a distinct research axis from the test-time compute scaling work that has dominated recent attention. For the normalization subfield specifically, the paper suggests that future effort should go into understanding what shape properties drive the erf vs. tanh performance gap (Section 7.2 shows it is not simply slope, but does not identify what it is), rather than into developing ever-more-elaborate schemes for recovering normalization's optimization benefits.

Follow-Up Research This Work Enables

Theoretical analysis of why erf outperforms tanh. The paper's most tantalizing open question is also its most actionable for theory-minded researchers. Section 7.2 demonstrates that fitting tanh(εx) to minimize L1 distance from erf(x) (optimal ε ≈ 1.205) improves performance—82.6% → 82.7% on ViT-B, 45.48 → 45.13 FID on DiT-L—but does not close the gap to erf itself (82.8%, 43.94 FID). This rules out the simplest hypothesis (erf is just a scaled version of tanh) and leaves the mechanism unexplained. A strong follow-up would analyze the higher-order differences between erf and tanh: their Taylor expansions (erf is odd with only odd-order terms, tanh shares this property but with different coefficients), their behavior in the tail (erf approaches its asymptote as exp(-x²)/x, tanh as 2exp(-2x)), or their response to typical activation distributions observed during training. Concretely, one could instrument a ViT-B trained with LayerNorm to record activation histograms at each layer, then compute the expected gradient norm under erf vs. tanh vs. scaled-tanh transformations applied to those distributions. If erf produces more uniform gradient flow across the activation range, or better preserves the relative ordering of moderate-magnitude activations, that would provide a mechanistic explanation testable by designing new functions that exaggerate the beneficial property. The paper's property analysis framework (Section 3) provides the experimental template for such studies—vary one geometric property while holding others constant and measure the effect.

Gradient-based optimization of the function shape itself. The paper's search over 16 discrete candidates (Table 7) is inherently limited—the space of smooth, S-shaped, zero-centered, bounded, monotonic functions is continuous and infinite-dimensional. A natural next step is to parameterize the function shape itself with a flexible learnable form (e.g., a small neural network, a spline with learnable control points, or a weighted combination of basis functions) and optimize it jointly with the model parameters, or meta-learn it across tasks. The key challenge is ensuring the four property constraints (zero-centeredness, boundedness, center sensitivity, monotonicity) are maintained during optimization, which could be enforced through architectural constraints (e.g., parameterizing the function as clip(g(x), -1, 1) where g(x) is an odd, monotonic, center-sensitive base) or through regularized objectives. A strong experiment: take the 16 candidate functions from Table 7 as initialization points for a continuous optimization over function space (parameterized as a weighted sum of these basis functions), and see whether gradient-based search discovers functions that outperform erf, or whether erf is a local optimum that gradient methods rediscover. A negative result—continuous optimization cannot beat erf—would be as informative as a positive one, suggesting that erf represents a fundamental limit rather than just the best of a small discrete set.

Hybrid Derf-normalization architectures that combine fitting capacity with regularization. The paper's diagnostic in Section 6.1 establishes that normalization layers provide stronger fitting (lower training loss) while point-wise functions provide stronger generalization (lower validation error relative to training loss). This naturally suggests that combining them—rather than choosing one or the other—might outperform either pure approach. A specific experimental design: take a ViT-B and replace LayerNorm with Derf in only a subset of layers, varying which layers use normalization and which use Derf. The hypothesis is that early layers (closer to the input) might benefit from normalization's stronger optimization to establish good representations, while later layers (closer to the output) might benefit from Derf's regularization to prevent overfitting. Alternatively, one could apply Derf in addition to LayerNorm rather than instead of it—a two-stage transformation Derf(LayerNorm(x)) or LayerNorm(Derf(x))—to see whether the combination captures the benefits of both. The paper already has the measurement protocol (evaluation-mode training loss from Table 13) to diagnose whether a hybrid improves fitting capacity, generalization, or both. A strong follow-up would produce a "mixing ratio" curve showing how performance varies as the fraction of layers using Derf increases from 0 to 1, identifying an optimal interior point if one exists.

Large-scale language model evaluation to determine whether Derf's benefits scale. The GPT-2 (124M) result in Table 12 is the paper's most ambiguous finding: Derf matches LayerNorm at 2.94 validation loss (ΔLN = 0.00) but does not outperform it. This could mean that language modeling is a domain where point-wise functions provide no advantage, or it could mean that 124M parameters is too small for the regularization benefit to manifest—perhaps at billion-parameter scales where overfitting to training statistics becomes a larger concern, Derf would pull ahead. The DiT scaling results in Table 9 provide some evidence for the latter hypothesis: on DiT-XL/2, DyT actually underperforms LayerNorm (20.83 vs. 19.94 FID), while Derf maintains and widens its lead (18.92 vs. 19.94), suggesting that function choice interacts with scale. A critical follow-up would train a Llama-style architecture at 1B, 3B, and 7B parameters with both LayerNorm and Derf, using a controlled training budget (identical data, identical hyperparameters except for per-layer α initialization which Table 21 shows may need tuning), and measure both validation loss and downstream task performance. The key question: does the ΔLN for language modeling remain zero at all scales, or does it become positive at larger scales? A persistent zero result would establish that language modeling is fundamentally different from vision, speech, and DNA in ways that make point-wise normalization replacements neutral rather than beneficial. A positive result at larger scales would make Derf immediately relevant to LLM training, where even a small per-layer efficiency gain compounds across hundreds of layers and trillions of training tokens.

Characterization of training dynamics under Derf vs. LayerNorm. The paper reports final performance but provides no training curves, convergence analysis, or stability diagnostics for Derf compared to normalization layers. The property analysis in Section 3 studies failure modes—functions that diverge during training—but does not characterize the trajectory of functions that succeed. Several practically important questions are unanswered: does Derf converge faster or slower than LayerNorm? Is it more or less sensitive to learning rate? Does it require the same number of warmup steps? Does the gradient norm evolution differ in ways that would affect large-scale training stability? A detailed training dynamics study—plotting loss curves, gradient norm distributions, and activation statistics over the course of training for ViT-B, DiT-B/4, and GPT-2 with LayerNorm, DyT, and Derf—would provide the practical guidance practitioners need to adopt Derf without discovering training instabilities the hard way. The property analysis provides the right methodology (Section 3) but applies it only to function viability, not to function quality within the viable regime.

Extending the property analysis to identify additional relevant function properties beyond the first four. The paper's four properties—zero-centeredness, boundedness, center sensitivity, monotonicity—were identified through controlled experiments that showed functions violating them underperform or fail. But the observed performance spread among functions satisfying all four properties (Table 7: 82.8% to 81.4% top-1) demonstrates that these are necessary but not sufficient conditions for optimal performance. There must be additional geometric properties that discriminate among the viable candidates. Candidate properties to test: tail heaviness (how quickly the function approaches its asymptote—erf approaches as exp(-x²), tanh as exp(-2x), arctan as 1/x); linear-region width (the input range over which the function is approximately linear, which differs across erf, tanh, and satursin even when their center slopes are matched); curvature (the second derivative in the transition region, which affects how gradients flow for activations moving between the linear and saturating regimes); and higher-order smoothness (erf is infinitely differentiable, while clipped functions have discontinuous derivatives at the clipping boundaries). Each could be tested using the methodology from Section 3: take a base function satisfying all four properties, systematically vary the candidate property through parametric transformations, and measure the effect on ViT-B accuracy. Identifying which additional properties matter would transform the function search from an empirical fishing expedition into a principled optimization over a characterized design space.

Practical Applications and Downstream Use Cases

Drop-in replacement for normalization in standard Transformer training pipelines. The most immediate application is straightforward: replace nn.LayerNorm with Derf in any existing Transformer codebase. The paper provides evidence that this substitution works across ViT, DiT, wav2vec 2.0, HyenaDNA, Caduceus, and GPT-2 (Tables 8–12), with consistent improvements or parity and only minor recipe adjustments (remove zero initialization for DiT, tune α initialization for GPT-2). The computational cost is identical or slightly lower than LayerNorm—at 15K hidden dimension, forward pass times are equal (0.42 ms), backward pass times are equal or slightly faster for Derf (0.73 vs. 0.74 ms, Table 27)—so this is a pure accuracy improvement at zero or negative cost. For an organization training ViT-Large models on ImageNet-1K, the +0.7% top-1 improvement over LayerNorm (Table 8) translates to a meaningful reduction in error rate (from 16.9% to 16.2%, a ~4% relative error reduction) without any change to training time or infrastructure. The paper's implementation (available at the linked repository) provides a reference Triton kernel that practitioners can integrate directly.

Improved image generation quality in diffusion models without additional sampling cost. The DiT results (Table 9) are the paper's strongest quantitative improvements in absolute terms: Derf reduces FID by −1.70 on DiT-B/4, −1.97 on DiT-L/4, and −1.02 on DiT-XL/2 relative to LayerNorm. These are substantial FID improvements for architectural changes that don't affect sampling cost—unlike classifier-free guidance or increased sampling steps which trade compute for quality, Derf improves the model itself. For production diffusion models where FID is a key quality metric, adopting Derf provides a "free" quality boost. The DiT-XL/2 result is particularly notable because it shows Derf maintaining advantage at larger scales where DyT actually degrades relative to LayerNorm (20.83 vs. 19.94 FID). The modification required—removing zero initialization and sweeping learning rate (Table 18)—is a one-time cost during training recipe setup, not an ongoing inference cost.

DNA sequence modeling where small accuracy improvements have outsized scientific impact. The DNA modeling results (Table 11) show Derf improving HyenaDNA by +0.5% and Caduceus by +0.4% average accuracy over GenomicBenchmarks. In genomic applications, these small absolute improvements can translate to meaningful downstream consequences—identifying a regulatory variant that a baseline model misses, or reducing false positives in variant effect prediction. Derf's drop-in nature is particularly valuable here because genomic models often have specialized architectures (HyenaDNA uses long-convolution operations, Caduceus uses bidirectional state-space layers) where normalization choices are not obvious; the paper's finding that Derf works regardless of whether the original model used LayerNorm or RMSNorm (HyenaDNA +0.5% over LN, Caduceus +0.4% over RMSNorm; Table 25 shows Derf outperforms both normalization variants on both architectures) means practitioners can adopt Derf without needing to reason about which normalization variant is best for their architecture.

On-device and edge deployment where removing cross-element communication reduces memory and synchronization overhead. The paper's efficiency analysis (Appendix F, Figure 13) shows that Derf and LayerNorm have comparable runtime at 15K hidden dimension, but the architecture of the computation is fundamentally different: LayerNorm requires gathering mean and variance across the channel dimension, which involves reduction operations and synchronization barriers, while Derf applies independent scalar operations to each element. On hardware with constrained memory bandwidth or limited parallel reduction support (edge GPUs, mobile accelerators, FPGA deployments), this architectural difference may translate to larger wall-clock savings than the Triton benchmarks on server GPUs suggest. The paper does not benchmark on edge hardware, but the structural property—no cross-element communication, fully element-wise—is architecturally favorable for deployment scenarios where memory access patterns dominate runtime. A practitioner deploying a ViT-based classifier on a mobile device could replace all LayerNorm layers with Derf and potentially see improved throughput even if the FLOP count is identical, because the operation is embarrassingly parallel with no reduction steps.

When to Prefer This Method

The paper implicitly defines a decision rule through its experimental scope and results, though it does not state one explicitly. The evidence supports the following:

• Prefer Derf when you are training a Transformer architecture (ViT, DiT, wav2vec 2.0, state-space DNA models, GPT-style language models) and want to either: (a) improve final task performance with zero or negative computational cost overhead, or (b) simplify the architecture by removing cross-element communication requirements. The paper demonstrates improvements or parity across all tested domains except GPT-2 where performance is equal, and the computational cost is identical or lower than LayerNorm (Appendix F). The only architecture-specific adjustments needed are removing zero initialization for DiT-style models and potentially tuning α initialization for language models (Table 21).

• Prefer traditional normalization (LayerNorm or RMSNorm) when you need maximal compatibility with existing training infrastructure, cannot tolerate any hyperparameter re-tuning, or are training a non-Transformer architecture not tested in the paper. The paper's case for Derf on Transformers is strong, but it provides no evidence for CNNs (where BatchNorm remains dominant), RNNs, or MLP-only architectures. If your training pipeline is brittle to architectural changes and the marginal improvement from Derf (+0.5% top-1 on ViT-B, −1.97 FID on DiT-L/4) does not justify the risk of unexpected training dynamics, sticking with the well-characterized normalization baseline is prudent.

• Prefer Derf over DyT in all tested settings. The paper consistently shows Derf outperforming DyT across ViT (82.8% vs. 82.5%), DiT (63.23 vs. 63.71 FID on -B/4, 43.94 vs. 45.66 on -L/4, 18.92 vs. 20.83 on -XL/2), wav2vec 2.0 (1.93 vs. 1.95), DNA (+0.4–0.5% over DyT), and GPT-2 (2.94 vs. 2.97 validation loss). Since the computational costs of Derf and DyT are identical (Table 27), there is no reason to choose DyT over Derf in any domain tested. The DyT baseline is strictly dominated.