FNet: Mixing Tokens with Fourier Transforms

ArXiv: 2105.03824

🎯 Pitch

A standard, unparameterized Fourier Transform—just a simple 2D FFT—replaces self-attention in a Transformer encoder and still achieves up to 97% of BERT’s GLUE accuracy while training 80% faster on GPUs. On long-sequence benchmarks, this zero-parameter mixing mechanism matches the accuracy of the most sophisticated efficient Transformers yet runs faster than all of them.

---

1. Executive Summary

This paper studies whether the self-attention sublayers in Transformer encoder architectures can be replaced with simpler, faster linear transformations for token mixing, using BERT models pretrained on C4 and evaluated on GLUE and the Long Range Arena (LRA) benchmark. The authors introduce FNet, a model that replaces the self-attention sublayer with a standard, unparameterized Fourier Transform (applying a 2D DFT along the sequence and hidden dimensions, then keeping only the real part). FNet achieves 92–97% of BERT's accuracy on GLUE while training 80% faster on GPUs and 70% faster on TPUs at 512-length inputs, and on the LRA benchmark it matches the accuracy of the most accurate efficient Transformers while outpacing all of them on GPUs — establishing that unstructured, parameter-free token mixing competes with learned attention only when the feed-forward sublayers remain to perform nonlinear processing.

2. Context and Motivation

The Core Problem: Self-Attention Is the Bottleneck in Transformer Encoders

The fundamental question this paper tackles is whether the self-attention mechanism — widely considered the essential innovation of the Transformer architecture — is actually necessary for competent text understanding, or whether it can be replaced with something dramatically simpler and faster. This matters because self-attention is both the defining feature of Transformers and their primary computational bottleneck.

The standard self-attention mechanism (Vaswani et al., 2017) scales quadratically with sequence length: for an input of $N$ tokens, computing attention requires $O(N^2)$ operations and $O(N^2)$ memory. As the authors note in Table 1, a single self-attention layer in BERT-Base performs $2N^2 d_h + 4N d_h^2$ operations per forward pass (where $d_h$ is the hidden dimension). For a 512-token input, this is roughly 98 GFLOPS per example — operations that are dominated by the $N^2$ term at longer sequence lengths. This quadratic scaling is not merely a theoretical concern: Table 4b shows that the standard Transformer runs out of memory (OOM) when sequence lengths reach 4096 on 8 V100 GPUs, making it impractical for tasks requiring long-range dependencies like document classification, genome sequence analysis, or high-resolution image processing.

The paper frames this problem with unusual directness. Rather than asking "how can we make attention more efficient?" — the question that spawned the entire efficient Transformer literature — they ask the more radical question: "Do we really need the flexibility, and associated cost, of attention?" (Section 2.2). This reframing is the paper's central intellectual move. It challenges the pervasive intuition, backed by numerous probing studies (Tenney et al., 2019; Vig and Belinkov, 2019; Clark et al., 2019; Voita et al., 2019), that attention's token-dependent, context-aware mixing patterns are what make Transformers work. If simpler, even parameter-free mechanisms can achieve comparable results, then the field may have over-invested in the complexity of attention.

The Gap: No Systematic Study of Radical Simplification

Prior to this work, the literature had explored two broad directions for addressing attention's computational cost, but neither had directly investigated whether attention could be replaced wholesale by simple, unstructured linear transformations:

Direction 1: Sparsifying or approximating attention while preserving its structure. Most "efficient Transformer" work (Child et al., 2019; Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020; Wang et al., 2020; Kitaev et al., 2020; Roy et al., 2021; Katharopoulos et al., 2020; Choromanski et al., 2021; Peng et al., 2021) starts from the premise that attention is fundamentally valuable and seeks to reduce its cost while keeping its essential character. These approaches use sparse attention patterns (limiting which tokens attend to which others), low-rank approximations, or kernel-based linearization of the softmax. While many achieve $O(N)$ theoretical complexity, the paper points out a critical practical issue:

"some of those works achieve O(N) scaling of attention, this complexity often hides large constants, which make them less scalable in practice than FNet" (Section 1).

For example, models like Longformer, ETC, and BigBird are $O(N)$ in the input length but quadratic in the number of global tokens, and those global tokens must be numerous enough to maintain accuracy. The result is that many efficient Transformers remain memory-intensive and do not actually achieve the speedups their asymptotic complexity suggests — as the LRA benchmark results from Tay et al. (2021a) demonstrate, the Performer (the fastest efficient Transformer) still uses 2.4× more memory than FNet at 8192 sequence length on GPUs (Table 4b).

Direction 2: Questioning whether attention weights need to be learned at all. A smaller but important thread of work had chipped away at the assumption that attention must be fully learned and token-dependent. Tay et al. (2020a) introduced the Synthesizer model, which replaced token-token dot-product attention with learned attention weights that are either parameterized directly or conditioned only on individual tokens (not token pairs). The Random Synthesizer variant uses fixed random attention patterns. Their finding — that learned but token-independent attention weights are highly expressive — directly inspired the authors' Linear and Random baselines. You et al. (2020) pushed further, replacing encoder attention weights with unparameterized Gaussian distributions in machine translation, showing minimal degradation provided cross-attention remained learnable. Raganato et al. (2020) found that replacing all but one attention head per layer with fixed, non-learnable positional patterns caused little to no accuracy loss.

However, these works stop short of completely eliminating the attention mechanism. The Synthesizer still uses multiple attention heads with softmax projections. The Gaussian attention work still computes weighted token mixtures using the attention architecture. Raganato et al. still retain one learned head per layer. None of them asks: what if we throw out the entire self-attention sublayer and replace it with a single, fixed, unparameterized linear transformation?

The paper explicitly positions itself in this gap:

"we investigate whether simpler token mixing mechanisms can wholly replace the relatively complex self-attention layers in Transformer encoder architectures." (Section 1)

Direction 3: MLP-Mixer (contemporaneous work). Tolstikhin et al. (2021) introduced MLP-Mixer for vision, which replaces self-attention with MLPs applied alternately to the spatial (token) and channel (hidden) dimensions. This is structurally very similar to the Linear baseline in this paper, and the authors acknowledge the connection. The key difference: MLP-Mixer uses learned parameters for spatial mixing (a dense linear layer applied along the sequence dimension), whereas FNet's Fourier Transform has no learnable parameters at all. The paper frames this as a more extreme test of whether learned mixing is necessary:

"in contrast to MLP-Mixer, FNet has no learnable parameters that mix along the spatial dimension." (Section 1)

Why This Gap Matters: Practical and Scientific Stakes

The importance of this investigation has both practical and scientific dimensions:

Practical stakes. The paper identifies three concrete deployment scenarios where replacing attention would yield immediate benefits (Section 1, Section 5):

• Speed: If attention can be replaced with a faster mixing mechanism, training and inference latency drop directly. The authors show that FNet's Fourier sublayer alone performs forward and backward passes an order of magnitude faster than the self-attention sublayer (Table 8: 11 ms vs. 136 ms per batch on GPU for Base models). However, the overall training speed gain is 70–80% rather than 10× because the feed-forward sublayers — which all models share — consume the majority of FLOPs.

• Memory footprint: Self-attention's $O(N^2)$ memory cost is what causes the Transformer to OOM at 4096 tokens on 8 V100 GPUs. FNet uses the Fast Fourier Transform (FFT), which has $O(N \log N)$ memory requirements and a lighter constant factor. Table 4b shows FNet using 2.2 GB peak memory at 2048 tokens versus 12.2 GB for the Transformer — a 5.5× reduction. This matters enormously for deployment on resource-constrained edge devices or for processing long documents without model sharding.

• Model size: The mixing sublayer in FNet has zero learnable parameters. Compared to BERT-Base's 112M parameters, FNet-Base has only 83M — a 26% reduction that comes entirely from removing the query, key, value, and output projection matrices in attention (Table 1). For mobile deployment or distillation into student models, every megabyte counts.

Scientific stakes. Beyond practical benefits, the paper probes a fundamental question about what makes Transformer architectures work. The standard narrative — supported by probing studies — is that attention layers learn to compose syntactic and semantic relationships by flexibly attending to different tokens in different contexts. If a fixed Fourier Transform (which applies the same mixing weights to every input regardless of content) can recover 92–97% of BERT's accuracy, then either:

1. The feed-forward sublayers are doing far more of the representational work than previously believed, with the mixing sublayer merely needing to provide some mechanism for tokens to exchange information (any mechanism, not necessarily a learned one); or
2. The specific inductive bias of the Fourier Transform — globally mixing all tokens with sinusoidal weights — happens to approximate something structurally useful about how language requires information to be combined.

The authors lean toward the first interpretation:

"That such a simple linear transformation works at all is surprising, and suggests that, for at least some NLP problems, attention may not be the principal component driving the performance of Transformers." (Section 1, contributions)

This challenges the field's heavy investment in attention mechanism design and opens the door to searching for mixing mechanisms that are optimized purely for computational efficiency rather than for expressivity.

Where Prior Approaches Fall Short: A Taxonomy of Limitations

The paper identifies specific shortcomings in existing approaches that motivate their more radical simplification:

Efficient Transformers hide large constants. While many efficient Transformers achieve $O(N)$ or $O(N \log N)$ asymptotic complexity, the authors argue this is misleading in practice. Table 4b shows the Performer — the fastest efficient Transformer benchmarked by Tay et al. (2021a) — achieving only 1.3× speedup over the vanilla Transformer at 512 sequence length on GPUs, compared to FNet's 2.0×. At 8192 tokens, the Performer is 2.0× faster than the Transformer while FNet is 4.0× faster (extrapolating from the trend). The gap widens with sequence length because the constants in sparse attention implementations — indexing overhead, custom CUDA kernels for gather/scatter operations, the cost of global tokens — dominate at moderate lengths.

Prior simplification work retained attention's architectural scaffolding. The Synthesizer, Gaussian attention, and fixed-pattern attention works all kept the multi-head attention architecture intact: they replaced how attention weights are computed but kept the weighted-sum aggregation, the head structure, the softmax normalization, and the learned output projection. The authors' Linear baseline strips this down to just two matrix multiplications — one along the sequence dimension and one along the hidden dimension — with no heads, no softmax, and no token-token interactions beyond what the matrix multiplication provides. The FNet goes further by removing even the learned parameters from the sequence mixing.

No prior work compared linear mixing to attention at BERT scale in a transfer learning setting. The Synthesizer and related works primarily evaluated on machine translation or smaller-scale benchmarks. This paper pre-trains on the massive C4 dataset (Raffel et al., 2020) with BERT-Base and BERT-Large configurations (12 and 24 layers, respectively), then fine-tunes on GLUE — the standard BERT evaluation protocol. This makes the comparison directly relevant to practitioners using BERT-style models in production.

Fourier Transforms had been used as computational accelerators, never as first-class replacements. The paper carefully distinguishes FNet from prior uses of Fourier Transforms in neural networks (Section 2.1). Previous work used FFTs to speed up convolutions in CNNs (Mathieu et al., 2014; Pratt et al., 2017), to approximate dense layers for parameter reduction (Cheng et al., 2015; Sindhwani et al., 2015), or to linearize attention via random Fourier features (Performer; Choromanski et al., 2021). The authors argue that:

"we believe our work is the first to wholly replace particular neural network sublayers with a Fourier Transform" (Section 1).

The distinction matters because prior uses treated the Fourier Transform as a means to an end — a computational shortcut to implement something else (a convolution, a dense layer, an attention kernel). FNet treats the Fourier Transform as the end itself — the mixing mechanism, with no attempt to approximate attention. The frequency-domain mixing is not an approximation of attention; it is an alternative mechanism that happens to work.

The concurrent, unpublished work by Backurs et al. (2021) — which the authors acknowledge via personal communication — independently developed a similar FFT-based neural model, suggesting this idea's time had come.

How This Paper Positions Itself

The paper positions FNet not as a universal replacement for attention but as a point on a spectrum of mixing mechanisms ranging from no mixing at all (the FF-only baseline, which fails catastrophically) to fully learned, token-dependent mixing (BERT attention, which is most accurate but slowest). The key axis is:

$\text{No mixing (FF-only)} \rightarrow \text{Structured, fixed mixing (FNet)} \rightarrow \text{Learned mixing (Linear)} \rightarrow \text{Token-dependent learned mixing (BERT)}$

The paper's central empirical claim is that moving from the right end of this spectrum to the middle (structured, fixed mixing) sacrifices surprisingly little accuracy while gaining substantial speed — and that moving only slightly back toward the right end (the FNet-Hybrid models, which add just 2 self-attention layers) recovers 97–99% of BERT's accuracy while retaining most of the speed gains.

This framing is important because it avoids the straw-man position that "attention is unnecessary." Instead, the paper shows that attention is one of several viable mixing mechanisms, that different points on the spectrum are optimal under different speed-accuracy-memory budgets, and that the field's singular focus on attention may have crowded out exploration of simpler alternatives that are more practical for many deployment scenarios.

The paper also positions itself relative to the efficient Transformer literature with a provocative claim: rather than making attention more efficient, perhaps we should be looking for mechanisms that are efficient by design and happen to work for text:

"Based on this, we argue that rather than seeking more efficient approximations of the attention, there may be more value in seeking out completely new mixing mechanisms." (Section 1, contributions)

This is not merely a methodological preference — it is a bet that the inductive bias of attention (token-dependent, context-aware mixing) is not uniquely suited to text understanding, and that other mixing patterns (global, frequency-based, convolutional, etc.) may prove equally effective when combined with the powerful nonlinear processing of feed-forward layers.

3. Technical Approach

3.1 Reader Orientation

This paper proposes a modified Transformer encoder architecture where the self-attention sublayer is surgically removed and replaced with a simple linear transformation — specifically, a 2D discrete Fourier Transform with no learnable parameters — that uniformly mixes information across all tokens before the feed-forward sublayer processes the result. The core problem it solves is that self-attention scales quadratically with sequence length, making Transformers slow and memory-hungry on long inputs; the "shape" of the solution is a spectrum of mixing mechanisms ranging from parameter-free structured transforms (FNet) to learned dense linear mixing (Linear) to full attention (BERT), where users can trade speed for accuracy by choosing their point on this spectrum or by building hybrid models that use attention only in a few layers.

3.2 Big-Picture Architecture (Diagram in Words)

FNet follows the same encoder block structure as BERT, with one surgical replacement:

1. Embedding Layer — Identical to BERT: word embeddings + position embeddings + segment type embeddings are summed to produce an input tensor of shape (batch, sequence_length, hidden_dimension). Position embeddings are technically unnecessary because the Fourier Transform encodes positional information through its twiddle factors, but they are kept "to allow for a cleaner comparison with BERT" (Section 3.2).

2. Encoder Blocks (× N) — Each block contains two sublayers with residual connections and layer normalization:

   • Fourier Mixing Sublayer (replaces self-attention): Applies a 2D DFT to the input — one 1D DFT along the sequence dimension and one 1D DFT along the hidden dimension — then extracts the real part. This mixes every token with every other token using fixed sinusoidal coefficients. There are zero learnable parameters here.
   • Feed-Forward Sublayer (identical to BERT): A two-layer MLP with GELU activation that processes each token position independently. This is where all the nonlinear processing and learned computation lives.

3. Pooler Layer — Identical to BERT: a dense layer applied to the first token's output, followed by tanh activation, producing a fixed-size representation for classification tasks.

4. Task-Specific Output Projection — Depends on the downstream task; standard BERT fine-tuning heads.

The key architectural invariant is that the Fourier sublayer handles cross-token information mixing (letting information flow between positions) while the feed-forward sublayer handles per-token representation learning (transforming what each position represents). This division of labor is identical to the attention/feed-forward split in Transformers, but the mixing mechanism is now fixed and global rather than learned and input-dependent.

3.3 Roadmap for the Deep Dive

• First, the mathematical definition of the Discrete Fourier Transform and how it is applied as a 2D mixing operation — this is the core computation that replaces attention, so understanding it mechanically is prerequisite to everything else.
• Second, the FNet encoder block structure and how information flows through it — the equation, the residual connections, the real-part extraction, and why the ordering of operations matters — to establish precisely what the model computes.
• Third, the implementation strategy for computing the DFT (FFT vs. matrix multiplication) and how the choice depends on hardware and sequence length — because the practical speed claims hinge on using the right algorithm in the right regime.
• Fourth, the baseline models (Linear, Random, FF-only) and how they isolate specific properties of the mixing mechanism — to understand which aspects of FNet's design actually matter for accuracy.
• Fifth, the FNet-Hybrid architecture and how attention layers are selectively re-introduced — because this bridges the full spectrum from pure Fourier mixing to pure attention.
• Sixth, the pre-training and fine-tuning configuration — because replication requires knowing exactly which hyperparameters and datasets were used.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architectural simplification paper whose core idea is that the self-attention sublayer in Transformer encoders can be replaced with an unparameterized Fourier Transform, and that this substitution preserves most of the model's accuracy while dramatically improving speed and memory efficiency.

---

The Discrete Fourier Transform as a Token Mixing Operation

The Fourier Transform is a mathematical operation that decomposes a signal into its constituent frequency components. For a discrete sequence of $N$ values $\{x_n\}_{n=0}^{N-1}$, the Discrete Fourier Transform (DFT) produces $N$ complex-valued outputs $\{X_k\}_{k=0}^{N-1}$ where each $X_k$ represents the amplitude and phase of a particular frequency in the original signal. The paper adapts this as a token mixing mechanism by applying it to the sequence of token embeddings.

The mathematical definition (Equation 1 in the paper):

$X_k = \sum_{n=0}^{N-1} x_n e^{-\frac{2\pi i}{N} nk}, \quad 0 \leq k \leq N-1$

where $x_n \in \mathbb{R}^{d_h}$ is the embedding vector at sequence position $n$, $X_k \in \mathbb{C}^{d_h}$ is the complex-valued output at frequency index $k$, $N$ is the sequence length, $i = \sqrt{-1}$ is the imaginary unit, and $e^{-\frac{2\pi i}{N} nk}$ is the "twiddle factor" — a complex number on the unit circle that determines how much position $n$ contributes to frequency $k$.

What it computes, operationally: for each output frequency index $k$, the DFT computes a weighted sum of all $N$ input token embeddings. The weight for token $x_n$ is the complex exponential $e^{-\frac{2\pi i}{N}nk}$, which rotates in the complex plane as $n$ increases — tokens at different positions get multiplied by different complex numbers, creating a frequency-specific mixing pattern. Importantly, the same set of mixing weights is used for every input sequence regardless of its content; these weights are purely a function of the positions $n$ and the frequency index $k$, with no dependence on the actual token values.

Why this form: the DFT is chosen because it has three properties that make it an effective drop-in replacement for attention. First, every output $X_k$ depends on every input $x_n$, achieving global mixing — no token is isolated from any other, which is the key functional requirement for replacing attention. Second, the DFT has structure: the twiddle factors are not random numbers but follow a sinusoidal pattern that decomposes the signal into orthogonal frequency components. This structured mixing turns out to be empirically crucial — the Random baseline, which uses unstructured random matrices for mixing, achieves only 56.6 average GLUE score versus FNet's 76.7, showing that which linear transformation is used matters enormously. Third, the DFT admits a fast algorithm: the Fast Fourier Transform (FFT) computes it in $O(N \log N)$ time rather than the $O(N^2)$ of naive matrix multiplication, making it scalable to long sequences.

---

The FNet Fourier Sublayer: 2D DFT with Real Extraction

The Fourier sublayer in FNet does not simply apply a 1D DFT along the sequence dimension. Instead, it applies a 2D DFT — one 1D DFT along the sequence dimension and one 1D DFT along the hidden dimension — and then extracts only the real part. This is the central equation of the paper (Equation 3):

$y = \Re\left(\mathcal{F}_{\text{seq}}\left(\mathcal{F}_h(x)\right)\right)$

where $x \in \mathbb{R}^{N \times d_h}$ is the input tensor (sequence length $N$, hidden dimension $d_h$), $\mathcal{F}_h$ is a 1D DFT applied independently to each position's hidden vector (mixing the hidden dimension), $\mathcal{F}_{\text{seq}}$ is a 1D DFT applied independently to each hidden channel across the sequence (mixing the sequence dimension), and $\Re(\cdot)$ extracts the real part of the complex-valued output.

What it computes, operationally: the computation proceeds in three steps. Step 1 — for each of the $N$ token positions, take its $d_h$-dimensional embedding vector and compute its 1D DFT along the hidden dimension, producing a complex vector of the same length where each component is a mixture of all hidden dimensions. Step 2 — for each of the $d_h$ hidden channels, take its $N$-dimensional sequence of values across positions and compute its 1D DFT along the sequence dimension, producing a complex vector where each component is a mixture of all positions. Step 3 — discard the imaginary components, keeping only the real parts.

The result $y \in \mathbb{R}^{N \times d_h}$ has the same shape as the input $x$, satisfying the interface contract of the self-attention sublayer it replaces: it accepts a real-valued tensor of token embeddings and returns a real-valued tensor of the same shape, where each output position's embedding now contains information mixed from all input positions.

Why apply DFT along the hidden dimension? The paper notes that the relative ordering of $\mathcal{F}_{\text{seq}}$ and $\mathcal{F}_h$ is "immaterial because the two 1D DFTs commute." This means applying the DFT along the hidden dimension first, then the sequence dimension, produces the same result as the reverse order. The reason to include hidden-dimension mixing at all is empirical: the authors report that using only a 1D DFT along the sequence dimension "yielded some training speed gains but hurt accuracy" (Appendix A.3). The 2D DFT provides more thorough mixing — not only does each output position depend on all input positions, but within each position, each hidden channel depends on all other hidden channels. This additional mixing gives the feed-forward sublayer a richer representation to work with.

Why extract only the real part? The DFT produces complex numbers, but the feed-forward sublayers and output layers in BERT-style architectures are designed for real-valued inputs. The authors found that "FNet was less accurate and less stable during training if only the real part of the DFT was used throughout the computation" — meaning, extracting the real part after each 1D DFT rather than only at the end. The final placement of $\Re(\cdot)$ matters: the real part of the product of two DFTs is not the same as the product of the real parts of two DFTs, because imaginary components can interact to produce real results. Extracting the real part only after both DFTs have been applied preserves these cross-terms. The authors also tried extracting the absolute value instead: this "led to a significantly less accurate model" (Appendix A.3), suggesting that the sign information in the real part carries important representational content.

---

The DFT Matrix Formulation and the FFT Algorithm

While Equation (1) defines the DFT mathematically, the paper discusses two distinct computational approaches for implementing it: direct matrix multiplication and the Fast Fourier Transform (FFT).

The DFT as matrix multiplication (Equation 2):

The DFT can be expressed as multiplication by a Vandermonde matrix $W \in \mathbb{C}^{N \times N}$:

$W_{nk} = \frac{e^{-\frac{2\pi i}{N} nk}}{\sqrt{N}}$

where $n, k = 0, \ldots, N-1$ index the rows and columns, $e^{-\frac{2\pi i}{N}nk}$ is the twiddle factor, and the division by $\sqrt{N}$ normalizes the transform. For an input sequence $x$ of length $N$, the DFT output is simply $X = Wx$ — a matrix-vector multiplication costing $O(N^2)$ operations.

When applied to a 2D input tensor of shape $(N, d_h)$, the 2D DFT requires two matrix multiplications: right-multiplying by $W_h \in \mathbb{C}^{d_h \times d_h}$ (mixing the hidden dimension) and left-multiplying by $W_{\text{seq}} \in \mathbb{C}^{N \times N}$ (mixing the sequence dimension). The total complexity is $O(N^2 d_h + N d_h^2)$ — which, notably, has the same asymptotic form as the operations count for the Linear baseline (Table 1: $N^2 d_h + N d_h^2$), but with complex rather than real arithmetic.

The Fast Fourier Transform (FFT):

The FFT — specifically the Cooley-Tukey algorithm (Cooley and Tukey, 1965) as implemented in FFTW (Frigo and Johnson, 2005) — reduces the DFT computation from $O(N^2)$ to $O(N \log N)$ by recursively decomposing the transform. For a sequence of length $N = N_1 N_2$, the Cooley-Tukey algorithm expresses the $N$-point DFT in terms of $N_1$ smaller DFTs of size $N_2$, each costing $O(N_2^2)$, plus $O(N)$ additional work for combining results. Choosing $N_2$ small (typically 2 for the radix-2 FFT) yields the $O(N \log N)$ complexity.

For the 2D DFT, the FFT is applied twice. The total complexity for FNet with FFT is $O(N d_h \log N + N d_h \log d_h)$ (Table 1), where the first term comes from the sequence-dimension FFT ($d_h$ independent 1D FFTs of length $N$) and the second from the hidden-dimension FFT ($N$ independent 1D FFTs of length $d_h$). This is asymptotically better than both naive DFT ($O(N^2 d_h + N d_h^2)$) and self-attention ($O(N^2 d_h + N d_h^2)$), though the constants differ and the $\log$ factors mean the advantage is most pronounced at long sequence lengths.

Hardware-dependent implementation strategy:

The paper makes a crucial practical observation: the faster algorithm depends on the hardware. On GPUs, the FFT is faster than matrix multiplication for all sequence lengths considered (512–8192 tokens), because GPU FFT implementations (cuFFT) are highly optimized and GPU matrix multiplication for small dimensions cannot fully utilize the massive parallelism. On TPUs, the crossover depends on sequence length: for sequences ≤ 4096 tokens, it is faster to precompute and cache the DFT matrix and compute the transform via matrix multiplication, because TPUs are "even more highly optimized for matrix multiplications than GPUs" (Section 3.3). For longer sequences (> 4096), the FFT becomes faster on TPUs because the $O(N^2)$ matrix multiplication cost overtakes the FFT's $O(N \log N)$ advantage despite the TPU's matrix multiplication bias.

Consequently, the paper's GPU FNet implementation always uses the FFT, while the TPU implementation uses matrix multiplication for sequences up to 4096 and switches to FFT for longer sequences. This hardware-adaptive strategy is essential for achieving the reported speedups in practice, and it means that FNet's efficiency advantage over the Linear model — which always uses matrix multiplication — grows with sequence length.

A note on numerical precision: The authors found that while computing the DFT via matrix multiplication with "HIGHEST XLA precision" was required to very accurately reproduce the FFT's numerical output, "DEFAULT XLA precision was sufficient to facilitate accurate model convergence" (Appendix A.3). That is, the model is robust to small numerical differences in how the DFT is computed, which allows using faster, lower-precision matrix multiplications without accuracy loss.

---

The FNet Encoder Block and Information Flow

Each FNet encoder block follows the same Pre-LayerNorm Transformer structure as BERT, with the self-attention sublayer replaced by the Fourier sublayer. The full forward pass through one block is (expressed in the code from Appendix A.7):

1. Fourier mixing: mixing_output = self.fourier_layer(x)

   This computes $y = \Re(\mathcal{F}_{\text{seq}}(\mathcal{F}_h(x)))$ as described above. The Fourier sublayer has no learnable parameters — it is a pure function that applies the same transformation to every input regardless of content.

2. Residual connection + layer norm: x = LayerNorm(x + mixing_output)

   The mixing output is added back to the original input (the residual connection, which allows gradients to flow directly through the block even if the Fourier transform provides noisy gradients) and then normalized. The layer norm uses a small epsilon of $10^{-12}$ for numerical stability.

3. Feed-forward: feed_forward_output = self.ff_layer(x, deterministic)

   The feed-forward sublayer is identical to BERT's: a two-layer MLP with GELU activation. Specifically:

   • First dense layer: projects from hidden dimension $d_h$ to intermediate dimension $d_{\text{ff}} = 4d_h$, with weights initialized from $\mathcal{N}(0, 0.02^2)$.
   • GELU activation: applies the Gaussian Error Linear Unit nonlinearity element-wise.
   • Second dense layer: projects back from $4d_h$ to $d_h$, with same initialization.
   • Dropout: applied during training with rate specified by the model configuration (the standard BERT-Base rate is 0.1).

4. Residual connection + layer norm: output = LayerNorm(x + feed_forward_output)

   The feed-forward output is added back to the normalized mixing output, then normalized again. This dual-residual structure (one after mixing, one after feed-forward) is identical to the Transformer encoder block.

Why the residual connections matter more for FNet: In a Transformer, the self-attention sublayer has learned parameters that can adapt to make its output compatible with the input for effective residual addition. In FNet, the Fourier sublayer has no learned parameters, so the mixing pattern is fixed. The residual connection is therefore critical — it ensures that even if the Fourier Transform scrambles the representations in an unhelpful way, the original token embeddings are preserved and can be recovered by the feed-forward sublayer. The feed-forward sublayer sees the sum of the original embedding and the Fourier-mixed embedding, which gives it both the local (per-token) and global (mixed) information it needs.

The feed-forward sublayer as the primary locus of learning: A crucial design implication is that in FNet, all cross-token learning is effectively pushed into the feed-forward sublayers. The Fourier Transform provides a fixed, global information highway through which tokens can exchange information, but it is the feed-forward sublayers that must learn to interpret and utilize this mixed information. This explains why FNet achieves 92–97% of BERT's accuracy: the feed-forward sublayers, which constitute the majority of parameters (the mixing sublayer has none), are powerful enough to extract useful patterns from even a fixed, unstructured mixing of token information. The paper's observation that "the overall training speed is impeded by the feed-forward sublayers that all models share" (Section 4.1) takes on deeper meaning here: because the feed-forward sublayers dominate both computation and learning, replacing attention with a cheaper mixing mechanism accelerates the model without eliminating its primary learning engine.

---

Why the Fourier Transform Encodes Positional Information

The paper makes a subtle but important observation: "Because of the positional information encoded by the Fourier Transform in Equation (1) (see $n$, $k$ indices), FNet performs just as well without position embeddings" (Section 3.2). This is a non-obvious property of the DFT that merits explanation.

The DFT matrix $W_{nk} = e^{-\frac{2\pi i}{N}nk}/\sqrt{N}$ depends on the absolute position $n$ through the twiddle factor. For a fixed frequency index $k$, as $n$ varies from $0$ to $N-1$, the twiddle factor rotates around the unit circle at a rate proportional to $k$. This means that the DFT's output at frequency $k$ — which is a weighted sum $\sum_n x_n e^{-\frac{2\pi i}{N}nk}$ — is sensitive to where each token appears in the sequence. A token at position 5 contributes to frequency $k$ with a different complex weight than the same token at position 20, and the difference encodes positional information.

In contrast, an ordinary (non-cyclic) matrix multiplication $y = Wx$ with an arbitrary weight matrix $W$ does not necessarily encode positional information — the weights $W_{\text{out}, n}$ for a given output position $n$ could be completely independent of $n$ and depend only on $n$'s value. The DFT's specific structure (the Vandermonde form with roots of unity) ensures that the weights have a systematic positional dependence.

Despite this built-in positional encoding, the authors retain the standard position embeddings from BERT "to allow for a cleaner comparison." This is a conservative design choice that ensures any performance differences between BERT and FNet are attributable to the mixing mechanism itself, not to differences in how positional information is incorporated.

---

The Linear Baseline: Learned Dense Mixing

To isolate the effect of parameterization from the effect of structure, the paper introduces the Linear encoder baseline. This model replaces each self-attention sublayer with two learned, dense linear transformations:

$y = W_{\text{seq}} \cdot x \cdot W_h$

where $W_{\text{seq}} \in \mathbb{R}^{N \times N}$ is a learned matrix that mixes the sequence dimension, $W_h \in \mathbb{R}^{d_h \times d_h}$ is a learned matrix that mixes the hidden dimension, and $x \in \mathbb{R}^{N \times d_h}$ is the input. There are no attention heads, no softmax, no query-key-value projections — just two matrix multiplications.

What it computes: right-multiplying by $W_h$ linearly combines the hidden dimensions for each token independently (producing per-token feature transformations), and left-multiplying by $W_{\text{seq}}$ linearly combines token representations across the sequence (producing per-hidden-channel cross-token mixing). The result is that each output position's embedding is a learned linear combination of all input positions' embeddings, with the combination weights being the entries of $W_{\text{seq}}$, which are learned during training but fixed at inference time (not input-dependent).

Parameter count and complexity: The Linear model has fewer parameters than BERT-Base (94M vs. 112M in Base configuration; see Table 1) because it eliminates the query, key, value, and output projection matrices that multi-head attention requires. However, it has more parameters than FNet (83M) because the mixing matrices $W_{\text{seq}}$ and $W_h$ are learned. The operations per layer are $N^2 d_h + N d_h^2$ (Table 1), which is less than BERT's $2N^2 d_h + 4N d_h^2$ but more than FNet's $N d_h \log N + N d_h \log d_h$ when the FFT is used.

Relationship to prior work: The authors note that "our Linear model is similar to the MLP-Mixer (Tolstikhin et al., 2021) (for vision) and also the Random Synthesizer (Tay et al., 2020a), but simplifies the latter model further by removing the multiple heads and softmax projections." The key simplification relative to the Synthesizer is that the Linear model does not use separate attention heads — it's a single global linear mixing followed by a single hidden-dimension linear mixing, without any of the architectural scaffolding (multiple parallel attention heads, softmax normalization, learned output projection per head) that attention-based models retain even when simplifying attention weight computation.

Why this baseline matters: The Linear model isolates the importance of learnedness in the mixing weights. If FNet (fixed, structured weights) performs comparably to the Linear model (learned, unstructured weights), then the specific pattern of mixing weights is less important than the existence of mixing — the feed-forward sublayers can adapt to whatever mixing pattern is provided. Conversely, if the Linear model substantially outperforms FNet, then learning the mixing weights matters. The empirical result — Linear-Base achieves 77.0 GLUE average vs. FNet-Base's 76.7 — suggests that learning the mixing weights provides only a marginal 0.3-point benefit on GLUE, despite the Linear model having 10M more parameters and a much larger memory footprint (Table 4b).

Training instability at scale: The Linear-Large model "severely underperforms its Base counterpart on the GLUE benchmark due to training instabilities" (Section 4.1), achieving only 59.8 average score. The authors attribute this to "gradient blow up" (Section 4.1) and note that "the Linear model and BERT were less stable than the models with no parameters in their mixing sublayers, namely the FNet, Random and FF-only models." This is an important practical finding: parameter-free mixing mechanisms are not only faster but also more stable during training, likely because the fixed mixing weights cannot drift into regimes that produce exploding gradients.

---

The Random and FF-Only Baselines: Isolating the Importance of Structure and Mixing

The paper includes two additional baselines that serve as ablation studies for specific properties of the mixing mechanism:

Random encoder: Each self-attention sublayer is replaced with two constant, random matrices — one for the sequence dimension and one for the hidden dimension — initialized at random and never updated during training. The operations and parameter count (83M, from the feed-forward and embedding layers only) are identical to FNet; the only difference is that $W_{\text{seq}}$ and $W_h$ are random Gaussian matrices rather than the DFT matrix.

The Random model achieves 56.6 average GLUE score (Table 2), which is substantially worse than FNet's 76.7 but still far above the FF-only model's 49.3. This demonstrates that structure matters: not just any mixing transformation will do — the specific structure of the DFT (orthogonal, sinusoidal, global) is far more effective than an arbitrary random transformation. The Random model's 56.6 score does, however, show that even random mixing is better than no mixing at all, confirming that the feed-forward sublayers can extract some signal from nearly any cross-token information channel.

FF-only encoder: The self-attention sublayer is completely removed. Each encoder block contains only the feed-forward sublayer with its residual connection and layer norm. This model has zero token mixing — each position is processed independently through the feed-forward layers and the only cross-token interaction occurs at the final pooling layer over the [CLS] token.

The FF-only model achieves only 49.3 average GLUE score and "severely underperforms all other models" (Section 4.1). On binary classification tasks (QNLI, SST-2, RTE), it scores approximately 50% — indicating that the model fails to learn the task and is essentially guessing. The STS-B task reports "FAIL," meaning the model's predictions had zero or negative correlation with human judgments. This baseline establishes the necessity of token mixing: without some mechanism for tokens to exchange information, the model cannot perform tasks that require comparing or combining information from different parts of the input.

Together, the Random and FF-only baselines carve out the design space: token mixing is necessary (FF-only fails), and the mixing must be structured (Random underperforms), but the mixing need not be learned (FNet approaches learned mixing performance) and need not be input-dependent (the DFT's fixed weights work remarkably well).

---

The FNet-Hybrid Architecture: Reintroducing Attention Selectively

The paper explores a spectrum between pure FNet and pure BERT through FNet-Hybrid models, which retain the Fourier sublayer in most encoder blocks but replace the Fourier sublayer with self-attention in a small number of layers. The default Hybrid configuration uses "the final two Fourier sublayers of FNet with self-attention sublayers" (Section 4.1).

Design space explored (Appendix A.5, Table 9):

The authors varied (1) the number of attention layers (0, 2, 4, 6) and (2) their position in the stack (BOTTOM, MIDDLE, MIXED, TOP). Key findings:

• Position matters: Placing attention at the TOP (final layers) gave the best accuracy. For 2 attention layers: TOP achieved 0.526 MLM accuracy and 0.738 NSP accuracy, compared to BOTTOM's 0.497 and 0.733, MIDDLE's 0.499 and 0.686, and MIXED's 0.509 and 0.727 after 100k pre-training steps. The authors hypothesize that attention in the final layers is most valuable because these layers produce the representations used for downstream task prediction, and attention's input-dependent mixing may be especially important for task-specific adaptation.

• Diminishing returns: Adding more attention layers improves accuracy but with decreasing marginal benefit: 0 attention layers → 0.486 MLM; 2 attention layers → 0.526 MLM (+0.040); 4 attention layers → 0.539 MLM (+0.013); 6 attention layers → 0.546 MLM (+0.007). The first two attention layers recover roughly 75% of the gap between pure FNet and full BERT.

• Speed-accuracy trade-off: The 2-attention-layer Hybrid trains at 198 ms/batch (GPU) compared to 169 ms/batch for pure FNet and 305 ms/batch for BERT-Base (Table 3). This is a 1.5× speedup over BERT while achieving 80.6 GLUE average — 97% of BERT-Base's 83.3.

The Hybrid architecture is conceptually important because it demonstrates that attention and Fourier mixing are complementary, not mutually exclusive. The authors position this as evidence that "while attention can improve accuracy, it may not be necessary to use in every layer" (Section 1, contributions). Most of the representational work can be done by the more efficient Fourier mixing; attention is only needed in a few strategic positions to recover near-BERT accuracy.

---

Pre-Training Configuration

The paper adopts the same fixed model configurations as Devlin et al. (2019) with modifications to the dataset:

Dataset: C4 (Raffel et al., 2020) — the Colossal Clean Crawled Corpus — rather than the original BERT's BooksCorpus + English Wikipedia. C4 is substantially larger and more diverse, providing a more challenging pre-training task. The authors use a 32,000-token SentencePiece vocabulary (Kudo and Richardson, 2018) trained on a 100-million-sentence subset of C4. The switch to SentencePiece (a subword tokenizer) from BERT's original WordPiece is noted but not discussed in detail; it is a common choice in more recent pre-training work.

Pre-training objectives: Standard BERT objectives — Masked Language Modeling (MLM) and Next Sentence Prediction (NSP). In MLM, 15% of input tokens are randomly masked and the model must predict them; the loss is cross-entropy over the vocabulary. In NSP, the model receives two sentences and must predict whether the second follows the first in the original corpus; the loss is binary cross-entropy. The total loss is the sum of MLM loss and NSP loss.

Model sizes (Table 6): The paper sweeps multiple model configurations, varying hidden dimension $d_h$, number of layers, and whether attention is included:

For all models: feed-forward size is fixed at $4d_h$ and the number of self-attention heads (where applicable) is fixed at $d_h / 64$, following Turc et al. (2019). FNet-Hybrid models always contain exactly 2 self-attention sublayers, placed at the top (final two layers), except for models with only 2 total layers where no Hybrid variant is tested.

Training hyperparameters (from Devlin et al., 2019, adapted for C4):

• TPU experiments: Batch size 256, trained on $4 \times 4$ TPU v3 chips, 1 million training steps.
• GPU experiments: Batch size 64, trained on 8 V100 chips, 1 million training steps.
• Optimizer: AdamW (the specific learning rate schedule, warmup steps, and weight decay are not explicitly stated but are "lifted from Devlin et al. (2019)," which uses a peak learning rate of $10^{-4}$, 10,000 warmup steps, and linear decay).
• Learning rate for smaller models: The authors found that "smaller model architectures benefited from larger learning rates," so they selected the best result using two learning rates: $10^{-3}$ and $10^{-4}$.
• All models trained for 1 million steps: This is a fixed budget rather than early stopping, ensuring fair speed comparisons.

Pre-training metrics (Table 5): BERT-Base achieves MLM accuracy 0.68, NSP accuracy 0.86, and total loss 1.76. FNet-Base achieves MLM accuracy 0.58, NSP accuracy 0.80, and total loss 2.45. The 10-point MLM accuracy gap and 0.69 loss gap indicate that BERT is genuinely better at the pre-training tasks — the Fourier Transform cannot match attention's ability to model token dependencies. However, this pre-training gap narrows during fine-tuning: FNet recovers to within 8% of BERT on GLUE, suggesting that some of the pre-training advantage is specific to the MLM/NSP objectives rather than transferable language understanding.

Why the training configuration may favor BERT: The authors acknowledge that "the training configuration is lifted from Devlin et al. (2019), it may be slightly biased towards the BERT attention model" (Appendix A.1). Hyperparameters optimized for attention-based models might not be optimal for FNet, meaning FNet's reported accuracy is a lower bound on what it could achieve with FNet-specific hyperparameter tuning.

---

Fine-Tuning Configuration

GLUE benchmark (Wang et al., 2018): The standard 9-task natural language understanding benchmark (MNLI, QQP, QNLI, SST-2, CoLA, STS-B, MRPC, RTE; WNLI is excluded following Devlin et al., 2019). Each task has a separate fine-tuning run with task-specific hyperparameters.

Fine-tuning protocol:

• Learning rate sweep: The paper performs 3 trials for Base models and 6 trials for Large models at each base learning rate (the specific learning rates swept are not stated explicitly, but standard BERT fine-tuning sweeps typically cover $\{2\times 10^{-5}, 3\times 10^{-5}, 5\times 10^{-5}\}$).
• Best result reported: For each task, the best accuracy across all trials and learning rates is reported, which "reflects our observation that BERT-Large was less stable than BERT-Base, as noted in Devlin et al. (2019)."
• No early stopping during pre-training: Pre-training runs for the full 1 million steps without validation-based early stopping, ensuring the speed comparisons are based on equal compute budgets.

Metrics (Table 2):

• MNLI: matched/mismatched accuracy (two evaluation sets).
• QQP, MRPC: average of accuracy and F1.
• STS-B: Spearman correlation.
• All others: accuracy.
• Average: mean over all 8 tasks (excluding WNLI), with "FAIL" cases excluded from the mean — notably, the FF-only model FAILs on STS-B, meaning its average of 49.3 is computed over the remaining 7 tasks only.

The fine-tuning results are the paper's primary accuracy benchmark, and the key numbers — FNet-Base at 76.7 (92% of BERT-Base's 83.3) and FNet-Large at 81.9 (97% of BERT-Large's 84.7) — are computed from the averages in Table 2.

---

Long Range Arena (LRA) Benchmark Configuration

For evaluating long-sequence scaling, the paper uses the LRA benchmark (Tay et al., 2021a), which consists of 5 tasks requiring sequence lengths from 1024 to 4096 tokens:

• ListOps: Hierarchical list operations (sequence length ~2000).
• Text: Document-level text classification (sequence length ~4000).
• Retrieval: Document retrieval (sequence length ~4000).
• Image: CIFAR-10 image classification flattened into 1024-pixel sequences.
• Pathfinder: Long-range spatial reasoning on 32×32 images (sequence length 1024).
• Path-X: Extremely long-range spatial reasoning on 128×128 images (sequence length 16384).

Hyperparameter adaptation (Appendix A.6): The paper encountered discrepancies between the configurations described in Tay et al. (2021a) and the code repository. Key adaptations:

• Image task (CIFAR-10): The authors swept over number of layers in $\{1, 2, 3, 4\}$ and found that 1 layer worked best for all models — contradicting Tay et al.'s suggestion of 3 layers.
• Pathfinder task: A base learning rate of 0.001 (as given in the code repository) yielded better results than the 0.01 indicated in the paper.
• Embedding dimension and batch size: Small sweeps were performed over these parameters, which are not listed in Tay et al.
• Best results reported: The authors note that "results for certain tasks — Text and Retrieval in particular — can vary quite a bit between runs, especially for the Transformer," and report the best results across runs.

Hardware: LRA experiments run on $4 \times 4$ TPU v3 chips for accuracy results and on 8 V100 GPUs for speed/memory benchmarking. The GPU benchmarks use the same LRA Text classification task but sweep sequence lengths from 512 to 16384 to measure scaling behavior.

---

Summary of Design Choices and Their Justifications

• 2D DFT over 1D DFT: The hidden-dimension DFT is included because it "yielded some training speed gains but hurt accuracy" when omitted (Appendix A.3). The marginal accuracy benefit is worth the small computational cost.
• Real-part extraction only at the end of the 2D DFT: Extracting the real part after each 1D DFT caused instability and accuracy degradation; doing it only at the end preserves complex interactions that produce real-valued output.
• FFT on GPUs, matrix multiplication on TPUs for short sequences: Hardware-adaptive implementation that maximizes speed on each platform.
• Position embeddings retained despite being unnecessary: Enables clean comparison with BERT by keeping everything except the mixing mechanism identical.
• Pre-training on C4 rather than Wikipedia+BooksCorpus: Larger, more diverse dataset provides a more challenging test of the mixing mechanism's expressivity.
• 1 million fixed training steps: Equal compute budget for fair speed comparisons; no early stopping that would advantage models that converge faster.
• Multiple learning rates for smaller models: Accounts for the observation that smaller architectures benefit from larger learning rates, preventing the learning rate from being a confounding factor.
• 2 attention layers at the top for Hybrid models: Empirically best position (Table 9), likely because final layers produce task-critical representations where input-dependent mixing is most valuable.
• Best-of-N trials for GLUE fine-tuning: Addresses instability in BERT-Large and provides fair upper-bound comparisons across models with different stability characteristics.

4. Key Insights and Innovations

Innovation 1: Self-Attention Is a Sufficient But Not Necessary Mechanism for Token Mixing — a Restructuring of the Field's Architectural Assumptions

The paper's most fundamental intellectual contribution is not FNet itself but the reframing of what token mixing requires. Prior to this work, the field operated under an implicit hierarchy: self-attention was the gold-standard mixing mechanism because it is input-dependent, context-aware, and learns which tokens to attend to; efficient Transformers sought to preserve these properties while reducing cost (sparse patterns, low-rank approximations, kernel linearization); and work questioning attention's necessity (Synthesizer, Gaussian attention) still retained the multi-head architecture with softmax projections, treating attention as the substrate to be simplified rather than the problem to be replaced.

The authors collapse this hierarchy by asking a question that — in retrospect — should have been obvious: if the feed-forward sublayers contain the majority of parameters and perform all nonlinear computation, how much of the representational work is actually being done by the attention mechanism versus merely being enabled by it? The answer, per their empirical results, is that attention's primary role may be far simpler than assumed: it provides a mechanism for tokens to exchange information so that the feed-forward sublayers can learn cross-token patterns. Any mechanism that provides this connectivity — structured, unstructured, learned, or fixed — suffices, provided it mixes tokens globally and deterministically enough for the feed-forward layers to adapt.

This is a conceptual shift from attention as compute to attention as connectivity. The field had been optimizing the wrong thing — making the mixing mechanism more expressive — when the bottleneck was never mixing expressivity but mixing existence. The FF-only baseline (49.3 GLUE average, failing entirely on STS-B and scoring at chance on binary tasks) proves that mixing is necessary; the Random baseline (56.6 GLUE average) proves that the mixing must be structured; and FNet's 76.7 GLUE average (92% of BERT-Base) proves that the structure need not be learned. The 0.3-point gap between FNet-Base (fixed DFT) and Linear-Base (learned dense mixing) at 77.0 makes this point starkly: learning the mixing weights buys almost nothing for the Base configuration, despite adding 11M parameters and a much larger memory footprint.

This finding is fundamental rather than incremental because it changes what the field should investigate. If attention's primary value is connectivity rather than computation, then the search space for mixing mechanisms should prioritize computational efficiency (speed, memory, scalability) over expressivity (token-dependence, multi-head structure, learned attention patterns). The paper's provocation — "rather than seeking more efficient approximations of the attention, there may be more value in seeking out completely new mixing mechanisms" (Section 1) — follows directly from this reframing. The efficient Transformer literature had been solving the wrong optimization problem: preserving attention's properties while reducing cost, rather than finding the cheapest mechanism that achieves connectivity.

The evidence for this reframing is not a single table but a pattern across the paper's experimental design. Table 2 shows a monotonic but shallow relationship between mixing mechanism complexity and accuracy: FF-only (0 mixing parameters) → Random (0 learned mixing parameters, unstructured) → FNet (0 learned mixing parameters, structured) → Linear (learned mixing parameters, unstructured) → BERT (learned, input-dependent mixing). The jumps between adjacent points on this spectrum are uneven: the Random → FNet gap (20.1 points) dwarfs the FNet → Linear gap (0.3 points), which in turn is smaller than the Linear → BERT gap (6.3 points). This pattern suggests that the structure of mixing matters greatly (DFT vs. random), the learnedness of mixing matters negligibly (DFT vs. learned dense), and the input-dependence of mixing matters moderately (learned dense vs. attention). But the dominant factor — the 27.4-point gap from FF-only to FNet — is simply the presence of structured mixing.

Innovation 2: The Discovery That the Fourier Transform's Fixed Sinusoidal Basis Functions as an Implicit Positional Encoding and Global Information Highway

While the paper's architectural reframing (Innovation 1) is its conceptual contribution, the specific choice of the Fourier Transform as the mixing mechanism yields a technical insight that was neither obvious ex ante nor guaranteed to work: a fixed, unparameterized linear transformation with a specific mathematical structure — sinusoidal global mixing — can serve as both a positional encoding mechanism and a cross-token information channel simultaneously, without any learning.

This is surprising for two reasons. First, positional encoding in Transformers has been treated as a separate design problem from token mixing. The original Transformer (Vaswani et al., 2017) added sinusoidal position embeddings to token embeddings before the first layer; BERT made these learned; later work explored relative position biases, rotary position embeddings (RoPE), and ALiBi. In all cases, positional information is injected at the embedding level and then preserved through the self-attention layers, which are not themselves designed to encode position (they are permutation-equivariant without position embeddings). FNet inverts this: the mixing mechanism itself encodes position through the DFT's twiddle factors $e^{-\frac{2\pi i}{N}nk}$, which depend on absolute position $n$. The authors demonstrate this empirically by noting that FNet "performs just as well without position embeddings" (Section 3.2) — a property no attention-based model can claim without architectural modification.

Second, the DFT's specific basis functions — complex exponentials at integer frequencies — have no obvious connection to linguistic structure. A sinusoidal basis seems more naturally suited to signal processing (where signals genuinely have frequency-domain structure) than to language (where token relationships are syntactic and semantic, not periodic). That such a basis works at all — recovering 92% of BERT's accuracy — suggests either that the feed-forward sublayers are remarkably adept at extracting linguistic patterns from frequency-domain representations, or that the DFT's global mixing pattern happens to distribute information in a way that makes cross-token patterns learnable even in an unnatural basis. The authors do not fully adjudicate between these hypotheses, but the result is empirically robust: across GLUE tasks, LRA tasks, and multiple model sizes, the DFT consistently works.

This finding is significant beyond FNet because it opens a design space that the field had not been exploring: mixing mechanisms chosen for their mathematical structure rather than their learned behavior. The DFT is orthogonal (up to normalization), globally connected, and efficiently computable via the FFT. Other structured transforms — Hadamard, Hartley, Discrete Cosine — live in the same design space, and the paper's experiments with alternatives (Appendix A.3) suggest this space is fruitful: the Hartley Transform matches DFT's accuracy, the Hadamard Transform degrades only ~2%, and the DCT degrades ~4%. This pattern suggests that the specific transform matters, but that a class of structured, orthogonal, global linear transformations may all be viable mixing mechanisms. The paper's survey is self-described as "cursory" (Section 5), implying that the design space is largely unexplored.

The evidence for this innovation is distributed across the paper's experimental results. Table 2 shows FNet achieving 76.7 GLUE average without position embeddings being essential. Figure 2 and Figure 3 (Appendix A.2) show FNet defining the Pareto frontier for smaller, faster models on both GPUs and TPUs — meaning the DFT is not just accurate enough but computationally efficient enough to beat learned alternatives at fixed speed budgets. Table 4b shows FNet's memory scaling advantage growing with sequence length (2.2 GB vs. 12.2 GB for the Transformer at 2048 tokens), a direct consequence of the FFT's $O(N \log N)$ complexity versus attention's $O(N^2)$.

Innovation 3: The Conceptual Separation of Mixing Expressivity from Task Performance — and the Implication That Feed-Forward Sublayers Are the Primary Locus of Learning in Transformer Encoders

A subtle but important insight emerges from the paper's experimental design that the authors articulate only partially: the feed-forward sublayers, not the mixing sublayers, are the primary engine of representational learning in Transformer encoders. FNet isolates this property by completely fixing the mixing mechanism and observing that the model still learns competently — implying that the feed-forward sublayers are powerful enough to interpret and utilize information mixed through an arbitrary, fixed transformation.

This inverts the standard narrative about Transformer architectures. The probing literature (Tenney et al., 2019; Clark et al., 2019; Voita et al., 2019) focused overwhelmingly on what attention layers learn — which syntactic dependencies are captured at which depths, which attention heads specialize for which relationships. The implicit assumption was that attention layers are where the interesting representational work happens, with feed-forward layers performing routine nonlinear transformations. FNet's results suggest the opposite: feed-forward layers (which contain the majority of parameters — the mixing sublayer has none, while each feed-forward block has $8d_h^2$ parameters from its two dense layers) may be doing the heavy lifting, with attention merely providing a mechanism for those layers to access information from other tokens.

This interpretation is supported by the paper's own data in a way the authors do not fully exploit. Table 3 shows that the Fourier sublayer alone is 12.2× faster than self-attention for Base models on GPUs (11 ms vs. 136 ms per batch for forward+backward passes), yet the overall model is only 1.8× faster (169 ms vs. 305 ms). The feed-forward sublayers dominate total computation: even if the mixing sublayer were infinitely fast, the model could not be more than ~2.5× faster than BERT because the feed-forward layers — identical across all models — consume the majority of FLOPs. But more importantly, if the feed-forward layers also dominate learning, then making the mixing mechanism cheaper is the right optimization: you preserve the learning engine while reducing the cost of the connectivity layer.

The FNet-Hybrid results (Table 2) sharpen this insight. Adding just 2 self-attention layers at the top of FNet recovers 97% of BERT-Base's accuracy (80.6 vs. 83.3) and 99% of BERT-Large's (83.6 vs. 84.7). This suggests that most of the model's representational capacity comes from the feed-forward layers shared across all encoder blocks, with attention providing a relatively small (3-8%) accuracy boost that is concentrated in the final layers where task-specific representations are formed. The Hybrid configuration is not a compromise — it is closer to the optimal architecture the paper's analysis implicitly argues for: efficient fixed mixing in most layers, learned input-dependent mixing only where it adds the most value.

This insight is incremental in its empirical basis but fundamental in its implications. It does not introduce a new technique but rather recasts the role of existing components in a way that changes architectural design priorities. If feed-forward layers are the primary learners, then research should focus on improving them (better activations, wider intermediates, structured sparsity) rather than on ever-more-elaborate attention mechanisms. If mixing is primarily about connectivity, then the mixing mechanism should be optimized for speed and memory rather than expressivity. The paper does not make these arguments explicitly, but they follow directly from the evidence it presents.

The evidence chain: Table 5 shows BERT-Base's MLM accuracy (0.68) substantially exceeding FNet-Base's (0.58) — a 10-point pre-training gap — suggesting that attention genuinely provides better pre-training signal. Yet Table 2 shows this gap narrowing to 6.6 GLUE points after fine-tuning (83.3 vs. 76.7). The pre-training advantage does not fully transfer, implying that some of what attention learns during pre-training is specific to the MLM objective rather than general linguistic competence. Meanwhile, the feed-forward layers — which are identical in structure between BERT and FNet — are sufficient to close most of the gap during task-specific fine-tuning.

Innovation 4: The Demonstration That Hardware-Aware Algorithm Selection (FFT vs. Matrix Multiplication for the DFT) Is a First-Class Design Concern That Can Reverse Speed Rankings Between Architectures

A practical but intellectually significant contribution is the paper's explicit treatment of hardware-dependent algorithm choice as a model architecture decision, not merely an implementation detail. The observation that the DFT is faster to compute via matrix multiplication on TPUs for sequences ≤4096 but faster via FFT on GPUs for all lengths (Section 3.3) is not just an engineering note — it has consequences for how models should be compared and deployed.

This matters because it exposes a subtlety in the efficient Transformer literature that the paper's LRA comparisons bring into focus. Table 4b shows FNet outperforming the Performer (the fastest efficient Transformer benchmarked by Tay et al., 2021a) on GPUs at all sequence lengths, with the gap widening from 1.5× at 512 tokens to 2.0× at 8192 tokens. Yet Table 7 (Appendix A.2) shows the Performer being faster than FNet on TPUs at sequence lengths ≥2048. The paper's honest reporting of this reversal — and its attribution to differences in GPU vs. TPU FFT implementations — is methodologically important. It implies that speed comparisons between architectures are not architecture-properties alone but architecture-hardware interactions, and that the "fastest" model depends on the deployment target.

This insight is incremental (it does not change the theoretical understanding of mixing mechanisms) but practically significant for two reasons. First, it means that FNet's speed advantage is likely to grow as TPU FFT implementations improve — the paper notes that "we suspect that FNet will only become more performant on TPUs as the TPU implementation of the FFT improves" (Section 3.3). The current TPU results are a lower bound. Second, it suggests that the hardware-aware adaptive strategy the paper uses (FFT on GPUs, matrix multiplication on TPUs for short sequences, FFT on TPUs for long sequences) is itself a contribution — a template for how future mixing mechanisms should be implemented with platform-specific optimizations rather than a one-size-fits-all approach.

The evidence is in Table 7's training speed numbers: on TPUs at 512 tokens, FNet with matrix multiplication achieves 9.5 steps/s (1.2× faster than the Transformer's 8.0), while FNet with FFT achieves 8.6 steps/s (1.1×). At 2048 tokens, the matrix multiplication variant drops to 6.1 steps/s (3.6×) while the FFT variant drops to 3.2 steps/s (1.9×). The crossover point — where FFT becomes faster than matrix multiplication — is at approximately 4096 tokens on TPUs, compared to essentially 0 tokens on GPUs. An implementation that always used FFT or always used matrix multiplication would be suboptimal on one platform or at one sequence length; the adaptive strategy is necessary to achieve the paper's reported speedups.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All transfer learning experiments use two pre-training/fine-tuning setups. Pre-training is performed on the C4 dataset (Raffel et al., 2020), a large, diverse corpus of web text, replacing BERT's original BooksCorpus + English Wikipedia. A 32,000-token SentencePiece vocabulary (Kudo and Richardson, 2018) is trained on a 100-million-sentence subset of C4. Fine-tuning evaluation is on the GLUE benchmark (Wang et al., 2018), specifically the 8-task validation split (WNLI excluded, following Devlin et al., 2019) with per-task metrics as defined in the original benchmark: accuracy for MNLI, QNLI, SST-2, CoLA, RTE; F1/accuracy average for QQP and MRPC; Spearman correlation for STS-B. Long-sequence evaluation uses the Long Range Arena (LRA) benchmark (Tay et al., 2021a) with 5 tasks spanning sequence lengths from 1024 to 16384 tokens (ListOps, Text, Retrieval, Image, Pathfinder, Path-X).

• Base model(s). The primary base models are BERT-Base and BERT-Large configurations from Devlin et al. (2019): BERT-Base has 12 encoder layers, hidden dimension $d_h = 768$, 12 attention heads, and ~112M parameters; BERT-Large has 24 layers, $d_h = 1024$, 16 heads, and ~339M parameters. FNet variants replace the self-attention sublayer with a Fourier sublayer in every encoder block while keeping all other components (feed-forward sublayers, embedding layers, pooler) identical. Additional model sizes are explored for the speed-accuracy trade-off curves (Figure 2, Figure 3), ranging from tiny (2 layers, $d_h = 128$, ~5M parameters) to Large (24 layers, ~238M parameters for FNet-Large), with feed-forward size fixed at $4d_h$ and attention heads (where applicable) at $d_h/64$ — configurations given in full in Table 6. The authors argue that BERT is "representative of the capabilities of many contemporary LLMs" (paraphrased from Section 1 motivation), and the specific choice of BERT's fixed configuration enables a clean controlled comparison where only the mixing sublayer changes.

• Metrics. Pre-training metrics (Table 5) include MLM accuracy (fraction of masked tokens correctly predicted), NSP accuracy (fraction of next-sentence predictions correct), and total loss (sum of MLM and NSP cross-entropy losses). GLUE fine-tuning metrics follow the standard per-task conventions and are reported as an average over 8 tasks (WNLI excluded, and any "FAIL" cases excluded from the average). Training speed is measured in milliseconds per batch with batch sizes of 256 on TPU ($4 \times 4$ v3 chips) and 64 on GPU (8 V100 chips), reported alongside GFLOPS per example for a forward pass (Table 3). Inference speed is measured similarly in milliseconds per batch. Peak memory usage during training is measured in GB (Table 4b). For the LRA benchmark, accuracy is per-task classification/correlation accuracy as defined in Tay et al. (2021a), with average computed over the 5 completed tasks only (Path-X excluded since all models either fail due to memory or perform at chance).

• Baselines. Five model variants are systematically compared, forming the spectrum of mixing mechanism complexity described in Section 3. First, BERT (Devlin et al., 2019): the standard Transformer encoder with multi-head self-attention in every layer — the upper bound on accuracy and the baseline for speed comparisons. Second, Linear: replaces each self-attention sublayer with two learned, dense linear transformations (one along the sequence dimension, one along the hidden dimension) with no attention heads or softmax — this tests whether learned mixing without input-dependence or multi-head structure suffices. Third, FNet: replaces each self-attention sublayer with an unparameterized 2D DFT (Equation 3) — the primary proposed model. Fourth, Random: identical to FNet in structure but with the DFT matrices replaced by constant random matrices initialized once and never updated — this tests whether the structure of the DFT matters or any fixed mixing works. Fifth, FF-only: completely removes the mixing sublayer, leaving only the feed-forward sublayer per encoder block — this tests whether token mixing is necessary at all. Additionally, FNet-Hybrid replaces the final two Fourier sublayers with self-attention sublayers (with other configurations explored in Appendix A.5, Table 9) — testing the complementarity of Fourier and attention mixing. For LRA comparisons, results from Tay et al. (2021a) are quoted for 10 efficient Transformer models (Local Attention, Sparse Transformer, Longformer, Linformer, Reformer, Sinkhorn Transformer, Synthesizer, BigBird, Linear Transformer, Performer), with the Performer (Choromanski et al., 2021) used as the primary efficiency baseline. The paper also runs its own Transformer and Linear baselines on LRA using Tay et al.'s codebase for controlled comparison.

• Generation budget / compute accounting. For the GLUE experiments, compute is measured in wall-clock training time to a fixed step budget: all models are pre-trained for exactly 1 million steps at the same batch sizes (256 on TPU, 64 on GPU), making speed comparisons direct ("BERT trains in 305 ms/batch, FNet in 169 ms/batch on GPU" from Table 3). No early stopping is used, ensuring all models consume identical optimizer step counts. For the speed-accuracy trade-off curves (Figure 2, Figure 3), speed is training steps per second averaged over 100k steps, and accuracy is MLM accuracy on a held-out validation set after 1M steps. For LRA, compute is measured in training steps per second and inference milliseconds per batch at fixed batch sizes on identical hardware (8 V100 GPUs or $4 \times 4$ TPU v3 chips). The FLOPs comparison uses GFLOPS per example for a forward pass (Table 3, last column): 62 GFLOPS for FNet-Base vs. 98 GFLOPS for BERT-Base (63% of BERT's FLOPs). The paper also reports peak memory usage during training (Table 4b) as a separate resource dimension. For mixing sublayer isolation (Table 8), all other sublayers are removed to measure pure mixing cost.

• Cross-validation / statistical protocol. For GLUE fine-tuning, the paper performs 3 trials per base learning rate for Base models and 6 trials for Large models, reporting the best result across all trials and learning rates. This choice reflects the observation that BERT-Large is less stable during fine-tuning, as noted in Devlin et al. (2019), and the authors found that "different BERT runs with the same base learning rate could yield slightly different results" (Section 4.1). The best-result protocol provides an upper-bound comparison across models with different stability characteristics — a choice that favors BERT (since it exhibits more variance) but is consistent with standard practice. For LRA, the authors note that "results for certain tasks — Text and Retrieval in particular — can vary quite a bit between runs, especially for the Transformer; we report the best results" (Appendix A.6). No cross-validation or statistical significance testing is reported. For the speed-accuracy trade-off curves, each model size uses the best of two learning rates ($10^{-3}$ and $10^{-4}$) as the smaller models benefited from larger learning rates, with a single training run per configuration. The pre-training metrics in Table 5 are from single runs.

Main Quantitative Results

GLUE Transfer Learning: FNet Recovers 92–97% of BERT's Accuracy While Training 70–80% Faster

Headline accuracy results (Table 2). On the 8-task GLUE validation benchmark, FNet-Base achieves an average score of 76.7 compared to BERT-Base's 83.3 — representing 92% of BERT's accuracy. FNet-Large achieves 81.9 compared to BERT-Large's 84.7 — representing 97% of BERT's accuracy. The gap narrows at the Large scale, which the authors attribute not to FNet catching up in absolute capability but to "BERT-Large being more stable during training" (paraphrased from Section 4.1 discussion of Large model GLUE results). Specifically, BERT-Large shows only a 1.4-point improvement over BERT-Base on the validation split (83.3 to 84.7), notably smaller than the ~2.5-point improvement Devlin et al. (2019) achieved on the test split, suggesting "reduced headroom" that compresses the absolute differences between models. The per-task pattern shows FNet's weakness concentrated on tasks requiring complex pairwise token interactions: on MNLI (84/81 for BERT-Base vs. 72/73 for FNet-Base — a 12-point gap), QQP (87 vs. 83 — a 4-point gap), and QNLI (91 vs. 80 — an 11-point gap). In contrast, FNet is competitive on single-sentence or shallow-reasoning tasks: SST-2 (93 vs. 95, slightly outperforming BERT-Base) and CoLA (73 vs. 69, BERT-Base leads).

Baseline comparisons on GLUE. The Linear-Base model achieves 77.0 — only 0.3 points above FNet-Base despite having 11M more learnable parameters and a much larger memory footprint (Table 4b shows Linear models using significantly more GPU memory than FNet at all sequence lengths). This near-identical performance is one of the paper's most striking results: learning the mixing weights provides negligible benefit for the Base configuration on GLUE. However, the Linear model fails catastrophically at Large scale: Linear-Large scores only 59.8, dramatically worse than Linear-Base's 77.0 and FNet-Large's 81.9. The authors attribute this to "gradient blow up" and note that "the Linear model and BERT were less stable than the models with no parameters in their mixing sublayers" (Section 4.1) — parameter-free mixing mechanisms like FNet's DFT provide implicit training stability that becomes increasingly valuable at scale.

The Random model scores 56.6 — 20.1 points below FNet-Base, demonstrating that the structure of the mixing transformation matters enormously. The FF-only model scores 49.3 and fails on STS-B (where the score is reported as "FAIL," meaning the model's predictions had zero or negative correlation with human judgments), and scores at chance (~50%) on binary classification tasks QNLI, SST-2, and RTE (Table 2 shows QNLI 52, SST-2 48, RTE 54). This confirms that token mixing is necessary for any task beyond single-sentence classification.

FNet-Hybrid bridges the gap at minimal speed cost. FNet-Hybrid-Base, with only 2 self-attention sublayers at the top of the model, achieves 80.6 GLUE average — 97% of BERT-Base's 83.3. FNet-Hybrid-Large achieves 83.6 — 99% of BERT-Large's 84.7. The speed penalty is modest: FNet-Hybrid-Base trains at 198 ms/batch on GPU versus FNet-Base at 169 ms/batch (a 17% slowdown) while still being 1.5× faster than BERT-Base at 305 ms/batch (Table 3). Examining per-task Hybrid performance: on MNLI, FNet-Hybrid-Base achieves 78/79 versus BERT-Base's 84/81 — recovering about half the gap between pure FNet (72/73) and BERT. On QQP, the Hybrid (85) nearly matches BERT (87). On QNLI, the Hybrid (88) substantially closes the gap from FNet (80). This pattern suggests that attention in the final layers is particularly important for complex pairwise reasoning tasks (natural language inference in MNLI, question-answer relationships in QNLI), which aligns with the probing literature's finding that higher Transformer layers encode more task-specific representations.

Pre-training vs. fine-tuning gap (Table 5). BERT-Base substantially outperforms FNet-Base on pre-training metrics: MLM accuracy 0.68 vs. 0.58 (a 10-point gap), NSP accuracy 0.86 vs. 0.80 (a 6-point gap), and total loss 1.76 vs. 2.45. The 10-point MLM gap is notable given that MLM is the primary pre-training objective. However, this pre-training advantage does not fully transfer to downstream tasks: on GLUE, the gap narrows to 6.6 points (83.3 vs. 76.7). The authors do not deeply analyze this finding, but it implies either that some of BERT's MLM advantage captures pre-training-specific patterns not useful for downstream tasks, or that FNet's representations are more amenable to task-specific adaptation during fine-tuning despite being weaker during pre-training. The pattern holds at Large scale: BERT-Large's MLM accuracy (0.72) exceeds FNet-Large's (0.63) by 9 points, but the GLUE gap is only 2.8 points (84.7 vs. 81.9). The Linear-Large model's pre-training metrics (MLM 0.65, NSP 0.85) look reasonable — better than FNet-Large's 0.63 and 0.82 — yet its GLUE score collapses to 59.8, confirming that pre-training metrics are an unreliable proxy for downstream performance, especially when training instability is a factor.

Speed results (Table 3). On GPUs (8 V100 chips, batch size 64), FNet-Base trains at 169 ms/batch — 1.8× faster than BERT-Base at 305 ms/batch — and FNet-Base inference runs at 46 ms/batch — 1.8× faster than BERT-Base at 82 ms/batch. On TPUs ($4 \times 4$ v3 chips, batch size 256), FNet-Base trains at 128 ms/batch — 1.7× faster than BERT-Base at 213 ms/batch — and inference runs at 23 ms/batch — 1.4× faster than BERT-Base at 32 ms/batch. The speedup is larger on GPUs than TPUs, which the authors attribute to the highly optimized GPU FFT implementation and TPUs' comparative advantage in matrix multiplications (Section 3.3, Table 8 discussion). The FF-only model is the fastest (1.9× training speedup on GPU, 2.0× inference speedup on TPU), establishing an upper bound on speed gains achievable by removing attention. FNet captures most of this bound — 1.8× vs. 1.9× training on GPU, 1.7× vs. 1.8× on TPU — indicating that the Fourier sublayer adds negligible overhead beyond the feed-forward layers that all models share. The Linear model is slightly slower than FNet (199 ms/batch training on GPU vs. 169 ms/batch) because its dense matrix multiplications cost more than the FFT, especially as sequence length grows beyond 512 (see LRA speed results below).

GFLOPS analysis (Table 3, final column). FNet-Base requires 62 GFLOPS per forward pass compared to BERT-Base's 98 GFLOPS — a 37% reduction. The Linear-Base and Random-Base models require 71 GFLOPS, reflecting their matrix multiplication cost versus FNet's FFT cost. The FF-only model requires 59 GFLOPS, which represents the purely feed-forward FLOPs shared by all models. This reveals that the mixing sublayer accounts for approximately 39 GFLOPS in BERT-Base (98 − 59) versus only 3 GFLOPS in FNet-Base (62 − 59) — a 13× reduction in mixing cost that translates to only a 1.6× reduction in total FLOPs because the feed-forward sublayers dominate. At Large scale: FNet-Large requires 217 GFLOPS versus BERT-Large's 337 GFLOPS — a 36% reduction, nearly identical to the Base-scale reduction.

Speed-accuracy trade-off curves (Figure 2 for GPU, Figure 3 in Appendix A.2 for TPU). When sweeping model sizes from 5M to 339M parameters, clear Pareto efficiency frontiers emerge. On GPUs (Figure 2): for smaller, faster models (left side of the plot, corresponding to configurations with 2–4 layers and $d_h = 128$–256), FNet (yellow squares) and Linear (red triangles) define the Pareto frontier — at a given speed, they achieve higher MLM accuracy than BERT (blue circles) at that speed, and at a given accuracy, they are faster. For larger, slower models (right side, configurations with 8–12 layers and $d_h = 512$–768), BERT and FNet-Hybrid (green stars) define the frontier. The crossover occurs around 100–150 ms/batch (GPU) and 0.55–0.60 MLM accuracy, corresponding to model sizes of roughly 30–40M parameters. This implies that for resource-constrained deployments (mobile, edge, low-latency serving), FNet or Linear models are strictly preferable to BERT at the same parameter count, while for high-accuracy deployments, BERT or FNet-Hybrid remain the best choice. The TPU trade-off curves (Figure 3) show qualitatively identical trends, confirming the finding is not hardware-specific. A notable detail: the Linear model's advantage over FNet at small scales (visible in the leftmost points of Figure 2) disappears at larger scales, consistent with the Linear-Large collapse on GLUE.

Long Range Arena: FNet Matches the Most Accurate Efficient Transformers While Being Significantly Faster on GPUs

Accuracy results (Table 4a). On the LRA benchmark, FNet achieves an average accuracy of 55.30 across the 5 completable tasks, compared to 55.83 for the paper's own Transformer baseline and 54.16 for the Linear baseline. Quoted results from Tay et al. (2021a) show the Synthesizer at 52.88, BigBird at 55.01, and the Transformer at 54.39. The differences between the top models are small — the range from 54.39 to 55.83 spans only 1.44 points — making it difficult to declare any model definitively more accurate. The paper appropriately concludes that FNet is "competitive with the most accurate of the efficient Transformers" (Section 4.2) rather than claiming superiority. The Path-X task defeats all models: the Transformer runs out of memory (OOM), and all other models (including FNet) perform no better than random chance, so this task is excluded from averages.

Per-task, FNet shows strengths and weaknesses: on Text classification (65.11), it matches the best-performing Linear Transformer (65.90) and Performer (65.40), substantially outperforming the Transformer (61.54 for the paper's run, 64.27 from Tay et al.). On Retrieval (59.61), it is competitive with BigBird (59.29) and Sparse Transformer (59.59). On Image (38.67), it underperforms the Transformer (41.51 for the paper's run, 42.44 from Tay et al.) and the best model (Sparse Transformer at 44.24). On ListOps (35.33), it trails the Reformer (37.27) and matches the Synthesizer (36.99). On Pathfinder (77.80), it outperforms most models including the Transformer (71.40 from Tay et al., 80.38 from the paper's run — note the substantial discrepancy between the paper's Transformer run and Tay et al.'s, which the authors attribute to hyperparameter differences). The Linear baseline (54.16) underperforms FNet on average, suggesting that for long-sequence tasks, the DFT's structure provides an advantage over learned dense mixing.

Speed results on GPUs (Table 4b). The training speed comparison is the paper's strongest LRA result. On 8 V100 GPUs at sequence length 512, FNet achieves 43 steps/second — 2.0× faster than the Transformer (21 steps/s) and 1.5× faster than the Performer (28 steps/s). As sequence length increases, FNet's advantage grows: at 1024 tokens, FNet is 2.3× faster than the Transformer (24 vs. 10 steps/s) and 1.6× faster than the Performer (24 vs. 15 steps/s). At 2048 tokens, FNet is 3.2× faster than the Transformer (14 vs. 4 steps/s) and 1.6× faster than the Performer (14 vs. 9 steps/s). At 4096 tokens, the Transformer goes OOM, while FNet achieves 7 steps/s versus the Performer's 4 steps/s (1.8× faster). At 8192 tokens, FNet achieves 4 steps/s versus the Performer's 2 steps/s (2.0× faster). The Linear model is also competitive but scales worse: at 2048 tokens, the Linear model achieves 9 steps/s versus FNet's 14 steps/s, and at 4096 tokens, both achieve 4 steps/s before the Linear model goes OOM at 8192. This demonstrates the FFT's $O(N \log N)$ advantage over dense matrix multiplication's $O(N^2)$ at long sequence lengths.

Memory results on GPUs (Table 4b, right side). FNet's peak memory usage is lower than all alternatives at every sequence length. At 512 tokens: FNet 0.8 GB, Transformer 1.6 GB, Performer 1.1 GB. At 2048 tokens: FNet 2.2 GB, Transformer 12.2 GB, Performer 3.1 GB — FNet uses 5.5× less memory than the Transformer. At 4096 tokens: FNet 3.9 GB, Performer 5.5 GB (Transformer OOM). At 8192 tokens: FNet 7.4 GB, Performer 10.4 GB. The Linear model also shows lower memory than the Transformer but higher than FNet: at 2048 tokens, Linear uses 2.8 GB versus FNet's 2.2 GB. The memory advantage comes from two sources: FNet has no learnable parameters in its mixing sublayer (eliminating optimizer state storage for those parameters), and the FFT's compute pattern requires less activation memory than attention's $O(N^2)$ attention matrices.

Inference speed on GPUs (Table 4c). On the LRA Text classification task, FNet delivers consistently faster inference than both the Transformer and the Performer at all sequence lengths. At 512 tokens: FNet 8 ms/batch versus Transformer 12 ms/batch (1.5× faster) and Performer 11 ms/batch (1.4× faster). At 2048 tokens: FNet 23 ms/batch versus Transformer 76 ms/batch (3.4× faster) and Performer 32 ms/batch (1.4× faster). At 4096 tokens: FNet 43 ms/batch versus Transformer 244 ms/batch (5.7× faster) and Performer 60 ms/batch (1.4× faster). At 8192 tokens: FNet 83 ms/batch versus Performer 116 ms/batch (1.4× faster) — the Transformer goes OOM. At 16384 tokens: FNet 164 ms/batch versus Performer 238 ms/batch (1.5× faster). The speedup multiples over the Transformer grow impressively with sequence length (1.5× → 2.3× → 3.4× → 5.7×), directly reflecting the FFT's $O(N \log N)$ complexity versus attention's $O(N^2)$. The Linear model is also competitive on inference at shorter lengths (9 ms/batch at 512, 1.4× faster than the Transformer) but scales worse than FNet at long lengths (72 ms/batch at 4096 vs. FNet's 43 ms/batch — 1.7× slower).

Speed results on TPUs (Table 7, Appendix A.2). The TPU picture is more nuanced. For training at sequence lengths ≤ 1024, FNet using matrix multiplication is competitive with or faster than the Performer (9.5 vs. 9.2 steps/s at 512; 9.1 vs. 8.4 steps/s at 1024). At 2048 tokens, the Performer overtakes FNet (6.9 vs. 6.1 steps/s for matrix multiplication, 3.2 steps/s for FFT). At 4096 tokens, the Performer is substantially faster (4.2 vs. 3.0 steps/s for matrix multiplication, 1.6 steps/s for FFT). At 8192 tokens, the Performer achieves 2.2 steps/s versus FNet's 0.8 steps/s (both FFT and matrix multiplication variants converge here). The paper attributes this TPU disadvantage to the "less optimized FFT implementation on TPUs relative to GPUs" (paraphrased from Section 3.3), and expresses expectation that this gap will narrow. For inference on TPUs, a similar pattern holds: at 512 tokens, FNet with matrix multiplication (6.0 ms/batch) is comparable to the Performer (6.1 ms/batch) and slightly faster than the Transformer (7.0 ms/batch). At 4096 tokens, the Performer (17.5 ms/batch) is faster than FNet with matrix multiplication (40.7 ms/batch) and FNet with FFT (58.8 ms/batch). However, the paper notes that the Performer's speed advantage at 4096 tokens (2.3× over FNet-mat) shrinks when considering that FNet is more accurate on the LRA benchmark overall (Table 4a): the Performer averages 51.41 versus FNet's 55.30.

Memory results on TPUs (Table 7, bottom). FNet has the lowest peak memory usage on TPUs at all sequence lengths. At 512 tokens: FNet 0.8 GB, Transformer 1.1 GB, Performer 1.0 GB. At 4096 tokens: FNet 2.0 GB (FFT) / 2.2 GB (mat), Transformer OOM, Performer 3.0 GB. At 16384 tokens: FNet with FFT uses 6.3 GB versus the Performer's 9.6 GB. This memory advantage is hardware-independent: it holds on both GPUs and TPUs, and grows with sequence length.

Ablation Studies and Robustness Checks

Fourier sublayer design choices (Appendix A.3). Real-part extraction position: Extracting the real part after each 1D DFT (rather than only at the end of the 2D DFT as in Equation 3) "was less accurate and less stable during training." Extracting the absolute value instead of the real part "led to a significantly less accurate model." These findings confirm that the specific computation $y = \Re(\mathcal{F}_{\text{seq}}(\mathcal{F}_h(x)))$ with real extraction only at the end is important — likely because complex intermediate interactions between the two DFTs produce meaningful cross-terms that are discarded if real parts are extracted prematurely.

1D vs. 2D DFT (Appendix A.3). Using only a 1D DFT along the sequence dimension (omitting hidden-dimension mixing) "yielded some training speed gains but hurt accuracy" — the authors do not quantify the accuracy degradation but state that this variant "still significantly outperformed the (no token mixing) FF-only model," confirming that token mixing is the primary function of the Fourier sublayer and hidden-dimension mixing provides additional benefit at moderate cost.

Alternative transforms to the DFT (Appendix A.3). Three alternatives were tested. The Discrete Cosine Transform (DCT) "underperformed FNet (∼4% accuracy degradation)." The Hadamard Transform "yielded less accurate results (∼2% accuracy degradation)" despite being slightly faster than the DFT. The Hartley Transform, defined as $H = \Re\{F\} - \Im\{F\}$ (the real part minus the imaginary part of the Fourier Transform), "matched the Fourier Transform on GLUE (76.7 vs. 76.7)." This is a notable negative result for the DCT and Hadamard, and an intriguing positive result for the Hartley — it suggests that transforms closely related to the DFT (the Hartley is essentially a real-valued variant) work well, and that the DFT is not uniquely effective but belongs to a class of structured, orthogonal, global transforms that serve as viable mixing mechanisms. The paper does not deeply analyze why the Hartley matches the DFT while the DCT (also real-valued) underperforms — likely, the Hartley's relationship to the DFT (preserving the full frequency spectrum in real form) is closer to the DFT's complex structure than the DCT's cosine-only basis.

Adding learnable parameters to the Fourier sublayer (Appendix A.3). Multiple attempts to inject learned parameters were "either detrimental or inconsequential, and generally slightly slowed the model." Specifically: element-wise multiplication with a learnable matrix, regular matrix multiplication with learnable sequence and hidden matrices, applying learnable weights before or after the DFT (including wrapping the DFT with learned pre- and post-multiplications, analogous to an inverse-DFT sandwich), and adding complex learnable weights to the DFT matrix itself. The complex-weight variant "was stable but did not yield any accuracy gains, suggesting that the DFT is locally optimal in some sense." This is a striking finding: adding parameters to the Fourier sublayer does not help. The DFT's fixed weights already occupy a point in the optimization landscape that additional parameters cannot improve upon — or the feed-forward layers are sufficiently powerful to adapt to the fixed DFT basis, making additional mixing parameters redundant. The authors do not explore this interpretation in depth but the result is methodologically important for anyone attempting to "improve" FNet by parameterizing the Fourier sublayer.

FNet block structure modifications (Appendix A.3). Several variants that attempt to mimic convolution more closely (sandwiching Fourier, feed-forward, inverse Fourier sublayers with residuals only at the final output) "degraded accuracy and lead to a more unstable model during training." Adding extra feed-forward sublayers or swapping them for simpler dense layers "did not help either." The standard Transformer encoder block structure (mixing → residual → layer norm → feed-forward → residual → layer norm) appears to be important independent of the mixing mechanism, and FNet's best configuration is the most conservative one: plug the DFT directly into the standard Transformer block without structural modifications.

FNet-Hybrid attention layer count and position (Appendix A.5, Table 9). The Hybrid model ablations test the number of attention sublayers (0, 2, 4, 6 in a 12-layer Base configuration) and their position (BOTTOM, MIDDLE, MIXED, TOP). Headline findings: more attention improves accuracy with diminishing returns (MLM accuracy: 0 attention layers → 0.486, 2 layers → 0.526, 4 layers → 0.539, 6 layers → 0.546 — per 100k steps), and position matters substantially. TOP placement (final layers) achieves 0.526 MLM accuracy for 2 attention layers versus BOTTOM at 0.497, MIDDLE at 0.499, and MIXED at 0.509. The TOP advantage is consistent: at the final encoder layers, attention's input-dependent mixing is most valuable because these layers produce the representations directly consumed by downstream task heads. The Hybrid configuration used in the main experiments (2 attention layers at TOP) is therefore not arbitrary but represents the empirically optimal position within the explored design space. The authors do not test more than 6 attention layers or configurations with attention in every layer except the first few (i.e., inverse Hybrid), limiting the exploration to the low-attention regime.

Hardware-dependent DFT implementation (Section 3.3, Table 7). The choice between FFT and matrix multiplication for the DFT is not treated as a hyperparameter but as a hardware-adaptive implementation strategy. On GPUs, FFT is faster at all sequence lengths. On TPUs, matrix multiplication "cached DFT matrix" is faster for sequences ≤ 4096, and FFT is faster for longer sequences. The paper demonstrates this with Table 7 training speed numbers: at 2048 tokens on TPU, FNet with matrix multiplication achieves 6.1 steps/s versus 3.2 steps/s with FFT — a 1.9× difference that flips the speed ranking. At 8192 tokens, both variants converge to 0.8 steps/s, suggesting that at very long lengths, the FFT's asymptotic advantage is offset by TPU implementation inefficiencies. The paper also finds that "DEFAULT XLA precision was sufficient to facilitate accurate model convergence" even though "HIGHEST XLA precision was required to very accurately reproduce FFT" — a practical finding that allows faster, lower-precision matrix multiplication-based DFT without accuracy loss.

Mixing sublayer speed isolation (Appendix A.4, Table 8). When all other model components are removed to measure pure mixing sublayer speed, FNet's advantage over attention is dramatic. For Base models on GPU: the Fourier sublayer (FFT) completes training passes at 11 ms/batch versus self-attention's 136 ms/batch — a 12.2× speedup. For Large models: 18 ms/batch versus 404 ms/batch — a 22.2× speedup. The Linear model's mixing sublayer is also fast but less dramatic: 36 ms/batch for Base (3.7× speedup over attention) and 103 ms/batch for Large (3.9×). The gap between mixing sublayer speedup (12.2× for FNet-Base) and overall model speedup (1.8×, Table 3) reveals that the feed-forward sublayers — which all models share — consume the majority of wall-clock time. This highlights a design tension: further speeding up the mixing sublayer yields diminishing returns because the feed-forward layers dominate total computation. The implication is that the next bottleneck to address is the feed-forward sublayer itself, not the mixing mechanism.

Learning rate sensitivity for smaller models (Section 4.1, Figure 2). The speed-accuracy trade-off curves (Figure 2, Figure 3) use the best of two learning rates ($10^{-3}$ and $10^{-4}$) per model, based on the observation that "smaller model architectures benefited from larger learning rates." Without this adjustment, smaller FNet and Linear models would appear less accurate at a given speed — the learning rate sweep ensures the comparison is fair across model sizes with different optimal hyperparameters. The paper does not report how much this adjustment matters quantitatively.

LRA hyperparameter adaptation (Appendix A.6). The authors found discrepancies between the LRA configurations described in Tay et al. (2021a) and the accompanying codebase, requiring several adaptations. For the CIFAR-10 (Image) task, the authors swept over the number of layers in {1, 2, 3, 4} and found 1 layer worked best for all models — contradicting Tay et al.'s recommendation of 3 layers. For Pathfinder, a learning rate of 0.001 (codebase default) outperformed 0.01 (paper specification). Small sweeps were also performed over embedding dimension and batch size, which are not listed in Tay et al. These adaptations mean the LRA accuracy comparisons are not perfectly controlled — FNet and the authors' Transformer/Linear baselines benefit from the hyperparameter sweeps, while the efficient Transformer results quoted from Tay et al. may reflect suboptimal hyperparameters. The authors acknowledge this with the caveat that "the accuracy comparisons between our runs and those from Tay et al. should be performed with the caveat that we found that results for certain tasks — Text and Retrieval in particular — can vary quite a bit between runs" (Appendix A.6). This somewhat weakens the strength of the LRA accuracy claims but does not affect the speed and memory comparisons (which are measured directly on identical hardware at controlled sequence lengths).

Position embeddings not strictly necessary for FNet (Section 3.2). The paper notes that "FNet performs just as well without position embeddings" because the DFT's twiddle factors $e^{-\frac{2\pi i}{N}nk}$ natively encode absolute position. However, all experiments include position embeddings for clean comparison with BERT, so this claim is based on preliminary experiments rather than the paper's main results. An explicit ablation showing FNet's GLUE scores with and without position embeddings is not provided, nor is the exact accuracy difference quantified. This is a missed opportunity: demonstrating that FNet matches or exceeds its reported accuracy without position embeddings would strengthen the claim that the DFT provides intrinsic positional encoding, and would further reduce the model's parameter count (the position embedding matrix is $N \times d_h$, which for BERT-Base's 512 × 768 = ~393K parameters is a small but non-trivial fraction of the 83M total).

Critical Assessment

Claim: FNet achieves 92–97% of BERT's accuracy on GLUE while training 80% faster on GPUs and 70% faster on TPUs. The experimental evidence for this claim is strong and well-documented. Table 2 directly supports the accuracy percentages: FNet-Base's 76.7 average is 92.1% of BERT-Base's 83.3, and FNet-Large's 81.9 is 96.7% of BERT-Large's 84.7. Table 3 directly supports the speed claims: 1.8× faster training on GPU (169 vs. 305 ms/batch), 1.7× on TPU (128 vs. 213 ms/batch). However, two important qualifications apply. First, the speed gains are measured at the specific batch sizes (64 on GPU, 256 on TPU) and sequence lengths (512) used in the paper. At different batch sizes or sequence lengths, the speedup multiples may differ — the LRA results (Table 4b) show that the speedup grows with sequence length on GPUs but shrinks on TPUs (Table 7). Second, the 80%/70% numbers from the abstract are GPU/TPU training speedups specifically, not inference speedups (which are 80%/40% respectively per Table 3). This matters for deployment scenarios where inference latency is the primary concern.

Claim: FNet matches the accuracy of the most accurate efficient Transformers on LRA while being the fastest on GPUs across all sequence lengths. The accuracy claim is supported with appropriate caution. Table 4a shows FNet at 55.30 average versus BigBird at 55.01 and the Transformer at 54.39–55.83. The differences are small enough that ranking is uncertain, and the paper correctly claims competitiveness rather than superiority. The speed claim on GPUs is strongly supported: Table 4b shows FNet outperforming both the Transformer and the Performer (the fastest efficient Transformer from Tay et al., 2021a) at all sequence lengths from 512 to 8192, with the speedup multiple growing from 1.5× over the Performer at 512 to 2.0× at 8192. The claim is hardware-specific: on TPUs (Table 7), the Performer is faster than FNet at sequence lengths ≥ 2048 for training and ≥ 4096 for inference. The paper is transparent about this limitation but frames it as a temporary condition ("we suspect that FNet will only become more performant on TPUs as the TPU implementation of the FFT improves"). Whether this optimism is warranted depends on external factors (TPU FFT development) that the paper cannot control. A reader evaluating FNet for TPU deployment should rely on Table 7, not the GPU-centric claims in the abstract.

Claim: The feed-forward sublayers, not the attention mechanism, are the primary drivers of representational learning in Transformer encoders. This is the paper's most provocative implicit claim, and the evidence for it is suggestive but incomplete. The key data point is that FNet (fixed, unparameterized mixing) achieves 92% of BERT's accuracy while BERT's attention sublayer is 12.2× more expensive in isolation (Table 8). However, this does not directly demonstrate that feed-forward layers are the primary learners — it demonstrates that either the feed-forward layers can compensate for weaker mixing, or that the DFT happens to provide mixing patterns that are nearly as useful as learned attention. To distinguish these hypotheses, one would need experiments that test whether FNet's feed-forward layers learn different representations than BERT's (e.g., through probing or representational similarity analysis), or experiments that vary feed-forward capacity independently of mixing mechanism capacity. The paper provides no such analysis. The FF-only baseline (49.3 GLUE) and Random baseline (56.6 GLUE) demonstrate that the mixing mechanism matters greatly, which partially undermines the "feed-forward layers do everything" interpretation. A more precise version of the claim would be: for tasks within the GLUE distribution, a fixed global mixing mechanism combined with deep feed-forward layers can approximate most of what learned attention provides, but the residual gap (8% for Base, 3% for Large) reflects capabilities that require input-dependent mixing.

Claim: The Linear model achieves comparable accuracy to FNet while being much slower and more memory-intensive. This is strongly supported by the Base-scale results: Linear-Base (77.0) is only 0.3 points above FNet-Base (76.7) on GLUE (Table 2), yet Linear-Base has 11M more parameters (94M vs. 83M, Table 1), trains slower (199 vs. 169 ms/batch on GPU, Table 3), and uses more memory (Table 4b). The Linear model's advantage in learned mixing is real but negligible at Base scale. However, the Linear-Large result (59.8 GLUE average, Table 2) complicates the narrative: the near-parity breaks at scale due to training instability. This suggests the comparison is not purely about mixing mechanism quality — at larger scales, parameter-free mixing provides a stability advantage that learned mixing lacks. The paper does not investigate whether training instability is inherent to learned dense mixing at scale or could be mitigated through better initialization, normalization, or optimization. A fairer comparison would explore stabilizing the Linear-Large model (e.g., through gradient clipping, learning rate warmup, or architectural regularization) to determine whether the Base-scale near-parity extends to Large scale under optimized training.

Missing experiments that would strengthen the paper. Several experiments are notably absent. (1) The paper never combines FNet's Fourier mixing with the revision mechanisms or PRM-guided search explored in other works — while these are not the paper's focus, testing whether FNet's representations are compatible with standard test-time techniques would increase its practical relevance. (2) The paper provides no analysis of what the Fourier Transform's mixing patterns actually look like in practice (e.g., visualizing which frequency components in which layers correlate with specific linguistic phenomena). Probing experiments analogous to those conducted on BERT (Clark et al., 2019; Tenney et al., 2019) would reveal whether FNet's internal representations encode syntactic and semantic information similarly to BERT, or whether it solves tasks through fundamentally different representational strategies. (3) The paper provides no distillation experiments despite explicitly suggesting this as future work (Section 5). Distilling a larger BERT teacher into a smaller FNet student would test whether FNet's accuracy ceiling is higher when guided by a stronger model. (4) The position embedding ablation is mentioned but not quantified. (5) The paper does not evaluate FNet on any generation tasks (only classification/regression), limiting the scope of its claims about mixing mechanisms. (6) The difficulty-dependent analysis that characterizes more recent compute-optimal scaling work is entirely absent — the paper treats all GLUE tasks as a flat benchmark without investigating which task characteristics make attention more or less necessary. (7) No experiment varies the feed-forward capacity independently to test whether larger feed-forward layers can compensate for weaker mixing (or vice versa).

Test set size and statistical reliability. The GLUE validation set used has 8 tasks with varying sizes (MNLI has ~10K matched + ~10K mismatched examples; RTE has only 277). The averages reported in Table 2 do not include confidence intervals, and the best-of-N-trials protocol (3 trials for Base, 6 for Large) selects the maximum, potentially overestimating each model's expected performance. The 0.3-point gap between FNet-Base and Linear-Base may not be statistically significant given the small trial counts and the use of maximum rather than mean performance. LRA accuracy results are equally fragile: the paper notes that Text and Retrieval accuracy "can vary quite a bit between runs" (Appendix A.6), and single-task differences of 1–3 points between top models (Table 4a) are unlikely to be statistically reliable. The speed and memory results (Tables 3, 4b, 4c, 7) are measured quantities and do not suffer from statistical noise, making them the paper's most robust contributions.

Hardware specificity of speed claims. The speed results are measured on specific hardware configurations: 8 V100 GPUs and $4 \times 4$ TPU v3 chips. V100 GPUs are a mature but not cutting-edge architecture (introduced 2017); newer GPUs (A100, H100) have different FFT and matrix multiplication performance characteristics. The paper's finding that FFT is uniformly faster on GPUs while matrix multiplication is faster on TPUs for shorter sequences may not generalize to other hardware. The paper's value is in establishing the hardware-adaptive implementation strategy as a principle, not in providing universal speedup numbers.

The random baseline deserves more attention. The Random model's 56.6 GLUE average (Table 2) is 20.1 points below FNet but 7.3 points above the FF-only model's 49.3. This substantial gap between random and structured mixing is one of the paper's most interesting findings but receives minimal analysis. The DFT's specific structure — orthogonal, sinusoidal, globally connected — matters enormously. Understanding why it matters (do the sinusoidal basis functions happen to align with linguistic patterns? Is orthogonality the key property? Would any orthogonal transform work equally well?) would substantially strengthen the paper's contribution. The Hartley Transform matching the DFT (Appendix A.3) provides a partial answer, suggesting that the class of transforms related to the Fourier basis works well. But the DCT's underperformance (4% degradation) indicates that not all sinusoidal transforms are equal. The paper does not investigate what specific property of the DFT (complex basis? negative frequencies? specific phase structure?) drives the advantage.

Pre-training data and configuration bias toward BERT. The pre-training configuration is "lifted from Devlin et al. (2019)" and "may be slightly biased towards the BERT attention model" (Appendix A.1). This is a frank acknowledgment, but its implications are unexplored. If BERT's hyperparameters (learning rate schedule, warmup, dropout rates, weight decay, layer initialization) were optimized for attention-based models, then FNet's reported accuracy is a lower bound on what it could achieve with FNet-specific optimization. The finding that smaller models benefit from larger learning rates (used in Figure 2) hints at the magnitude of this effect. A full hyperparameter sweep for FNet could potentially narrow the 8% GLUE gap further.

Scale limitations. The largest configuration tested is BERT-Large scale (24 layers, ~340M parameters for BERT, ~238M for FNet-Large). Modern LLMs operate at 1B–100B+ parameters. The paper's claims about the feed-forward sublayers doing the "heavy lifting" may not generalize to extreme scales where attention's expressivity becomes necessary to coordinate information across very deep networks. The Linear-Large instability (collapse from 77.0 to 59.8) is a warning sign: as scale increases, the interaction between mixing mechanism and training dynamics changes qualitatively, and parameter-free mixing's stability advantage may be more important than any representational properties of the specific mixing transform.

The paper's most robust contributions are the speed and memory results, not the accuracy parity. The paper's lasting value is likely not "FNet achieves 92% of BERT's accuracy" — that number depends on the specific benchmark, model scale, and training configuration — but rather the demonstration that a parameter-free, structured linear transform can serve as a viable mixing mechanism, combined with the careful hardware-adaptive implementation strategy that maximizes its efficiency. The framing that "rather than seeking more efficient approximations of the attention, there may be more value in seeking out completely new mixing mechanisms" (Section 1) is supported by the evidence, but the paper's own exploration of this design space is limited to the DFT and three alternatives (Hartley, Hadamard, DCT). Whether other transforms (wavelets, learned orthogonal matrices, structured sparse patterns) offer better speed-accuracy trade-offs remains an open question that the paper motivates but does not answer.

6. Limitations and Trade-offs

Difficulty Estimation Is Not Separated from Problem Solving — the Paper Provides No Practical Mechanism for Knowing When to Prefer FNet Over BERT

The assumption or constraint. The paper's central claim is that FNet offers a better speed-accuracy trade-off than BERT for specific deployment regimes — particularly smaller, faster models (Figure 2, left side of the Pareto frontier). However, the paper provides no mechanism for a practitioner to determine, for a given task or input, whether FNet's accuracy is sufficient or whether BERT's additional expressivity is necessary. The paper treats the GLUE benchmark and LRA benchmark as flat evaluation surfaces, without any difficulty-conditioned analysis: all inputs receive the same mixing mechanism regardless of their complexity.

This matters because the GLUE average masks substantial per-task variance. Table 2 shows FNet-Base scoring 95 on SST-2 (slightly outperforming BERT-Base's 93) but only 63 on RTE (versus BERT-Base's 69) and 76 on MRPC (versus BERT-Base's 83). FNet-Large scores 88 on MRPC (exceeding BERT-Large's 86) but only 69 on RTE (versus BERT-Large's 66 — here FNet actually leads). The pattern is not uniform: FNet excels on some tasks and underperforms on others, and a practitioner deploying FNet in production has no way to know a priori which tasks or inputs will suffer from the loss of attention. The paper does not provide a difficulty estimator, a confidence calibration analysis, or any per-example analysis that would enable selective routing (using FNet for easy examples, escalating to BERT for hard ones).

The consequence. Without difficulty estimation or selective routing, a practitioner must choose between (a) deploying FNet universally and accepting degraded performance on the subset of tasks/inputs where attention matters, or (b) deploying BERT universally and paying the speed/memory penalty on all inputs, including those where FNet would have sufficed. The paper's speed-accuracy trade-off curves (Figure 2) are averaged over all examples; they do not reveal whether the accuracy gap is concentrated in a small fraction of hard examples (which could potentially be routed to a larger model) or distributed broadly (making universal FNet deployment uniformly worse).

This is a significant practical gap because the paper's framing — "FNet achieves 92% of BERT's accuracy" — implies a uniform 8% degradation, but the per-task results in Table 2 show degradation ranging from roughly -2% (SST-2, where FNet outperforms BERT) to roughly -13% (MNLI, with a 12-point gap on the matched set). A practitioner deploying FNet for natural language inference would experience substantially worse performance than the 92% headline number suggests, while a practitioner deploying for sentiment analysis would experience no degradation at all. The paper provides no guidance for making this deployment decision.

What evidence exists in the paper. The per-task GLUE breakdown in Table 2 is the only evidence for task-dependent accuracy variation. There is no per-example analysis, no confidence calibration, and no investigation of which input characteristics (sentence length, syntactic complexity, reasoning depth) correlate with performance degradation. The LRA benchmark results (Table 4a) show similar variance: FNet scores 65.11 on Text classification (matching the best models) but only 38.67 on Image classification (underperforming the Transformer by roughly 3 points). The paper does not discuss these task-level differences or attempt to explain them.

Mitigation status. Not addressed. The paper acknowledges in passing that the gap between BERT and FNet "shrinks to just 3% for Large models" (Section 4.1), attributing this to training stability rather than task characteristics, but never investigates which tasks benefit from attention and why. The Hybrid model (Section 4.1, Appendix A.5) could serve as a partial mitigation — using attention on a small subset of layers recovers most of the accuracy gap — but the paper does not explore whether a dynamic Hybrid (applying attention only when needed) is feasible.

---

All Results Are on a Single Model Family (BERT) and Two Benchmarks (GLUE, LRA) — with No Evidence That Findings Generalize to Other Architectures, Tasks, or Data Modalities

The assumption or constraint. The paper evaluates FNet exclusively on BERT-style encoder architectures pre-trained on C4 and fine-tuned on GLUE (classification/regression tasks) or LRA (long-sequence classification). This implicitly assumes that the findings about Fourier mixing transfer to other model families (decoder-only, encoder-decoder), other pre-training objectives (autoregressive language modeling, span corruption), other downstream tasks (generation, question answering with long outputs, structured prediction), and other data modalities (code, speech, video).

The authors explicitly restrict their scope:

"Throughout this work we have restricted our focus to encoders. FNet decoders can be designed by 'causally' masking the Vandermonde matrix, but a lower level implementation is required to introduce causal masking to FFTs. How to adapt Fourier mixing for encoder-decoder cross-attention is an open question as evidence suggests that cross-attention may be crucial to performance." (Section 5)

They also acknowledge that their tasks "do not require generation so we leave FNet decoders and encoder-decoder setups to future work." This is a candid boundary on the current results, but it is a consequential one: many of the most important Transformer applications (machine translation, text summarization, dialogue, code generation) require decoder or encoder-decoder architectures, and the paper provides zero evidence that Fourier mixing works in those settings.

The consequence. A practitioner working on a generation task (e.g., summarization, translation) or using a decoder-only architecture (e.g., GPT-style models) cannot infer from this paper whether replacing attention with Fourier mixing would work. The authors explicitly flag that cross-attention "may be crucial to performance" (citing You et al., 2020), and the paper provides no mechanism for cross-attention with Fourier mixing. This means FNet is currently inapplicable to a large fraction of production Transformer workloads.

Even within the encoder-only domain, the results are tied to BERT's specific architecture and pre-training recipe. The paper uses BERT's fixed configurations (12/24 layers, 768/1024 hidden dimensions, GELU activations) and pre-training objectives (MLM + NSP), trained on C4. Whether FNet's accuracy parity would hold with different pre-training objectives (e.g., ELECTRA-style replaced token detection), different model shapes (deeper but narrower, wider but shallower), or different corpora (code, multilingual text, scientific literature) is unknown. The paper's finding that the Linear-Large model collapses (59.8 GLUE, Table 2) while FNet-Large does not (81.9) suggests that the interaction between mixing mechanism and training dynamics is sensitive to scale and model family — FNet's stability advantage at BERT-Large scale may not generalize to other configurations.

What evidence exists in the paper. The paper's results are exclusively on BERT-Base/Large configurations (Table 6) evaluated on GLUE (Table 2) and LRA (Table 4). The speed-accuracy trade-off curves (Figure 2, Figure 3) sweep model sizes within the BERT family but do not test alternative architectures (e.g., ALBERT, RoBERTa, ELECTRA, T5 encoder) or alternative pre-training objectives. The paper's exploration of alternative transforms (Appendix A.3) is limited to the Hartley, Hadamard, and DCT as drop-in replacements for the DFT in the FNet architecture — not an exploration of how these transforms perform in different architectural contexts.

Mitigation status. Partially acknowledged but not addressed. The authors are transparent about the encoder-only scope and decoder/encoder-decoder as future work (Section 5). They do not discuss generalization to other encoder architectures, pre-training objectives, or modalities. The paper's suggestion that "the FNet encoder could be used as a drop in replacement in a Transformer as other works have successfully demonstrated; see, for example, (Zaheer et al., 2020; Guo et al., 2021)" (Section 5) hand-waves toward broader applicability but provides no evidence.

---

The Speed Gains Are Bottlenecked by the Feed-Forward Sublayers — and the Paper Provides No Path to Reducing That Bottleneck While Preserving Accuracy

The assumption or constraint. The paper demonstrates that FNet's Fourier sublayer is dramatically faster than self-attention in isolation: 12.2× faster for Base models on GPU (11 ms vs. 136 ms per batch, Table 8) and 22.2× faster for Large models (18 ms vs. 404 ms). However, the overall model speedup is only 1.8× for Base and 1.8× for Large on GPU (Table 3) because the feed-forward sublayers — which are identical across BERT, FNet, Linear, Random, and FF-only models — consume the majority of total FLOPs and wall-clock time.

The GFLOPS breakdown (Table 3, last column) quantifies this: FNet-Base requires 62 GFLOPS per forward pass versus BERT-Base's 98 GFLOPS, a 37% reduction. But the FF-only model (which removes the mixing sublayer entirely) requires 59 GFLOPS — meaning the mixing sublayer accounts for only 3 GFLOPS in FNet-Base versus 39 GFLOPS in BERT-Base. The Fourier sublayer is 13× cheaper than self-attention, but total model FLOPs drop by only 37% because the feed-forward sublayers ($4d_h^2$ parameters and FLOPs per layer) dominate. The authors acknowledge this implicitly:

"FNet's overall training speed is impeded by the feed-forward sublayers that all models share." (Section 4.1)

The consequence. FNet's speed advantage is bounded above by roughly 2× (the ratio of total FLOPs: 98/62 ≈ 1.6, plus some additional benefit from reduced memory pressure and faster backward passes). No amount of optimization to the Fourier sublayer — making it infinitely fast, even — can reduce total model latency below the feed-forward cost. This means FNet is not a path to order-of-magnitude speedups over BERT; it is a path to a 1.5–2× constant-factor improvement, after which further gains require addressing the feed-forward bottleneck.

For practitioners, this has a direct implication: if a 2× speedup is insufficient for their deployment constraints, FNet alone does not solve their problem. They would need to combine FNet with feed-forward optimizations (e.g., quantization, pruning, distillation, Mixture-of-Experts) that are orthogonal to the mixing mechanism. The paper provides no analysis of how FNet interacts with these techniques — for example, whether FNet's representations are more or less amenable to quantization than BERT's, or whether distilled FNet students converge as well as distilled BERT students.

The speedup numbers are also batch-size-dependent and sequence-length-dependent. The paper's headline 1.8× GPU training speedup is measured at batch size 64, sequence length 512. At different batch sizes (which affect GPU utilization and memory bandwidth saturation) or sequence lengths (which change the relative cost of mixing vs. feed-forward), the speedup multiple may differ. The LRA results (Table 4b) show FNet's training speedup over the Transformer growing from 2.0× at 512 tokens to 3.2× at 2048 tokens — the advantage grows because the mixing cost becomes a larger fraction of total compute as sequence length increases. But at BERT's standard 512-token length, the mixing cost is a relatively small fraction, capping the speedup.

What evidence exists in the paper. Table 3 (GFLOPS column) and Table 8 (mixing sublayer isolation) directly quantify the bottleneck. The FF-only model's speed (Table 3: 162 ms/batch training on GPU, 1.9× faster than BERT-Base) establishes the upper bound on speed gains from optimizing the mixing sublayer — FNet's 169 ms/batch is close to this bound. Figure 2 shows that the speed-accuracy frontier is defined by different models at different speed regimes, but even the fastest models (small FNet and Linear variants) are only 3–4× faster than BERT-Base, not 10× or 100×.

Mitigation status. Not addressed. The paper does not propose any method for reducing feed-forward cost, does not analyze how FNet composes with orthogonal efficiency techniques, and does not discuss the constant-factor nature of its speedup. The suggestion that FNet is suitable as "a lightweight, distilled student model" (Section 5) implicitly acknowledges that further compression is desirable, but the paper provides no distillation experiments.

---

The LRA Benchmark Reveals That FNet's Accuracy and Speed Advantages Are Hardware-Dependent — and the Paper's Strongest Speed Claims Are GPU-Specific

The assumption or constraint. The paper's headline speed claims — "training 80% faster on GPUs and 70% faster on TPUs" (abstract), and FNet being the fastest model on LRA "across all sequence lengths on GPUs" (Section 1) — hide a significant hardware dependence. On GPUs, FNet is uniformly faster than all efficient Transformers at all sequence lengths (Table 4b, Table 4c). On TPUs, the story reverses: the Performer is faster than FNet at sequence lengths ≥ 2048 for training and ≥ 4096 for inference (Table 7).

The root cause is the relative maturity of FFT implementations on each platform. The authors state:

"We suspect that FNet will only become more performant on TPUs as the TPU implementation of the FFT improves." (Section 3.3)

This is a forward-looking statement, not a current fact. At the time of writing, the TPU FFT is slow enough that the paper's TPU implementation uses matrix multiplication for sequences ≤ 4096 — effectively reverting to $O(N^2)$ complexity for the DFT on the platform where matrix multiplications are most optimized. The practical consequence is that FNet's efficiency advantage is not hardware-independent; it is strong on GPUs, modest on TPUs for short sequences, and reversed on TPUs for long sequences (relative to the Performer).

The consequence. A practitioner deploying on TPUs — which are Google's primary ML accelerator and widely used for Transformer training — cannot expect FNet to be the fastest option at long sequence lengths. Table 7 shows that at 2048 tokens on TPU, the Performer trains at 6.9 steps/s versus FNet-mat at 6.1 steps/s (1.1× slower) and FNet-FFT at 3.2 steps/s (2.2× slower). At 4096 tokens, the Performer is at 4.2 steps/s versus FNet-mat at 3.0 steps/s (1.4× slower) and FNet-FFT at 1.6 steps/s (2.6× slower). The advantage flips depending on hardware choice.

The paper's framing of this as a temporary condition that will resolve as TPU FFT improves is speculative. Hardware FFT implementations improve slowly (cuFFT took years to reach its current performance), and there is no guarantee that TPU FFT will close the gap with GPU FFT or with TPU matrix multiplication in the near term. A practitioner making deployment decisions today must evaluate FNet based on current TPU performance, not projected future improvements.

The inference speed picture on TPUs (Table 7, bottom) is similarly nuanced. At 512 tokens, FNet-mat inference (6.0 ms/batch) is comparable to the Performer (6.1 ms/batch). At 4096 tokens, the Performer (17.5 ms/batch) is 2.3× faster than FNet-mat (40.7 ms/batch). The paper argues that FNet compensates with higher accuracy (55.30 vs. 51.41 LRA average, Table 4a), but this is an accuracy-for-speed trade-off that depends on the practitioner's priorities — it is not a Pareto improvement.

What evidence exists in the paper. Table 4b and Table 4c provide GPU speed/memory results showing FNet's dominance. Table 7 (Appendix A.2) provides TPU results showing the Performer's advantage at longer lengths. The comparison between these tables reveals the hardware dependence. The paper's TPU implementation strategy (matrix multiplication for ≤ 4096, FFT for longer) is described in Section 3.3, but the practical implications — that the DFT is effectively $O(N^2)$ on TPUs for sequences up to 4096 — are not prominently discussed.

Mitigation status. Partially acknowledged. The paper notes the hardware dependence in Section 3.3 and Appendix A.2, and expresses optimism about TPU FFT improvements. However, the abstract and introduction emphasize the GPU results without equivalent prominence for the TPU caveats. The statement that FNet is "significantly faster... across all sequence lengths on GPUs (and across relatively shorter lengths on TPUs)" is present in the abstract but the qualification about longer TPU sequences is easy to miss.

---

The Paper Does Not Investigate Whether Fourier Mixing Preserves Linguistic Structure in Ways That Matter for Interpretability, Fairness, or Robustness

The assumption or constraint. The paper evaluates FNet solely on accuracy and speed, with no analysis of what the model learns, whether its representations encode linguistic structure similarly to BERT, or whether the choice of mixing mechanism affects properties beyond accuracy — such as robustness to adversarial examples, fairness across demographic groups, calibration of confidence estimates, or interpretability of attention patterns.

This is an omission relative to the probing literature the paper itself cites (Tenney et al., 2019; Clark et al., 2019; Voita et al., 2019). Those works established that BERT's attention layers learn interpretable syntactic and semantic patterns (subject-verb agreement, coreference, semantic roles) at different depths. The paper's finding that a fixed Fourier Transform can recover 92% of BERT's accuracy raises the question: does FNet learn the same linguistic patterns through a different mechanism, or does it solve tasks through entirely different representational strategies that happen to produce correct answers on GLUE? The paper provides no probing experiments, no representational similarity analysis, and no investigation of what the frequency-domain representations actually encode.

The consequence. If FNet solves tasks through fundamentally different (and less interpretable) mechanisms than BERT, this has implications for deployment in high-stakes settings where model decisions need to be explainable. BERT's attention weights can be visualized and inspected; FNet's frequency-domain mixing provides no analogous interpretability hook. The DFT's mixing weights are fixed sinusoids — they do not reveal which tokens the model "attended to" in making a prediction. A practitioner in a regulated domain (healthcare, finance, legal) who needs to explain model decisions may find FNet's opacity disqualifying, even if its accuracy is comparable.

Additionally, the paper provides no robustness evaluation. Does FNet's fixed mixing make it more or less vulnerable to adversarial perturbations? More or less calibrated in its confidence estimates? More or less prone to exploiting spurious correlations? The paper does not investigate any of these questions, leaving the practitioner with no information about how the choice of mixing mechanism affects model behavior beyond aggregate accuracy.

What evidence exists in the paper. None. There are no probing experiments, no adversarial robustness tests, no calibration analysis, and no fairness evaluations. The paper is purely an accuracy-and-speed study. This is not a hidden flaw — the paper's scope is explicitly limited to demonstrating that Fourier mixing works — but it is a consequential omission for any practitioner considering deployment.

Mitigation status. Not addressed. The paper does not discuss interpretability, robustness, or fairness, and does not flag these as limitations or future work. The probing literature is cited in Section 2.2 to motivate the investigation of whether attention is necessary, but the paper never closes the loop by probing FNet to see what it learns instead.

---

The Training Instability of the Linear-Large Model Is Unexplained — and Raises Unanswered Questions About the Scalability of Learned Dense Mixing Versus Fixed Structured Mixing

The assumption or constraint. The Linear-Large model — which replaces self-attention with learned dense matrix multiplications — collapses to a GLUE average of 59.8 (Table 2), far below Linear-Base's 77.0 and FNet-Large's 81.9. The paper attributes this to "gradient blow up" and "training instabilities" (Section 4.1), and notes that "the Linear model and BERT were less stable than the models with no parameters in their mixing sublayers, namely the FNet, Random and FF-only models."

However, the paper provides no investigation of why learned dense mixing becomes unstable at scale while fixed Fourier mixing does not, and no attempt to mitigate the instability (through gradient clipping, modified initialization, learning rate scheduling, normalization changes, or architectural modifications). The finding is reported as an empirical observation without analysis.

The consequence. The Linear-Large collapse undermines a clean interpretation of the paper's core finding. If learned mixing were strictly better or equal to fixed mixing (as the Base-scale near-parity at 77.0 vs. 76.7 suggests), then the takeaway would be: "learning the mixing weights provides negligible benefit, so use the fastest fixed mixing mechanism." But the Large-scale collapse suggests a different story: learned mixing can be better than fixed mixing in principle, but is harder to optimize at scale. The near-parity at Base scale may reflect not the equivalence of learned and fixed mixing, but rather the difficulty of optimizing learned mixing at any scale — with the optimization failure becoming catastrophic at Large scale.

If this interpretation is correct, then FNet's advantage over the Linear model is not primarily about the representational adequacy of fixed mixing, but about the optimization benefits of removing parameters from the mixing sublayer. A better-optimized Linear model (with appropriate stabilization techniques) might substantially outperform FNet at both Base and Large scales. The paper provides no evidence either way, leaving a fundamental ambiguity about what drives the result.

The instability also raises questions about scaling further. The BERT-Large configuration (24 layers, 1024 hidden) is large by 2019 standards but small by current standards. If learned dense mixing becomes unstable at 24 layers, what happens at 48, 96, or 128 layers — the regime of modern large language models? Does Fourier mixing remain stable indefinitely (since it has no parameters to drift), or does it encounter its own scaling challenges (e.g., the fixed mixing pattern becoming a bottleneck for very deep representations)? The paper provides no evidence and no analysis.

What evidence exists in the paper. Table 2 reports the Linear-Large GLUE score (59.8). Table 5 shows Linear-Large pre-training metrics (MLM 0.65, NSP 0.85) that look reasonable — better than FNet-Large's 0.63 and 0.82 — suggesting the collapse occurs during fine-tuning rather than pre-training. Table 3 shows Linear-Large training speed (397 ms/batch on TPU, 1.3× faster than BERT-Large). No further diagnostics are provided.

Mitigation status. Not addressed. The paper does not attempt to stabilize the Linear-Large model, does not perform any gradient analysis, and does not discuss whether the instability is fundamental (inherent to learned dense mixing at scale) or contingent (fixable with better optimization). The Linear model is abandoned at Large scale rather than debugged. This is a missed opportunity: understanding why learned dense mixing fails would illuminate whether fixed mixing is genuinely a superior design choice or merely a more easily optimized one.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper causes a reframing of the token mixing problem rather than a paradigm shift in the Kuhnian sense. It does not introduce a new learning algorithm, a new optimization technique, or a new class of tasks. Instead, it challenges the field's implicit assumption that token mixing must be learned and input-dependent to be effective, and demonstrates that a much simpler question — "what is the cheapest mechanism that provides sufficient cross-token connectivity for feed-forward layers to do their work?" — yields practical gains that the attention-centric literature had overlooked.

The magnitude of this reframing is moderate but genuine. Before this work, the efficient Transformer literature operated within a narrow design space: preserve attention's structural properties (token-dependence, multi-head architecture, softmax normalization) while reducing its computational cost through sparsity, low-rank approximation, or kernelization. The paper's contribution is to show that this entire design space may be solving the wrong optimization problem. The FF-only baseline (49.3 GLUE average, failing on STS-B entirely) proves that token mixing is necessary. The Random baseline (56.6 GLUE) proves that structure matters. FNet's 76.7 GLUE average — achieved with zero learned mixing parameters — proves that learnedness matters surprisingly little, and that input-dependence (the defining feature of attention) is not the indispensable property the field had assumed. The 0.3-point gap between FNet-Base (76.7) and Linear-Base (77.0) on GLUE is the paper's single most subversive data point: learning the mixing weights, at the cost of 11M additional parameters and a substantially larger memory footprint, buys almost nothing for the Base configuration on this benchmark.

This reframing shifts research priorities in two concrete ways. First, it makes mixing mechanism design a search for computational efficiency rather than representational expressivity. The paper's exploration of alternative transforms — Hartley matches DFT accuracy, Hadamard degrades ~2%, DCT degrades ~4% (Appendix A.3) — suggests a design space of structured, orthogonal, global linear transforms that is largely unexplored. The FFT's $O(N \log N)$ complexity and highly optimized GPU implementations make the DFT a strong initial choice, but the paper's thesis is that the specific transform is less important than getting the cheapest possible mechanism that provides structured global mixing. This directs attention away from the ever-more-elaborate attention approximations (sparse patterns, random features, low-rank factorizations) and toward fundamentally different mathematical operators chosen for their computational properties.

Second, it elevates the feed-forward sublayer from a supporting role to the primary locus of learning in encoder architectures. The paper does not make this argument explicitly, but the evidence compels it. Table 8 shows the Fourier sublayer completing forward and backward passes 12.2× faster than self-attention for Base models on GPU (11 ms vs. 136 ms per batch), yet the overall model is only 1.8× faster (Table 3). The feed-forward sublayers dominate both FLOPs (59 of 62 GFLOPS in FNet-Base, Table 3) and training time. If these sublayers also dominate learning — as FNet's strong performance with fixed mixing suggests — then the field's preoccupation with attention mechanism design has been optimizing the wrong component. The Hybrid results reinforce this: adding just 2 attention sublayers at the top of FNet recovers 97–99% of BERT's accuracy (Table 2), suggesting that attention's primary value is in the final layers where task-specific representations form, not in the intermediate layers where the bulk of representation learning occurs.

The paper resolves a latent tension in the probing literature that it does not directly address. Prior work (Tenney et al., 2019; Clark et al., 2019; Voita et al., 2019) demonstrated that BERT's attention layers encode rich syntactic and semantic structure. The natural interpretation was that attention is responsible for learning this structure. FNet's results suggest an alternative: attention reveals structure that the feed-forward layers have already encoded, and provides a mechanism for routing information between tokens that makes that structure accessible. If this interpretation is correct — and the paper provides suggestive but not definitive evidence — then the probing literature's findings about attention are partly findings about the feed-forward layers that attention makes visible. This would reconcile the apparent contradiction between "attention learns rich linguistic structure" and "replacing attention with a fixed Fourier Transform preserves 92% of accuracy": the feed-forward layers encode the structure, and attention (or the DFT, or any global mixing mechanism) merely provides the information routing that makes that structure usable for downstream computation.

Which research directions become more attractive? The paper implicitly argues that searching for efficient mixing mechanisms is now a first-class research problem, not a derivative of the attention-approximation literature. Convolutional mixing for text, structured sparse transforms (wavelets, Walsh-Hadamard, learned orthogonal matrices with fast transforms), and hardware-adaptive mixing mechanisms that choose different transforms based on sequence length or platform all become viable targets. The paper also makes feed-forward sublayer optimization more attractive: if these sublayers are the primary learners, then improving them (better activations, structured sparsity, wider intermediates with conditional computation) may yield larger gains than further attention improvements.

Which directions become less attractive? The paper casts doubt on the research program of making attention more efficient while preserving its exact functional form. If a fixed DFT can recover 92% of BERT's accuracy, then the marginal value of a slightly more efficient attention approximation — preserving token-dependence and multi-head structure at 80% of the cost rather than 60% — is unclear. The LRA results make this point empirically: FNet (55.30 average, Table 4a) is competitive with or exceeds the accuracy of BigBird (55.01), the Synthesizer (52.88), and the Performer (51.41), while being substantially faster on GPUs (Table 4b). The efficient Transformer literature's decade-long investment in attention-specific optimizations may have been solving a problem — "how to make attention cheap" — that the paper suggests is better solved by asking "what is the cheapest mechanism that provides connectivity?"

Follow-Up Research This Work Enables

Probing FNet's internal representations to determine whether it solves tasks through the same linguistic strategies as BERT, or through fundamentally different mechanisms that happen to produce correct answers on GLUE. The paper cites the probing literature (Tenney et al., 2019; Clark et al., 2019; Voita et al., 2019) to motivate the question of whether attention is necessary, but never probes FNet to see what replaces attention. A strong follow-up would replicate the structural probing experiments from Tenney et al. — measuring whether FNet's hidden states at each layer encode syntactic depth, part-of-speech, dependency relations, and coreference — and compare the depth-wise encoding profiles to BERT's. If FNet encodes syntactic structure at similar depths (e.g., subject-verb agreement emerges in middle layers for both models), this would support the hypothesis that feed-forward layers are the primary encoders of linguistic structure, with the mixing mechanism merely routing information. If FNet encodes structure at different depths or through different representational geometries, this would suggest that the mixing mechanism shapes how structure is learned even if the end-task accuracy is similar. The LRA Text classification task (65.11 for FNet, matching BERT, Table 4a) would be a natural testbed: document-level classification requires aggregating information across long ranges, and probing would reveal whether FNet achieves this through frequency-domain aggregation (low frequencies capturing document-level themes) or through a different mechanism.

Investigating whether the Linear-Large collapse is an optimization failure (fixable with better training strategies) or a fundamental representational limitation of learned dense mixing at scale. The paper reports that Linear-Large "severely underperforms its Base counterpart on the GLUE benchmark due to training instabilities" (GLUE average 59.8, Table 2), but provides no diagnostics and no stabilization attempts. A strong follow-up would systematically test stabilization strategies: gradient clipping, different initialization schemes (orthogonal initialization for the mixing matrices, scaled identity initialization), learning rate warmup ratios, layer normalization placement, and architectural modifications (factorizing the $N \times N$ sequence mixing matrix into lower-rank components). The key question is whether a stabilized Linear-Large can exceed FNet-Large's 81.9 GLUE average. If yes, then learned mixing has representational advantages that are currently masked by optimization difficulty, and the field should invest in making it trainable rather than abandoning it for fixed transforms. If no — if even a perfectly stabilized Linear-Large plateaus near FNet's accuracy — then fixed mixing is genuinely sufficient, and the research priority shifts to finding the most efficient fixed transform. The pre-training metrics in Table 5 are suggestive: Linear-Large achieves MLM accuracy 0.65 and NSP accuracy 0.85 during pre-training (better than FNet-Large's 0.63 and 0.82), but collapses during fine-tuning to 59.8 GLUE. This pattern — strong pre-training, catastrophic fine-tuning — is unusual and warrants investigation of whether the learned mixing matrices overfit to pre-training objectives in ways that fixed transforms cannot.

Exploring the design space of structured, orthogonal, global linear transforms as token mixing mechanisms, using the paper's Hartley/Hadamard/DCT results as a starting point. The paper cursorily tests three alternatives to the DFT (Appendix A.3) and finds the Hartley Transform matches DFT accuracy, the Hadamard degrades ~2%, and the DCT degrades ~4%. This suggests a class of transforms that work well, but the paper does not characterize the class. A strong follow-up would systematically evaluate transforms along axes that might determine effectiveness: orthogonality (is the DFT's unitary property important, or do non-orthogonal transforms like random projections work?), global connectivity (do transforms that mix all tokens outperform those with block-diagonal structure?), frequency-domain properties (do transforms that decompose signals into frequency components outperform those with no frequency interpretation?), and computational efficiency (what is the Pareto frontier of accuracy vs. FLOPS for candidate transforms?). Candidate transforms to test: the Discrete Sine Transform, the Chebyshev Transform, learned orthogonal matrices parameterized via the Cayley transform or matrix exponential (with fast structured multiplication), random orthogonal matrices (drawn from the Haar distribution), and wavelet transforms (which provide multi-scale, sparse mixing unlike the DFT's dense global mixing). The evaluation should include BERT-Base-scale pre-training on C4 with GLUE fine-tuning (replicating the paper's setup) and LRA benchmarking for long-sequence behavior. The Hartley Transform matching the DFT (both 76.7 GLUE) is the paper's most intriguing unreplicated result — if confirmed, it implies that the DFT's complex-valued computation (with real-part extraction) is unnecessary, and a purely real-valued transform suffices, which would simplify implementation and potentially improve speed on hardware where complex arithmetic is not well-optimized.

Testing FNet as a lightweight student model in a knowledge distillation pipeline from a larger BERT or T5 teacher, following the paper's explicit suggestion (Section 5). The paper hypothesizes that FNet "will be effective as a lightweight, distilled student model deployed in resource-constrained settings such as production services or on edge devices" (Section 5), but provides no distillation experiments. A strong follow-up would replicate the DistilBERT setup (Sanh et al., 2019): pre-train a BERT-Base teacher, then distill into FNet-Base (and smaller FNet variants from Table 6) using a combination of soft label distillation (matching the teacher's output distribution), hidden-state distillation (matching the teacher's intermediate representations at selected layers), and attention-map distillation (if applicable for the 2 attention layers in FNet-Hybrid). The key metric is whether a distilled FNet student can approach the teacher's accuracy more closely than the paper's independently pre-trained FNet-Base (76.7 GLUE vs. BERT-Base teacher at 83.3). If distillation narrows the gap — say, to 80+ GLUE for FNet-Base — this would demonstrate that FNet's accuracy ceiling is higher than the paper's independent pre-training results suggest, and that the speed-accuracy trade-off is even more favorable when a strong teacher is available. The paper's speed-accuracy trade-off curves (Figure 2) show FNet defining the Pareto frontier for smaller models; distillation could shift this frontier upward, making FNet strictly dominant over BERT at small scales. The TinyBERT-style two-stage distillation (task-agnostic pre-training distillation followed by task-specific fine-tuning distillation) would be a natural extension, and would test whether FNet's fixed mixing mechanism is compatible with standard distillation techniques that were developed for attention-based students.

Evaluating FNet on autoregressive language modeling and sequence-to-sequence tasks to determine whether Fourier mixing transfers to decoder and encoder-decoder architectures — or whether the paper's encoder-only scope reflects a fundamental limitation. The paper explicitly restricts its scope to encoders (Section 5), noting that "FNet decoders can be designed by 'causally' masking the Vandermonde matrix, but a lower level implementation is required to introduce causal masking to FFTs." A strong follow-up would implement causal Fourier mixing by modifying the FFT to operate on prefixes: for each position $n$, compute the DFT over tokens $0$ through $n$ only, preventing the model from attending to future tokens. This can be done either via online FFT (recomputing the transform at each position, which is $O(N^2 \log N)$ total — defeating the purpose) or via a causal Vandermonde matrix multiplication with a lower-triangular mask (which is $O(N^2)$ but may be acceptable at modest sequence lengths). The experiment would pre-train an FNet decoder on a standard autoregressive language modeling corpus (e.g., OpenWebText, the Pile) and evaluate perplexity and zero-shot downstream task performance against a GPT-2 or GPT-Neo baseline with matched parameter counts and training FLOPs. For encoder-decoder tasks, the experiment would pair an FNet encoder with a causal FNet decoder (or a standard Transformer decoder with cross-attention to the FNet encoder's outputs) and evaluate on machine translation (WMT) or summarization (CNN/DailyMail). The paper's hedging — "evidence suggests that cross-attention may be crucial to performance" (Section 5, citing You et al., 2020) — implies that even if encoder-side Fourier mixing works, the encoder-decoder interface may require attention. Testing this boundary directly would establish whether FNet is an encoder-specific curiosity or a general architectural principle that extends to the full Transformer family. A negative result (FNet decoders substantially underperform Transformer decoders at matched FLOPs) would be informative: it would suggest that the mixing requirements differ between bidirectional encoding (where global mixing suffices) and autoregressive decoding (where position-dependent, content-aware mixing is more important).

Measuring whether FNet's fixed mixing confers robustness advantages — to adversarial perturbations, distribution shift, and overfitting — relative to BERT's learned attention. The paper provides no robustness evaluation, but there are theoretical reasons to expect differences. FNet's mixing weights are fixed and independent of input content, meaning an adversary cannot exploit attention patterns to craft targeted perturbations (as has been demonstrated for attention-based models in the adversarial NLP literature). The Fourier basis is global and smooth, potentially providing implicit regularization against high-frequency noise in the input (since high-frequency noise maps to high-frequency DFT coefficients that the feed-forward layers can learn to ignore). A strong follow-up would evaluate FNet and BERT on standard adversarial robustness benchmarks for NLP (TextFooler, BERT-Attack, adversarial GLUE), on out-of-distribution generalization (training on one domain of GLUE and evaluating on another), and on few-shot fine-tuning (training on small subsets of GLUE tasks to measure sample efficiency and overfitting). The hypothesis: FNet's parameter-free mixing acts as a strong regularizer, making it more robust to adversarial attacks and less prone to overfitting on small datasets, at the cost of reduced capacity on large, in-distribution datasets where BERT's learned attention can exploit more complex patterns. If confirmed, this would position FNet not just as a faster alternative to BERT but as a preferred model for safety-critical or data-scarce deployments where robustness matters more than a few points of in-distribution accuracy. The LRA benchmark provides a starting point: the paper reports that "results for certain tasks — Text and Retrieval in particular — can vary quite a bit between runs, especially for the Transformer" (Appendix A.6). Quantifying this variance for FNet vs. BERT across multiple random seeds would be a simple first test of whether fixed mixing provides more stable training.

Practical Applications and Downstream Use Cases

On-device and edge deployment of text classification and understanding models where memory and latency constraints preclude BERT-sized attention models. The paper's Table 4b shows FNet using 0.8 GB peak memory at 512 tokens on GPUs versus BERT's 1.6 GB — a 2× reduction. At 2048 tokens, FNet uses 2.2 GB versus BERT's 12.2 GB — a 5.5× reduction. The small FNet variants in Table 6 (e.g., 2 layers, $d_h = 256$, 10M parameters, ~13M for FNet) are small enough to run on mobile devices or embedded systems where attention's $O(N^2)$ memory cost is prohibitive. For applications like on-device email classification, SMS spam detection, or document triage — where the model must process variable-length text with hard latency bounds and no cloud fallback — FNet's fixed mixing provides predictable latency (no dynamic attention patterns to optimize for) and low memory (enabling larger batch sizes or concurrent inferences). The speed-accuracy trade-off curves (Figure 2) show that at the 15M parameter scale, FNet achieves higher MLM accuracy than BERT at the same training speed — meaning a practitioner building a lightweight on-device classifier should prefer FNet over BERT at that scale, not merely accept FNet as a degraded but faster alternative. The paper's finding that FNet "performs just as well without position embeddings" (Section 3.2) is practically relevant here: removing the position embedding matrix ($N \times d_h$, ~393K parameters for standard BERT-Base) further reduces model size for memory-constrained deployments, though this ablation is not quantified in the paper.

Batch inference pipelines for long-document processing where sequence length exceeds 512 tokens and attention's quadratic scaling becomes the dominant cost. Table 4c shows FNet inference at 4096 tokens running at 43 ms/batch on GPUs versus the Transformer's 244 ms/batch — a 5.7× speedup. At 8192 tokens, FNet runs at 83 ms/batch while the Transformer runs out of memory entirely. For applications like legal document review, scientific literature screening, or long-form content moderation — where documents routinely span thousands of tokens and must be processed in bulk — FNet's $O(N \log N)$ scaling directly translates to infrastructure cost savings. A pipeline that previously required distributing documents across multiple GPUs to fit attention's memory footprint can now run on a single GPU with FNet. The LRA Text classification accuracy (65.11 for FNet, matching the best models at 64–66, Table 4a) suggests this speedup does not come at the cost of task performance for document-level understanding. The Linear model's inference speed degradation at long lengths (72 ms/batch at 4096 tokens vs. FNet's 43 ms/batch, Table 4c) makes this a case where FNet's FFT advantage is most pronounced — for long-document processing, FNet is not just faster than BERT but faster than the learned-mixing alternative as well, making it the dominant choice on GPUs.

Distillation of large Transformer models into small FNet students for deployment in production services with strict latency budgets. The paper explicitly suggests this use case (Section 5): "we suspect that FNet will be effective as a lightweight, distilled student model deployed in resource-constrained settings such as production services or on edge devices." The paper does not provide distillation results, but the speed-accuracy trade-off curves (Figure 2) show that small FNet models (2–4 layers, $d_h = 128$–256, 4–15M parameters) define the Pareto frontier for training speeds above ~250 steps/s on GPU. A production service currently serving a distilled BERT-Base model (e.g., DistilBERT at 66M parameters) could potentially switch to a distilled FNet at comparable accuracy but with lower latency (FNet-Base inference at 46 ms/batch vs. BERT-Base at 82 ms/batch on GPU, Table 3 — roughly 1.8× faster inference) and lower memory (0.8 GB vs. 1.6 GB at 512 tokens, Table 4b). The Hybrid architecture provides a middle ground: a distilled FNet-Hybrid-Base (88M parameters, 97% of BERT-Base accuracy per Table 2) with 2 attention layers at the top could serve as a drop-in replacement for BERT-Base with 1.5× training speedup (198 vs. 305 ms/batch on GPU, Table 3) and 1.6× inference speedup (51 vs. 82 ms/batch), with minimal accuracy degradation. The Hybrid's attention layers would also preserve the interpretability benefits of attention weights for the final task-specific representations, which may be important for debugging or compliance in production settings.

Large-scale pre-training data generation or filtering where the model must process billions of documents and throughput dominates accuracy considerations. The paper's pre-training metrics (Table 5) show FNet-Base achieving MLM accuracy 0.58 versus BERT-Base's 0.68 — a 10-point gap on the pre-training objective. For applications where the model's role is to filter or score training data (e.g., identifying high-quality documents for a larger model's pre-training corpus, filtering toxic content, or scoring text for educational appropriateness), this accuracy gap may be acceptable given the throughput gains. At 1.8× training speedup on GPUs (Table 3), an FNet-based data filter could process 1.8× more documents in the same wall-clock time, with the 10-point MLM accuracy gap potentially tolerable if the filter is used for coarse selection rather than fine-grained annotation. The memory advantage (Table 4b) would also allow larger batch sizes, further increasing throughput. This use case leverages FNet's strengths (speed, memory) while minimizing the impact of its weakness (reduced representation capacity for complex token-token interactions), since data filtering is typically a simpler task than the fine-grained natural language inference or semantic similarity tasks where FNet shows its largest accuracy gaps (MNLI, QQP, QNLI in Table 2).

When to Prefer This Method

The paper does not articulate an explicit decision rule or trade-off matrix against named alternatives beyond the general speed-accuracy trade-off curves in Figure 2. The evidence, however, supports the following conditional preferences, grounded in the paper's results:

• Prefer FNet over BERT when the deployment is GPU-based, sequence lengths are long (≥2048 tokens), and the task is one where FNet's GLUE per-task accuracy gap is small (SST-2, CoLA, MRPC — where FNet-Base trails BERT-Base by 0–7 points, or even leads on SST-2 and MRPC for Large models per Table 2). Under these conditions, FNet provides 2–5.7× speedups (Tables 4b, 4c) with minimal accuracy cost.

• Prefer FNet over the Linear model when sequence length exceeds 512 tokens or memory footprint is a binding constraint. Table 4b shows FNet's memory use at 2048 tokens is 2.2 GB versus Linear's 2.8 GB (21% less), and the training speed gap widens from 1.2× at 512 tokens (43 vs. 34 steps/s) to 1.6× at 2048 tokens (14 vs. 9 steps/s). At BERT-Base scale, the accuracy difference is 0.3 GLUE points (Table 2), so FNet strictly dominates Linear for long sequences on GPUs.

• Prefer FNet-Hybrid over pure FNet when accuracy requirements demand >92% of BERT-Base performance and the deployment can tolerate the 17% training slowdown (198 vs. 169 ms/batch on GPU, Table 3) from 2 attention layers. Table 2 shows FNet-Hybrid-Base at 80.6 GLUE (97% of BERT) versus FNet-Base at 76.7 (92%). For Large models, the Hybrid advantage is smaller (83.6 vs. 81.9, 1.7 points) but still meaningful relative to the 2.8-point gap to BERT-Large.

• Prefer the Performer (or other efficient Transformers with linearized attention) over FNet on TPUs when sequence lengths exceed 2048 tokens and training throughput is the primary metric. Table 7 shows the Performer at 6.9 steps/s versus FNet at 6.1 steps/s (mat) or 3.2 steps/s (FFT) at 2048 tokens — a 1.1–2.2× TPU training speed advantage that grows at 4096 tokens. However, this preference may be temporary: the paper attributes the TPU gap to immature FFT implementations and expects it to shrink (Section 3.3). For GPU deployments, FNet is faster than the Performer at all sequence lengths (Table 4b), so this TPU-specific preference does not generalize.

• Prefer BERT (standard Transformer) over FNet when the task requires complex pairwise token reasoning (natural language inference, question-answer matching) and accuracy is the dominant metric with no computational constraints. Table 2 shows FNet-Base trailing BERT-Base by 12 points on MNLI matched accuracy (72 vs. 84) and 11 points on QNLI (80 vs. 91). The FNet-Hybrid recovers about half this gap (78 vs. 84 on MNLI, 88 vs. 91 on QNLI), but for maximum accuracy on these task types, full attention remains superior. This preference weakens at Large scale, where the gap shrinks (FNet-Large 78/76 on MNLI vs. BERT-Large 88/88 is a 10–12 point gap, while FNet-Hybrid-Large 79/80 narrows it to 8–9 points), but the direction is consistent.