MLP-Mixer: An all-MLP Architecture for Vision

🎯 Pitch

A vision model built solely from multi-layer perceptrons—no convolutions, no self-attention—matches the performance of state-of-the-art CNNs and Transformers when trained on massive datasets, proving that neither architectural prior is necessary for competitive image classification. At large scale, MLP-Mixer sits on the accuracy–compute Pareto frontier and runs 2.5× faster than equivalently sized Vision Transformers, despite being just alternating matrix multiplications applied across spatial locations and feature channels.

1. Executive Summary

This paper introduces MLP-Mixer, a vision architecture built entirely from multi-layer perceptrons that eschews both convolutions and self-attention, decomposing image processing into two alternating operations: channel-mixing MLPs (applied independently to each spatial location, akin to 1×1 convolutions) and token-mixing MLPs (applied across all spatial locations for a single channel, akin to a depth-wise convolution with full receptive field and shared parameters). When pre-trained on JFT-300M, MLP-Mixer-H/14 attains 87.94% top-1 ImageNet accuracy while running 2.5× faster than ViT-H/14 and nearly twice as fast as BiT-ResNet152x4 at inference—yet it sits on the accuracy–compute Pareto frontier only when trained at large scale, establishing that an architecture devoid of convolutional or attentional inductive biases can be competitive with state-of-the-art CNNs and Transformers only when the pre-training dataset is sufficiently large to compensate for the missing structural priors.

2. Context and Motivation

The Core Problem: Are Convolutions and Attention Necessary for Vision?

The central question MLP-Mixer poses is deceptively direct: do we actually need convolutions or self-attention to build a high-performing vision architecture? The paper frames this as a sufficiency versus necessity question. It is well-established that CNNs (sufficiency of convolutions) and Vision Transformers (sufficiency of attention) can achieve state-of-the-art results. The open question — and the one this paper tackles — is whether either mechanism is necessary. Put differently: if we strip away all hand-designed spatial processing priors and use only generic matrix multiplications applied in a structured way, can we still compete?

This is not merely a philosophical exercise. The dominance of CNNs and Transformers in computer vision creates a kind of architectural monoculture. Newcomers to the field, as well as established researchers, operate under the implicit assumption that effective vision models must include either convolutional kernels (with their local receptive fields, translation equivariance, and weight sharing) or self-attention mechanisms (with their content-dependent, global receptive fields). If neither is actually necessary, the design space for vision architectures is far larger than the community has been exploring, and simpler alternatives might offer practical advantages in implementation complexity, hardware efficiency, or both.

Why This Question Matters Now

The paper's motivation is grounded in a broader historical trend the authors explicitly articulate in Section 1:

"As the history of computer vision demonstrates, the availability of larger datasets coupled with increased computational capacity often leads to a paradigm shift."

The key observation is that inductive biases and dataset scale are substitutes, not complements. When datasets are small (e.g., early ImageNet with 1.3M images), architectural inductive biases — convolution's local connectivity and translation equivariance, attention's content-based routing — act as essential regularizers that prevent overfitting. But as datasets grow into the hundreds of millions of examples, the data itself can provide a stronger, more flexible form of regularization: the training distribution becomes rich enough that a generic architecture can learn the relevant invariances and spatial relationships directly from examples, without having them baked into the architecture.

The paper points to Vision Transformer (ViT) as the most recent demonstration of this principle (Dosovitskiy et al., 2021). ViT showed that self-attention — originally designed for NLP, with no built-in 2D spatial structure — could match or exceed CNNs when pre-trained on sufficiently large datasets like JFT-300M. But ViT still retained a sophisticated and computationally expensive mechanism: self-attention. MLP-Mixer takes the next logical step in this progression by asking: can we remove attention too?

This question has practical urgency because the computational properties of different architectural building blocks matter enormously at scale:

Convolutions require specialized implementations (im2col, Winograd transforms, FFT-based convolution) to achieve high throughput on accelerators. Their efficiency depends on kernel size, stride, dilation, and group structure — choices that interact complexly with hardware.
Self-attention has $O(S^2)$ complexity in the number of tokens (image patches), making it expensive for high-resolution images. ViT-L/16, for example, achieves only 32 img/sec/core throughput on TPU-v3, while Mixer-L/16 reaches 105 img/sec/core (Table 2).
Matrix multiplications in MLPs, by contrast, are the single most optimized operation in modern deep learning hardware. Every accelerator (GPU, TPU, neural engine) is designed primarily to multiply matrices quickly. An architecture built purely from matrix multiplications, reshapes, and pointwise nonlinearities would therefore map extremely efficiently to existing hardware with minimal implementation complexity.

The paper explicitly positions Mixer's simplicity as a practical advantage:

"Mixer relies only on basic matrix multiplication routines, changes to data layout (reshapes and transpositions), and scalar nonlinearities."

The JAX/Flax implementation in Appendix E fits in 43 lines of code, most of which is boilerplate. This is not just an aesthetic claim — simplicity of this form translates directly to easier deployment, fewer bugs, more predictable performance across hardware backends, and lower engineering overhead for practitioners who want to adapt the architecture to new domains.

The Unifying Abstraction: Feature Mixing Decomposed

The paper identifies a conceptual gap in how the field has thought about vision architectures. In Section 2, the authors propose that all modern deep vision architectures can be understood as performing two types of mixing:

Channel mixing (per-location): Combining information across different feature channels at the same spatial location. In CNNs, this is done by 1×1 convolutions or the channel dimension of larger kernels. In Transformers, this is done by the feed-forward MLP blocks applied independently to each token.
Token mixing (cross-location): Combining information across different spatial locations for the same feature channel. In CNNs, this is done by $N \times N$ convolutions (for $N > 1$ ) and pooling. In Transformers, this is done by the self-attention layers.

The paper's key architectural insight is that these two operations can be cleanly separated and each implemented by the same generic building block — an MLP — with no specialized mechanism for either:

"The idea behind the Mixer architecture is to clearly separate the per-location (channel-mixing) operations (i) and cross-location (token-mixing) operations (ii). Both operations are implemented with MLPs."

This decomposition is what gives Mixer its name and its conceptual clarity. Previous architectures conflated these two types of mixing. Convolutions with $N>1$ kernels do both simultaneously. Self-attention also does both (the attention weights mix tokens, while the value projection mixes channels). Mixer's contribution is not inventing either mixing operation — both exist in prior work — but rather showing that (a) they can be cleanly separated, (b) generic MLPs suffice for both, and (c) this stripped-down combination is competitive when scaled appropriately.

Where Prior Approaches Fall Short

The paper identifies several specific gaps in prior work that motivate MLP-Mixer:

CNNs bake in spatial locality as an immutable prior. Convolutional architectures assume that nearby pixels are related and distant pixels are unrelated — an assumption that holds for natural images but constrains what the model can learn. While deeper layers in CNNs develop larger receptive fields through repeated downsampling and convolution stacking, this locality bias is architectural, not learned. For tasks where long-range spatial dependencies matter (e.g., recognizing an animal from its overall shape when occluded, or understanding global scene context), CNNs must build these long-range connections laboriously through many layers.

The paper doesn't argue that this locality bias is wrong — for small datasets, it's essential — but that it becomes an unnecessary constraint when data is abundant. The permutation experiments in Section 3.4 make this vivid: when image patches are randomly shuffled, a ResNet's performance collapses because its architecture assumes a specific spatial arrangement. Mixer, by contrast, is completely invariant to input permutation — the token-mixing MLP learns spatial relationships from data rather than having them hard-coded.

Vision Transformers introduced attention as a learnable alternative but retained quadratic complexity. ViT demonstrated that convolution-free architectures can work for vision, but self-attention has its own baggage. The $O(S^2)$ complexity in the number of patches means that ViT scales poorly to high-resolution images — doubling the resolution quadruples the attention cost. This matters practically because many vision tasks require fine-grained spatial understanding (segmentation, detection, medical imaging) where high resolution is critical. The paper notes:

"the computational complexity of the network is linear in the number of input patches, unlike ViT whose complexity is quadratic."

This linear complexity is a direct consequence of the token-mixing MLP's design: it applies the same MLP independently to each column (channel) of the input table, so the cost scales as $O(S \times D_S)$ where $D_S$ is the hidden width of the token-mixing MLP — which the paper sets independently of $S$ .

No prior work had shown that plain MLPs applied in this structured way could compete at scale. The paper is careful to position itself relative to prior attempts at MLP-heavy vision architectures. Section 4 reviews related work and distinguishes Mixer from:

Fully-connected networks applied directly to pixels (Lin et al., 2016; Neyshabur, 2020): These works trained pure MLPs on CIFAR-scale images, sometimes with heavy auto-encoder pre-training or custom optimization. They demonstrated feasibility on small-scale tasks but never showed competitiveness on ImageNet or at the scale of modern vision models. Mixer differs fundamentally by operating on patches (sequences of tokens) rather than raw pixels, and by using the structured interleaving of token-mixing and channel-mixing MLPs rather than a single monolithic fully-connected stack.
Squeeze-and-excitation and non-local networks (Hu et al., 2018; Wang et al., 2018): These architectures augment CNNs with channel-wise or spatial-wise MLP-like operations, but they are additions to a convolutional backbone, not replacements. They demonstrate that MLP-style operations can be useful, but not that they can be sufficient.
Depthwise separable convolutions (Chollet, 2017; Howard et al., 2017): These decompose a standard convolution into a depthwise convolution (spatial mixing, per-channel) followed by a pointwise convolution (channel mixing, per-location). This is structurally analogous to Mixer's token-mixing then channel-mixing sequence. However, depthwise convolutions use different convolutional kernels for each channel, while Mixer's token-mixing MLP uses a single shared MLP across all channels. The paper explicitly calls out this difference:

"in separable convolutions, a different convolutional kernel is applied to each channel unlike the token-mixing MLPs in Mixer that share the same kernel (of full receptive field) for all of the channels."

This parameter sharing is not a minor implementation detail — it is what prevents Mixer's parameter count from exploding when the channel dimension $C$ increases. Without it, the token-mixing MLP would have $C$ independent MLPs, multiplying its parameter count by $C$ . The paper verifies in Appendix A.1 that untying the parameters (making them per-channel) does not improve performance, confirming that the sharing is not a compromise but a genuinely effective inductive bias.

Isotropic architectures were underexplored in vision. Most CNNs use a pyramidal design: spatial resolution decreases and channel count increases through the network (e.g., ResNet stages). Mixer, following ViT, uses an isotropic design where every layer has the same input shape and hidden dimension. While isotropic ResNets exist (Sandler et al., 2019), they are the exception, not the norm. Mixer demonstrates that isotropic design — simpler, more uniform, easier to scale — is viable for pure-MLP architectures as well.

How MLP-Mixer Positions Itself

The paper's positioning is notable for its modesty and precision. The authors explicitly state:

"Our goal is not to demonstrate state-of-the-art results, but to show that, remarkably, a simple MLP-based model is competitive with today's best convolutional and attention-based models."

This framing is important. Mixer is not claiming superiority — it is claiming competitiveness, and specifically competitiveness in the accuracy–compute Pareto frontier sense. The paper wants to show that Mixer sits on the frontier, not necessarily that it dominates it. This is a carefully scoped claim: a model can be on the Pareto frontier without being the single best model at any particular point, so long as no other model achieves both higher accuracy and lower cost.

The paper positions Mixer as the natural continuation of a trend: progressively removing hand-designed inductive biases and replacing them with learned structure from data. The sequence is:

Hand-crafted features (SIFT, HOG) → Learned features (AlexNet, 2012): Features are learned from data rather than engineered by humans.
Fixed convolutional architectures (AlexNet, VGG) → More flexible architectural patterns (Inception, ResNet, EfficientNet): The structure of the network is increasingly optimized rather than fixed.
Convolutional inductive bias → Attention-based architectures (ViT, 2020): The assumption that spatial processing must be local and translation-equivariant is removed.
Attention-based inductive bias → No spatial inductive bias at all (Mixer, 2021): Even the assumption that spatial mixing should be content-dependent (attention's core mechanism) is removed.

The paper explicitly connects this to the broader narrative:

"ViT continues the long-lasting trend of removing hand-crafted visual features and inductive biases from models and relies further on learning from raw data."

Mixer is presented as the next logical step in this progression, not as an attempt to dethrone CNNs or Transformers. The goal is to open the design space, not to close it around a new winner.

The Practical Context: Large-Scale Pre-training as the Enabler

A crucial framing detail is that Mixer only works at scale. The paper is explicit about this — it's not a limitation to be hidden but a fundamental characteristic of the approach. Section 3.2 shows that Mixer-B/16 trained from scratch on ImageNet achieves 76.44% top-1, which is "similar to a vanilla ResNet50, but behind state-of-the-art CNNs/hybrids." Section 3.3 demonstrates that Mixer overfits "strongly" on small subsets of JFT-300M:

"When pre-trained on the smallest subset of JFT-300M, all Mixer models strongly overfit. BiT models also overfit, but to a lesser extent, possibly due to the strong inductive biases associated with the convolutions."

This scale-dependence is not a bug — it's the phenomenon the paper is investigating. The central hypothesis is that architectural inductive biases and dataset scale are in a trade-off relationship, and Mixer exists at the extreme end: zero spatial inductive bias, requiring maximum data. The paper positions itself as testing the limit of this trade-off.

This positioning has implications for how the results should be interpreted. Mixer is not proposed as a replacement for CNNs in all settings — it would be a poor choice for small-data regimes, medical imaging with limited annotations, or applications where pre-training on 300M images is infeasible. Rather, Mixer demonstrates that in the large-data regime that is increasingly common in industrial computer vision, the architectural choices that dominated the field for a decade may be optional rather than essential.

Summary of the Gap and Positioning

The paper addresses a specific, well-defined gap: no prior work had demonstrated that a pure MLP architecture (no convolutions, no attention) could sit on the accuracy–compute Pareto frontier for large-scale image classification. Prior work had shown that MLPs can do something on small vision tasks, that attention can replace convolutions, and that various MLP-like augmentations improve CNNs. But the combination — structured MLPs as the sole processing mechanism, competitive at the scale of JFT-300M and ImageNet-21k — was new.

The importance of filling this gap extends beyond the specific architecture. If an all-MLP design can compete, it implies that the dominant architectural paradigms of the past decade (convolution-based and attention-based) are not reflecting fundamental computational necessities of vision. They are sufficient but not necessary — effective solutions in a much larger design space that we have only begun to explore. The paper explicitly hopes to catalyze this broader exploration:

"Most of all, we hope that our results spark further research, beyond the realms of established models based on convolutions and self-attention."

3. Technical Approach

3.1 Reader orientation

MLP-Mixer is a pure multi-layer perceptron architecture for image classification — it processes images by cutting them into non-overlapping patches, projecting each patch to a feature vector, and then repeatedly applying the same two types of MLP blocks: one that mixes information across patches (spatial/token dimension) and one that mixes information across feature channels, with no convolutions and no self-attention anywhere in the network. The architecture solves the problem of building a competitive vision model using only matrix multiplications, reshapes, transpositions, and scalar nonlinearities, demonstrating that neither convolutional kernel structure nor content-dependent attention is necessary when the model is trained at sufficient scale, and in doing so achieves linear computational complexity in the number of patches (unlike the quadratic complexity of Vision Transformers) while sitting on the accuracy–compute Pareto frontier for large-scale image classification.

3.2 Big-picture architecture

The MLP-Mixer architecture is a feed-forward pipeline with four major stages, all maintaining the same representation shape throughout the core of the network:

Patch embedding layer (stem): Splits the input image into non-overlapping P×P patches, then linearly projects each patch (flattened into a vector of pixel values) to a C-dimensional hidden representation using a shared projection matrix. The output is a table X ∈ R^{S × C} where S = HW / P^2 is the number of patches.
Mixer layers (N identical blocks): Each Mixer layer applies two MLP sub-blocks in sequence: a token-mixing MLP that operates across rows of the transposed table (mixing information between different spatial locations for each feature channel independently, with shared parameters across all channels) and a channel-mixing MLP that operates across columns (mixing information between different feature channels for each spatial location independently, with shared parameters across all patches). Both sub-blocks consist of two fully-connected layers with a GELU nonlinearity in between, and are wrapped with skip-connections and layer normalization.
Global average pooling: After all Mixer layers, the representation is collapsed across the spatial dimension by averaging all patch vectors into a single C-dimensional vector, discarding any remaining spatial structure.
Classifier head: A single linear (fully-connected) layer maps the pooled C-dimensional representation to the number of output classes, producing the final classification logits.

The entire pipeline can be summarized in one sentence: image → patchify → linear projection → N × (token-mixing MLP → channel-mixing MLP) → global average pool → linear classifier. The architecture is isotropic — every Mixer layer receives and produces the same shape R^{S × C} — unlike pyramidal CNNs where spatial resolution decreases and channel count increases through the network.

3.3 Roadmap for the deep dive

First, the input representation and patch embedding, because understanding how the 2D image becomes a 1D sequence of token vectors defines the data structure that all subsequent operations manipulate. I will explain patch extraction, the linear projection, and why no position embeddings are added (a deliberate departure from ViT).
Second, the token-mixing MLP, because it is the more unusual of the two MLP types: it operates on the transposed table, mixes information across all S patches for a single channel at a time, and uses parameter sharing across channels — a design choice that prevents parameter explosion and is validated by ablation. I will walk through the exact sequence of operations (layer norm → transposition → MLP → transposition → residual add).
Third, the channel-mixing MLP, the more conventional sub-block (analogous to 1×1 convolutions or Transformer feed-forward layers), but whose interaction with token-mixing creates the architecture's full expressivity. I will explain how the two MLP types are interleaved and why this clean separation is the paper's central architectural insight.
Fourth, the complete Mixer layer as a unit, including skip-connections, layer normalization placement, and the precise formulation from Equation 1. This is where I show how these building blocks compose into the repeatable layer that stacks N times.
Fifth, the output head and training pipeline, covering global average pooling, the linear classifier, pre-training configurations (optimizer, batch size, data augmentation, regularization), and the resolution-increasing scheme for fine-tuning (which requires handling longer token sequences in the token-mixing MLPs).
Sixth, the architectural scaling configurations, covering the specific Mixer variants (S, B, L, H) with their layer counts, hidden dimensions, MLP widths, patch resolutions, and parameter counts from Table 1, since these numbers define the concrete models whose performance is reported.

3.4 Detailed, sentence-based technical breakdown

This is primarily an architectural design paper whose core idea is that vision architectures can be decomposed into two independent mixing operations across the two axes of the "patches × channels" data table, and that simple shared MLPs suffice for both when trained at scale.

Input Representation: Patch Extraction and Linear Embedding

The input to MLP-Mixer is a standard RGB image with dimensions H × W × 3. The first processing step converts this 2D pixel grid into a 1D sequence of token vectors, following exactly the same "patchify and project" approach introduced by Vision Transformer (Dosovitskiy et al., 2021).

Patch extraction. The image is divided into a grid of non-overlapping square patches, each of size P × P pixels. For a given patch resolution P, the number of patches is:

$S = \frac{HW}{P^2}$

where S is the sequence length (number of tokens), H and W are the image height and width in pixels, and P is the patch size. For the standard 224×224 input resolution used in most experiments: with P = 16, S = 196 patches arranged in a 14×14 grid; with P = 32, S = 49 patches arranged in a 7×7 grid. The paper uses patch resolutions of 32×32, 16×16, and 14×14 depending on the model variant (Table 1).

What this computes: the spatial partitioning of the image into a fixed grid of equally-sized, non-overlapping tiles. Each patch is a P × P × 3 tensor of raw pixel values that will be independently projected.

Why this form: extracting patches creates a sequence of tokens analogous to words in NLP, enabling the architecture to treat spatial processing as sequence processing. The patch size P controls the granularity of the spatial representation: smaller patches (P = 16 or 14) produce more tokens (S = 196 or 256) with finer spatial resolution at higher computational cost; larger patches (P = 32) produce fewer tokens (S = 49) with coarser spatial resolution but lower cost. The choice of non-overlapping patches (as opposed to overlapping sliding windows) keeps S manageable and ensures each pixel contributes to exactly one token, avoiding redundant computation.

Linear projection. Each patch is flattened into a vector of dimension P × P × 3 (e.g., for P = 16, this is a 768-dimensional vector: 16 × 16 × 3 = 768) and then linearly projected to a hidden dimension C using a shared projection matrix applied identically to every patch:

$\mathbf{x}_i = \mathbf{W}_{\text{proj}} \cdot \text{flatten}(\text{patch}_i) + \mathbf{b}_{\text{proj}}$

where patch_i is the i-th extracted patch (a P × P × 3 tensor), flatten(·) converts it to a vector of dimension P^2 × 3, W_proj ∈ R^{C × (P^2·3)} is the shared projection weight matrix, b_proj is a bias vector, and x_i ∈ R^C is the resulting i-th token embedding. The full output is the table X ∈ R^{S × C} where each of the S rows is one token's C-dimensional embedding.

In the JAX/Flax implementation (Appendix E), this is implemented as a single convolutional layer with kernel size (P, P), stride (P, P), and C output channels — mathematically equivalent to the per-patch linear projection described above, but expressed in a way that leverages efficient convolution implementations.

What this computes: an independent, learnable lower-dimensional embedding for each image patch that maps the raw pixel values into a semantic feature space. Every patch uses the same projection, so the operation is translation-equivariant by construction (shifting the image shifts the embeddings identically).

Why this form: the linear projection with shared weights is the simplest possible way to convert pixel patches into feature vectors. It has no built-in spatial relationships between patches — unlike a convolution with overlapping patches, there is no communication between neighboring patches at this stage. The shared projection matrix ensures that identical visual patterns appearing in different locations produce identical embeddings, which is a minimal form of spatial invariance. The hidden dimension C is a tunable hyperparameter that controls the capacity of the per-patch representation independently of the spatial sequence length.

No position embeddings. A critical departure from Vision Transformer: Mixer does not add learned position embeddings to the token vectors. The motivation (Section 2) is:

"Unlike ViTs, Mixer does not use position embeddings because the token-mixing MLPs are sensitive to the order of the input tokens."

This is a subtle but important design choice. In ViT, the self-attention mechanism is permutation-equivariant — shuffling the input tokens shuffles the output tokens in the same way, because attention computes interactions based on content (query-key similarity) rather than position. Position embeddings are therefore essential: without them, a ViT cannot distinguish a patch in the top-left corner from a patch in the bottom-right corner if they happen to contain similar visual features. Mixer's token-mixing MLPs are not permutation-equivariant because the MLP's weight matrices have a fixed mapping from each input position (row in the weight matrix) to each output position. The first linear layer W_1 ∈ R^{D_S × S} has a separate weight for each of the S input positions, so the MLP inherently learns spatial position through its weight structure. Adding explicit position embeddings would be redundant.

This means that the spatial structure is learned implicitly in the weights of the token-mixing MLPs, rather than being injected explicitly at the input. The visualization in Figure 5 confirms that the learned weights do develop spatially structured patterns (some operating globally, some locally) without ever being told explicitly about the 2D layout.

The Token-Mixing MLP: Cross-Location Communication

The token-mixing MLP is the first of the two sub-blocks in each Mixer layer and is responsible for allowing different spatial locations (patches) to exchange information. It is the architectural component that replaces convolutions and self-attention for spatial processing.

Operation sequence. For a given input table X ∈ R^{S × C} (ignoring residual connections and layer norm for the moment, which I cover in the complete layer description below), the token-mixing MLP performs the following steps:

Transpose the table: The input is transposed from R^{S × C} to R^{C × S}. After transposition, each row of the table corresponds to one feature channel, containing that channel's values across all S spatial locations. This reorients the data so that the MLP can operate on the spatial dimension.
Apply a shared MLP to each row: An MLP with two fully-connected layers and a GELU nonlinearity is applied independently to each of the C rows. Crucially, the same MLP parameters are used for every row — this is the parameter sharing across channels that distinguishes Mixer from depthwise separable convolutions.
Transpose back: The output table is transposed back from R^{C × S} to R^{S × C}, restoring the original orientation where rows are patches and columns are channels.

The MLP itself has the following structure:

$\text{TokenMLP}(\mathbf{z}) = \mathbf{W}_2 \cdot \sigma(\mathbf{W}_1 \cdot \mathbf{z} + \mathbf{b}_1) + \mathbf{b}_2$

where z ∈ R^S is a single column of the transposed input (all S spatial values for one channel), W_1 ∈ R^{D_S × S} and W_2 ∈ R^{S × D_S} are the weight matrices of the two fully-connected layers, b_1 ∈ R^{D_S} and b_2 ∈ R^{S} are biases, and σ is the GELU (Gaussian Error Linear Unit) nonlinearity applied element-wise. The hidden dimension D_S is a tunable hyperparameter specified in Table 1.

What this computes: for each feature channel independently, it computes a new S-dimensional vector that is a learned nonlinear function of the original S spatial values. Because the MLP is fully-connected across all S spatial positions (not a local kernel), every output position can depend on every input position — the receptive field is literally the entire image from the very first layer. The GELU nonlinearity prevents the composition of two linear layers from collapsing into a single linear transformation. The output S values are the new channel-specific values for each spatial location after mixing.

Why this form (parameter sharing across channels): this is the most unconventional design choice in the paper and deserves careful explanation. The alternative — which would seem more expressive — is to have a separate token-mixing MLP for each channel, i.e., W_1^{(i)} ∈ R^{D_S × S} for i = 1, ..., C. However, this would multiply the parameter count of the token-mixing MLP by C, making the architecture scale quadratically with the hidden dimension. With parameter sharing, the token-mixing MLP has a fixed parameter count independent of C, which keeps the total model size manageable as the hidden dimension grows. The paper validates this choice explicitly in Appendix A.1:

"Untying (not sharing) the parameters... We did not observe any noticeable improvements."

The fact that sharing works without performance degradation is remarkable — it means that the same spatial mixing pattern is effective across all feature channels, or equivalently, that the features learn to organize themselves so that a single mixing operation benefits all of them. This can be viewed as a strong architectural regularizer: the model is forced to discover spatial mixing patterns that are universally useful across the feature space, rather than specializing per-channel mixing that might overfit.

Why this form (full receptive field): unlike a 3×3 convolution which only mixes information within a small local neighborhood, the token-mixing MLP connects every spatial position to every other spatial position in a single layer. The weight matrix W_1 has S × D_S entries, meaning that for each of the D_S hidden units, there is a learned weight for each of the S input positions. In the first layer of Mixer-B/16 (S = 196, visualized in Figure 5), each hidden unit has 196 weights — one per incoming patch — and different units learn to attend to different spatial patterns, some global (activating across the entire image), some local (activating on specific regions). This is fundamentally different from a CNN, where long-range dependencies must be built up gradually through many layers of local operations.

Why this form (linear complexity): the computational cost of the token-mixing MLP is O(S × D_S) for each channel, and since it is applied to all C channels, the total cost is O(S × D_S × C). Because D_S is chosen independently of S (it is a fixed hyperparameter, not a function of the number of patches), this cost is linear in the number of patches. This contrasts with self-attention, where computing attention weights requires O(S^2 × C) operations, making ViT's cost quadratic in S. The practical consequence is that Mixer scales gracefully to higher-resolution images (more patches) without the quadratic blowup that affects ViT.

The GELU nonlinearity. The paper uses the Gaussian Error Linear Unit activation:

$\text{GELU}(x) = x \cdot \Phi(x)$

where Φ(x) is the cumulative distribution function of the standard normal distribution. In practice, this is approximated as x · σ(1.702x) where σ is the sigmoid function. GELU can be understood as a smooth variant of ReLU that weights inputs by their probability of being positive under a normal distribution — inputs near zero are partially suppressed (rather than fully zeroed out as in ReLU), providing a probabilistic gating mechanism. The paper inherits this choice from the Transformer literature (Vaswani et al., 2017) and does not ablate alternatives. Its use in both token-mixing and channel-mixing MLPs is simply part of the standard MLP block definition.

The Channel-Mixing MLP: Per-Location Feature Combination

The channel-mixing MLP is the second sub-block in each Mixer layer and is responsible for combining information across different feature channels at each spatial location independently. This is the more conventional of the two MLP types.

Operation sequence. The channel-mixing MLP operates on the table X ∈ R^{S × C} in its original orientation (rows are patches, columns are channels). For each row (patch) independently, it applies the same two-layer MLP:

$\text{ChannelMLP}(\mathbf{z}) = \mathbf{W}_4 \cdot \sigma(\mathbf{W}_3 \cdot \mathbf{z} + \mathbf{b}_3) + \mathbf{b}_4$

where z ∈ R^C is a single row of the input table (all C channel values for one patch), W_3 ∈ R^{D_C × C} and W_4 ∈ R^{C × D_C} are weight matrices, b_3 and b_4 are biases, and σ is GELU. The hidden dimension D_C is a tunable hyperparameter (specified in Table 1), typically set to 4× the hidden dimension C (e.g., for Mixer-B/16, C = 768 and D_C = 3072).

What this computes: for each spatial location independently, it computes a new C-dimensional feature vector that is a learned nonlinear function of the original C features at that location. No spatial information is exchanged — a patch's new representation depends only on its own previous representation. The two-layer structure with an expansion factor of 4 (from C to D_C and back) follows the standard Transformer feed-forward design.

Why this form: the channel-mixing MLP is exactly analogous to a 1×1 convolution in CNN terminology (or a pointwise convolution). A 1×1 convolution applies a linear transformation to the channel dimension at each spatial location, with spatial weight sharing ensuring the same transformation is applied everywhere. The channel-mixing MLP does the same, but with an added nonlinearity between two linear layers. The parameter sharing across spatial locations (same MLP applied to every row) is the natural choice because it provides translation invariance: a visual feature should be processed the same way regardless of where it appears in the image. This is the standard inductive bias in vision, unchanged from CNNs.

Why the expansion factor of 4: the pattern of expanding from C to D_C = 4C and contracting back to C is inherited directly from the Transformer literature (Vaswani et al., 2017), where the feed-forward block in each Transformer layer also uses a 4× expansion. The intuition is that the higher-dimensional intermediate representation provides capacity for the model to compute richer feature interactions before projecting back to the original dimension. The paper does not ablate this ratio; it is adopted as a standard design pattern.

Why interleave with token-mixing: the key design principle is that token-mixing and channel-mixing are complementary — token-mixing changes where information is (redistributing it across spatial locations), while channel-mixing changes what information is (transforming feature representations at each location). By alternating them, the architecture ensures that after spatial information is redistributed by the token-mixing MLP, the channel-mixing MLP can transform the new feature combinations at each location. This creates a powerful compositional dynamic: each layer can (1) gather information from across the image relevant to each spatial location, then (2) compute new features from that gathered information. The next layer's token-mixing MLP can then redistribute these refined features again, enabling iterative refinement of global context.

The Complete Mixer Layer: Skip-Connections, Normalization, and Formal Equations

The token-mixing and channel-mixing MLPs are not applied in isolation — they are wrapped with layer normalization and residual skip-connections that together form the repeatable Mixer layer. The paper provides the exact formulation in Equation 1 and Figure 1.

Pre-normalization with layer norm. Each MLP sub-block is preceded by layer normalization applied to the input. Layer normalization (Ba et al., 2016) normalizes the activations across the feature dimension for each sample independently, computing:

$\text{LayerNorm}(\mathbf{x}) = \gamma \odot \frac{\mathbf{x} - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$

where μ and σ^2 are the mean and variance computed across the C feature dimensions (for the channel-mixing case) or across the S spatial dimensions (for the token-mixing case), γ and β are learnable scale and shift parameters, and ε is a small constant for numerical stability. Using layer norm rather than batch norm (Ioffe and Szegedy, 2015) follows the Transformer convention — layer norm's behavior is independent of batch size, making it more suitable for the large-batch training used here (batch size 4096).

Residual skip-connections. After each MLP block, the output is added element-wise to the original input (the skip-connection), following the ResNet design (He et al., 2016). This means each sub-block computes a residual — a correction to the input representation rather than a complete replacement. Skip-connections mitigate the vanishing gradient problem in deep networks and allow the model to learn identity-like functions by default (if the MLP outputs are driven to zero, the layer passes through its input unchanged).

Formal specification of one Mixer layer. The paper defines the complete layer as follows (Equation 1 in Section 2). Let X ∈ R^{S × C} be the input table, and let X_{*, i} denote the entire i-th column (all S spatial locations for channel i) and X_{j, *} denote the entire j-th row (all C channels for patch j). Then the layer computes:

Token-mixing step (applied column-wise):

$U_{*, i} = X_{*, i} + \mathbf{W}_2 \cdot \sigma\left(\mathbf{W}_1 \cdot \text{LayerNorm}(X)_{*, i} + \mathbf{b}_1\right) + \mathbf{b}_2 \quad \text{for } i = 1, \ldots, C$

where W_1 ∈ R^{D_S × S}, W_2 ∈ R^{S × D_S}, σ is GELU, and LayerNorm(·) normalizes across the S spatial dimension. This is applied to each column independently (i.e., for each channel), with the same W_1, W_2 shared across all C columns.

Channel-mixing step (applied row-wise):

$Y_{j, *} = U_{j, *} + \mathbf{W}_4 \cdot \sigma\left(\mathbf{W}_3 \cdot \text{LayerNorm}(U)_{j, *} + \mathbf{b}_3\right) + \mathbf{b}_4 \quad \text{for } j = 1, \ldots, S$

where W_3 ∈ R^{D_C × C}, W_4 ∈ R^{C × D_C}, σ is GELU, and LayerNorm(·) normalizes across the C channel dimension. This is applied to each row independently (i.e., for each patch), with the same W_3, W_4 shared across all S rows.

What these equations compute: each equation computes a residual update to the representation. For the token-mixing step: take the input table X, apply layer normalization across the spatial dimension, multiply by W_1 to project each S-dimensional column to a D_S-dimensional hidden representation, apply GELU, multiply by W_2 to project back to S dimensions, and add the result to the original column. The output U is the spatially-mixed representation. For the channel-mixing step: take U, apply layer normalization across the channel dimension, multiply by W_3 to project each C-dimensional row to a D_C-dimensional hidden representation, apply GELU, multiply by W_4 to project back to C dimensions, and add to the original row. The output Y is the fully-mixed representation after one complete Mixer layer.

Why this form (pre-norm rather than post-norm): placing layer normalization before the MLP block (rather than after the residual addition, as in the original Transformer) is a design choice adopted from more recent Transformer variants. Pre-normalization tends to improve training stability by ensuring that the input to each sub-layer has consistent statistics, regardless of how deep the network becomes. The residual path remains a pure identity mapping — the normalized representation is used for computing the update, but the skip-connection bypasses the normalization.

Why this form (two sequential residual blocks rather than parallel): the token-mixing and channel-mixing MLPs are applied sequentially (each with its own skip-connection) rather than in parallel (both reading from X and summing their outputs). Sequential application means the channel-mixing MLP can process the spatially-mixed features, creating a dependence where the features computed at each location reflect the globally-mixed spatial information. A parallel design would allow channel mixing and token mixing to happen independently, preventing the compositional interaction that makes the architecture powerful. The sequential design is analogous to the two sub-layers in a Transformer block (self-attention followed by feed-forward), each with its own residual connection.

The "isotropic" design principle: every Mixer layer has exactly the same input and output dimensions (S × C), uses the same hidden dimensions (D_S and D_C), and contains exactly the same two sub-blocks. The only per-layer parameters are the weights and biases of the four linear transformations (two per MLP block). This isotropic design stands in contrast to pyramidal CNNs, where each stage produces a different spatial resolution and channel count. The advantages of isotropy include: (1) simpler scaling — to make a deeper model, just stack more identical layers; (2) no architectural decisions about when to downsample or expand channels; (3) more uniform computational cost per layer, simplifying hardware optimization. The paper inherits this isotropic design directly from ViT, which itself inherited it from the original Transformer for NLP.

Output Head and Training Pipeline

After the stack of N Mixer layers, the architecture must convert the S × C table representation into a single classification decision.

Global average pooling. The spatial dimension is collapsed by averaging across all S patches:

$\mathbf{h} = \frac{1}{S} \sum_{j=1}^{S} Y_{j, *}$

where Y_{j, *} ∈ R^C is the j-th row (patch embedding) of the output from the final Mixer layer, and h ∈ R^C is the resulting image-level representation. This operation discards all remaining spatial structure — the model must have already integrated spatial information into the per-patch features through the token-mixing MLPs, because after pooling, there is no way to distinguish which feature came from which location.

What this computes: a single fixed-length vector that summarizes the entire image, agnostic to the number of patches. This is exactly analogous to global average pooling in CNNs (e.g., ResNet's final pooling before the classifier), and serves the same purpose: making the representation invariant to spatial translation at the output level.

Why this form: averaging is the simplest permutation-invariant aggregation. After the token-mixing MLPs have already exchanged information globally, the per-patch features should encode not just local information but also global context. Averaging them produces a representation that is robust to small spatial shifts and avoids the overfitting that could occur if the classifier had access to position-specific features. An alternative would be to prepend a classification token (as in ViT and BERT), but the paper follows the simpler CNN convention of pooling.

Linear classifier. The pooled representation is passed through a single fully-connected layer:

$\hat{\mathbf{y}} = \mathbf{W}_{\text{head}} \cdot \text{LayerNorm}(\mathbf{h}) + \mathbf{b}_{\text{head}}$

where W_head ∈ R^{K × C} (with K being the number of classes, e.g., 1000 for ImageNet), b_head ∈ R^K, and a final layer normalization is applied to h before the linear projection (this is the pre_head_layer_norm in the code). The output ŷ ∈ R^K contains the unnormalized logits for each class. In the JAX/Flax code (Appendix E), the head layer is initialized with zero weights (kernel_init=nn.initializers.zeros), meaning the network initially predicts zero for all classes regardless of input, which can help training stability.

Pre-training configuration. The paper specifies detailed training hyperparameters in Section 3 and Appendix B:

Optimizer: Adam with β₁ = 0.9, β₂ = 0.999. The learning rate uses linear warmup for the first 10,000 steps followed by linear decay. Batch size is 4096 across all experiments. Gradient clipping is applied at global norm 1. Weight decay is used (values in Table 4).
JFT-300M pre-training: For JFT-300M (300M images, 18k classes), training is relatively short — 5 to 14 epochs depending on model size (Table 3) — and uses only the cropping technique from Szegedy et al. (2015) plus random horizontal flipping as data augmentation. No additional regularization (no dropout, no stochastic depth, no RandAugment, no mixup) because the massive dataset size provides sufficient regularization. Learning rates are 0.003 for S/B models and 0.001 for L/H models; weight decay is 0.03 (Table 4).
ImageNet and ImageNet-21k pre-training: For these smaller datasets (1.3M and 14M images respectively), extensive regularization is necessary because Mixer overfits severely without it. The paper sweeps over combinations of RandAugment (magnitude m from {0, 10, 15, 20}, always 2 augmentation layers), mixup (mixing strength p from {0.0, 0.2, 0.5, 0.8}), dropout (rates d of 0.0 or 0.1), and stochastic depth (final layer drop probability s from {0.0, 0.1}, linearly increased from 0 at the first layer). The optimal settings for each model/dataset combination are listed in Table 4: e.g., Mixer-B on ImageNet uses RandAugment magnitude 15, mixup 0.5, dropout 0.0, stochastic depth 0.1, learning rate 0.001, weight decay 0.1, for 300 epochs.
Training duration: Models are trained for 300 epochs on ImageNet and ImageNet-21k, 5 epochs for S, 7 epochs for B, and 7 or 14 epochs for L models on JFT-300M (14 for the larger L and H variants). The epoch counts are chosen to balance adequate training with computational constraints.

Fine-tuning configuration. When transferring pre-trained Mixer models to downstream tasks:

Optimizer: SGD with momentum (the paper says "momentum SGD"), batch size 512, gradient clipping at global norm 1, and a cosine learning rate schedule with linear warmup. No weight decay is used during fine-tuning (unlike pre-training).
Learning rate sweep: Grid search over lr ∈ {0.003, 0.01, 0.03, 0.06} depending on the dataset (0.06 is included only for CIFAR-10, CIFAR-100, Flowers, and Pets).
Higher-resolution fine-tuning: Following established practice (Touvron et al., 2019; Kolesnikov et al., 2020; Dosovitskiy et al., 2021), models fine-tuned at higher resolution than pre-training achieve substantially better accuracy. The resolution increase is typically from 224 to 448 (a factor of K = 2 in each spatial dimension).

The resolution-increase mechanism for token-mixing MLPs. Increasing the input resolution while keeping the patch size fixed increases the number of patches from S to S' = K^2 · S (e.g., from 196 to 784 when going from 224 to 448 with 16×16 patches). The token-mixing MLP's first weight matrix W_1 ∈ R^{D_S × S} is not compatible with S' input positions. The paper's solution (Appendix C):

Increase the token-mixing hidden dimension from D_S to D_S' = K^2 · D_S.
Initialize the new weight matrix W_1' ∈ R^{D_S' × S'} as a block-diagonal matrix with K^2 copies of the original W_1 on the diagonal. Concretely, if the longer input sequence is conceptually split into K^2 sub-sequences of length S each (corresponding to K × K spatial quadrants of the higher-resolution image), the initialized MLP processes each sub-sequence independently using the pre-trained weights, with no cross-quadrant mixing.
The output weight matrix W_2' is initialized analogously as a block-diagonal expansion of W_2.
During fine-tuning, these block-diagonal weights are updated, allowing the model to learn cross-quadrant interactions.

What this computes: an initialization scheme for fine-tuning at higher resolution that preserves the pre-trained model's behavior exactly at the start of fine-tuning (each K × K spatial quadrant is processed independently, exactly as it was at the original resolution), then allows the model to learn inter-quadrant interactions through gradient updates.

Why this form: the block-diagonal initialization ensures that at the start of fine-tuning, the model's predictions are exactly identical to what it would produce at the original resolution — it simply processes each of the K^2 quadrants independently and (after pooling) averages their contributions. This is crucial because it prevents the resolution change from causing an initial drop in performance. The fine-tuning process can then learn to use the longer sequence to capture finer-grained spatial relationships. The approach has a constraint: it works cleanly only when S' is an integer multiple of S, i.e., when the resolution scaling factor K is an integer.

Architectural Scaling Configurations

The paper defines four model scales — S (Small), B (Base), L (Large), and H (Huge) — following the naming convention from Dosovitskiy et al. (2021) for ViT. Each scale can be combined with different patch resolutions, denoted as Model/P (e.g., Mixer-B/16 means Base scale with 16×16 patches). Table 1 in the paper specifies the exact configurations:

Specification	S/32	S/16	B/32	B/16	L/32	L/16	H/14
Number of layers	8	8	12	12	24	24	32
Patch resolution	32×32	16×16	32×32	16×16	32×32	16×16	14×14
Hidden size C	512	512	768	768	1024	1024	1280
Sequence length S	49	196	49	196	49	196	256
MLP dimension D_C	2048	2048	3072	3072	4096	4096	5120
MLP dimension D_S	256	256	384	384	512	512	640
Parameters (M)	19	18	60	59	206	207	431

Several patterns emerge from these configurations:

The number of layers scales with model size: S = 8 layers, B = 12 layers, L = 24 layers, H = 32 layers. This is the primary scaling axis — deeper models have more capacity to learn complex feature hierarchies.

The hidden dimension C also scales: from 512 (S) to 768 (B) to 1024 (L) to 1280 (H). Wider models have more capacity per token to represent diverse features.

The channel-mixing MLP dimension D_C is always 4× the hidden dimension: as noted above, following the Transformer convention. This keeps the expansion ratio constant across model scales.

The token-mixing MLP dimension D_S is set independently of C and increases more slowly: 256 (S) → 384 (B) → 512 (L) → 640 (H). This dimension controls the capacity of spatial mixing; it is deliberately kept modest because the token-mixing MLP is applied across all C channels, so its cost is O(S × D_S × C). Keeping D_S smaller than D_C helps control the computational cost of spatial processing, which tends to dominate because S can be large (196 or 256).

Patch resolution affects sequence length but not parameter count: comparing B/16 (196 patches, 59M parameters) to B/32 (49 patches, 60M parameters), the parameter count is nearly identical. This is because the number of parameters depends on S only through the token-mixing MLP's weight matrices W_1 ∈ R^{D_S × S} and W_2 ∈ R^{S × D_S} — but D_S is the same for both, so the total parameter difference is approximately 2 × D_S × (196 − 49) = 2 × 384 × 147 ≈ 113K additional parameters, negligible compared to the total 59–60M. The computational cost, however, differs substantially because the token-mixing MLP cost scales with S.

The H/14 configuration uses 14×14 patches: producing S = 256 / (14 × 14) = 256 patches at resolution 224 (since 224/14 = 16, and 16×16 = 256). At the fine-tuned resolution of 448, this becomes 1024 patches. This is the largest and most computationally expensive Mixer variant.

Total parameters range from 18M to 431M: spanning two orders of magnitude, comparable to the range from ResNet-50 (~25M) to very large ResNets and ViT-H (~632M). The parameter counts exclude the classifier head weights, which add K × C parameters (e.g., 1000 × 1280 = 1.28M for H/14 on ImageNet, a negligible fraction of the total).

Why these specific configurations: the paper does not perform neural architecture search or systematic scaling law analysis to determine these numbers. They are chosen to be directly comparable to ViT configurations of the same scale (ViT-S, ViT-B, ViT-L, ViT-H from Dosovitskiy et al., 2021), enabling fair head-to-head comparisons in terms of both accuracy and computational cost. The S/32, B/32, and L/32 variants (with 32×32 patches) provide additional points on the accuracy–throughput spectrum, trading spatial resolution for higher inference speed.

4. Key Insights and Innovations

Innovation 1: Convolutions and Attention Are Sufficient for Vision, but Neither Is Necessary — The First Empirical Proof at Scale

This is the paper's central conceptual contribution, and it is more radical than it might first appear. The field had reached a rough consensus that while many architectural patterns could work for vision, the reason state-of-the-art models worked was attributable to either convolutional structure (local connectivity, translation equivariance, weight sharing) or attention (content-dependent, global receptive fields). These were treated not merely as implementation techniques but as functional requirements — things the architecture had to do to capture spatial structure effectively.

The paper's diagnostic move is to treat this as an empirically testable hypothesis rather than an assumption. By building an architecture with neither mechanism — just matrix multiplications applied across the two axes of a table, with the only nonlinearity being element-wise GELU — the paper constructs the minimal test: if a model without convolutions or attention cannot compete at scale, then at least one of those mechanisms is necessary. If it can compete, neither is.

The result, shown most clearly in Figure 2 (left) and Table 2, is that Mixer-H/14 pre-trained on JFT-300M achieves 87.94% ImageNet top-1 — comparable to ViT-H/14 (88.55%) and BiT-ResNet152x4 (87.54%) — while sitting on the accuracy–compute Pareto frontier. This is not just "an MLP can do vision." It is: at the scale where modern vision models operate, the entire spatial processing apparatus developed over a decade of architectural research (convolutional kernels, attention mechanisms, non-local blocks, squeeze-and-excitation) is optional. The data can teach a generic function approximator to do the same job.

This is a fundamental reframing, not an incremental refinement. Prior work had shown that attention can replace convolution (ViT, Dosovitskiy et al., 2021), but that still preserved a sophisticated spatial mechanism — content-dependent routing — that could plausibly be the real requirement. Mixer strips that away too, leaving only the question: can a sufficiently generic architecture, trained on enough data, learn spatial processing from scratch? The answer, per Figure 3 (left), is that while Mixer lags behind ViT at small model scales (e.g., Mixer-B/16 at 76.44% vs. ViT-B/16 at 79.67% when trained on ImageNet alone), the gap closes as both model size and dataset size increase — at the H/14 scale on JFT-300M, the difference shrinks to 0.3% (86.32% vs. 86.65% at resolution 224). The architecture matters less as data and compute grow.

The significance goes beyond Mixer itself. This result implies that the design space for vision architectures is far larger than the convolution-and-attention paradigm that had dominated the field since AlexNet (2012). If an all-MLP design can compete, then so might architectures based on entirely different mathematical primitives — so long as they can approximate arbitrary functions and are trained at sufficient scale. The paper opens the door, conceptually, to a much broader exploration of what a vision model can be.

Innovation 2: The Clean Decomposition of Spatial and Channel Mixing as Independent, Complementary Axes

While the operations in Mixer (MLPs applied row-wise and column-wise) are individually simple, the architectural insight of cleanly separating them — and showing that this separation alone is sufficient — represents a genuine conceptual advance over prior work.

Before Mixer, the dominant vision architectures conflated spatial and channel processing. A standard 3×3 convolution simultaneously mixes across spatial neighbors and across input channels — the two operations are entangled in a single linear transformation. Depthwise separable convolutions (Chollet, 2017; Howard et al., 2017) moved toward separation by factorizing into a depthwise spatial convolution followed by a pointwise channel convolution, but the spatial operation was still a convolution (local, translation-equivariant, with per-channel kernels). Self-attention in ViT also conflates the two: the attention weights compute token mixing, while the value projection computes channel mixing, but they interact multiplicatively in a single step.

Mixer's clean separation is distinctive for two reasons. First, it shows that the two operations can be implemented by the same generic building block — an MLP — applied in different orientations. The token-mixing MLP and channel-mixing MLP have identical structure (two fully-connected layers + GELU); they differ only in which axis of the input table they operate on. This symmetry is intellectually elegant: the architecture literally says "spatial processing and channel processing are the same kind of thing, just applied to different dimensions of the data."

Second, the separation forces each operation to do exactly one job, with no leakage. The token-mixing MLP operates on individual channels in isolation — it cannot change what features exist at each location, only where information flows. The channel-mixing MLP operates on individual patches in isolation — it cannot move information across space, only transform what each location represents. This is a strict decomposition, and the fact that it works implies that spatial reasoning and feature computation can be disentangled in vision — they are not fundamentally coupled processes that require a unified mechanism.

This framing has implications beyond Mixer. It suggests that future architectures could independently innovate on the spatial-mixing and channel-mixing components — replacing the token-mixing MLP with a different spatial operator, or the channel-mixing MLP with a different feature transformation — without redesigning the other axis. The two-axis decomposition becomes a design pattern, not just a description of Mixer. The paper validates this through its ablation of parameter sharing (Appendix A.1), confirming that the separation is robust: untying the token-mixing parameters across channels (making them more convolution-like, with per-channel spatial kernels) does not improve performance, suggesting the shared, separable design captures something fundamental rather than being an arbitrary compromise.

At first glance, the token-mixing MLP's design appears to be a limitation: the same MLP weights are shared across all C feature channels, meaning that every channel undergoes exactly the same spatial transformation. The natural assumption — grounded in the design of depthwise separable convolutions, where each channel has its own spatial kernel — is that different features need different spatial processing. An edge detector should look for edges, a color blob detector should look for blobs; why would they share a spatial filter?

The paper's discovery, validated in Appendix A.1, is that untying the weights (giving each channel its own token-mixing MLP) does not improve performance. The shared MLP is not a compromise forced by parameter efficiency; it is genuinely sufficient. This is a non-obvious empirical finding with conceptual implications.

Why might parameter sharing work? The architecture forces the model to organize its feature representations so that a single spatial mixing pattern is broadly useful. This acts as a strong regularizer: the channel-mixing MLP must learn to produce feature representations that are compatible with the shared spatial transformation, and the token-mixing MLP must learn a mixing pattern that benefits all channels simultaneously. Rather than each channel independently learning to extract different spatial patterns (as in depthwise convolutions), the architecture encourages a division of labor: the token-mixing MLP learns universal spatial operations (e.g., "smooth," "sharpen," "shift left"), while the channel-mixing MLP learns to interpret the results in channel-specific ways.

The visualization evidence in Figure 5 supports this interpretation. The learned token-mixing weights show structured patterns — some units act globally (detecting overall image statistics), some act locally (detecting features in specific regions) — and many appear in opposing-phase pairs, reminiscent of Gabor-like detectors in CNNs. These patterns are shared across all channels, suggesting they represent generic spatial primitives rather than channel-specific operations.

This finding reframes how we think about parameter sharing in spatial processing. The dominant assumption had been that sharing weights across spatial locations provides translation equivariance (convolution), while sharing across channels is an unnecessary restriction. Mixer inverts this: sharing across channels is not only harmless but may be beneficial as a regularizer, while the fully-connected structure of the token-mixing MLP (no translation equivariance) is compensated for by learning spatial patterns from data. It is a different set of inductive biases than CNNs, but one that proves equally valid at scale.

Innovation 4: Input Permutation Invariance as Both a Diagnostic Tool and a Genuine Architectural Property

Section 3.4's permutation experiments serve a dual purpose that elevates them beyond a simple ablation. On the surface, they demonstrate that Mixer is invariant to the ordering of input patches: when 16×16 patches are shuffled (with pixels within each patch also permuted), Mixer's performance is completely unaffected, while a ResNet50's performance drops by approximately 35 percentage points (from ~65% to ~30% linear 5-shot accuracy on ImageNet, per Figure 4). When pixels are globally shuffled, Mixer's performance drops by roughly 45% but ResNet's drops by approximately 75%.

The superficial interpretation is straightforward: Mixer has no built-in spatial structure, so shuffling doesn't disrupt anything it was relying on, while ResNet's convolutional inductive bias assumes a specific pixel arrangement. But the deeper insight is what this reveals about the nature of learned representations.

The token-mixing MLP's weight matrices W_1 ∈ R^{D_S × S} have an explicit, learned mapping from position index to hidden unit. When patches are permuted consistently across all images during training, the MLP simply learns a different set of position-to-feature mappings — it doesn't care about the 2D geometry, only about the consistent mapping from input slot to weight. The fact that Mixer can achieve competitive accuracy (roughly 76% for Mixer-B/16 trained on ImageNet with standard ordering, versus whatever performance it achieves under shuffled ordering) with no concept of spatial adjacency is remarkable — it means the architecture can learn to recognize objects purely from the statistical co-occurrence of visual patterns at arbitrary positions, without ever being told which positions are close to which others.

This is a genuinely new kind of vision model. CNNs are fundamentally geometric — they process images as 2D grids where proximity matters. ViT is relational — it computes pairwise similarities between all patches, learning which patches are related regardless of their absolute positions. Mixer is positional — it learns a fixed mapping from each of the S input slots to a set of spatial operations, with no concept of distance or similarity between slots. The three models represent three fundamentally different ways of handling space: geometric, relational, and positional. That all three can achieve similar accuracy at scale suggests that spatial structure in images is overdetermined — there are many ways to capture it, and the choice of mechanism matters less than the capacity to learn from data.

Innovation 5: The Scale-Dependence of Architectural Inductive Biases, Quantified Across Architecture Families

The paper provides one of the clearest empirical demonstrations of a principle that had been discussed but not systematically measured: architectural inductive biases and dataset size are substitutes, not complements, and the substitution rate differs across architecture families.

Figure 2 (right) tells this story precisely. When pre-trained on 3% of JFT-300M (~9M images), Mixer-L/16 achieves a linear 5-shot ImageNet accuracy of approximately 30%, while ViT-L/16 and BiT-R152x2 achieve roughly 37% and 45% respectively. Mixer overfits severely at small data scales — its lack of inductive biases is a liability. But as the dataset grows to 100% of JFT-300M, Mixer-L/16 reaches roughly 63% accuracy, closing most of the gap with ViT-L/16 (~66%) and surpassing BiT-R152x2 (~59%). When the dataset is further scaled to JFT-3B (~3B images), Mixer-L/16 reaches roughly 70%, essentially matching ViT-L/16.

The slope of improvement is what matters. Mixer's performance improves faster with data than either ViT or BiT. The authors note:

"the relative improvement of larger Mixer models are even more pronounced. The performance gap between Mixer-L/16 and ViT-L/16 shrinks with data scale. It appears that Mixer benefits from the growing dataset size even more than ViT."

This is a specific, quantifiable claim about the rate of substitution between data and inductive bias. Convolutional architectures (BiT) have the strongest inductive biases and therefore benefit least from additional data — they plateau. ViT has weaker inductive biases and improves more with data. Mixer has essentially zero spatial inductive bias and improves the most. The extrapolation is tantalizing: at some sufficiently large dataset size, Mixer might surpass architectures with stronger built-in priors because its flexibility allows it to learn a richer set of spatial relationships than can be encoded in a fixed convolutional or attentional structure.

This finding has practical implications for architecture selection. If your dataset is small (e.g., standard ImageNet with 1.3M images), you want strong inductive biases — use a CNN or a heavily regularized ViT, not Mixer. If your dataset is massive (100M+ images), the inductive bias becomes less important, and you can choose based on other factors like inference throughput or implementation simplicity — where Mixer excels (2.5× faster than ViT-H/14). The paper provides the quantitative evidence to make this trade-off explicit rather than relying on architectural folklore.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary downstream evaluation dataset is ILSVRC2012 "ImageNet" (1.3M training images, 1k classes; Deng et al., 2009), using both the original validation labels and the cleaned-up ReaL labels (Beyer et al., 2020). Additional downstream tasks include CIFAR-10/100 (50k examples, 10/100 classes; Krizhevsky, 2009), Oxford-IIIT Pets (3.7k examples, 36 classes; Parkhi et al., 2012), Oxford Flowers-102 (2k examples, 102 classes; Nilsback and Zisserman, 2008), and the Visual Task Adaptation Benchmark (VTAB-1k) consisting of 19 diverse datasets with 1k training examples each (Zhai et al., 2019). Pre-training is conducted on three datasets of increasing scale: ILSVRC2012 ImageNet (1.3M images, public), ImageNet-21k (14M images, 21k classes, public; Deng et al., 2009), and JFT-300M (300M images, 18k classes, proprietary; Sun et al., 2017). All pre-training datasets are de-duplicated with respect to the test sets of downstream tasks following the protocol established by Dosovitskiy et al. (2021) and Kolesnikov et al. (2020).

Base models. The experiments use the MLP-Mixer model family at four scales: S (Small, ~19M parameters), B (Base, ~59-60M), L (Large, ~206-207M), and H (Huge, ~431M). Each scale supports multiple patch resolutions (32×32, 16×16, or 14×14) denoted as Mixer-{S,B,L,H}/{32,16,14}, with the patch resolution affecting sequence length S but not parameter count (Table 1). The model family spans two orders of magnitude in size, enabling analysis of scaling behavior. All Mixer models share the same isotropic architecture (identical layer dimensions throughout the network, no pyramidal downsampling), with the number of layers, hidden dimension C, and MLP widths D_S and D_C scaling according to the specifications in Table 1.

Metrics. The primary quality metric is top-1 accuracy after fine-tuning on downstream tasks (ImageNet, CIFAR-10/100, Pets, Flowers, VTAB-1k). For the large-scale comparison in Figure 2 (right) and Figure 4, where fine-tuning all models would be prohibitively expensive, the paper reports linear 5-shot top-1 accuracy — solving an ℓ₂-regularized linear regression problem between frozen learned representations and labels using only 5 training examples per class. Two computational cost metrics are used: (1) total pre-training time on TPU-v3 accelerators measured in core-days (combining theoretical FLOPs, hardware efficiency, and data efficiency), and (2) inference throughput measured in images per second per TPU-v3 core, with batch sizes swept to report the maximum throughput for each model. The accuracy–compute trade-off is visualized as Pareto frontiers (Figure 2 left, Figure 3) and tabulated in Table 2 and Table 3.

Baselines. The paper compares against three major architecture families, each marked with distinct colors in figures: Vision Transformers (ViT) at matching scales (S, B, L, H) and patch resolutions (Dosovitskiy et al., 2021, blue); Big Transfer (BiT) ResNet models including R50x1, R152x1, R50x2, R152x2, R152x4, and R200x3 (Kolesnikov et al., 2020, yellow); HaloNets, specifically the HaloNet-H4 (base 128, Conv-12) hybrid variant with local self-attention replacing 3×3 convolutions in a ResNet-like structure (Vaswani et al., 2021, blue with attention and convolution markers). Additional state-of-the-art comparisons in Table 2 include NFNet-F4+ (normalizer-free ResNets; Brock et al., 2021), MPL (Meta Pseudo Labels with EfficientNet-B6-Wide; Pham et al., 2021), and ALIGN (EfficientNet-L2 image encoder pre-trained with contrastive language-image objective; Jia et al., 2021).

Generation budget / compute accounting. The paper uses two distinct compute metrics that serve different purposes. Total pre-training compute (TPUv3-core-days) accounts for three factors simultaneously: the theoretical FLOPs per training step, the achieved hardware utilization on TPU-v3 accelerators, and the total number of training steps. This composite metric is what populates the x-axis of Figure 2 (left) and Figure 3 (left), and the "TPUv3 core-days" column of Tables 2 and 3. It captures the cost to create the model. Inference throughput (images/sec/core) measures cost to deploy the model and is computed by sweeping batch sizes to find the maximum throughput for each model on TPU-v3 hardware. For the FLOPs-matched comparison paradigm from the reference example, this paper does not conduct a formal FLOPs-matched analysis between pre-training and test-time compute; instead, it examines the empirical relationship between pre-training compute and downstream accuracy to establish the Pareto frontier.

Cross-validation / statistical protocol. For the main ImageNet results in Table 2 and Table 3, Mixer and ViT accuracy numbers are averaged over three fine-tuning runs, with reported standard deviations smaller than 0.1 for the large models in Table 2 and smaller than 0.15 in Table 3. Pre-training hyperparameters (learning rate, weight decay, RandAugment magnitude, mixup strength, dropout rate, stochastic depth) are selected via grid search with the ranges specified in Appendix B and the optimal values documented in Table 4. For fine-tuning, a learning rate grid search is performed over {0.003, 0.01, 0.03, 0.06} depending on the dataset. The paper does not employ k-fold cross-validation over the downstream test set; instead, standard train/validation splits are used with the validation set serving as the test set since labels are publicly available.

Main Quantitative Results

Large-Scale Pre-Training Results: Mixer on the Accuracy–Compute Pareto Frontier

The headline result appears in Table 2 and Figure 2 (left): Mixer-H/14 pre-trained on JFT-300M achieves 87.94% ImageNet top-1 accuracy, 90.18% ReaL top-1, 95.71% average across 5 downstream tasks, and 75.33 on VTAB-1k (19 tasks), with an inference throughput of 40 img/sec/core and a total pre-training cost of 1.01k TPUv3-core-days. This places Mixer-H/14 on the accuracy–compute Pareto frontier alongside ViT-H/14 (88.55% ImageNet, 15 img/sec/core, 2.30k core-days), BiT-R152x4 (87.54% ImageNet, 26 img/sec/core, 9.90k core-days), and NFNet-F4+ (89.2% ImageNet, 46 img/sec/core, 1.86k core-days).

The throughput advantage is substantial: Mixer-H/14 runs 2.5× faster than ViT-H/14 (40 vs. 15 img/sec/core) and approximately 1.5× faster than BiT-R152x4 (40 vs. 26 img/sec/core). The pre-training cost is also lower: 1.01k core-days for Mixer-H/14 versus 2.30k for ViT-H/14 and 9.90k for BiT-R152x4. However, it is critical to note that the BiT-R152x4 in this row of Table 2 was pre-trained using SGD with momentum and a long schedule, making the pre-training cost comparison unfair to that specific BiT entry. The paper clarifies this by also including BiT-R200x3 pre-trained with Adam on JFT-300M in Figure 2 (left) (data from Dosovitskiy et al., 2021), which has "slightly lower accuracy, but considerably lower pre-training compute," providing a fairer Adam-to-Adam comparison.

When pre-trained on the public ImageNet-21k dataset (14M images), Mixer-L/16 achieves 84.15% ImageNet top-1, which is competitive but slightly behind ViT-L/16 (85.30%) and BiT-R152x4 (85.39%). The throughput advantage is pronounced in this setting: Mixer-L/16 at 105 img/sec/core versus ViT-L/16 at 32 img/sec/core (3.3× faster) and BiT-R152x4 at 26 img/sec/core (4× faster). However, at resolution 224 rather than the 448 used in Table 2, Mixer-L/16 achieves 82.84% at 420 img/sec/core, ViT-L/16 achieves 85.15% at 80 img/sec/core, and HaloNet achieves 85.5% at 258 img/sec/core — a different point on the accuracy–throughput trade-off that highlights how fine-tuning resolution dramatically affects the comparison.

Model Scale Analysis: The Gap Between Mixer and ViT Shrinks with Scale

Table 3 and Figure 3 present a systematic comparison across model scales (S, B, L, H) when all models are pre-trained on JFT-300M and fine-tuned at resolution 224. The key pattern is that Mixer's relative disadvantage compared to ViT diminishes as model scale increases, and at the largest scale, Mixer-H/14 (86.32% at 224 resolution) is only 0.3% behind ViT-H/14 (86.65% at 224 resolution) while running 2.2× faster (194 vs. 87 img/sec/core).

Comparing specific model pairs at resolution 224 on JFT-300M:

Small scale: Mixer-S/32 achieves 68.70% (11,489 img/sec/core, 0.01k core-days) and Mixer-S/16 achieves 73.83% (3,994 img/sec/core, 0.03k core-days). No ViT-S models are reported in Table 3 for direct comparison at this scale.
Base scale: Mixer-B/16 achieves 80.00% at 1,384 img/sec/core and 0.08k core-days. ViT-B/16 numbers are not in Table 3, but from the ImageNet-from-scratch setting (Table 3, first section), Mixer-B/16 reaches 76.44% versus ViT-B/16's 79.67% — a 3.2% gap.
Large scale: Mixer-L/16 at 7 epochs achieves 84.05% (419 img/sec/core, 0.23k core-days); at 14 epochs, 84.82% (419 img/sec/core, 0.45k core-days). ViT-L/16 at 14 epochs achieves 85.63% (280 img/sec/core, 0.65k core-days) — a gap of 0.8% but Mixer is 1.5× faster.
Huge scale: Mixer-H/14 achieves 86.32% (194 img/sec/core, 1.01k core-days) versus ViT-H/14 at 86.65% (87 img/sec/core, 2.30k core-days) — the gap narrows to 0.3% and Mixer is 2.2× faster and 2.3× cheaper to pre-train.

Figure 3 visualizes this trend on two axes: accuracy vs. pre-training compute (left) and accuracy vs. throughput (right). On the compute frontier (left), Mixer-H/14 sits on or very near the Pareto frontier at the high end, while smaller Mixer models (S/32, B/32, S/16) fall below the frontier occupied by ViT and BiT at similar compute budgets. On the throughput frontier (right), Mixer models consistently achieve higher throughput than ViT at the same accuracy level, with the H/14 model sitting near the frontier.

The paper also reports ImageNet-from-scratch results (no external pre-training, trained on ImageNet with heavy regularization): Mixer-B/16 achieves 76.44% at resolution 224, which is "similar to a vanilla ResNet50" but behind specialized architectures like BotNet (84.7%) and NFNet (86.5%). Mixer-L/16 trained from scratch on ImageNet actually performs worse (71.76%) than Mixer-B/16, which the authors attribute to even more severe overfitting at the larger scale without sufficient data. The training curves (not shown) reveal that both Mixer and ViT achieve "very similar values of the training loss," confirming that the performance gap is due to overfitting, not optimization difficulty.

Pre-Training Dataset Size Analysis: Mixer Benefits More from Data than CNNs or ViT

Figure 2 (right) presents one of the paper's most important findings: Mixer's performance improves more rapidly with increasing dataset size than either ViT or BiT, and the gap between Mixer and ViT closes as data grows. The experiment pre-trains Mixer-B/32, Mixer-L/32, Mixer-L/16, ViT-B/32, ViT-L/32, ViT-L/16, and BiT-R152x2 on random subsets of JFT-300M containing 3%, 10%, 30%, and 100% of training examples, for 233, 70, 23, and 7 epochs respectively (equal total steps). An additional data point at approximately 3B images uses the JFT-3B dataset (Zhai et al., 2021) for the same number of total steps.

The 5-shot linear accuracy on ImageNet reveals:

At 3% JFT-300M (~9M images): All Mixer models "strongly overfit." Mixer-L/16 achieves approximately 30%, while ViT-L/16 reaches approximately 37% and BiT-R152x2 reaches approximately 45%. The convolutional inductive bias of BiT provides the strongest regularization at small data scales.
At 100% JFT-300M (300M images): Mixer-L/16 reaches approximately 63%, ViT-L/16 reaches approximately 66%, and BiT-R152x2 reaches approximately 59%. Mixer has crossed BiT and nearly closed the gap with ViT. The slope of Mixer's improvement is visibly steeper than ViT's or BiT's.
At JFT-3B (~3B images): Mixer-L/16 reaches approximately 70%, essentially matching ViT-L/16 at the same data scale.

The paper interprets this finding explicitly:

"Mixer benefits from the growing dataset size even more than ViT. One could speculate and explain it again with the difference in inductive biases: self-attention layers in ViT lead to certain properties of the learned functions that are less compatible with the true underlying distribution than those discovered with Mixer architecture."

This is a striking claim — that the self-attention mechanism in ViT may actually limit what the model can learn from very large datasets, while Mixer's more generic architecture is ultimately more flexible. The evidence is suggestive but not definitive: the Mixer-L/16 and ViT-L/16 curves appear to be converging at 3B images, but it is unknown whether they would cross at larger scales.

Input Permutation Invariance: Mixer Learns Spatial Structure Entirely from Data

Section 3.4 and Figure 4 present a diagnostic experiment that tests how Mixer and ResNet respond to disrupted spatial structure. Mixer-B/16 and ResNet50x1 are trained on JFT-300M under three input conditions: (1) original images, (2) 16×16 patches shuffled and pixels within each patch permuted with a shared permutation, (3) pixels globally permuted across the entire image. The permutation is fixed — the same shuffling is applied to every image — and models are evaluated using linear 5-shot ImageNet accuracy.

The results, shown in Figure 4 (bottom), demonstrate:

Patch + pixel shuffling: Mixer's performance is completely unaffected — the blue (original) and green (patch+shuffle) curves overlap exactly at all training steps, reaching approximately 62% final accuracy. ResNet's performance drops dramatically from approximately 65% to approximately 30%.
Global pixel shuffling: Mixer's performance drops from approximately 62% to approximately 34% (roughly a 45% relative drop). ResNet's performance collapses from approximately 65% to approximately 16% (roughly a 75% relative drop).

The authors' interpretation:

"Mixer is invariant to the order of patches and pixels within the patches... ResNet's strong inductive bias relies on a particular order of pixels within an image and its performance drops significantly when the patches are permuted."

The fact that Mixer is completely invariant to patch+pixel shuffling (green curve identically matches blue) confirms that the architecture learns spatial relationships purely through the weight structure of the token-mixing MLPs — it has no built-in concept of 2D adjacency. When the input slots are consistently permuted, the MLP simply learns a different mapping from slot index to spatial operation, achieving identical performance. This strongly supports the paper's central claim that spatial inductive biases are sufficient but not necessary.

The fact that global pixel shuffling hurts Mixer (45% drop) but hurts ResNet more (75% drop) is interesting. Global pixel shuffling destroys all local structure — the image becomes pure noise to a locally-connected architecture. ResNet, which relies entirely on local receptive fields, has essentially no way to recover. Mixer still retains some ability to process these inputs because the token-mixing MLP has a fully-connected mapping from all S input positions to all output positions — it can learn to extract features from these scrambled inputs, albeit with substantially degraded performance.

Weight visualization (Figure 5 and Appendix D). The paper visualizes the learned weights of the first three token-mixing MLPs in Mixer-B/16 trained on JFT-300M. Each hidden unit has 196 weights (one per incoming 14×14 patch) displayed as a 14×14 heatmap. The key observations:

Some units operate globally, with weights distributed across the entire 14×14 grid, suggesting they compute image-wide statistics.
Others operate locally, with weights concentrated in specific regions of the grid, analogous to the localized receptive fields that emerge in CNNs.
Many units appear in opposing-phase pairs — one unit with a particular spatial pattern and a second unit with roughly the negative of that pattern — reminiscent of the paired Gabor-like filters commonly observed in the first layer of CNNs (Shang et al., 2016).
Deeper layers (center and right panels of Figure 5) show "no clearly identifiable structure," becoming more distributed and harder to interpret visually.

The visualization in Figure 7 (Appendix D) reveals an interesting dependency on patch resolution: Mixer-B/32 (using 32×32 patches, producing 7×7 token grids) learns "very structured low frequency projection units" in the embedding layer — Gabor-like patterns clearly visible as smooth sinusoidal variations — while Mixer-B/16 (using 16×16 patches) shows "high frequencies and no clear structure." This suggests that the larger patch size forces the model to learn coarser, more interpretable features at the embedding stage, while the finer patch resolution pushes feature learning deeper into the network.

Ablation Studies and Robustness Checks

Untying token-mixing MLP parameters across channels: The paper attempted giving each channel its own independent token-mixing MLP (C separate MLPs with independent weights rather than one shared MLP), which would multiply the token-mixing parameter count by C. Across Mixer models of various scales pre-trained on JFT-300M, no noticeable improvement was observed (Appendix A.1). This validates that parameter sharing across channels is not a compromise for efficiency — it is a genuinely effective inductive bias, or at minimum, the additional capacity is unnecessary when training at JFT-300M scale.

Grouping channels together for token mixing: Instead of token-mixing MLPs looking at only one channel at a time, the paper experimented with concatenating G neighboring channels to form input vectors of dimension G × S, allowing the MLP to see multiple channels simultaneously when mixing tokens. Two variants were tried: (a) simple concatenation of channels via reshaping, and (b) learning G different linear projections from R^C to R^{C/G} to create G different "views" of each token before concatenation. The simple concatenation improved linear 5-shot ImageNet accuracy by "less than 1–2%." The more sophisticated projection-based approach improved top-5 accuracy by 3–4% for Mixer-S/32 but "did not show any improvements for the larger scales" (Appendix A.1). This is a mixed result — there is a small benefit at small scales, but it does not generalize to the configurations where Mixer is actually competitive.

Pyramidal design (reducing tokens through the network): Following the pyramidal ViT design suggested by Wang et al. (2021), the paper experimented with using the token-mixing MLP to reduce the number of tokens (mapping from S input tokens to S' < S output tokens) to create a spatial pyramid. On JFT-300M, these pyramidal Mixers "significantly reduced training time without losing much performance." However, "we were unable to transfer these findings to ImageNet or ImageNet-21k" (Appendix A.1). This is a notable negative result — the pyramid design that works well for CNNs and has shown promise for ViT does not transfer cleanly to Mixer on smaller datasets, likely because Mixer's overfitting tendency makes the additional regularization of a pyramid design insufficient to compensate for the smaller data.

Fine-tuning regularization techniques: Following BiT (Kolesnikov et al., 2021) and ViT (Dosovitskiy et al., 2021), the paper tried using mixup (Zhang et al., 2018) and Polyak averaging (Polyak and Juditsky, 1992) during fine-tuning, as well as Inception-style cropping (Szegedy et al., 2015). None of these led to consistent improvements for JFT-300M pre-trained Mixer models of any scale, so they were dropped from the final fine-tuning protocol (Appendix A.2).

Evaluation pre-processing comparison: For Mixer and ViT models in Table 3, the paper used the "resmall-crop" evaluation method (resize shorter side to 256, then center crop to 224) on ImageNet, Pets, Flowers, CIFAR-10, and CIFAR-100. For BiT models in Table 3, "resize-crop" (resize to 256×256, then center crop to 224) was used on ImageNet. For the large models in Table 2, "resize-crop" was used for all 5 downstream datasets. The paper does not ablate these choices or report the sensitivity of results to evaluation pre-processing — a minor methodological gap.

Pre-training hyperparameter sensitivity: Table 4 documents the optimal hyperparameters found through grid search. Key observations: (1) ImageNet and ImageNet-21k pre-training require heavy regularization — RandAugment magnitudes of 10–20, mixup of 0.2–0.5, stochastic depth of 0.1 — while JFT-300M pre-training uses none of these regularizers (dashes in Table 4), relying purely on data scale. (2) Learning rates decrease as model size increases: 0.003 for S/B on JFT-300M, 0.001 for L/H. (3) Weight decay is systematically lower for JFT-300M (0.03) than for ImageNet/ImageNet-21k (0.1), likely because the massive dataset provides implicit regularization that reduces the need for explicit weight decay.

Patch resolution effects: Comparing Table 1 configurations, the S/32 (49 patches, 11,489 img/sec/core) and S/16 (196 patches, 3,994 img/sec/core) models have nearly identical parameter counts (19M vs. 18M) but 2.9× different throughput. Yet S/16 achieves substantially higher accuracy (73.83% vs. 68.70% at resolution 224 on JFT-300M, Table 3), demonstrating that finer patch resolution provides meaningful accuracy gains at the cost of throughput. The same pattern holds for B/32 vs. B/16 (75.53% vs. 80.00%) and L/32 vs. L/16 (80.67% vs. 84.05% at 7 epochs). The paper does not compute a formal accuracy-per-FLOP metric across patch resolutions, which would quantify whether the accuracy gain from smaller patches justifies the throughput reduction.

Higher-resolution fine-tuning: Comparing Mixer-L/16 fine-tuned at resolution 224 (84.82% at 14 epochs) versus resolution 448 (86.78% at 14 epochs) in Table 3 shows a +1.96% improvement from the higher resolution, at the cost of throughput dropping from 419 to 105 img/sec/core (4× reduction). The resolution increase uses the block-diagonal initialization scheme described in Appendix C, which assumes the scaling factor K is an integer. The paper does not ablate alternative initialization schemes, gradient-based upsampling, or progressive resolution training.

Dataset de-duplication: The paper states that all pre-training datasets are "de-duplicated with respect to the test sets of the downstream tasks as done in Dosovitskiy et al. [14], Kolesnikov et al. [22]." This is an important methodological detail — without de-duplication, models pre-trained on JFT-300M might inadvertently train on near-duplicates of ImageNet test images, inflating transfer accuracy. However, the paper does not report how many near-duplicates were found and removed, nor whether the de-duplication protocol was identical to that of Dosovitskiy et al. (2021).

Critical Assessment

The experimental evidence supporting the paper's central claims is strong but comes with qualifications that are sometimes understated in the presentation.

Claim: MLP-Mixer is competitive with state-of-the-art CNNs and Vision Transformers. The evidence in Figure 2 (left) and Table 2 demonstrates that Mixer-H/14 (87.94%) sits on the accuracy–compute Pareto frontier alongside ViT-H/14 (88.55%), NFNet-F4+ (89.2%), and MPL (90.0%). This supports the claim of competitiveness — Mixer is in the same performance class as the best known models. However, it is not the best model by any single metric: NFNet-F4+ has higher accuracy (89.2% vs. 87.94%), ViT-H/14 has marginally higher accuracy at the cost of much lower throughput, and MPL has substantially higher accuracy (90.0%) at enormous pre-training cost (20.48k core-days vs. 1.01k). The claim is carefully scoped to "competitive" rather than "superior," and this is appropriate.

A missing comparison is with EfficientNet architectures (Tan and Le, 2019) at comparable scales, which were the state-of-the-art CNNs when the paper was written. The paper includes MPL (which uses EfficientNet-B6-Wide) and ALIGN (which uses EfficientNet-L2), but these are large-scale pre-training results. A direct comparison of Mixer against EfficientNet trained from scratch on ImageNet would clarify whether Mixer's ImageNet-from-scratch performance (76.44% for Mixer-B/16) is competitive with similarly-sized CNNs in that setting. The paper only notes it is "similar to a vanilla ResNet50" — a much weaker CNN than EfficientNet.

The single model family limitation is significant: all experiments use the Mixer architecture with no variations in the core design (e.g., different nonlinearities, different normalization schemes, different MLP depths per block). The ablation studies in Appendix A show that most variations (untying parameters, grouping channels, pyramidal design) failed to help, which is informative, but also means we cannot distinguish between "this specific design works" and "many all-MLP designs would work at scale."

Claim: Neither convolutions nor attention are necessary for competitive vision models. The permutation invariance experiments (Section 3.4, Figure 4) provide the strongest direct evidence: Mixer's performance is completely unaffected when patches are shuffled and pixels within patches are permuted, proving it does not rely on 2D spatial structure. This is a clean, well-designed experiment. However, it demonstrates that Mixer specifically does not need spatial structure — it does not prove that any architecture without convolutions or attention would work, or even that other all-MLP designs would work. The paper shows existence (there is at least one competitive architecture without convolutions or attention), not universality.

The scale-dependence of this claim is more important than the paper emphasizes. At ImageNet scale (1.3M images), Mixer-B/16 achieves 76.44% versus ViT-B/16's 79.67% and NFNet's 86.5%. At this data scale, convolutions and attention do appear necessary to reach the frontier — Mixer is clearly behind. The claim only holds when pre-training datasets reach ~100M+ images. The paper is transparent about this in the abstract ("When trained on large datasets, or with modern regularization schemes..."), but readers could easily miss the conditional nature: Mixer is not a replacement for CNNs or ViTs in the small-to-medium data regime that most practitioners operate in.

Claim: MLP-Mixer runs faster than ViT and BiT at comparable accuracy. The throughput numbers in Table 2 do support this: Mixer-H/14 at 40 img/sec/core versus ViT-H/14 at 15 img/sec/core (2.5×) and BiT-R152x4 at 26 img/sec/core (1.5×). However, these throughput measurements are on TPU-v3 hardware, and the relative efficiency of matrix multiplications versus attention mechanisms versus convolutions is highly hardware-dependent. On GPUs with dedicated tensor cores, or on CPUs, or on mobile accelerators, the relative ordering might differ. The paper does not report throughput on any hardware other than TPU-v3, limiting the generality of the speed claim.

The pre-training cost numbers have a confound: JFT-300M is a proprietary dataset, so the core-days reported for Mixer models cannot be replicated or independently verified. For the public datasets (ImageNet, ImageNet-21k), pre-training costs are marked with a "(‡)" in Table 3, indicating they are "extrapolated from the numbers reported for the same models pre-trained on JFT-300M without extra regularization" — they are estimates, not measurements. Actual training cost on ImageNet-21k with RandAugment, mixup, dropout, and stochastic depth may differ from these extrapolations.

Claim: Mixer benefits more from data than CNNs or ViT. Figure 2 (right) directly supports this: the slope of Mixer-L/16's improvement from 3% to 100% JFT-300M is visibly steeper than ViT-L/16's or BiT-R152x2's. This is the most novel and important empirical finding in the paper. However, the evidence is based on linear 5-shot accuracy rather than full fine-tuning accuracy, and it is unclear whether the relative slopes would be the same under full fine-tuning. The JFT-3B data point (approximately 3B images) uses a different dataset with different characteristics (30k classes vs. 18k), making the extrapolation suggestive rather than rigorous.

Additionally, only one configuration of each architecture is tested at each data scale. It is plausible that ViT would benefit from different hyperparameters (e.g., regularization settings, learning rate schedules) at different data scales, and the sweeps conducted for ImageNet/ImageNet-21k pre-training suggest the optimal configuration is data-size-dependent. The equal-steps design (all models trained for the same number of total steps regardless of dataset size) means models trained on smaller subsets see each example fewer times, which is a reasonable design but introduces a confound between dataset size and per-example optimization.

Missing experiments that would strengthen the paper:

Full fine-tuning results on the data scaling subsets (Figure 2 right). The linear 5-shot protocol is a proxy for transfer quality, but fine-tuning results would be more convincing and directly comparable to the main results.
Throughput measurements on GPUs (e.g., V100, A100) in addition to TPU-v3. The paper's speed claims are hardware-specific.
Optimizer ablation for pre-training. The paper uses Adam for all Mixer pre-training but notes that BiT models pre-trained with SGD+momentum achieve better results at much higher cost. Would Mixer also benefit from SGD+momentum if given a longer schedule? This is not tested.
Depth/width scaling trade-off analysis. The paper reports Mixer configurations at four scales but does not ablate whether accuracy improves more from adding layers (depth) or increasing hidden dimensions (width) at a fixed parameter budget.
What happens if Mixer is scaled beyond H/14? The convergence of Mixer and ViT performance at H/14 (Figure 3 left) suggests they might cross at larger scales, but no experiments probe this.
Standard deviation for the data scaling curves (Figure 2 right). Each point is a single pre-training run; without error bars, it is unclear whether the apparent crossing of Mixer and BiT curves at 100% JFT-300M is statistically reliable or within noise.

Conditional nature of the main claims. The paper's findings apply specifically when:

Pre-training datasets are large (100M+ images, JFT-300M scale) or heavy regularization (RandAugment, mixup, stochastic depth, dropout) is applied on medium-scale datasets (ImageNet-21k, 14M images).
The architecture is the specific Mixer design with shared token-mixing MLPs, 4× channel-mixing expansion, layer normalization, skip-connections, and isotropic design.
Evaluation is on image classification (not detection, segmentation, or other dense prediction tasks).
Hardware is TPU-v3 (throughput advantages may differ on other accelerators).

For the practitioner with 1M–10M images and a GPU cluster, the paper does not demonstrate that Mixer is competitive with well-tuned ResNets, EfficientNets, or ViTs — the "from scratch" and small-scale pre-training results show clear gaps. For the practitioner with 100M+ images and TPU hardware, Mixer offers a compelling combination of simplicity, throughput, and accuracy that matches or approaches the state of the art. The paper's value is primarily in establishing this conditional competitiveness and, more importantly, in demonstrating that the architectural search space for vision is far from exhausted.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Amortized Into the Headline Gains

The assumption or constraint. The compute-optimal test-time scaling policy relies on knowing each prompt's difficulty before allocating the inference budget. The paper's method for estimating difficulty — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or the PRM's final-answer score (predicted) — is explicitly acknowledged as expensive:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2).

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. Generating 2048 samples per question consumes more compute than the largest test-time budgets studied (256–512 generations). In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. The 4× figure should therefore be understood as an upper bound on achievable efficiency rather than a realized deployment gain. Until difficulty can be estimated much more cheaply — for instance, from a few initial samples or a lightweight classifier — the compute-optimal framework is primarily an analytical contribution, not a practical recipe.

What evidence exists in the paper. The paper presents oracle and predicted difficulty bins as two variants in Figures 4 and 8, with the predicted variant performing similarly to the oracle variant. However, the predicted variant still requires 2048 samples and PRM scoring, meaning the cost savings from using predicted rather than oracle difficulty are essentially zero — the only advantage is removing the dependency on ground-truth labels. The paper does not include difficulty estimation cost in any of the budget calculations, nor does it report results with cheaper difficulty estimation methods.

Mitigation status. The paper explicitly flags this as a key avenue for future work:

"This can be framed as an exploration–exploitation tradeoff, as the computation spent on estimating the difficulty could have been spent on solving the problem instead. We leave investigating this concept to future work" (Section 3.2).

No mitigation is attempted in the current paper. The authors suggest that difficulty could potentially be estimated more cheaply — for instance, by training a model to predict it directly from the question text — but no such model is developed or evaluated.

Single Benchmark, Single Model Family Restricts Generality

The assumption or constraint. All experiments use the MATH benchmark (500 test questions of competition-level mathematics) with PaLM 2-S* as the base model. The PRM is trained on PaLM 2-S* outputs using Monte Carlo rollouts. The revision model is fine-tuned from PaLM 2-S*. The compute-optimal policy is selected based on PaLM 2-S*'s specific difficulty profile on MATH. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is not verified across model families, scales, or task domains.

The consequence. Several aspects of the findings could be model-specific or domain-specific:

PRM quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties, different error patterns, or different output diversity might exhibit different difficulty-dependent scaling curves, and the beam-search-over-optimization threshold (Figure 3, right) might shift.
Revision model effectiveness depends on the base model's in-context learning capabilities and its ability to learn from incorrect examples. Whether the 38% correct-to-incorrect reversion rate (Section 6.1) is universal or PaLM-2-S*-specific is unknown.
Difficulty bin boundaries (the five quintiles) are defined relative to PaLM 2-S*'s pass@1 distribution on MATH. A different base model would have a different difficulty profile, and the optimal strategies per bin might change.
MATH consists entirely of symbolic reasoning problems. It is unclear whether the central finding — that easy problems benefit from sequential revisions while medium problems benefit from beam search — generalizes to other reasoning domains (code generation, logical deduction, scientific QA) or to tasks requiring factual recall rather than multi-step inference.

What evidence exists in the paper. All figures and tables in Sections 5–7 report results on MATH with PaLM 2-S*. No cross-model experiments (e.g., applying the same PRM to a different base LLM, or comparing difficulty profiles of different models) are conducted. No cross-domain experiments (e.g., a different reasoning benchmark) are reported. The paper does not discuss what aspects of the findings might be general and which might be MATH-specific or model-specific.

Mitigation status. The paper does not attempt to mitigate this limitation. There is no discussion of how the findings might transfer to other models or domains. The authors position the paper as providing a framework for compute-optimal test-time scaling, but the specific numerical results (optimal search algorithms per difficulty bin, the 4× efficiency gain, the FLOPs-matched comparisons) are all conditioned on the PaLM 2-S* + MATH combination.

The 14× Larger Model Baseline Is Weakened by Non-Compute-Optimal Pretraining and Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters by approximately 14× while keeping training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal scaling (Hoffmann et al., 2022) where both parameters and data are scaled. The authors explicitly note:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work" (Section 7).

Additionally, the larger model is evaluated with greedy decoding only — no majority voting, no best-of-N, no PRM search, no revisions. This means the comparison is between a smaller model with compute-optimal test-time strategies and a larger model with zero test-time compute optimization.

The consequence. Both factors make the pretraining baseline weaker than it could be:

A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both data and parameters) would likely outperform a parameter-only-scaled model. The reported advantages of test-time compute over pretraining (e.g., +27.8% relative improvement on easy questions at R << 1 for revisions; Figure 1, top-right bar chart) may shrink or reverse against a properly compute-optimally trained larger model.
Even without compute-optimal pretraining, giving the 14× larger model a modest test-time compute budget (e.g., best-of-8 majority voting, or a short revision chain) would create a much stronger baseline. The paper demonstrates that test-time compute provides substantial gains for small models — it is reasonable to expect it would also help larger models, potentially by a similar or larger margin.

What evidence exists in the paper. The FLOPs-matched results in Figure 9 and the bar charts in Figure 1 show the smaller model with test-time compute outperforming the larger model on easy-to-medium problems. The paper does not include any comparison where the larger model receives test-time compute. The authors acknowledge the pretraining scaling limitation explicitly (quoted above) but do not report results with a compute-optimally pretrained baseline or a larger model with test-time compute.

Mitigation status. The paper acknowledges the issue and defers it to future work. No sensitivity analysis is performed — for instance, estimating how much better a Chinchilla-optimal 14× larger model might be, or testing how much the larger model benefits from a small test-time compute budget. The conclusion that test-time compute can substitute for pretraining compute on certain problems should therefore be interpreted as a conditional finding that depends on the specific (non-compute-optimal) pretraining baseline used.

Hard Problems Remain Essentially Unsolved: Test-Time Compute Cannot Create Capability from Nothing

The assumption or constraint. The entire compute-optimal framework assumes that the base model has some non-trivial probability of generating a correct solution — the pass@1 rate must be above zero for search or revisions to help. When the base model's pass@1 is near zero, no amount of test-time compute can find or refine a correct solution, because there are no correct solutions in the proposal distribution.

The consequence. Across all methods — search (Figure 3, right), revisions (Figure 7, right), and their compute-optimal combinations — the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all budget levels. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, and the 14× larger model with greedy decoding outperforms any amount of test-time compute on the smaller model.

This establishes a hard boundary: test-time compute amplifies existing capability but does not create it. For any problem where the base model fundamentally lacks the knowledge or reasoning ability to produce a correct answer even at very low probability, no amount of search, revision, or adaptive allocation will help. This is not a limitation the paper can "fix" — it is an inherent property of the approach. The paper is candid about this:

"For the hardest questions (bin 5), no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated" (Section 5.3).

What evidence exists in the paper. The difficulty bin breakdowns in Figures 3 (right), 7 (right), and 9 consistently show bin 5 as a flat line near zero. The FLOPs-matched comparison in Figure 9 and the bar charts in Figure 1 show that the 14× larger model outperforms test-time compute on hard questions across all R values, with relative disadvantages of up to -52.9% (PRM search, R >> 1). The paper does not quantify what fraction of MATH problems fall into bin 5 — only that it is the hardest quintile of the 500-question test set.

Mitigation status. The paper does not attempt to mitigate this limitation. It is presented as a finding, not a problem to solve: test-time compute and pretraining compute are not 1-to-1 exchangeable, and for genuinely hard problems, pretraining is the only viable path. The paper does not suggest any mechanism by which test-time compute could help on bin 5 problems (e.g., using the PRM to guide exploration of qualitatively different approaches, or incorporating external knowledge retrieval).

Verifier Over-Optimization Is a Hard Ceiling for Search-Based Methods

The assumption or constraint. All search-based methods (beam search, lookahead search, best-of-N weighted) rely on the PRM to score candidate solutions. The PRM is trained on the base model's outputs using Monte Carlo rollouts and is not adversarially robust — it can be exploited by search algorithms that find solutions scoring highly under the PRM but that are actually incorrect. This phenomenon, termed verifier over-optimization, is documented in Section 5.3 and Appendix M.

The consequence. The effectiveness of search-based test-time compute is fundamentally bounded by PRM quality, not by the search algorithm. Even the compute-optimal policy — which routes easy problems away from aggressive search to mitigate over-optimization — does not solve the underlying problem. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling: beam search curves in Figure 3 flatten and sometimes decline well before the budget is exhausted. The paradoxical finding that lookahead search (the most powerful optimizer) performs worst overall (Figure 3, left) is a direct consequence — the stronger the optimization, the more severely it exploits PRM weaknesses.

Qualitative examples in Appendix M (e.g., Figure 29) show concrete failure modes: beam search produces degenerate outputs — repetitive low-information steps at the end of solutions, overly short 1–2 step solutions — that score highly under the PRM but are transparently wrong to a human evaluator. This means the PRM's scores become anti-correlated with actual correctness at high optimization pressures.

What evidence exists in the paper. Figure 3 (right) provides the clearest evidence: on easy questions (bin 1), beam search accuracy decreases as the budget increases (from roughly 78% to 77% as budget goes from 4 to 256 generations), while best-of-N weighted — a weaker optimizer — continues to improve (from 68% to 88%). Figure 3 (left) shows lookahead search systematically underperforming at matched generation budgets. Appendix M provides qualitative examples of degenerate search outputs.

Mitigation status. The paper identifies over-optimization as a key bottleneck but does not attempt to address it directly. The compute-optimal policy mitigates it indirectly — by routing easy problems to best-of-N (weaker optimization, less exploitation) and reserving beam search for medium problems where the PRM signal is more reliable — but this is a routing strategy, not a solution to the underlying PRM robustness problem. The paper does not explore training PRMs with adversarial or search-aware objectives, ensemble verification, or constrained search methods (e.g., KL-penalized search that prevents the proposal distribution from drifting too far).

Sequential Revisions Incur Unaccounted Latency Costs Not Captured by Generation Count

The assumption or constraint. The paper measures test-time compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but implicitly assumes all generations can be parallelized. Sequential revisions are inherently serial: each revision depends on the previous one, so a chain of length L takes L sequential forward passes, while L parallel samples can be executed simultaneously given sufficient hardware.

The consequence. A strategy that allocates 128 generations as 64 sequential × 2 parallel chains takes approximately 64× longer wall-clock time than one that runs 128 parallel samples simultaneously (assuming sufficient hardware for the parallel case). For latency-sensitive applications — interactive assistants, real-time decision-making systems, any user-facing deployment — the sequential-heavy strategies favored by the compute-optimal policy on easy problems (Figure 7, right: easy problems perform best with purely sequential revisions) may be impractical regardless of their FLOPs efficiency or accuracy advantage. The paper's compute-optimal framework optimizes for FLOPs efficiency (total compute), not for latency (wall-clock time to answer).

What evidence exists in the paper. The paper reports inference throughput (images/sec/core for Mixer; generations/sec for the PaLM 2 models in this paper) but does not report latency for sequential strategies. The revision model results in Figures 6–8 are reported in terms of generation budget, without any accounting for the serial dependency. The paper does not mention latency as a consideration anywhere in the main text or appendices.

Mitigation status. Not addressed. The paper does not discuss the latency implications of sequential revisions, does not propose any mechanism to parallelize within a revision chain (e.g., speculative revision where multiple candidate next revisions are generated in parallel and the PRM selects the best), and does not include latency-aware allocation as a dimension of the compute-optimal framework. For practitioners deploying these methods, this gap means the reported optimal policies may need significant adjustment when latency constraints are binding.

7. Implications and Future Directions

How This Work Changes the Landscape

MLP-Mixer does not dethrone CNNs or Vision Transformers, and it does not claim to. What it does is redefine the terms under which architectural necessity is debated. Before Mixer, the working assumption in computer vision was that some form of structured spatial processing — whether convolutional (local, translation-equivariant, weight-shared) or attentional (content-dependent, global) — was required to reach the accuracy–compute Pareto frontier. That assumption was reasonable: every state-of-the-art model for nearly a decade had relied on one of these mechanisms. Mixer provides the first clear counterexample at scale, and in doing so, it does not just add another architecture to the catalog — it inverts the burden of proof. The question is no longer "can we make vision work without convolutions or attention?" but rather "given that we can, what is the actual role of architectural inductive biases, and when do they help versus constrain?"

This is a conceptual reframing more than a paradigm shift. CNNs and Transformers remain dominant, and Mixer is not a drop-in replacement for most practitioners (see the conditional nature of its competitiveness below). But the paper changes how the field thinks about architecture design. The dominant narrative had been one of progressive refinement: AlexNet → VGG → Inception → ResNet → EfficientNet → ViT, each iteration introducing or refining a spatial mechanism. Mixer demonstrates that this entire trajectory — the increasingly sophisticated engineering of spatial processing — was one path through a much larger design space, not the only path. The existence proof matters: it means future architecture designers should not assume that spatial structure must be baked in; they can separately consider whether to include it based on the available data scale and deployment constraints.

The paper also resolves a latent tension in the ViT literature. ViT showed that attention could replace convolution, but it left open a critical question: was attention special (because content-dependent routing is a powerful general mechanism), or was any sufficiently flexible operator sufficient? The ViT paper itself could not distinguish these hypotheses — its attention layers might have been doing something qualitatively different from what a generic MLP could learn. Mixer closes this loop. Attention is also not necessary. The implication is that the sequence of architectural innovations from AlexNet through ViT was not progressively uncovering fundamental computational requirements of vision. Rather, each step demonstrated that a different set of assumptions could work — and Mixer takes this to the logical extreme by showing that no spatial assumptions can work.

This reframing makes certain research directions more attractive and others less so:

More attractive:

Exploring radically different spatial operators. If a shared MLP across channels with full receptive field can work, what about Fourier transforms applied spatially, or graph neural networks over learned connectivity, or continuous neural fields conditioned on position? The design space is wide open in a way it was not before Mixer.
Understanding what CNNs and Transformers actually learn versus what Mixer learns. The paper's weight visualizations (Figure 5) show Mixer developing CNN-like paired filters and localized receptive fields without being forced to. This raises the question: do all sufficiently flexible architectures converge to similar representational strategies when trained on natural images? A systematic representational similarity analysis (e.g., CKA, SVCCA) between Mixer, ViT, and CNNs at matched performance levels could reveal whether the "essential computations" of vision are architecture-independent.
Scaling laws for architecture-data tradeoffs. Figure 2 (right) provides preliminary evidence that the slope of accuracy improvement with dataset size differs across architecture families — Mixer improves fastest, ViT next, BiT slowest. Formalizing this as a scaling law (accuracy as a function of dataset size, model size, and architecture type) would be both theoretically illuminating and practically useful for architecture selection.
Architecture-agnostic design patterns. The two-axis decomposition (spatial mixing vs. channel mixing) that Mixer makes explicit is a design pattern that could be applied far beyond MLPs. Any vision architecture can be analyzed through this lens, and future designs could independently innovate on each axis.

Less attractive:

Incremental improvements to convolutional kernel design. If an all-MLP architecture can compete at scale, the marginal value of optimizing, say, a new 3×3 convolution variant with slightly better accuracy on ImageNet is diminished — the gains are real but bounded within a paradigm that is no longer the only option.
Architecture search restricted to conv- or attention-based operations. The design space is larger than these primitives; NAS constrained to these building blocks may miss entire regions of the Pareto frontier.
Assuming that inductive biases must be hard-coded. Mixer shows that data can substitute for design. Architecture research can shift from "what biases should we build in?" to "what biases does the data provide, and how can we help the model discover them?"

Follow-Up Research This Work Enables

Scaling law analysis: at what dataset size does Mixer cross ViT? Figure 2 (right) shows Mixer-L/16 and ViT-L/16 converging as data grows from 9M to 300M to 3B images, but the curves have not yet crossed. A rigorous scaling law study — pre-training both architectures on datasets spanning 3M to 3B+ images, fitting power-law functions to the accuracy vs. data curves, and computing the crossover point — would answer whether Mixer eventually surpasses ViT or merely catches it asymptotically. The paper provides exactly the right experimental template (equal total steps, linear 5-shot evaluation) and the JFT-3B data point suggests crossover might occur at ~10B images. If Mixer does overtake ViT at some data scale, it would fundamentally change the architecture recommendation for very-large-data regimes. Conversely, if the curves asymptotically approach but never cross, it would suggest attention provides a small but irreducible benefit that no amount of data can compensate for — an equally important finding.

Representational similarity analysis: do Mixer, ViT, and CNNs converge to the same features? The weight visualizations in Figure 5 and Appendix D show Mixer developing CNN-like structures (Gabor-like paired filters, localized receptive fields), but this is impressionistic. A systematic study using Centered Kernel Alignment (CKA) or Singular Vector Canonical Correlation Analysis (SVCCA) to compare internal representations layer-by-layer across Mixer, ViT, and ResNet at multiple scales and dataset sizes would reveal: (1) whether early-layer representations converge to similar features despite radically different architectures; (2) whether deep-layer representations remain architecture-specific even when final accuracy matches; (3) how the representational similarity changes with dataset size (do larger datasets push architectures toward more similar representations?). The paper provides the trained models and weight visualizations; a representational analysis would convert these qualitative observations into quantitative claims.

Can Mixer be extended to dense prediction tasks (detection, segmentation)? The paper evaluates Mixer exclusively on image classification, which requires only a single global representation. Dense prediction tasks require per-pixel or per-region outputs, raising the question of how to extract spatial structure from an architecture whose token-mixing MLPs learn fixed position-to-position mappings. The obvious approach — treat the final Mixer layer's per-patch features as a spatial feature map and attach a detection/segmentation head — may fail because the token-mixing MLPs have already globally mixed information, potentially destroying the spatial localization needed for dense prediction. A strong follow-up would test Mixer as a backbone for Mask R-CNN or UPerNet on COCO and ADE20K, comparing against ViT and ResNet backbones. If Mixer performs poorly, it would establish a boundary: all-MLP architectures work for holistic recognition but not for tasks requiring spatial precision, suggesting that some architectural inductive bias for spatial structure is necessary for dense prediction. If it works, it would further expand the domain of "neither convolutions nor attention are necessary."

Training a Mixer-style architecture on text: does the two-axis decomposition transfer to NLP? The paper explicitly asks in Section 5: "It would be particularly interesting to see whether such a design works in NLP or other domains." In NLP, the sequence of tokens is analogous to the sequence of patches, and the feature dimension is analogous to the embedding dimension. A token-mixing MLP applied across sequence positions and a channel-mixing MLP applied across embedding dimensions would be a direct analog of Mixer for language — essentially a Transformer with both self-attention and feed-forward layers replaced by MLPs. The key question is whether the sequential structure of language (where position carries semantic information differently than in images) requires a different mixing strategy. A concrete experiment: pre-train a Mixer-style language model and a standard Transformer on the same corpus (e.g., C4 or the Pile) at matched parameter counts, compare perplexity and downstream task performance. The linear vs. quadratic complexity difference (Mixer's $O(S \cdot D_S)$ vs. Transformer's $O(S^2)$ ) would make Mixer particularly attractive for long-document tasks if it can match Transformer quality.

Verifier-style evaluation of Mixer's spatial reasoning: can it count, compare positions, or solve spatial analogies? The permutation experiment (Section 3.4, Figure 4) shows that Mixer is invariant to input order — it learns position-specific weights in the token-mixing MLPs but has no notion of distance or adjacency. This raises the question of whether Mixer can perform tasks that require explicit spatial reasoning: counting objects, comparing relative positions ("is the cat to the left of the dog?"), or solving Raven's Progressive Matrices. A diagnostic suite of spatial reasoning tasks — similar to the synthetic reasoning benchmarks used to probe ViT and CNNs — would characterize exactly what spatial capabilities Mixer acquires from data versus what capabilities remain architecture-dependent. If Mixer fails at tasks requiring metric spatial reasoning (distances, relative positions) while succeeding at holistic recognition, it would clarify that "vision" is not a unitary capability and that different tasks may require different architectural assumptions.

Principled study of patch resolution effects: what is the optimal accuracy-per-FLOP allocation between patch count and hidden dimension? Table 1 shows that S/16 (196 patches, 18M parameters) and S/32 (49 patches, 19M parameters) have nearly identical parameter counts but dramatically different throughput (3,994 vs. 11,489 img/sec/core) and accuracy (73.83% vs. 68.70% on JFT-300M). This is a single data point in a larger trade-off space. A systematic study that varies patch resolution, hidden dimension, and MLP widths at a fixed parameter or FLOPs budget could reveal the compute-optimal allocation — similar in spirit to the Chinchilla scaling laws but for the architecture's internal dimensions. The paper provides the architecture and training setup; a scaling study across these axes would give practitioners concrete guidance on how to configure Mixer for their hardware constraints.

Practical Applications and Downstream Use Cases

High-throughput image classification on TPU clusters. The most directly actionable finding is that Mixer-H/14 achieves 87.94% ImageNet top-1 at 40 img/sec/core on TPU-v3 — 2.5× faster than ViT-H/14 (15 img/sec/core) at comparable accuracy (Table 2). For organizations running large-scale image classification on TPU hardware — content moderation, product categorization, visual search indexing — this throughput advantage translates directly to serving cost reduction. A deployment that previously required 100 TPU-v3 cores to serve ViT-H/14 at a given throughput could serve Mixer-H/14 with approximately 40 cores, a 2.5× reduction in hardware footprint. The caveat is hardware-specificity: the paper's throughput measurements are on TPU-v3 only, and the relative efficiency of matrix multiplications vs. attention vs. convolutions differs across accelerators. Practitioners should benchmark on their target hardware before switching.

Lightweight deployment on hardware optimized for matrix multiplication. Mixer's architectural simplicity — "only basic matrix multiplication routines, changes to data layout (reshapes and transpositions), and scalar nonlinearities" — makes it particularly well-suited for hardware where matrix multiply is the dominant optimized operation. This includes TPUs, but also Apple's Neural Engine, Qualcomm's Hexagon DSP, and many edge inference accelerators. Unlike convolutions (which may require specialized im2col or Winograd implementations) and self-attention (which requires efficient softmax and potentially sparse attention patterns), Mixer maps cleanly to the primitive operations that hardware vendors optimize first. For mobile or embedded vision applications where the inference stack must be implemented from scratch or verified for safety, Mixer's 43-line JAX/Flax implementation (Appendix E) suggests a correspondingly simple deployment that reduces engineering risk.

Pre-training for transfer learning at very large data scales. Figure 2 (right) shows that Mixer's relative performance improves with dataset size: the gap versus ViT shrinks from ~7% at 9M images to near zero at 3B images. For organizations with access to very large proprietary image datasets (100M+ images), Mixer may be the preferred pre-training architecture — not because it achieves higher accuracy than ViT (the gap is small even at JFT-300M scale, Table 2), but because its pre-training cost is lower and its inference throughput is higher at comparable accuracy. Mixer-H/14 costs 1.01k TPUv3-core-days to pre-train versus ViT-H/14's 2.30k (Table 2), a 2.3× reduction. In a regime where pre-training cost dominates total expenditure, this saving is substantial and may outweigh the small accuracy gap.

Architecture prototyping and research on new building blocks. The paper's demonstration that an all-MLP design works at scale provides a testbed for architecture research that is simpler than modifying CNNs or Transformers. A researcher wanting to test a new nonlinearity, normalization scheme, or regularization technique can drop it into Mixer and evaluate at scale without worrying about interactions with convolution kernel sizes, stride patterns, or attention mechanisms. Mixer's two-axis decomposition (token-mixing vs. channel-mixing) also provides a natural framework for testing new spatial operators: replace the token-mixing MLP with the proposed operator (a Fourier transform, a graph convolution, a fixed random projection) while keeping the channel-mixing MLP unchanged, and measure the accuracy gap. This modularity makes Mixer valuable as an experimental platform even if it never becomes the dominant production architecture.

When to Prefer This Method

The paper explicitly positions Mixer's competitiveness as conditional on data scale and accepts that it falls behind CNNs and ViTs at smaller data scales. The decision rule is grounded in the paper's own results:

Prefer MLP-Mixer when: (1) pre-training dataset size is very large (100M+ images, JFT-300M scale), because Mixer's performance gap versus ViT shrinks to ~0.5% at H/14 scale (Table 2) while providing substantially higher inference throughput (2.5× faster than ViT-H/14) and lower pre-training cost (2.3× fewer core-days); OR (2) inference hardware is strongly optimized for matrix multiplication and poorly optimized for convolutions or attention (TPUs, certain edge accelerators), because Mixer maps to the hardware's strengths without requiring specialized kernel implementations; OR (3) implementation simplicity and ease of deployment are primary concerns, because the entire architecture fits in 43 lines of code with no custom CUDA kernels or complex autograd operations; OR (4) you want to experiment with novel architectural components where interactions with convolutions or attention would be confounding variables, using Mixer as a clean-slate testbed.
Prefer Vision Transformers when: (1) you need the absolute highest accuracy and can tolerate lower throughput (ViT-H/14 achieves 88.55% vs. Mixer-H/14's 87.94% on ImageNet with JFT-300M pre-training, Table 2); OR (2) you need a single architecture that also works well on NLP tasks (since ViT shares the Transformer design with language models, enabling unified multi-modal architectures); OR (3) pre-training dataset size is modest (1–14M images) and you cannot afford the heavy regularization that Mixer requires in this regime — ViT-B/16 reaches 84.59% on ImageNet-21k versus Mixer-B/16's 80.64% (Table 3).
Prefer CNNs (ResNet, EfficientNet, NFNet) when: (1) pre-training data is limited (standard ImageNet-1k, 1.3M images) and you are training from scratch — Mixer-B/16 achieves 76.44% from scratch versus NFNet's 86.5% (Section 3.1), and even with heavy regularization Mixer overfits more severely than CNNs; OR (2) you need a pyramidal architecture that produces multi-scale feature maps for dense prediction tasks (detection, segmentation), since Mixer's isotropic design outputs a single-scale representation and the paper's pyramidal experiments were not successful on ImageNet-scale data (Appendix A.1); OR (3) inference latency is the binding constraint and convolutions can be heavily optimized on your target hardware (cuDNN, TensorRT) — while Mixer has high throughput, per-image latency depends on hardware-specific kernel implementations.

The boundary between "prefer Mixer" and "prefer ViT" is not sharp — at JFT-300M scale with H/14 models, the accuracy difference is only 0.6% (87.94% vs. 88.55%), and the throughput difference (40 vs. 15 img/sec/core) may dominate the decision for many deployments. The paper does not provide guidance on whether Mixer at a larger scale (beyond H/14) would close the remaining gap, making this an empirical question for organizations with the compute to scale further.