ArXiv: 2010.11929

🎯 Pitch

A pure Transformer matching state-of-the-art CNNs on ImageNet sounds impossibleβ€”until you feed it 300 million images. ViT shows that the right training scale can override the need for convolutional inductive biases, achieving top results with far less compute.

1. Executive Summary

This paper introduces the Vision Transformer (ViT), a model that applies a standard Transformer encoder directly to sequences of image patches for image classification, departing from the then-dominant paradigm of convolutional neural networks. The key finding is that while ViT underperforms comparable CNNs when trained on mid-sized datasets like ImageNet due to its lack of built-in inductive biases β€” translation equivariance and locality β€” it surpasses state-of-the-art convolutional architectures when pre-trained on sufficiently large datasets (ImageNet-21k with 14M images or JFT-300M with 303M images), establishing that large-scale training trumps inductive bias. The best model, ViT-H/14 pre-trained on JFT-300M, achieves 88.55% top-1 accuracy on ImageNet, 90.72% on ImageNet-ReaL, and 77.63% across the 19-task VTAB suite, while requiring substantially fewer computational resources to train than leading CNN baselines β€” 2.5k TPUv3-core-days versus 9.9k TPUv3-core-days for the comparable BiT-L ResNet β€” establishing that a pure transformer applied to patch sequences can match or exceed the state of the art only when paired with sufficient pre-training scale.

2. Context and Motivation

The Core Problem: Convolutional Neural Networks Had a Near-Monopoly on Vision

In early 2021, when this paper was published, the field of computer vision was dominated by convolutional neural networks (CNNs). This dominance was not accidental β€” it reflected decades of accumulated evidence that convolution's built-in assumptions about how images work were genuinely useful. The core problem the Vision Transformer paper addresses is a fundamental architectural question: is the convolutional inductive bias genuinely necessary for strong visual representation learning, or is it merely a convenient shortcut that becomes unnecessary when sufficient data is available?

Convolutional architectures encode two specific assumptions about images into their structure. First, locality: pixels that are close together in image space are more likely to be related than pixels that are far apart. A 3Γ—3 convolution only looks at a local neighborhood. Second, translation equivariance: if an object moves from the left side of an image to the right side, the convolution's output should move in a corresponding way, because the same filter is applied everywhere. These inductive biases constrain the space of functions a CNN can learn β€” which is helpful when data is limited (the model doesn't have to "rediscover" basic image properties from scratch) but potentially harmful when data is abundant (the biases might prevent the model from learning patterns that don't conform to the local, translation-equivariant assumptions).

The paper frames this tension explicitly in the introduction:

"When trained on mid-sized datasets such as ImageNet without strong regularization, these models yield modest accuracies of a few percentage points below ResNets of comparable size. This seemingly discouraging outcome may be expected: Transformers lack some of the inductive biases inherent to CNNs, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data."

The key question, then, is whether the convolutional inductive bias is an architectural necessity or a data-efficiency crutch. The paper's central thesis is that it's the latter β€” and that at sufficient scale, the flexibility of a pure attention-based architecture actually becomes an advantage because the model can learn whatever spatial patterns the data requires, rather than being constrained to only local, translation-equivariant ones.

Why This Problem Matters (and Why in 2020-2021 It Was Especially Urgent)

The practical stakes of this question were enormous for several reasons:

The NLP revolution left vision behind. By late 2020, Transformers (Vaswani et al., 2017) had completely transformed natural language processing. BERT (Devlin et al., 2019) had established the pre-train-then-fine-tune paradigm that dominated NLP benchmarks. GPT-3 (Brown et al., 2020) had demonstrated that scaling up Transformer models and datasets seemed to produce qualitatively new capabilities β€” emergent few-shot learning, reasoning-like behavior, and impressive generative fluency. Crucially, the NLP community had converged on a mostly unified architecture: the Transformer. This architectural consolidation meant that engineering efforts (optimized kernels, distributed training infrastructure, hardware co-design) could be amortized across virtually all NLP applications.

Computer vision had no such architectural consolidation. The state of the art was a diverse zoo of architectures: ResNets (He et al., 2016), EfficientNets (Tan & Le, 2019), ResNeXts (Xie et al., 2017), and various attention-augmented hybrids β€” each with their own optimized implementations, hyperparameter sensitivities, and hardware affinities. If a Transformer could work for vision with minimal modification, the entire NLP infrastructure stack (TPU kernels, distributed training recipes, scaling strategies) could be repurposed. This is not a minor practical consideration β€” the paper emphasizes this directly in Section 3:

"An advantage of this intentionally simple setup is that scalable NLP Transformer architectures – and their efficient implementations – can be used almost out of the box."

Scaling laws were pointing toward larger models and datasets. By 2020, evidence was accumulating across both NLP and vision that simply making models larger and training them on more data was a reliable path to better performance β€” often more reliable than clever architectural innovations. In NLP, Kaplan et al. (2020) had characterized power-law scaling relationships. In vision, Sun et al. (2017) had shown that CNN performance scaled with dataset size, and Kolesnikov et al. (2020) had demonstrated that large-scale pre-training on datasets like JFT-300M produced CNNs that transferred remarkably well to downstream tasks. The BiT (Big Transfer) paper had established that ResNets, when pre-trained on large datasets with the right normalization choices (Group Normalization instead of Batch Normalization, standardized convolutions), set a very strong baseline. What was unknown was whether these scaling trends would hold β€” or even accelerate β€” for architectures with fewer built-in assumptions.

The theoretical debate about inductive biases. There is a deep theoretical question lurking beneath this practical concern: what is the role of architectural priors versus learned structure? CNNs embody the hypothesis that vision requires hard-coded knowledge about spatial locality and translation equivariance. The success of attention in NLP β€” where word order matters but is not as rigidly structured as 2D spatial relationships β€” did not automatically imply that vision could work the same way. Images have a fundamentally different structure than text: 2D geometry, occlusions, scale variations, and fine-grained texture patterns. It was entirely plausible that a global attention mechanism that treats every patch as potentially related to every other patch would be overwhelmed by the quadratic complexity of pairwise relationships in images, or would learn spurious long-range correlations that don't generalize, or would simply fail to capture the local texture patterns that CNNs handle so naturally.

Prior Approaches and Where They Fell Short

The paper positions itself relative to four distinct lines of prior work, each of which attempted to bring attention mechanisms to vision but stopped short of a pure, unmodified Transformer:

1. Attention-augmented convolutional networks. This line of work (Wang et al., 2018; Bello et al., 2019; Hu et al., 2018; Carion et al., 2020) kept the CNN backbone intact and added self-attention as a supplement β€” for example, by inserting non-local blocks that allow feature maps at one spatial location to attend to feature maps at all other locations, or by using Transformers to process CNN output features for object detection (as in DETR, Carion et al., 2020). The limitation is conceptual and practical: the CNN backbone still does the heavy lifting of feature extraction, and the attention mechanism is an add-on rather than a replacement. You still need the CNN infrastructure; you don't get the NLP-style architectural unification.

2. Stand-alone self-attention replacing convolutions entirely. A more ambitious line of work attempted to replace convolutions with self-attention mechanisms β€” building an entire vision model out of attention operations rather than convolutional ones. Ramachandran et al. (2019) proposed "stand-alone self-attention" where every spatial location attends to a local neighborhood around it, mimicking the limited receptive field of a convolution. Wang et al. (2020a) pushed this further with axial attention, decomposing 2D attention into sequential row-wise and column-wise operations to reduce complexity. Zhao et al. (2020) explored various self-attention architectures for image recognition. These models demonstrated that attention could work for vision in principle, but the paper identifies a critical limitation:

"The latter models, while theoretically efficient, have not yet been scaled effectively on modern hardware accelerators due to the use of specialized attention patterns."

This is the key shortfall: these attention-based vision models used specialized attention patterns (local windows, axial decompositions, sparse patterns) that were not the standard global multi-head self-attention that Transformers use in NLP. Because the patterns were non-standard, they couldn't leverage the highly optimized Transformer implementations that existed for TPUs and GPUs. They required custom kernels and complex engineering, making them impractical to scale to the massive datasets (300M images) where the benefits of attention might outweigh its inductive bias disadvantage.

3. Patch-based Transformers at small scale. Cordonnier et al. (2020) came closest to the ViT approach: they extracted 2Γ—2 patches from an input image and applied full self-attention to those patches. This is architecturally very similar to ViT. However, there were two critical differences that the ViT paper highlights:

  • Scale of evaluation: Cordonnier et al. (2020) did not pre-train on large datasets. Their experiments were limited to smaller-scale settings where Transformers, lacking inductive biases, would struggle to match CNNs. They couldn't demonstrate the "large scale trumps inductive bias" result because they never tested at large scale.
  • Patch size limitation: Using 2Γ—2 patches means the effective sequence length is very large (e.g., a 224Γ—224 image becomes 12544 patches). The quadratic complexity of self-attention becomes prohibitive at this scale. ViT uses much larger patches (16Γ—16 or 14Γ—14), reducing the sequence length to manageable levels (196 or 256 tokens) and making large-scale training practical.

As the paper states:

"Cordonnier et al. (2020) use a small patch size of 2 Γ— 2 pixels, which makes the model applicable only to small-resolution images, while we handle medium-resolution images as well."

4. Image GPT (iGPT). Chen et al. (2020a) applied Transformers to images in a completely different way: they treated images as sequences of pixels (after aggressively reducing resolution and color depth) and trained them autoregressively as generative models, similar to how GPT-2/3 model text. They achieved 72% on ImageNet after fine-tuning β€” respectable but substantially behind state-of-the-art CNNs at the time (which were above 85%). The iGPT approach had several limitations that ViT aimed to overcome: (a) the pixel-level representation is extremely long (even reduced-resolution images produce thousands of tokens), making training expensive; (b) the generative pre-training objective may not be optimal for representation learning compared to supervised pre-training; (c) the low-resolution, reduced-color representation necessarily discards fine-grained visual information that matters for classification.

5. Large-scale CNN pre-training. Concurrently with the NLP revolution, the vision community had been pushing the boundaries of CNN pre-training. Mahajan et al. (2018) explored weakly supervised pre-training on billions of Instagram images. Kolesnikov et al. (2020) developed BiT β€” Big Transfer β€” which pre-trained ResNets on large datasets like ImageNet-21k and JFT-300M using Group Normalization and standardized convolutions, establishing strong baselines for transfer learning. Xie et al. (2020) trained EfficientNets with semi-supervised learning (Noisy Student) on JFT-300M, achieving state-of-the-art ImageNet accuracy. These works showed that CNNs benefit from scale, but they left open the question of whether CNNs were the best architecture at scale, or simply the default choice that everyone was scaling because it was the default choice.

How the Paper Positions Itself

The Vision Transformer paper positions itself as a direct challenge to the assumption that convolutional inductive biases are necessary for vision β€” or even helpful at scale. Its positioning can be understood along four dimensions:

Methodological minimalism as a philosophical stance. The paper's key methodological claim is that it applies a standard Transformer β€” not a modified or vision-specialized one β€” directly to image patches. This minimalism is both scientific and practical. Scientifically, it provides the cleanest test of the "inductive bias vs. scale" hypothesis: if you remove all vision-specific architectural choices (except the initial patchification, which is a necessary interface between the 2D image and the 1D Transformer), do you still get competitive performance at scale? Practically, minimal modification means maximal reuse of existing Transformer infrastructure β€” the TPU kernels, the distributed training systems, the hyperparameter schedules can all be borrowed from NLP.

The paper emphasizes this:

"In model design we follow the original Transformer (Vaswani et al., 2017) as closely as possible."

This is a deliberate contrast with prior attention-for-vision work, which introduced specialized attention patterns, local windows, axial decompositions, or hybrid CNN-attention architectures β€” each of which required domain-specific engineering.

Scale as the independent variable, not architecture. The paper's experimental design β€” and its most important intellectual contribution β€” is the treatment of dataset size as a controlled variable. Rather than asking "is the Vision Transformer a good model?" it asks "at what scale does the Vision Transformer become a good model, and why?" The comparisons in Figures 3 and 4 systematically vary pre-training dataset size from 1.3M images (ImageNet) through 14M (ImageNet-21k) to 303M (JFT-300M), revealing a crossover point: below some dataset size threshold, CNNs win because inductive bias compensates for limited data; above that threshold, ViT catches up and eventually surpasses. This reframes the narrative from "architecture A beats architecture B" to "architecture A and architecture B have different scaling properties, and the optimal choice depends on your data budget."

Pre-training compute as a comparable unit of cost. By reporting TPUv3-core-days for all models (their own ViTs and the CNN baselines), the paper positions computational cost as a first-class metric alongside accuracy. This is important because it prevents "unfair" comparisons where a model with more parameters or more training is claimed to be architecturally superior. When ViT-H/14 achieves 88.55% on ImageNet with 2.5k TPUv3-core-days versus BiT-L's 87.54% with 9.9k TPUv3-core-days, the comparison is not just about accuracy β€” it's about accuracy per unit of computation. This framing makes the practical case for ViT: even if you're not convinced about the philosophical merits of attention over convolution, ViT gives you better accuracy for less compute, so you should use it.

The hybrid model as an empirical bridge. The paper introduces a hybrid architecture β€” a CNN front-end that produces feature maps, followed by a Transformer that processes patches extracted from those feature maps β€” as a way to probe whether the benefits of convolution and attention are complementary. The result in Figure 5 captures a nuanced finding: hybrids slightly outperform pure ViT at small model sizes, but the gap vanishes at larger sizes. This suggests that convolutional feature extraction helps when the Transformer is capacity-limited, but that a sufficiently large Transformer can learn whatever useful representations the CNN would have provided β€” and possibly better ones, since the Transformer isn't constrained to the CNN's local receptive fields. The hybrid model thus serves as an empirical argument that the benefits of convolution are data- and compute-dependent, not absolute.

In essence, the Vision Transformer paper is not just proposing a new architecture β€” it's proposing a new way of thinking about architecture design in computer vision. The question is not "what operations should a vision model use?" but rather "given enough data and compute, what constraints can we remove and still have the model learn effectively?" This framing, which the NLP community had already adopted, was novel for vision and opened the door to the subsequent wave of Transformer-based vision models (DeiT, Swin, DINO, MAE) that built on ViT's foundation.

3. Technical Approach

3.1 Reader Orientation

This paper introduces the Vision Transformer (ViT), a system that performs image classification by treating an image as a sequence of patches β€” analogous to words in a sentence β€” and processing those patches through a standard Transformer encoder originally designed for natural language. The system solves the problem of applying Transformers to visual data without introducing vision-specific architectural modifications, and the "shape" of the solution is remarkably minimal: the only vision-specific operation is the initial slicing of an image into fixed-size patches and linearly projecting each patch into a vector, after which a completely generic Transformer processes the resulting sequence and produces a classification.

3.2 Big-Picture Architecture (Diagram in Words)

The Vision Transformer has five major components, connected in a feedforward pipeline:

  1. Patch Extraction and Flattening β€” takes a raw image and slices it into a grid of non-overlapping fixed-size patches, then flattens each 2D patch into a 1D vector of pixel values. This converts the 2D image into a 1D sequence suitable for a Transformer.

  2. Linear Patch Embedding β€” applies a trainable linear projection to each flattened patch vector, mapping it from patch-pixel-space into a D-dimensional embedding space that matches the Transformer's internal dimension. This is the learned representation of each patch's visual content.

  3. Classification Token and Position Embeddings β€” prepends a special learnable [class] token to the sequence (whose final state will serve as the image representation for classification) and adds learned position embeddings to all tokens so the Transformer knows where each patch came from in the original image.

  4. Transformer Encoder β€” a stack of L identical layers, each containing multi-headed self-attention followed by a multi-layer perceptron, with LayerNorm before each sublayer and residual connections around each sublayer. This is a completely standard Transformer encoder with no vision-specific modifications.

  5. Classification Head β€” takes the final state of the [class] token and passes it through a small MLP (during pre-training) or a single linear layer (during fine-tuning) to produce class predictions.

Information flows as follows: an image enters β†’ it is sliced into patches β†’ patches are flattened and linearly projected β†’ the [class] token is prepended and position embeddings are added β†’ the sequence passes through L Transformer layers β†’ the final [class] token representation is extracted β†’ the classification head produces class scores.

3.3 Roadmap for the Deep Dive

  • First, the patch embedding mechanism (Equation 1), because this is the only vision-specific operation in the entire model and defines how visual data enters the Transformer.
  • Second, the Transformer encoder equations (Equations 2–4), which show how the standard NLP Transformer processes the patch sequence without modification.
  • Third, the classification token design and why alternatives failed, including the empirical finding that global average pooling works equally well but requires a different learning rate.
  • Fourth, the hybrid architecture variant, which uses a CNN front-end to produce feature maps before the Transformer, and what this reveals about the complementarity of convolution and attention.
  • Fifth, the fine-tuning and resolution adjustment procedure, particularly the 2D interpolation of position embeddings and why this is one of only two points where 2D structure is manually injected.
  • Sixth, the model variants and their configurations, which are directly adopted from BERT and scaled up to create the Base, Large, and Huge families.
  • Seventh, the inductive bias analysis, explicitly cataloguing what inductive biases CNNs have that ViT lacks, and where minimal 2D structure is still injected.
  • Eighth, the pre-training and fine-tuning protocols, including optimizer choices, regularization, and the self-supervised masked patch prediction experiment.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architectural innovation paper whose core idea is that a standard Transformer applied to image patches can match or exceed convolutional networks when pre-trained on sufficient data. The technical contribution is not a new building block but rather the demonstration that existing building blocks β€” when composed in a deliberately minimal way β€” can unify vision and language architectures.

Patch Embedding: Converting an Image into a Sequence

The fundamental challenge in applying Transformers to images is that Transformers expect a 1D sequence of token embeddings as input, while images are 2D (or 3D, including color channels) grids of pixels. Naively treating every pixel as a token would produce sequence lengths of 50,176 for a 224Γ—224 image, making the quadratic self-attention cost prohibitive. The Vision Transformer solves this by coarsening the representation: instead of treating pixels as tokens, it treats patches β€” contiguous square regions of the image β€” as tokens.

The formal specification is given in Equation 1. Given an input image $x \in \mathbb{R}^{H \times W \times C}$ where $H$ is the height, $W$ is the width, and $C$ is the number of channels (3 for RGB), the image is reshaped into a sequence of flattened 2D patches $x_p \in \mathbb{R}^{N \times (P^2 \cdot C)}$:

z0=[xclass;xp1E;xp2E;⋯ ;xpNE]+Epos,E∈R(P2β‹…C)Γ—D,Epos∈R(N+1)Γ—D\mathbf{z}_0 = [\mathbf{x}_{\text{class}}; \mathbf{x}_p^1 \mathbf{E}; \mathbf{x}_p^2 \mathbf{E}; \cdots; \mathbf{x}_p^N \mathbf{E}] + \mathbf{E}_{\text{pos}}, \quad \mathbf{E} \in \mathbb{R}^{(P^2 \cdot C) \times D}, \quad \mathbf{E}_{\text{pos}} \in \mathbb{R}^{(N+1) \times D}

where $P$ is the patch size (typically 16Γ—16 or 14Γ—14 pixels), $N = HW/P^2$ is the number of patches (which is also the effective sequence length for the Transformer), $D$ is the constant latent vector size used throughout all Transformer layers, $\mathbf{E}$ is the trainable linear projection matrix that maps each flattened patch from its raw pixel space of dimension $P^2 \cdot C$ to the Transformer dimension $D$, $\mathbf{E}_{\text{pos}}$ is the learned position embedding matrix, and $\mathbf{x}_{\text{class}}$ is the special classification token embedding (prepended to the sequence).

What it computes: The equation describes three sequential operations. First, each flattened patch $\mathbf{x}_p^i$ (a vector of $P^2 \cdot C$ pixel values) is multiplied by the embedding matrix $\mathbf{E}$ to produce a $D$-dimensional patch embedding β€” this is a standard linear projection. Second, the special classification token $\mathbf{x}_{\text{class}}$ (a learnable vector of size $D$) is prepended to the sequence of patch embeddings, producing a sequence of $N+1$ vectors each of dimension $D$. Third, the position embedding matrix $\mathbf{E}_{\text{pos}}$ (containing $N+1$ learned vectors of size $D$, one per sequence position) is added element-wise to the sequence, providing positional information. The output $\mathbf{z}_0$ is the input to the first Transformer layer β€” a sequence of $N+1$ vectors, each of dimension $D$.

Why this form: The patch-based approach is a deliberate tradeoff between sequence length and granularity. The sequence length $N$ is inversely proportional to $P^2$, so doubling the patch size reduces the sequence length by a factor of 4. For a 224Γ—224 image with $P=16$, the sequence length is $224^2 / 16^2 = 196$ patches (plus the classification token gives 197), which is comparable to typical NLP sequence lengths and easily handled by standard Transformer implementations. If $P=2$ (as in prior work by Cordonnier et al., 2020), the sequence length would be 12,544, which is prohibitively long for standard self-attention. The linear projection $\mathbf{E}$ is a standard learned embedding layer β€” exactly analogous to a word embedding matrix in NLP β€” but operating on raw pixel vectors rather than token indices. The fact that this projection is learned (rather than fixed, e.g., a hand-crafted feature extractor) means the model can learn to represent whatever visual features are useful for the downstream task, including features that span the full patch. The addition of position embeddings (rather than, say, concatenation) follows the original Transformer design and ensures that positional information does not overwhelm the content information β€” the two are combined additively and the model can learn to weight them appropriately through subsequent attention operations.

An important design choice is what the position embeddings encode. The paper experiments with several alternatives (Appendix D.4): no position embeddings, 1D learned position embeddings (the default), 2D learned position embeddings (where separate X-coordinate and Y-coordinate embeddings are learned and concatenated), and relative position embeddings (where the offset between query and key positions parameterizes an attention bias). The finding is that all position embedding methods perform similarly:

"while there is a large gap between the performances of the model with no positional embedding and models with positional embedding, there is little to no difference between different ways of encoding positional information"

The authors speculate that because the Transformer operates on patches rather than pixels, the spatial dimensions are small enough (14Γ—14 rather than 224Γ—224) that learning spatial relations from scratch is easy regardless of how the position information is parameterized. This is a significant finding because it means the simplest approach β€” 1D learned position embeddings β€” suffices, keeping the model maximally simple.

The Transformer Encoder: Standard NLP Architecture Applied to Patches

Once the sequence of patch embeddings (plus classification token, plus position embeddings) is constructed, it is processed by a completely standard Transformer encoder. There is no modification whatsoever to the Transformer architecture. The encoder consists of $L$ identical layers, each applying multi-headed self-attention (MSA) followed by a multi-layer perceptron (MLP), with LayerNorm (LN) applied before each sublayer and residual connections after each sublayer β€” the "pre-norm" residual structure that had become standard in large Transformer training (Wang et al., 2019; Baevski & Auli, 2019):

zβ„“β€²=MSA(LN(zβ„“βˆ’1))+zβ„“βˆ’1,β„“=1…L\mathbf{z}'_\ell = \text{MSA}(\text{LN}(\mathbf{z}_{\ell-1})) + \mathbf{z}_{\ell-1}, \quad \ell = 1 \ldots L zβ„“=MLP(LN(zβ„“β€²))+zβ„“β€²,β„“=1…L\mathbf{z}_\ell = \text{MLP}(\text{LN}(\mathbf{z}'_\ell)) + \mathbf{z}'_\ell, \quad \ell = 1 \ldots L

where $\mathbf{z}_{\ell-1} \in \mathbb{R}^{(N+1) \times D}$ is the input to layer $\ell$ (the output of the previous layer, or $\mathbf{z}_0$ for the first layer), $\mathbf{z}'_\ell$ is the intermediate representation after the self-attention sublayer and its residual connection, $\mathbf{z}_\ell$ is the final output of layer $\ell$ after the MLP sublayer and its residual connection, LN is Layer Normalization (applied before each sublayer, hence the term "pre-norm"), MSA is multi-headed self-attention as defined in the original Transformer (Vaswani et al., 2017), and MLP is a two-layer fully-connected network with GELU non-linearity.

What it computes: At each layer, the sequence first passes through multi-headed self-attention. This operation allows every token (every patch embedding plus the classification token) to attend to every other token, computing pairwise similarity scores (via query-key dot products), converting those scores to attention weights via softmax, and using those weights to compute a weighted sum of value vectors from all tokens. The result is that each token's representation is updated to incorporate information from all other tokens in the sequence. The residual connection adds the original input $\mathbf{z}_{\ell-1}$ back to the attention output, ensuring gradient flow and allowing the model to learn incremental refinements. The output then passes through an MLP β€” two linear transformations with a GELU activation in between β€” which applies the same transformation to each position independently. Another residual connection adds the MLP input back to its output. LayerNorm is applied before each sublayer (not after), which empirically improves training stability for large Transformers.

Why this form: The key property of this architecture for vision is that self-attention is global from the very first layer. Unlike a CNN, where the receptive field starts small (3Γ—3) and grows gradually through downsampling, a ViT token at layer 1 can already attend to every other token in the image. This means the model can, in principle, integrate information across the entire image immediately β€” for example, relating a patch showing a bird's beak to a patch showing its tail feathers, even if they are on opposite sides of the image. The cost of this globality is the lack of built-in locality: the model must learn from data that nearby patches are often related and that translation equivariance is useful, rather than having these properties hard-coded. The pre-norm residual structure is inherited from NLP best practices and is not a vision-specific choice; the paper's philosophy is to change as little as possible from the standard recipe.

The MLP uses a GELU (Gaussian Error Linear Unit) non-linearity rather than the more common ReLU, following the BERT architecture (Devlin et al., 2019). GELU weights inputs by their value multiplied by the cumulative distribution function of the standard normal distribution evaluated at that value, providing a smooth, non-monotonic activation that empirically performs better than ReLU in Transformers, particularly for large models. The MLP has one hidden layer: the first linear layer expands the dimension from $D$ to $D_{\text{MLP}}$ (typically 4Γ— larger, e.g., 768 β†’ 3072 for ViT-Base), and the second linear layer projects back to $D$.

The Classification Token and Final Representation

The final classification prediction is produced from the output of the Transformer encoder by extracting only the state of the special classification token:

y=LN(zL0)\mathbf{y} = \text{LN}(\mathbf{z}^0_L)

where $\mathbf{z}^0_L$ is the output of the final Transformer layer at position 0 (the classification token's position in the sequence), LN is Layer Normalization, and $\mathbf{y}$ is the final image representation vector of dimension $D$.

What it computes: After $L$ layers of self-attention and MLP processing, the classification token has had the opportunity to attend to all patch tokens across all layers, accumulating information from the entire image. The final LayerNorm normalizes this representation, and the result $\mathbf{y}$ is the image-level feature vector. During pre-training, this vector is passed through a classification head consisting of an MLP with one hidden layer (with tanh non-linearity). During fine-tuning, a single zero-initialized linear layer $D \times K$ (where $K$ is the number of downstream classes) replaces the pre-training head.

Why this form: The classification token design is directly inherited from BERT's [CLS] token (Devlin et al., 2019). The intuition is that because this token attends to all patch tokens through self-attention, it can learn to aggregate task-relevant information from the entire image into its representation. However, the paper reveals an important nuance about this design choice in Appendix D.3. An initial attempt to use global average pooling (GAP) over all patch tokens β€” similar to how ResNets pool their final feature map β€” performed "very poorly." But further investigation showed this was not due to any inherent advantage of the classification token. Instead:

"the difference in performance is fully explained by the requirement for a different learning-rate"

When the learning rate was adjusted (from $8 \times 10^{-4}$ to $3 \times 10^{-4}$ for the GAP variant in the example shown in Figure 9), GAP achieved comparable performance to the classification token. This finding is important because it shows the classification token is a convenience, not a necessity β€” the architecture is robust to how the final representation is pooled, provided training hyperparameters are appropriately tuned.

The Hybrid Architecture: CNN Front-End + Transformer

As a bridge between pure CNNs and pure Transformers, the paper introduces a hybrid variant where the input to the Transformer is not raw image patches but feature maps produced by a CNN. Formally, the patch embedding projection in Equation 1 is applied to patches extracted from a CNN feature map rather than from the raw image. In the extreme case where patches have spatial size 1Γ—1 in the feature map, the input sequence is obtained by simply flattening the spatial dimensions of the feature map and projecting each feature vector to dimension $D$.

The paper experiments with two configurations to vary the sequence length: (1) taking the output of stage 4 of a regular ResNet50 (standard), or (2) removing stage 4, placing the same number of layers in stage 3 to keep the total layer count, and taking the output of this extended stage 3. Option (2) produces a 4Γ— longer sequence length because stage 3 operates at higher spatial resolution than stage 4, and consequently a more expensive ViT model (since self-attention cost is quadratic in sequence length).

What this design tests: The hybrid model probes whether convolutional local feature extraction is complementary to Transformer global integration. If the CNN front-end extracts useful local features and the Transformer handles long-range dependencies, the hybrid should outperform both pure CNNs and pure Transformers, especially at smaller scales where the Transformer might struggle to learn local patterns from scratch.

What the results show (Section 4.4, Figure 5): Hybrids slightly outperform pure ViT at small computational budgets, but the difference vanishes for larger models. This is a nuanced finding: convolution helps when the Transformer is capacity-limited, but a sufficiently large Transformer can learn whatever representations the CNN would have provided β€” and possibly better ones, since the Transformer is not constrained to local receptive fields. The paper notes that this result is "somewhat surprising, since one might expect convolutional local feature processing to assist ViT at any size." It suggests that at sufficient scale, the inductive bias provided by convolution is redundant at best and restrictive at worst.

Fine-Tuning at Higher Resolution and Position Embedding Interpolation

A practical challenge arises when fine-tuning on downstream tasks at higher resolution than pre-training. Pre-training typically uses 224Γ—224 images, but fine-tuning often benefits from higher resolution (e.g., 384Γ—384 or 512Γ—512). Because ViT keeps the patch size constant, higher resolution produces more patches and thus a longer sequence. For example, a 384Γ—384 image with 16Γ—16 patches produces a sequence of $384^2 / 16^2 = 576$ patches versus 196 patches at 224Γ—224 resolution.

The Vision Transformer can handle arbitrary sequence lengths (up to memory constraints) because self-attention has no built-in dependence on sequence position β€” it operates purely on the content and position embeddings. However, the pre-trained position embeddings $\mathbf{E}_{\text{pos}}$ were learned for the original $N+1$ positions (e.g., 197 positions for 224Β² with P=16). At higher resolution, there are more positions, and a naive approach (e.g., discarding the pre-trained position embeddings and learning new ones from scratch) would lose valuable pre-training information about spatial structure.

The solution is 2D interpolation: the pre-trained position embeddings are interpreted as corresponding to positions in a regular 2D grid (the original patch layout), and new position embeddings for the higher-resolution grid are produced by interpolating between the learned embeddings based on their 2D locations. For example, if the original grid is 14Γ—14 and the new grid is 24Γ—24, the embedding for position (i, j) in the new grid is a bilinear interpolation between the nearest learned embeddings in the original grid.

The paper notes (Section 3.2):

"Note that this resolution adjustment and patch extraction are the only points at which an inductive bias about the 2D structure of the images is manually injected into the Vision Transformer."

This is a carefully qualified statement. The initial patch extraction (slicing the image into a grid) imposes a 2D structure, and the position embedding interpolation during fine-tuning imposes another. Apart from these two interface points β€” which are necessary to convert between the 2D image domain and the 1D sequence domain β€” the Transformer itself has no knowledge of 2D geometry. All spatial relationships must be learned from data through the self-attention mechanism.

Model Variants and Configurations

The paper defines three ViT configurations, directly adopted from BERT (Devlin et al., 2019), as summarized in Table 1:

  • ViT-Base: 12 layers, hidden size $D = 768$, MLP hidden size 3072, 12 attention heads, 86M parameters. This is exactly the BERT-Base configuration.
  • ViT-Large: 24 layers, hidden size $D = 1024$, MLP hidden size 4096, 16 attention heads, 307M parameters. This is exactly the BERT-Large configuration.
  • ViT-Huge: 32 layers, hidden size $D = 1280$, MLP hidden size 5120, 16 attention heads, 632M parameters. This is a new, larger variant created for this paper by scaling up the BERT pattern.

The naming convention encodes the model size and patch size: ViT-L/16 means the Large variant with 16Γ—16 input patches. The patch size determines the sequence length: for a 224Γ—224 image, P=16 gives $224/16 = 14$ patches per side, producing $14 \times 14 = 196$ patches (plus the classification token = 197 tokens). The computational cost of self-attention scales quadratically with sequence length, so smaller patch sizes are substantially more expensive.

The attention head count determines how many independent attention operations run in parallel per layer. Each head operates on a subspace of dimension $D_h = D / \text{heads}$: for ViT-Base, each head has dimension $768/12 = 64$; for ViT-Large, each head has dimension $1024/16 = 64$; for ViT-Huge, each head has dimension $1280/16 = 80$. The outputs of all heads are concatenated and projected back to dimension $D$, keeping the total parameter count and computation constant regardless of head count.

Inductive Bias Analysis: What ViT Lacks and Where Structure Is Injected

The paper explicitly catalogues the inductive biases that ViT lacks relative to CNNs, and where minimal 2D structure is still injected (Section 3.1):

What CNNs have that ViT lacks:

  1. Locality: In a CNN, each neuron in early layers only receives input from a small spatial neighborhood (e.g., a 3Γ—3 convolution). The connectivity pattern forces the model to process local features first and combine them hierarchically β€” edges in early layers, textures in middle layers, object parts in late layers. ViT has no such constraint: at every layer, every patch can attend to every other patch. The model must learn from data that attending locally is often useful.

  2. Two-dimensional neighborhood structure: CNNs arrange their feature maps in a 2D grid and convolutions preserve this structure β€” a pixel at position (i, j) in one layer corresponds (approximately) to a region centered at (i, j) in the next layer. ViT flattens patches into a 1D sequence and has no built-in concept of 2D adjacency. The position embeddings provide positional information, but the attention mechanism itself doesn't know which patches are neighbors β€” it must learn that from the pattern of position embedding similarities.

  3. Translation equivariance: In a CNN, if an object shifts in the image, the feature maps shift correspondingly because the same filters are applied everywhere. This is a consequence of weight sharing in convolutions. ViT's MLP layers are translationally equivariant (they apply the same transformation to each position independently), which provides a weak form of equivariance, but the self-attention layers are not intrinsically translationally equivariant β€” the attention weights depend on the content and learned position embeddings, and there's no hard guarantee that shifting the input shifts the output correspondingly.

Where minimal 2D structure is injected:

  1. Patch extraction: Slicing the image into a regular grid of patches imposes a 2D topology β€” the model knows that patches come in rows and columns, even if that structure is not explicitly encoded. This is a necessary interface operation; without it, there would be no way to convert a 2D image into a 1D sequence.

  2. Position embedding interpolation during fine-tuning: As described above, when fine-tuning at higher resolution, the pre-trained position embeddings are interpolated in 2D space. This explicitly uses the 2D layout of patches and is the only other point where the model "knows" about image geometry.

The paper's position is that these minimal injections are sufficient. The Transformer learns all other spatial relationships β€” which patches are nearby, that translation invariance is useful, that objects have part-whole structure β€” purely from the data through the self-attention mechanism and the learned position embeddings. The evidence for this claim comes from the visualization in Figure 7 (center), which shows that the learned position embeddings naturally encode 2D distance: nearby patches have more similar position embeddings, and a clear row-column structure emerges without being explicitly programmed.

The Self-Supervised Pre-Training Experiment

The paper also explores self-supervised pre-training using a masked patch prediction objective, directly analogous to BERT's masked language modeling (Section 4.6, Appendix B.1.2). In this setup, 50% of patch embeddings are corrupted: 80% of corrupted patches are replaced with a learnable [mask] embedding, 10% are replaced with a random other patch embedding, and 10% are left unchanged. The model is trained to predict the mean 3-bit color (512 possible colors) of each corrupted patch from its final representation. This is trained for 1M steps (approximately 14 epochs) on JFT with batch size 4096, using Adam with learning rate $2 \times 10^{-4}$, linear warmup for 10k steps, and cosine decay. The self-supervised ViT-B/16 model achieves 79.9% on ImageNet β€” 2% better than training from scratch but 4% behind supervised pre-training. The paper reports that even 100k pre-training steps (rather than the full 1M) achieve similar gains, and that pre-training on ImageNet rather than JFT yields comparable improvements. This is presented as a preliminary exploration; the paper acknowledges that contrastive pre-training methods (Chen et al., 2020b; He et al., 2020) are promising alternatives left to future work.

Summary of Design Choices and Their Justifications

  • Patch-based tokenization over pixel-based: reduces sequence length from ~50,000 to ~200, making standard Transformer computation tractable. The patch size of 16Γ—16 is chosen as a practical tradeoff β€” large enough for efficiency, small enough to retain meaningful visual structure.
  • Linear patch projection over hand-crafted features or CNN-based features: preserves the end-to-end learned philosophy of Transformers. The embedding matrix $\mathbf{E}$ can learn to represent whatever features (edges, textures, colors) are useful for downstream tasks.
  • Standard Transformer encoder over specialized attention patterns (local, axial, sparse): maximizes reuse of existing NLP infrastructure (TPU kernels, distributed training code) and provides the cleanest test of whether pure attention can work for vision. This is a deliberate scientific choice, not a claim that specialized patterns are never useful.
  • Classification token over global average pooling: follows NLP convention and works well, but is not essential β€” GAP works equally well with appropriate learning rate tuning. The choice is one of convention, not necessity.
  • 1D learned position embeddings over 2D or relative embeddings: all methods work similarly (Table 8), so the simplest approach is used. This is a robustness finding: the model is not sensitive to how position information is encoded, provided it is encoded.
  • Pre-norm residual structure over post-norm: follows the training stability best practices established in large NLP Transformer training (Wang et al., 2019; Baevski & Auli, 2019).
  • GELU activation over ReLU: follows BERT, empirically better for large Transformers.
  • Adam optimizer over SGD: empirically better for pre-training (Table 7 in Appendix D.1), including for ResNets in the authors' setup. This is somewhat unconventional (ResNets are typically trained with SGD) but is justified by controlled comparison showing Adam pre-training leads to better fine-tuning transfer on most datasets.
  • 2D interpolation of position embeddings for resolution change: the only principled way to handle longer sequences without discarding pre-trained spatial knowledge. This is one of only two vision-specific design decisions in the entire model.

4. Key Insights and Innovations

Innovation 1: The Inductive Bias–Scale Tradeoff as an Empirical Principle

The Vision Transformer's most intellectually distinctive contribution is not the architecture itself β€” which is a standard Transformer β€” but the empirical demonstration that architectural inductive biases and dataset scale are fungible resources: the value of hard-coded structural assumptions decreases as the amount of training data increases, and at sufficient scale, a model with fewer built-in assumptions can outperform one with more.

This is a fundamental reframing of how the field should think about architecture design. Prior to ViT, the dominant assumption in computer vision β€” implicit in decades of CNN research β€” was that convolutional inductive biases (locality, translation equivariance, hierarchical spatial structure) were necessary for effective visual representation learning. The field had operationalized this assumption by treating CNNs as the default backbone and exploring attention only as a supplementary mechanism (Wang et al., 2018; Bello et al., 2019) or as a replacement that still preserved local, CNN-like connectivity patterns (Ramachandran et al., 2019; Zhao et al., 2020). Even the closest prior work (Cordonnier et al., 2020) used 2Γ—2 patches that made large-scale training impractical.

The ViT paper challenges this assumption not through theoretical argument but through a diagnostic experiment with dataset size as the controlled variable. Figure 3 shows the crossover: on ImageNet (1.3M images), ViT-Large underperforms ViT-Base and is beaten by BiT ResNets; on ImageNet-21k (14M images), ViT-Large roughly matches ViT-Base; on JFT-300M (303M images), ViT-Large overtakes ViT-Base by a substantial margin, and ViT-H/14 surpasses all CNN baselines. Figure 4 reinforces this with the few-shot linear evaluation on random JFT subsets of increasing size: ResNets outperform ViT at 9M and 30M samples, but ViT overtakes them at 90M+ samples.

The significance of this finding extends beyond the specific architectures compared. It establishes that scale is not merely a way to improve performance within an architecture class β€” it changes which architecture class is optimal. This is a qualitatively different claim from "bigger models work better." It implies that the architecture rankings derived from mid-sized benchmarks (like ImageNet-1k) may not hold at larger scales, and that the field's historical preference for CNNs may have been an artifact of the dataset sizes available at the time rather than a fundamental property of visual learning. This is a conceptual advance with practical consequences: it tells practitioners that if they plan to pre-train on a large dataset, they should consider architectures with fewer inductive biases, not because inductive biases are harmful per se, but because they constrain the space of learnable functions in ways that become limiting when data is abundant.

The paper's framing of this as "large scale training trumps inductive bias" (Section 1) is deliberately provocative. A more precise formulation β€” supported by the hybrid model results in Figure 5 β€” is that inductive biases provide a head start in the low-data regime, but become a ceiling in the high-data regime. The hybrid models (CNN front-end + Transformer) outperform pure ViT at small model sizes but the gap vanishes at larger sizes, suggesting that convolutional feature extraction helps when the Transformer lacks capacity to learn local patterns from scratch, but that a sufficiently large Transformer can learn these patterns autonomously β€” and may learn better ones that don't conform to the local, translation-equivariant template.

This insight has had a lasting impact on the field. The subsequent wave of vision Transformer research (DeiT, Swin Transformer, DINO, MAE) largely accepted ViT's core premise β€” that Transformers without heavy vision-specific inductive biases are viable for vision β€” and focused on improving data efficiency, self-supervised pre-training, and hierarchical architectures, rather than questioning whether pure attention could work at all.

Innovation 2: Architectural Minimalism as a Scientific and Engineering Strategy

The paper's second major innovation is methodological rather than architectural: it demonstrates that minimal modification of a domain-agnostic architecture, combined with scale, is a viable β€” and potentially superior β€” alternative to domain-specific architectural engineering. This is as much a statement about how to do research as it is about model design.

Prior to ViT, the dominant approach to improving vision models was to design vision-specific operations. CNNs themselves are the product of decades of incremental architectural innovation: local receptive fields, weight sharing, pooling, residual connections, bottleneck blocks, depthwise separable convolutions, squeeze-and-excitation modules, and dozens of other carefully engineered components. The attention-for-vision literature continued this tradition by designing specialized attention patterns β€” local windows (Parmar et al., 2018), axial decompositions (Ho et al., 2019; Wang et al., 2020a), criss-cross attention (Huang et al., 2020) β€” each of which required custom implementations that couldn't leverage the highly optimized Transformer kernels developed for NLP.

The ViT paper makes a deliberate break from this tradition. Its core methodological claim, stated explicitly in Section 3, is:

"In model design we follow the original Transformer (Vaswani et al., 2017) as closely as possible. An advantage of this intentionally simple setup is that scalable NLP Transformer architectures – and their efficient implementations – can be used almost out of the box."

This is not merely a convenience β€” it is a scientific argument. By minimizing vision-specific modifications, the paper isolates the variable of interest: the Transformer architecture itself. If the model had introduced a novel attention pattern, a specialized normalization scheme, or a vision-specific tokenization method, it would be unclear whether any observed improvement came from the Transformer's self-attention mechanism or from the domain-specific engineering. The minimal-modification approach provides the cleanest possible test of whether self-attention, unassisted by convolutional structure, is sufficient for visual representation learning.

The practical payoff of this minimalism is demonstrated in the computational efficiency comparisons (Table 2, Figure 5). ViT-H/14 requires 2.5k TPUv3-core-days to pre-train versus 9.9k for BiT-L (ResNet152x4) β€” a ~4Γ— reduction β€” while achieving better accuracy. This efficiency advantage comes directly from the architectural minimalism: because ViT is a standard Transformer, it benefits from years of NLP infrastructure investment, including optimized self-attention kernels, efficient data parallelism strategies, and memory-efficient attention implementations. The vision-specific attention architectures that preceded ViT (stand-alone self-attention, axial attention) could not access this infrastructure because their non-standard attention patterns required custom β€” and often substantially slower β€” implementations.

The paper also makes a subtler methodological contribution through what it doesn't optimize. The authors do not perform an extensive hyperparameter search over Transformer configurations. They adopt BERT's exact layer counts, hidden sizes, and MLP ratios (Table 1). They don't experiment with alternative activation functions, normalization placements, or attention head configurations. This restraint is unusual for a paper introducing a new vision architecture and reinforces the message: the standard recipe works, and the community's effort is better spent on scaling and data than on architectural micro-optimization. Appendix D.2 does explore scaling different Transformer dimensions (depth, width, MLP size, patch size), finding that scaling all dimensions proportionally β€” rather than over-optimizing any single one β€” yields the most robust improvements, further supporting the "don't over-engineer" philosophy.

This innovation has had a lasting influence on research methodology in the field. The subsequent success of models like CLIP (Radford et al., 2021), which applied a standard Transformer to text and a standard ViT to images with minimal cross-modal engineering, and MAE (He et al., 2022), which applied standard masked autoencoding to ViT with minimal vision-specific design, can be seen as direct descendants of ViT's minimal-modification philosophy.

Innovation 3: The Global Self-Attention Baseline and the Attention Distance Analysis

ViT establishes a new baseline capability for vision architectures: global information integration from the very first layer, without any of the architectural scaffolding (local windows, hierarchical structure, specialized attention patterns) that prior attention-for-vision models relied on. The paper's analysis of how the model uses this capability β€” through the attention distance metric β€” provides a novel diagnostic for understanding what self-attention learns in the absence of convolutional structure.

Prior attention-based vision models had constrained self-attention to be local, axial, or sparse (Parmar et al., 2018; Ho et al., 2019; Child et al., 2019; Ramachandran et al., 2019) β€” in part because naive global attention seemed computationally infeasible for pixel-level representations, and in part because the field assumed that local processing was necessary for vision. The ViT paper's patch-based tokenization makes global attention tractable (196 tokens for 16Γ—16 patches at 224Γ—224 resolution, versus 50,176 for pixel-level attention), enabling the first controlled study of what a pure global-attention vision model actually learns.

The attention distance analysis (Section 4.5, Figure 7 right, and Appendix D.7, Figure 11) reveals a nuanced picture that challenges simple narratives. Rather than attending globally at all layers β€” which might be expected from an architecture with no locality constraints β€” ViT shows heterogeneous attention behavior across heads and layers. Some attention heads in the lowest layers attend to most of the image, demonstrating that the model does indeed use its global integration capability early. Other heads in the same low layers have consistently small attention distances, meaning they attend primarily to nearby patches β€” a behavior that resembles convolutional local processing but is learned from data rather than hard-coded. As depth increases, attention distance increases for all heads, with most heads attending broadly across the image in the second half of the network.

This finding is significant because it demonstrates emergent locality: the model learns to process local features in early layers not because it is forced to, but because local processing is genuinely useful for vision. The fact that some heads remain global even in early layers suggests that the model also learns when to override local processing in favor of long-range integration β€” for example, to relate the beak and tail feathers of a bird that are spatially separated in the image (as shown in the attention map visualizations in Figure 6 and Appendix Figure 14). This adaptive balance between local and global processing β€” learned entirely from data β€” may explain why pure ViT catches up to CNNs at scale: it can use local processing when that's optimal and global processing when that's optimal, without being constrained to either.

The comparison with hybrid models (Figure 11, bottom) provides further evidence. Hybrids, which have a CNN front-end that performs explicit local feature extraction, show less localized attention in their Transformer layers β€” the CNN has already done the local processing, so the Transformer's attention heads are more uniformly global. This suggests that the localized attention in pure ViT is indeed serving a similar function to the CNN's early convolutional layers: extracting local features before integrating them globally.

Conceptually, this innovation establishes that global self-attention is a superset of local processing, not an alternative to it. A model with global attention can learn to attend locally when that's beneficial, but a model constrained to local attention cannot learn to integrate globally when that's needed. This framing β€” that removing constraints expands the space of learnable functions without removing any functions that the constrained architecture could express β€” provides a theoretical justification for the paper's empirical finding that scale favors the less-constrained architecture.

Innovation 4: The Negative Result That Position Embedding Design Doesn't Matter (and Why That Matters)

The paper includes a crisp negative result that is easy to overlook but has significant implications: the specific design of position embeddings β€” 1D learned, 2D learned, relative, or shared across layers β€” makes essentially no difference to ViT's performance, with all variants performing similarly (Appendix D.4, Table 8). The only thing that matters is that some position information is provided; removing position embeddings entirely causes a substantial accuracy drop.

This result is innovative not because it proposes a new position encoding method, but because it establishes that at the patch level, the 2D spatial structure of images is simple enough that any reasonable encoding suffices. The authors hypothesize:

"since our Transformer encoder operates on patch-level inputs, as opposed to pixel-level, the differences in how to encode spatial information is less important. More precisely, in patch-level inputs, the spatial dimensions are much smaller than the original pixel-level inputs, e.g., 14 Γ— 14 as opposed to 224 Γ— 224, and learning to represent the spatial relations in this resolution is equally easy for these different positional encoding strategies."

This is a subtle but important insight about the relationship between representation granularity and architectural complexity. At the pixel level (as in iGPT, Chen et al., 2020a), the spatial structure is complex β€” 224Γ—224 = 50,176 positions with fine-grained relationships β€” and position encoding design might matter substantially. At the patch level with 14Γ—14 = 196 positions, the problem is sufficiently low-dimensional that even a simple learned 1D embedding can recover the 2D structure, as demonstrated by the position embedding similarity visualization in Figure 7 (center), which shows that the 1D embeddings naturally learn a 2D distance metric with clear row-column structure.

The practical implication is methodological: future work on vision Transformers need not invest effort in sophisticated position encoding schemes. The standard learned 1D embeddings are sufficient, and the community can focus attention on other design choices (patch size, architecture scaling, training recipes) that have larger impacts. This finding has held up remarkably well in subsequent work β€” most ViT variants (DeiT, Swin, MAE, DINO) use simple learned or sinusoidal position embeddings without substantial modification.

Innovation 5: The Training-Inference Cost Asymmetry as a Deployment Advantage

While not the paper's headline contribution, the empirical computational cost analysis (Appendix D.5, Figure 12) reveals a practically important property of ViT relative to CNNs: ViT models are substantially more memory-efficient at inference time, with the largest models fitting much larger batch sizes on a single accelerator core than comparable ResNets.

Figure 12 (right) shows that ViT-H/14 can fit a per-core batch size roughly an order of magnitude larger than ResNet152x4 at the same input resolution. At 224Γ—224, ViT-H/14 fits ~500 images per core versus ~50 for ResNet152x4. This memory efficiency advantage stems from the Transformer's uniform computation pattern: self-attention and MLP operations have consistent memory access patterns that map well to TPU/GPU hardware, whereas CNNs' hierarchical structure (with different spatial resolutions at different layers and varying filter sizes) creates memory access patterns that are harder to optimize.

This finding is not merely an implementation detail β€” it has economic implications for deployment. A model that can process larger batches per accelerator requires fewer accelerators to serve a given throughput, directly reducing inference costs. Combined with the pre-training efficiency advantage (2.5k vs. 9.9k TPUv3-core-days for ViT-H/14 vs. BiT-L), this creates a compelling total-cost-of-ownership argument for ViT that extends beyond accuracy comparisons: ViT is cheaper to train and cheaper to deploy than comparable CNNs.

The innovation here is conceptual: the paper reframes the architecture comparison from a pure accuracy contest to a multi-objective optimization that includes training cost, inference throughput, and memory efficiency. This holistic framing is relatively uncommon in architecture papers (which typically focus on accuracy at a fixed parameter count or FLOP budget) and has become increasingly important as models move from research benchmarks to production deployment.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary pre-training datasets are ILSVRC-2012 ImageNet (1k classes, 1.3M images), ImageNet-21k (21k classes, 14M images), and JFT-300M (18k classes, 303M high-resolution images). Downstream evaluation uses ImageNet (original validation labels and cleaned ReaL labels), CIFAR-10/100, Oxford-IIIT Pets, Oxford Flowers-102, and the 19-task VTAB classification suite. All pre-training datasets are de-duplicated with respect to downstream test sets following Kolesnikov et al. (2020).

  • Base model(s). The paper evaluates three families: pure Vision Transformers (ViT-Base, ViT-Large, ViT-Huge, detailed in Table 1), modified ResNets denoted "BiT" (ResNet with Group Normalization replacing Batch Normalization and standardized convolutions, following Kolesnikov et al., 2020), and hybrid models that feed CNN feature maps into a ViT encoder. ViT configurations are directly adopted from BERT (Devlin et al., 2019): ViT-Base is BERT-Base (12 layers, D=768, 86M parameters), ViT-Large is BERT-Large (24 layers, D=1024, 307M parameters), and ViT-Huge is a newly scaled variant (32 layers, D=1280, 632M parameters). The naming convention appends patch size: ViT-L/16 means the Large variant with 16Γ—16 patches.

  • Metrics. The primary metrics are top-1 accuracy (%) on downstream datasets after fine-tuning, and few-shot linear accuracy obtained by solving a regularized least-squares regression that maps frozen representations of a subset of training images to {βˆ’1, 1}^K target vectors (producing a closed-form solution). Pre-training computational cost is measured in TPUv3-core-days (number of TPU v3 cores multiplied by training time in days) and exaFLOPs. Attention distance (analogous to CNN receptive field size) is computed as the average spatial distance between query and attended patches, weighted by attention weights.

  • Baselines. The primary CNN baseline is Big Transfer (BiT) (Kolesnikov et al., 2020), which pre-trains large ResNets with Group Normalization and standardized convolutions on the same datasets. The strongest BiT variant is ResNet152x4 (BiT-L). Additional state-of-the-art comparisons include Noisy Student (Xie et al., 2020), which trains EfficientNet-L2 with semi-supervised learning on ImageNet and JFT-300M. For VTAB, baselines include VIVI (ResNet co-trained on ImageNet and YouTube; Tschannen et al., 2020) and S4L (supervised plus semi-supervised learning on ImageNet; Zhai et al., 2019a). For the self-supervised experiment, the baseline is ViT trained from scratch on ImageNet.

  • Generation budget / compute accounting. There is no generation budget in this paper since it addresses supervised classification, not generative tasks. Compute is measured in two ways: (1) TPUv3-core-days for pre-training (all models trained on TPUv3 hardware, with the cost computed as number of cores Γ— training days), and (2) exaFLOPs for the controlled scaling study in Figure 5 (details in Appendix D.5). For inference efficiency, peak images per second per core and maximum per-core batch size are measured on TPUv3 hardware across various input resolutions (64, 128, 224, 384, 512 pixels). The paper uses 7 or 14 epochs of JFT pre-training for the main scaling study, with 14-epoch runs costing twice the compute of 7-epoch runs.

  • Cross-validation / statistical protocol. Fine-tuning results on ImageNet, ImageNet-ReaL, CIFAR-10/100, Pets, and Flowers are reported as mean and standard deviation over three fine-tuning runs (Table 2). For VTAB, results are reported with mean and standard deviation across the 19 tasks. Hyperparameter selection for fine-tuning uses small sub-splits of the training set as development sets (10% for Pets and Flowers, 2% for CIFAR, 1% for ImageNet), with final results trained on the full training set. For VTAB, a single hyperparameter setting (learning rate 0.01, 2500 steps) is selected via a small sweep over two learning rates and two schedules evaluated on 200-example validation sets. The paper does not use cross-validation for pre-training strategy selection; instead, it reports few-shot linear accuracy on-the-fly as a proxy evaluation during pre-training, and uses early stopping on validation accuracy for the data requirement experiments (Figure 4). Statistical significance testing between architectures is not formally reported.

Main Quantitative Results

Comparison to State of the Art (Table 2)

The headline result appears in Table 2. ViT-H/14 pre-trained on JFT-300M achieves 88.55% (Β±0.04) on ImageNet, 90.72% (Β±0.05) on ImageNet-ReaL, 99.50% on CIFAR-10, 94.55% on CIFAR-100, 97.56% on Oxford-IIIT Pets, 99.68% on Oxford Flowers-102, and 77.63% (Β±0.23) across VTAB's 19 tasks. This outperforms BiT-L (ResNet152x4, also pre-trained on JFT-300M) on all benchmarks: BiT-L achieves 87.54% on ImageNet, 90.54% on ImageNet-ReaL, 99.37% on CIFAR-10, 93.51% on CIFAR-100, 96.62% on Pets, 99.63% on Flowers, and 76.29% on VTAB. ViT-H/14 also matches or exceeds Noisy Student (EfficientNet-L2), which reports 88.4%/88.5% on ImageNet and 90.55% on ImageNet-ReaL.

The computational cost comparison is striking: ViT-H/14 requires 2.5k TPUv3-core-days versus 9.9k for BiT-L β€” approximately a 4Γ— reduction β€” and 12.3k for Noisy Student. The smaller ViT-L/16 achieves 87.76% on ImageNet with only 0.68k TPUv3-core-days, outperforming BiT-L while using ~14.5Γ— less pre-training compute. ViT-L/16 pre-trained on the public ImageNet-21k (0.23k TPUv3-core-days) achieves 85.30% on ImageNet β€” competitive but below the JFT-300M models β€” while being trainable on a standard cloud TPUv3 with 8 cores in approximately 30 days.

The VTAB breakdown in Figure 2 shows ViT-H/14 outperforming BiT-L on Natural tasks (7 tasks, including CIFAR, Pets, Flowers) and Structured tasks (8 tasks requiring geometric understanding, such as localization and counting), while performance on Specialized tasks (4 tasks: medical and satellite imagery) is similar between the top models. ViT-H/14 achieves 77.63% overall versus 76.29% for BiT-L and 72.72% for ViT-L/16 pre-trained on ImageNet-21k.

Pre-Training Data Requirements (Figures 3 and 4, Table 5)

Figure 3 presents the core scaling experiment: ViT models pre-trained on ImageNet (1.3M images), ImageNet-21k (14M), and JFT-300M (303M), then fine-tuned on ImageNet. When pre-trained on ImageNet, ViT-Large models underperform ViT-Base models despite their larger capacity β€” ViT-L/16 achieves 76.53% versus 77.91% for ViT-B/16 (Table 5). The BiT ResNet family (shaded region in Figure 3) outperforms all ViT variants at this data scale. When pre-trained on ImageNet-21k, the gap between ViT-Base and ViT-Large narrows: ViT-B/16 achieves 83.97%, ViT-L/16 achieves 85.15%. Only with JFT-300M pre-training does ViT-Large clearly dominate: ViT-B/16 reaches 84.15%, ViT-L/16 reaches 87.12%, and ViT-H/14 reaches 88.04% (without the Polyak averaging and 512-pixel resolution used in Table 2).

Figure 4 reinforces this with a controlled experiment training on random JFT subsets of 9M, 30M, 90M, and 303M images (no additional regularization, same hyperparameters, early stopping used). ViT-B/32, which is slightly faster than ResNet50x1, performs worse on the 9M subset but overtakes ResNet50x1 at 90M+ images. Similarly, ViT-L/16 overtakes ResNet152x2 at 90M+ images. The paper observes that ResNets "plateau sooner than ViT, which performs better with larger pre-training." ViT-b/32 (ViT-B with all hidden dimensions halved) consistently performs below ViT-B/32, confirming that model capacity matters alongside data scale.

This crossover pattern β€” CNNs better at small data, ViT better at large data β€” is the paper's central empirical finding. As the paper states: "This result reinforces the intuition that the convolutional inductive bias is useful for smaller datasets, but for larger ones, learning the relevant patterns directly from data is sufficient, even beneficial."

Controlled Scaling Study (Figure 5, Table 6)

Figure 5 shows transfer performance (average over 5 datasets: ImageNet, ImageNet-ReaL, CIFAR-10, CIFAR-100, Pets, Flowers) versus total pre-training compute in exaFLOPs for 7 ResNets, 6 Vision Transformers, and 5 hybrids. The key findings:

First, Vision Transformers dominate ResNets on the performance/compute trade-off. ViT uses approximately 2–4Γ— less compute to attain the same average accuracy. For example, ViT-L/16 at 7 epochs (783 exaFLOPs) achieves roughly equivalent transfer performance to ResNet152x2 at 14 epochs (1126 exaFLOPs).

Second, hybrids slightly outperform ViT at small computational budgets, but the difference vanishes for larger models. At ~100 exaFLOPs, R50x1+ViT-B/32 outperforms ViT-B/32 by a visible margin. At ~800–1700 exaFLOPs, R50x1+ViT-L/16 and ViT-L/16 converge to similar performance. This is described as "somewhat surprising, since one might expect convolutional local feature processing to assist ViT at any size."

Third, Vision Transformers appear not to saturate within the range tried. The performance curve for ViT (blue line in Figure 5) shows continued improvement up to the largest model (ViT-H/14 at 4262 exaFLOPs), whereas the ResNet curve shows signs of diminishing returns. The paper explicitly notes this "motivates future scaling efforts."

Detailed numbers in Table 6 show that ViT-H/14 at 14 epochs achieves the best performance on most datasets (88.08% ImageNet, 90.36% ImageNet-ReaL, 99.50% CIFAR-10, 94.71% CIFAR-100, 97.11% Pets, 99.71% Flowers) while using 4262 exaFLOPs, compared to ResNet200x3 at 14 epochs (87.22% ImageNet, 90.15% ReaL, 99.34% CIFAR-10, 93.53% CIFAR-100, 96.32% Pets, 99.04% Flowers) at 3306 exaFLOPs.

Inspection of Learned Representations (Section 4.5, Figures 6 and 7)

Patch embedding filters. Figure 7 (left) visualizes the top principal components of the learned linear embedding matrix E for ViT-L/32. The components "resemble plausible basis functions for a low-dimensional representation of the fine structure within each patch" β€” showing edges, color gradients, and texture-like patterns that look qualitatively similar to the filters learned in the first layer of a CNN, despite being trained through a different architecture and objective.

Position embedding structure. Figure 7 (center) shows the cosine similarity between position embeddings for ViT-L/32. The embeddings naturally encode 2D distance: nearby patches have more similar position embeddings, and a clear row-column structure emerges. Patches in the same row or column exhibit similar embeddings, and for larger grids, a sinusoidal structure is sometimes apparent (Appendix D). The paper notes: "That the position embeddings learn to represent 2D image topology explains why hand-crafted 2D-aware embedding variants do not yield improvements" β€” the model learns the 2D structure from 1D embeddings, making explicit 2D parameterization redundant. Appendix D.4 shows the position embedding similarity patterns depend on training hyperparameters (Figure 10), with different learning rates and weight decay producing different similarity structures, but all variants achieve comparable accuracy (Table 8).

Attention distance analysis. Figure 7 (right) and Figure 11 (Appendix D.7) plot the mean attention distance (in pixels, averaged across 128 example images) for each of the 16 attention heads at each of the 24 layers in ViT-L/16. Each dot represents one head at one layer. The key observations:

  • In the lowest layers, attention distance is highly variable across heads: some heads attend to most of the image (mean distance >100 pixels at layer 0), while others attend primarily to nearby patches (mean distance <20 pixels). This demonstrates that the model learns a mixture of local and global processing from data, with no hard architectural constraint forcing either behavior.
  • As depth increases, attention distance increases for all heads. In the second half of the network (layers 12–23), most heads attend broadly across tokens, with mean distances approaching 80–120 pixels (nearly the full image width of 224 pixels).
  • In the hybrid model (R50x1+ViT-L/16, Figure 11 bottom), the localized attention in early layers is "less pronounced," suggesting that "it may serve a similar function as early convolutional layers in CNNs." The CNN front-end has already performed local feature extraction, reducing the need for the Transformer to learn localized attention.

Attention map visualization. Figure 6 and Appendix Figure 14 show representative attention maps from the classification token to the input space, computed using Attention Rollout (Abnar & Zuidema, 2020) β€” recursively multiplying attention weight matrices across all layers to account for attention mixing. The model attends to image regions that are "semantically relevant for classification," such as the face and body of an animal, the petals of a flower, or the distinguishing features of an object.

Self-Supervision Results (Section 4.6)

The masked patch prediction experiment yields ViT-B/16 achieving 79.9% accuracy on ImageNet after self-supervised pre-training on JFT, compared to 77.9% when trained from scratch on ImageNet β€” a 2% improvement. This remains 4% behind supervised pre-training on JFT (84.15% from Table 5). The paper notes that 100k pre-training steps achieve similar gains to 1M steps, and that pre-training on ImageNet rather than JFT yields comparable improvements. Three prediction targets were tested: predicting mean 3-bit color (512 classes), predicting a 4Γ—4 downsampled version of the 16Γ—16 patch with 3-bit colors (16 predictions of 512 classes each), and L2 regression on the full patch (256 predictions on 3 RGB channels). All worked similarly, though L2 was slightly worse. The 50% corruption rate (with 80% mask, 10% random, 10% unchanged) outperformed BERT's 15% rate.

Ablation Studies and Robustness Checks

Optimizer choice: Adam vs. SGD for ResNet pre-training (Appendix D.1, Table 7): ResNets pre-trained on JFT with Adam achieve better fine-tuning transfer than those pre-trained with SGD. On average across 5 datasets, ResNet50x1 achieves 89.33% with Adam versus 88.79% with SGD; ResNet152x2 achieves 94.01% with Adam versus 93.72% with SGD. Adam outperforms SGD on 7 of 10 individual dataset-model combinations. This justifies the unconventional choice of Adam for CNN pre-training in the paper's setup. The absolute numbers are lower than Kolesnikov et al. (2020) because models are pre-trained for only 7 epochs, not 30.

Transformer shape scaling (Appendix D.2, Figure 8): Scaling different dimensions from a base configuration (8 layers, D=1024, MLP size=2048, patch size=32) reveals that scaling depth produces the largest improvements (visible up to 64 layers, with diminishing returns after 16), while scaling width produces the smallest changes. Decreasing patch size (thus increasing sequence length) shows "surprisingly robust improvements without introducing parameters." The paper concludes that compute is a better predictor of performance than parameter count, and that scaling all dimensions proportionally is the most robust strategy.

Classification token vs. global average pooling (Appendix D.3, Figure 9): An initial comparison showed the classification token substantially outperforming global average pooling (GAP). However, this was "fully explained by the requirement for a different learning-rate." When GAP is evaluated with learning rate 3Γ—10⁻⁴ instead of 8Γ—10⁻⁴ (the optimal rate for the classification token), both methods perform similarly, reaching comparable linear 5-shot accuracy on ImageNet. This demonstrates that the classification token is a convenience inherited from BERT, not an architectural necessity.

Position embedding variants (Appendix D.4, Table 8): Six position embedding strategies were compared on ViT-B/16 using ImageNet 5-shot linear accuracy: no position embedding (0.61382), 1D learned position embedding added to input (0.64206), 1D added at every layer (0.63964), 1D added at every layer with shared weights (0.64292), 2D learned position embedding added to input (0.64001), 2D added at every layer (0.64046), 2D added at every layer shared (0.64022), and relative position embeddings (0.64032). The maximum difference between any position embedding variant is approximately 0.003, while removing position embeddings entirely causes a ~0.029 drop. The conclusion: providing some position information is critical, but the specific form makes essentially no difference.

Axial attention variants (Appendix D.6, Figure 13): Axial-ViT-B/32 and Axial-ViT-B/16 (which replace each global self-attention block with sequential row-attention + MLP + column-attention + MLP blocks) achieve better accuracy-compute tradeoffs than standard ViT in terms of FLOPs: at comparable exaFLOPs, Axial-ViT-B/16 reaches higher linear 5-shot accuracy on ImageNet. However, the "naive implementation is extremely slow on TPUs" (Figure 13, right), making it impractical for large-scale training. AxialResNet also shows a reasonable accuracy/compute tradeoff in FLOPs but suffers from the same implementation inefficiency.

Pre-training dataset size robustness (Figures 3, 4): The crossover pattern where ViT outperforms ResNets at large scale but underperforms at small scale is replicated across multiple experimental setups: fine-tuning on ImageNet (Figure 3), linear few-shot evaluation on ImageNet (Figure 4), and fine-tuning on other datasets (Table 5). Table 5 shows that on CIFAR-10, CIFAR-100, Pets, and Flowers, ViT-L/16 surpasses ViT-B/16 only when pre-trained on JFT-300M; with ImageNet-21k pre-training, the Base and Large variants perform similarly. This consistency across different evaluation metrics and datasets strengthens the central scaling claim.

Fine-tuning resolution and Polyak averaging (Table 2 vs. Table 5): The best ImageNet results in Table 2 (88.55% for ViT-H/14, 87.76% for ViT-L/16) use two additional techniques beyond standard fine-tuning: higher resolution (512 for ViT-L/16, 518 for ViT-H/14) and Polyak averaging with factor 0.9999. Without these techniques (Table 5), ViT-H/14 achieves 88.04% and ViT-L/16 achieves 87.12% on ImageNet with JFT-300M pre-training. The additional techniques contribute approximately 0.5–0.6 percentage points.

Effect of training epochs on scaling (Table 6): Doubling pre-training epochs from 7 to 14 for select models (ViT-L/16, ViT-H/14, ResNet152x2, ResNet200x3, R50x1+ViT-L/16) consistently improves transfer accuracy. For ViT-L/16, 14 epochs yields 87.12% ImageNet vs. 86.30% at 7 epochs, while costing 1567 vs. 783 exaFLOPs. The relative improvement is larger for ViT than for ResNets: ResNet152x2 improves from 84.97% to 85.56% (0.59 points) while ViT-L/16 improves from 86.30% to 87.12% (0.82 points), suggesting ViT may benefit more from extended training.

ObjectNet evaluation (Appendix D.9): ViT-H/14 achieves 82.1% top-5 accuracy and 61.7% top-1 accuracy on ObjectNet, providing an out-of-distribution robustness evaluation following the setup in Kolesnikov et al. (2020). The paper does not report CNN baselines on this benchmark, making it a demonstration rather than a controlled comparison.

Critical Assessment

Claim 1: "Large scale training trumps inductive bias"

This is the paper's central claim β€” that at sufficient data scale, the lack of convolutional inductive biases ceases to be a disadvantage and may become an advantage. The evidence strongly supports this claim with two important caveats.

The positive evidence is multi-layered and internally consistent: Figure 3 shows the crossover from CNNs outperforming ViT on ImageNet (1.3M) to ViT overtaking on JFT (303M); Figure 4 replicates the crossover with controlled JFT subsets rather than different datasets; Figure 5 extends the finding to a compute-controlled scaling study; and Table 5 confirms the pattern across multiple downstream datasets. The consistency across evaluation protocols (fine-tuning, linear few-shot, different datasets) reduces the likelihood that the result is an artifact of a particular experimental setup.

However, two caveats limit the strength of this claim:

Caveat 1: The "trump" only applies to the specific ViT and CNN configurations compared. The paper compares ViT to BiT ResNets β€” a strong but specific CNN baseline using Group Normalization and standardized convolutions. The paper does not compare to EfficientNets (except Noisy Student as a state-of-the-art reference in Table 2, not in the controlled scaling study), RegNets, ConvNeXts (which postdate this paper), or other CNN families. The claim that "large scale training trumps inductive bias" is supported for ResNet-like CNNs but may not generalize to all possible convolutional architectures. A CNN with different inductive biases (different kernel sizes, different normalization, different connectivity patterns) might have different scaling properties. The paper's own hybrid results (Figure 5) show that CNNs plus Transformers can outperform pure Transformers at intermediate scales, suggesting that the optimal architecture is not simply "Transformer" but depends on the specific scale.

Caveat 2: The scale at which ViT overtakes CNNs is enormous β€” 300M images is far beyond what most practitioners can access. ViT-L/16 with ImageNet-21k (14M images) pre-training achieves 85.15% on ImageNet (Table 5), while the paper does not report a comparable ResNet trained on the same data in the scaling study. It is possible that at the 14M-image scale, the architectures are roughly tied (the "crossover" point), and the ViT advantage only clearly emerges at 300M images. For practitioners with access to ImageNet-21k but not JFT-300M, the architectural choice may be less clear-cut than the paper's headline suggests. The ViT-L/16 ImageNet-21k result (85.30% in Table 2) is strong but not dramatically beyond what large CNNs achieve with similar data.

Claim 2: ViT attains excellent results compared to state-of-the-art CNNs while requiring substantially fewer computational resources to train

The evidence supports a qualified version of this claim: ViT achieves better accuracy with less pre-training compute on the specific metrics reported, but the comparison is not fully controlled.

Table 2 clearly demonstrates that ViT-H/14 (2.5k TPUv3-core-days) outperforms BiT-L (9.9k TPUv3-core-days) and matches Noisy Student (12.3k TPUv3-core-days) on ImageNet and related benchmarks. The compute advantage is real and substantial β€” approximately 4Γ— less than BiT-L.

However, the comparison is weakened by the fact that ViT and the baselines use potentially different training recipes, optimizers, and regularization strategies that are not held constant. The paper acknowledges this explicitly:

"we note that pre-training efficiency may be affected not only by the architecture choice, but also other parameters, such as training schedule, optimizer, weight decay, etc."

The controlled scaling study in Figure 5 partially addresses this by comparing models under consistent training conditions (all pre-trained for 7 or 14 epochs on JFT with Adam), but Table 2 compares ViT against the published numbers for BiT and Noisy Student, which may use different training recipes. ViT uses Adam; BiT's published numbers use Adam as well, but the training schedules differ.

A missing comparison is: what accuracy would BiT-L achieve if trained with the same TPUv3-core-day budget as ViT-H/14 (2.5k)? If BiT-L could be trained for 2.5k TPUv3-core-days instead of 9.9k, would its performance approach ViT-H/14? The compute-controlled scaling study (Figure 5) provides a partial answer: at 2000–4000 exaFLOPs, ViT outperforms ResNets by a clear margin, but this uses 7-epoch or 14-epoch training rather than matching compute budgets directly. A compute-matched comparison where both architectures are given exactly the same FLOPs budget would be more rigorous.

Additionally, ViT-H/14 and ViT-L/16 benefit from fine-tuning at resolution 512/518 with Polyak averaging (Table 2), while the BiT-L baseline results are from Kolesnikov et al. (2020) β€” it is unclear whether BiT-L received equivalent resolution fine-tuning or model averaging. If these fine-tuning techniques are architecture-agnostic and would improve BiT-L as well, the comparison slightly favors ViT.

Claim 3: Vision Transformers appear not to saturate within the range tried, motivating future scaling efforts

The evidence in Figure 5 is suggestive but limited. The ViT performance curve in Figure 5 does continue to improve from the smallest to the largest model, without visible inflection toward a plateau. This contrasts with ResNets, which show a visibly shallower slope at the high-compute end. However, the ViT curve only contains 6 data points (ViT-B/32, B/16, L/32, L/16 at 7 epochs, plus L/16 and H/14 at 14 epochs), and the highest-compute point (ViT-H/14 at 4262 exaFLOPs) is only ~2.7Γ— more compute than the next point (ViT-L/16 at 14 epochs, 1567 exaFLOPs). With only 6 models spanning roughly two orders of magnitude in compute, detecting saturation would be difficult. The claim is plausible and consistent with NLP scaling trends, but the experimental evidence is more of an observation than a rigorous demonstration of non-saturating scaling.

Claim 4: Hybrids slightly outperform ViT at small computational budgets, but the difference vanishes for larger models

This is the strongest-controlled finding in the paper, with a significant limitation. Figure 5 clearly shows the hybrid advantage diminishing with scale: at ~100 exaFLOPs, the hybrid advantage is visually apparent; at ~800–1700 exaFLOPs, the curves converge. However, the paper only experiments with one specific hybrid configuration: ResNet50 as the CNN front-end. Different CNN depths, different feature extraction stages, or different ways of combining CNN and Transformer features might show different scaling behavior. The paper's conclusion that "convolutional local feature processing" is the relevant variable is confounded by the specific architectural choices: the hybrid models have more total layers (ResNet layers + Transformer layers) than the pure ViT models, and it is unclear whether the hybrid advantage at small scales comes from the CNN's local receptive fields specifically, or simply from having more total parameters and layers.

Moreover, the paper does not explore whether the converse hybrid β€” Transformer followed by CNN β€” would show similar scaling behavior, which would help isolate whether the benefit is specifically about early local processing or about architectural diversity in general.

Weaknesses in experimental design

Limited pre-training epoch range: Most models are trained for only 7 epochs on JFT-300M (Table 3). Given that JFT has 303M images, 7 epochs represents approximately 2.1B image presentations β€” a substantial amount of compute, but still only a small fraction of a full pass over the dataset. It is unclear whether the relative performance of ViT versus CNNs would change at longer training horizons (30+ epochs, as in Kolesnikov et al., 2020). The paper partially addresses this with 14-epoch runs for select models, but training all models for multiple epoch budgets would paint a more complete picture.

No normalization of the position embedding interpolation: The paper states that 2D interpolation of position embeddings is used for fine-tuning at higher resolution, but doesn't describe the interpolation method (bilinear, nearest-neighbor) or report ablations comparing interpolation methods. This is a minor point but matters for reproducibility.

Single hardware platform (TPUv3): All computational cost measurements are on TPUv3 hardware. The relative efficiency of ViT versus CNNs may differ on GPUs, where convolution operations are highly optimized (cuDNN) and attention implementations may have different performance characteristics. The paper's efficiency claims are hardware-dependent.

Limited few-shot evaluation: The few-shot linear evaluation (Figures 4, 8, 9) provides fast proxy metrics but may not fully capture the representation quality that emerges with full fine-tuning. While the paper focuses on fine-tuning results for its main claims, the scaling experiments in Appendix D rely heavily on few-shot metrics, which could miss nuanced differences between architectures.

VTAB evaluation at fixed resolution: The paper uses 384Γ—384 resolution for all VTAB tasks, whereas prior work (Kolesnikov et al., 2020) used task-specific resolutions. Using a fixed high resolution favors ViT, which benefits from higher resolution (more patches = more attention context), potentially making the comparison to BiT on VTAB slightly generous to ViT.

6. Limitations and Trade-offs

The Crossover Scale Requirement: ViT Only Wins With Enormous Pre-Training Datasets

The assumption or constraint. The paper's central claim β€” that large-scale training trumps inductive bias β€” is fundamentally conditional: ViT surpasses comparable CNNs only when pre-trained on datasets substantially larger than ImageNet. The paper demonstrates this crossover explicitly in Figure 3 and Figure 4, and states the condition clearly in the introduction:

"When trained on mid-sized datasets such as ImageNet without strong regularization, these models yield modest accuracies of a few percentage points below ResNets of comparable size."

The crossover point appears to fall somewhere between 14M and 300M images. At ImageNet scale (1.3M images), ViT-L/16 significantly underperforms ViT-B/16 (76.53% vs. 77.91%) and is beaten by BiT ResNets. At ImageNet-21k scale (14M images), the architectures are roughly comparable: ViT-L/16 achieves 85.15% on ImageNet after fine-tuning (Table 5). Only at JFT-300M scale (303M images) does ViT clearly dominate, with ViT-L/16 reaching 87.12%.

The consequence. The practical implication is stark: for the vast majority of practitioners who do not have access to 300M-image proprietary datasets, ViT may not offer an unambiguous advantage over well-tuned CNNs. The paper's headline result β€” ViT outperforms ResNets while requiring ~4Γ— less computation β€” applies specifically to the JFT-300M pre-training regime. A practitioner with access to ImageNet-21k (14M images, public) or a comparable dataset would find ViT-B/16 (83.97%) roughly tied with what a comparable ResNet would achieve, with no clear architectural winner. The ViT advantage is a luxury of organizations that can assemble web-scale image datasets β€” exactly the organizations least in need of the computational efficiency gains ViT offers.

Moreover, the exact crossover scale is not precisely characterized. The paper tests three points (1.3M, 14M, 303M images), leaving a gap of roughly 20Γ— in dataset size between ImageNet-21k and JFT-300M. Where exactly does ViT overtake ResNet? 30M? 100M? 200M? Without knowing this threshold, practitioners cannot determine whether their specific data budget is sufficient to justify choosing ViT over a CNN.

What evidence exists in the paper. Figure 3 systematically varies pre-training dataset across three scales, showing the crossover. Figure 4 provides finer granularity by training on random JFT subsets of 9M, 30M, 90M, and 303M images, revealing that ViT-B/32 overtakes ResNet50x1 at around 90M images and ViT-L/16 overtakes ResNet152x2 in a similar range. However, Figure 4 uses linear few-shot evaluation rather than full fine-tuning, which may not perfectly reflect transfer performance. Table 5 provides full fine-tuning results but only for the three main datasets (ImageNet, ImageNet-21k, JFT-300M), leaving the exact crossover point between 14M and 303M unspecified.

Mitigation status. Not addressed. The paper reports the crossover as a finding, not a problem to be solved. Section 5 (Conclusion) does not propose methods to reduce ViT's data requirements. The self-supervised pre-training experiment (Section 4.6) achieves only 79.9% on ImageNet β€” a 2% improvement over training from scratch but far from closing the gap with CNNs at the ImageNet scale. The authors explicitly flag this gap as future work, but provide no immediate mitigation. Subsequent work by other groups (e.g., DeiT by Touvron et al., 2021, which uses knowledge distillation to train ViT on ImageNet alone) would later address this limitation, but the paper itself offers no solution.

Single Model Family and Single Task Domain: No Evidence of Generality Beyond Classification

The assumption or constraint. Every experiment in the paper uses one base architecture (the standard Transformer encoder as implemented for BERT) and evaluates exclusively on image classification tasks. The paper acknowledges this scope limitation only obliquely in the conclusion:

"One is to apply ViT to other computer vision tasks, such as detection and segmentation. Our results, coupled with those in Carion et al. (2020), indicate the promise of this approach."

The paper does not test ViT on object detection, instance segmentation, semantic segmentation, depth estimation, keypoint detection, video understanding, or any other core vision task. The Carion et al. (2020) reference (DETR) uses a Transformer for detection but with a CNN backbone for feature extraction β€” not a pure ViT. The paper also does not test alternative Transformer variants (e.g., encoder-decoder, autoregressive decoders, different normalization schemes) to determine whether the findings are specific to the BERT-style encoder or generalize across Transformer architectures.

The consequence. Classification is a particular kind of vision task: it requires aggregating information across the entire image into a single label, which maps naturally to ViT's global self-attention. Dense prediction tasks (detection, segmentation) require spatially localized outputs at multiple scales β€” a fundamentally different requirement that global self-attention may not handle as naturally. The patch-based tokenization, which works well for 224Γ—224 classification, may not provide sufficient spatial resolution for pixel-level tasks, or may require impractically small patches (and thus quadratically more expensive attention). Without experiments on these tasks, a practitioner considering ViT for a detection or segmentation pipeline has no evidence about whether the architecture's advantages transfer. The paper's framing β€” "Vision Transformer for Image Recognition" β€” is accurate, but the title's implication that Transformers work broadly "for image recognition" may overstate what the experiments actually demonstrate.

Furthermore, the paper uses only one specific Transformer implementation: BERT's encoder with GELU activations, pre-norm residual structure, and classification token aggregation. It does not test whether the findings depend on these choices. Would a post-norm Transformer show the same scaling behavior? Would an autoregressive decoder-only Transformer (like GPT) work as well? The paper's claim is specifically about the BERT architecture applied to patches, not about Transformers in general.

What evidence exists in the paper. None beyond classification. All reported numbers β€” ImageNet, CIFAR-10/100, Pets, Flowers, VTAB (19 classification tasks) β€” are classification accuracy. The VTAB suite includes some structured tasks (e.g., localization via Clevr-Count and dSpr-Loc) that require spatial reasoning beyond simple classification, and ViT-H/14 achieves strong results on these (77.63% overall on Structured tasks, Figure 2), but these are still formulated as classification problems with discrete labels, not dense prediction. The paper does not report any detection or segmentation results.

Mitigation status. Not attempted. The conclusion flags detection and segmentation as "another challenge" and points to Carion et al. (2020) as evidence of promise, but does not perform any experiments. A reader evaluating ViT for non-classification tasks has no direct evidence from this paper to guide their decision.

The Cost of Difficulty Estimation Is Unaccounted for in Headline Efficiency Numbers

Note: This limitation uses "difficulty" as an analogy. Unlike the earlier paper analyzed in the reference example, ViT does not explicitly estimate prompt difficulty. However, a structurally analogous issue exists: the paper's computational cost comparisons between ViT and ResNets do not account for the cost of determining which architecture, model size, and pre-training recipe is appropriate for a given dataset and task β€” what we might call the "architecture selection overhead."

The assumption or constraint. The paper reports that ViT-H/14 requires 2.5k TPUv3-core-days to pre-train versus 9.9k for BiT-L, implying that a practitioner can simply choose ViT and realize a ~4Γ— compute savings. This assumes the practitioner already knows that ViT-H/14 is the right configuration for their data scale. In reality, determining the optimal ViT configuration requires running experiments across model sizes (Base, Large, Huge), patch sizes (32, 16, 14), pre-training datasets (ImageNet-21k vs. whatever proprietary data is available), and hyperparameters β€” costs that are not amortized into the headline numbers.

More subtly, the paper's finding that ViT only outperforms CNNs above some data threshold (somewhere between 14M and 300M images) means that a practitioner with an intermediate-sized dataset cannot simply assume ViT will work better. They would need to train both a ViT and a ResNet to determine which architecture is optimal for their specific data scale β€” effectively doubling the pre-training cost to make the decision. The paper's scaling curves (Figures 3, 4, 5) characterize this crossover at specific points, but these points depend on the specific architectures, training recipes, and datasets used in the paper and may not transfer to different settings.

The consequence. The 4Γ— training efficiency advantage is best understood as an upper bound achievable only when (a) the practitioner knows a priori that ViT is the right architecture, (b) has access to a dataset at the scale of JFT-300M, and (c) selects the correct ViT configuration without extensive hyperparameter search. For a practitioner exploring a new domain or dataset size, the total cost of arriving at a well-performing model includes both the architecture selection cost and the final training cost. The paper's headline efficiency numbers include only the latter.

What evidence exists in the paper. The paper does not report the total computational cost of the research project β€” the sum of all training runs that were performed to arrive at the final ViT configurations, including failed experiments, hyperparameter sweeps, and architecture ablations. Tables 3 and 6 report the cost of individual training runs but not the aggregate exploration cost. The scaling study (Figure 5) sweeps 18 model configurations (7 ResNets, 6 ViTs, 5 hybrids), each requiring substantial pre-training compute, and these sweep costs are not factored into any efficiency comparison. This is standard practice in ML research, but it means the reported efficiency numbers represent the cost of reproducing a known-good configuration, not the cost of discovering one.

Mitigation status. Not addressed. The paper does not discuss architecture selection overhead or propose methods to reduce it (e.g., scaling laws that predict ViT performance from smaller-scale experiments, or transferable architecture rankings that would allow practitioners to extrapolate from their data scale). The consistent use of BERT configurations (Table 1) is implicitly a form of mitigation β€” by adopting NLP configurations directly rather than searching over architecture space, the paper reduces the practitioner's search burden β€” but this is not framed or evaluated as a cost-reduction strategy.

Position Embedding Interpolation Is Underspecified and May Not Scale to Arbitrary Resolution Changes

The assumption or constraint. When fine-tuning at higher resolution than pre-training (e.g., 384Γ—384 or 512Γ—512 vs. 224Γ—224 pre-training), the paper states that "we perform 2D interpolation of the pre-trained position embeddings, according to their location in the original image" (Section 3.2). The paper provides no further details: the interpolation method (bilinear, bicubic, nearest-neighbor), whether the interpolated embeddings are frozen or fine-tuned, whether the interpolation is performed once or at each forward pass, and whether the classification token's position embedding is also interpolated or treated specially are all unspecified.

This is not merely a documentation gap β€” it is a potentially important implementation detail. Position embeddings encode spatial relationships that the model has learned during pre-training at a specific resolution. Interpolating them to a new grid makes the implicit assumption that the spatial relationships are smooth and can be represented at intermediate positions. For large resolution changes (e.g., 224β†’1024), the interpolation quality may degrade, and the model's spatial understanding may not transfer.

The consequence. A practitioner attempting to fine-tune ViT at a resolution substantially different from pre-training may encounter unexpected performance degradation that is not attributable to the architecture but to the underspecified position embedding handling. The paper demonstrates that resolution increase helps (Table 2 uses 512/518 for the best results, versus 384 for standard fine-tuning), but tests only a limited range of resolutions. Without ablations on interpolation method, a practitioner cannot know whether poor fine-tuning results at a new resolution indicate a fundamental limitation of the model or a suboptimal interpolation strategy.

More fundamentally, the reliance on interpolation reveals a tension in the paper's design philosophy. The paper's key methodological claim is that ViT requires minimal vision-specific engineering β€” only patch extraction and position embedding interpolation. But position embedding interpolation is a non-trivial vision-specific operation that the paper does not ablate, optimize, or even fully specify. If the interpolation method matters substantially, then ViT's simplicity claim is weaker than it appears; the model is still relying on a vision-specific operation, just one that is under-documented rather than celebrated.

What evidence exists in the paper. The paper reports resolution-dependent results (Table 2 vs. Table 5, showing a ~0.5 percentage point improvement from higher resolution with Polyak averaging) but provides no ablation of interpolation methods. Appendix D.4 extensively ablates position embedding designs (1D vs. 2D vs. relative, different insertion points) but does not ablate the interpolation procedure used during fine-tuning. The position embedding similarity visualization (Figure 7, center) shows the learned structure at the pre-training resolution (14Γ—14 for ViT-L/32), but does not show how this structure behaves after interpolation to higher resolutions.

Mitigation status. Not addressed. The interpolation method is mentioned only in passing and is never formalized, ablated, or discussed as a potential source of variation. The paper treats it as an implementation detail, but it is in fact one of only two vision-specific operations in the entire model. A practitioner reading the paper would need to consult the released code to determine the interpolation method, and would have no guidance on whether alternative methods would improve results.

No Accounting for Latency, Wall-Clock Time, or Real-World Deployment Constraints

The assumption or constraint. All computational cost comparisons in the paper use pre-training FLOPs (exaFLOPs) and TPUv3-core-days as the metric of efficiency. While Appendix D.5 provides inference speed measurements (images per second per core, Figure 12 left), and shows that ViT models have inference speeds "comparable to similar ResNets," the paper does not discuss wall-clock latency, throughput under batch-size constraints, or memory requirements for fine-tuning at high resolution.

This matters because ViT's computational profile differs qualitatively from CNNs in ways that affect deployment. Self-attention is quadratic in sequence length, meaning that the cost of processing a higher-resolution image grows with the square of the resolution (for fixed patch size), while CNN cost grows roughly linearly with pixel count. For a 512Γ—512 image with 16Γ—16 patches, ViT processes 1,024 tokens; for a 1024Γ—1024 image, it processes 4,096 tokens β€” a 4Γ— increase in sequence length producing a ~16Γ— increase in self-attention cost. A practitioner deploying ViT in a latency-sensitive application (e.g., real-time video processing, interactive applications) would need to understand this scaling, but the paper provides only inference measurements at fixed resolutions without discussing the resolution-latency tradeoff curve.

The consequence. The efficiency comparison between ViT and CNNs may invert in deployment scenarios. During pre-training, ViT can use large batch sizes (up to 4096 in the paper's experiments) where self-attention's parallelism advantages dominate. During inference with batch size 1 (common in interactive applications), the constant-factor efficiency of convolution kernels (heavily optimized over decades for GPU hardware) may give CNNs a wall-clock advantage even if ViT has lower theoretical FLOPs. The paper's inference measurements (Figure 12) use peak throughput across "a wide range of batch-sizes" rather than reporting batch-size-1 latency, which is the metric most relevant for interactive deployment.

Additionally, ViT's memory efficiency advantage (Figure 12, right, showing larger maximum per-core batch sizes) comes partly from the architecture and partly from the way the paper measures it. The largest per-core batch size is measured at a given resolution, but the memory footprint during fine-tuning with large models at high resolution (ViT-H/14 at 518Γ—518, producing a long sequence) may be substantially larger than at 224Γ—224, and the paper does not report memory requirements for its best fine-tuning configurations.

What evidence exists in the paper. Appendix D.5 provides inference throughput (Figure 12, left) showing ViT speeds comparable to ResNets at resolutions 64–512. For example, at 224Γ—224, ViT-L/16 processes approximately 500 images per second per core, similar to ResNet152x2. However, these measurements are peak throughput across batch sizes, not latency at batch size 1, and the paper does not report how throughput degrades at very small batch sizes. The paper also notes (Appendix D.5) that "the theoretical bi-quadratic scaling of ViT with image size only barely starts happening for the largest models at the largest resolutions," suggesting that the quadratic self-attention cost is not the dominant factor at the resolutions tested β€” but this observation is resolution-dependent and may not hold at 1024Γ—1024 or higher.

Mitigation status. Partially addressed. The paper provides more deployment-relevant measurements than many architecture papers (inference throughput, memory efficiency, per-core batch size), but does not discuss latency, batch-size-1 performance, or the resolution-latency tradeoff. The conclusion that ViT is "relatively cheap to pre-train" is well-supported, but the implicit extension to "ViT is efficient to deploy" requires assumptions about batch size and resolution that the paper does not validate.

The FLOPs-Matched Comparison Does Not Give Both Architectures Equal Optimization

Note: This paper predates the "FLOPs-matched comparison" terminology used in the Example, but the underlying methodological concern is identical: when comparing two architectures, are they given genuinely equal treatment in terms of training recipes, hyperparameter optimization, and regularization?

The assumption or constraint. The paper's central comparative claim β€” that ViT outperforms ResNets at a given computational budget (Figure 5) β€” assumes that both architectures are equally well-optimized under the training recipe used. Specifically, the paper trains all models with Adam (β₁=0.9, Ξ²β‚‚=0.999, batch size 4096, weight decay 0.1), a choice it acknowledges is unconventional for ResNets:

"ResNets are typically trained with SGD and our use of Adam as optimizer is quite unconventional."

Appendix D.1 justifies this by showing that Adam pre-training leads to better fine-tuning transfer than SGD pre-training for ResNets on JFT β€” but this comparison itself uses the paper's specific training recipe (7 epochs, batch size 4096, weight decay 0.1). It is possible that an SGD-based recipe with different hyperparameters (learning rate schedule, momentum, weight decay, number of epochs) would extract substantially better performance from the ResNet baselines, narrowing or eliminating the ViT advantage.

The paper also uses a consistent pre-training recipe across all models: 7 epochs for most, 14 for select larger variants. But the optimal number of pre-training epochs may differ between architectures. CNNs, with their stronger inductive biases, might converge faster and benefit less from extended pre-training, while ViT, learning spatial structure from scratch, might require more epochs. If so, a fixed-epoch comparison could be systematically biased in either direction depending on the epoch count chosen.

The consequence. The scaling curves in Figure 5, which show ViT dominating ResNets on the performance/compute trade-off, may reflect the interaction of architecture with a specific training recipe rather than a fundamental architectural advantage. If ResNets could achieve better performance with a different optimizer (e.g., SGD with a carefully tuned momentum schedule), a different regularization strategy, or a different number of pre-training epochs, the apparent ViT advantage might shrink or disappear. This is a standard challenge in architecture comparison, but the paper's strong claims ("Vision Transformers dominate ResNets on the performance/compute trade-off") rest on the assumption that the training recipe is equally fair to both architectures.

The hybrid model results add nuance to this concern. Hybrids (CNN front-end + Transformer) slightly outperform pure ViT at small computational budgets but converge to similar performance at large budgets (Figure 5). This could indicate that convolutions provide genuine benefits at small scale, or it could indicate that the training recipe is better suited to architectures with some convolutional structure. Without a more extensive hyperparameter search per architecture, these alternatives cannot be distinguished.

What evidence exists in the paper. Appendix D.1 (Table 7) provides a limited comparison of Adam vs. SGD for two ResNet variants (ResNet50x1 and ResNet152x2), showing Adam outperforms SGD on average. However, this comparison uses the same 7-epoch training schedule for both optimizers, the same batch size, and the same weight decay β€” parameters that interact with optimizer choice. The paper does not sweep SGD-specific hyperparameters (base learning rate beyond what's listed in Table 3 for Adam, momentum values beyond 0.9, learning rate decay schedules) to find the best SGD configuration before comparing to Adam. The claim that "Adam works slightly better than SGD for ResNets in our setting" is accurate, but "in our setting" may not represent the best achievable SGD performance.

Mitigation status. Partially addressed through transparency. The paper explicitly notes the optimizer choice caveat and provides Appendix D.1 as justification. The scaling study (Figure 5) uses a consistent recipe for both architectures, which is the standard approach for controlled comparisons. The paper's acknowledgment that "pre-training efficiency may be affected not only by the architecture choice, but also other parameters, such as training schedule, optimizer, weight decay, etc." (Section 4.2) demonstrates awareness of the limitation. However, no systematic architecture-specific hyperparameter optimization is performed, and the conclusions are stated more strongly than this caveat would warrant. A reader should interpret the ViT-vs-ResNet comparison as valid under the paper's training recipe but not necessarily as a fundamental architectural ranking that would hold under all possible training recipes.

7. Implications and Future Directions

How This Work Changes the Landscape

The Vision Transformer paper causes a paradigm shift in computer vision architecture design by demonstrating that the convolutional inductive bias β€” treated as essential for decades β€” is not a necessary ingredient for visual representation learning. It is a substitute for data, not a prerequisite. This reframes architecture design from "what operations should a vision model use?" to "given my data budget, what constraints can I safely remove?" The magnitude of this shift is evident in what followed: within two years of ViT's publication, Transformers became the default starting point for most vision research, and the field's architectural innovation shifted from designing better convolutions to designing better tokenization strategies (Swin Transformer, 2021), self-supervised objectives (MAE, 2022; DINO, 2021), and scaling recipes.

The paper resolves a latent contradiction that had been building in the literature. Prior work on attention for vision had reached an impasse: stand-alone self-attention models (Ramachandran et al., 2019; Zhao et al., 2020) demonstrated that attention could replace convolutions in principle, but their specialized local attention patterns (axial, criss-cross, local windows) made them impossible to scale efficiently on existing hardware. Simultaneously, the NLP community had demonstrated that standard global self-attention, combined with massive pre-training, produced qualitatively new capabilities. The contradiction was: is attention viable for vision, or isn't it? ViT resolves this by showing that the failure mode was not attention per se, but the combination of local attention patterns (which couldn't leverage NLP infrastructure) and insufficient pre-training scale (which couldn't overcome the missing inductive bias). The paper's findings reconcile the optimistic attention-for-vision results with the pessimistic scaling results: both were correct, but under different conditions. Specialized attention can work at small scale; standard attention can work at large scale; the impasse was an artifact of testing standard attention at the wrong scale.

The paper also redirects research attention from architectural novelty to scaling strategy. Prior to ViT, the dominant mode of contribution in vision architecture research was to propose a new operation, block, or connectivity pattern (ResNet blocks, DenseNet connections, SE modules, depthwise separable convolutions, squeeze-and-excitation). ViT's minimal-modification philosophy β€” deliberately not introducing vision-specific operations β€” made a powerful argument that the most impactful design choice was not the internal architecture but rather the pre-training data scale, the tokenization granularity (patch size), and the training recipe. This shift is visible in subsequent work: DeiT (Touvron et al., 2021) focused on training recipes and distillation rather than new operations; MAE (He et al., 2022) focused on the pre-training objective rather than the encoder architecture; scaling law papers (Zhai et al., 2022) treated ViT as a substrate for studying compute-optimal pre-training. The paper's appendix findings reinforce this shift β€” notably that all position embedding designs work similarly (Table 8) and that GAP works as well as the class token with proper learning rate tuning (Figure 9) β€” establishing that many apparently consequential design choices are in fact negligible, and that effort is better spent on scaling.

The paper makes CNN-centric research directions less attractive in two specific ways. First, the finding that hybrids (CNN + Transformer) lose their advantage at scale (Figure 5, where the hybrid advantage vanishes for larger models) suggests that engineering better CNN front-ends for Transformers is a dead end β€” the architecture itself subsumes whatever the CNN would provide, given sufficient capacity. Second, the finding that ViT appears not to saturate within the range tested (Figure 5, blue curve still climbing at 4262 exaFLOPs) while ResNets show diminishing returns shifts the burden of proof: a researcher proposing a new CNN architecture must now demonstrate not just that it outperforms prior CNNs, but that it outperforms ViT at the same compute budget, and ideally that it scales as favorably. The paper's memory efficiency results (Figure 12, right) further disadvantage CNNs: ViT's larger maximum per-core batch sizes make it the more practical choice for large-scale training regardless of accuracy.

However, the paper also opens entirely new research directions that were previously invisible because the field lacked a viable non-convolutional vision architecture at scale. The finding that ViT's learned position embeddings naturally recover 2D topology (Figure 7, center) raises the question of what other visual structures β€” occlusions, part-whole hierarchies, depth ordering β€” the model learns without being programmed to. The attention distance analysis (Figure 7, right) showing that some heads learn localized attention while others are globally integrated from early layers suggests that the model discovers a processing strategy intermediate between pure local and pure global β€” a strategy that had no name or mechanism in the CNN era but that can now be studied directly through attention maps. The paper's preliminary self-supervised result (79.9% on ImageNet for ViT-B/16 with masked patch prediction, Section 4.6) suggests that the NLP self-supervision playbook may transfer to vision β€” a hypothesis that MAE and subsequent work would confirm spectacularly.

Follow-Up Research This Work Enables

Self-supervised pre-training at scale with ViT as the backbone. The paper's preliminary masked patch prediction experiment (Section 4.6, 79.9% on ImageNet, 4% behind supervised pre-training on JFT) demonstrates that self-supervised ViT is viable but leaves a large gap to close. A direct follow-up would systematically compare BERT-style masked prediction against contrastive objectives (SimCLR, MoCo, BYOL) using ViT as the backbone, pre-trained on JFT-300M. The key measurement would be: at what pre-training dataset size does self-supervised ViT match supervised ViT's transfer performance? The paper's scaling framework (Figures 3 and 4) provides the template β€” train self-supervised ViT-B, ViT-L, and ViT-H on random JFT subsets of increasing size, evaluate linear few-shot and fine-tuning transfer, and locate the crossover where self-supervision catches up. A negative result (self-supervision never catches up, even at 300M images) would suggest that visual representation learning has a fundamental dependence on label signal that NLP's masked language modeling does not, sharpening our understanding of modality-specific pre-training dynamics. A positive result would open the door to training ViT on web-scale unlabeled image data (billions of images), potentially shifting the field's data bottleneck from labeled datasets to unlabeled ones.

ViT for dense prediction: detection and segmentation with minimal modification. The paper explicitly flags detection and segmentation as "another challenge" (Section 5) but performs no experiments. A direct extension would apply ViT as a backbone for Mask R-CNN or DETR-style object detection on COCO, with two architectural variants: (1) a hierarchical ViT that produces multi-scale feature maps (by extracting intermediate Transformer layer outputs and reshaping them to 2D), mimicking the feature pyramid that detection heads expect; (2) a pure patch-based approach where object proposals are predicted directly from patch tokens without reshaping, preserving ViT's 1D sequence abstraction. The key comparison would be against a ResNet backbone with identical detection heads and training schedule, measuring both box AP/mask AP and pre-training + fine-tuning compute. The paper's finding that ViT learns localized attention in early layers (Figure 7, right) predicts that the hierarchical variant should work well, since the early layers already compute local features that approximate what a CNN's early layers would produce. A negative result (ViT substantially underperforming ResNet on detection despite matching on classification) would reveal that global self-attention, while sufficient for image-level aggregation, lacks the spatial specificity needed for pixel-level tasks β€” a fundamental limitation that would redirect research toward hybrid architectures for dense prediction.

Scaling laws for ViT: optimal allocation of compute between model size, patch size, and pre-training data. The paper provides a controlled scaling study (Figure 5) but uses a fixed pre-training recipe (7 or 14 epochs on JFT-300M) and does not systematically vary data size against model size. Following the template of Kaplan et al. (2020) for language models and Hoffmann et al. (2022) for compute-optimal pre-training, a follow-up would train ViT models across a grid of parameter counts (5M to 1B), patch sizes (8, 16, 32), and pre-training dataset sizes (1M to 300M images) and fit power laws predicting transfer accuracy as a function of these variables. The key question: is there a compute-optimal patch size that depends on model capacity and data scale, or is the paper's choice of P=16/14 near-universal? The paper's Appendix D.2 (Figure 8) hints that decreasing patch size yields "surprisingly robust improvements without introducing parameters," suggesting that compute spent on longer sequences may be more efficient than compute spent on wider layers. A scaling law analysis would quantify this tradeoff and produce concrete recommendations: for a 100M-image dataset and a 100-exaFLOP budget, should a practitioner use ViT-B/16 (short sequence, large model) or ViT-B/8 (long sequence, same model)? This is the inference-time analog of the Chinchilla laws and would make ViT's computational efficiency advantage (Table 2) predictable rather than empirical.

Adversarial robustness and out-of-distribution generalization of ViT versus CNNs. The paper reports ObjectNet top-1 accuracy of 61.7% for ViT-H/14 (Appendix D.9) but does not compare to ResNet baselines on this benchmark and provides no analysis of robustness beyond this single number. A systematic follow-up would evaluate ViT and comparable ResNets (matched for ImageNet accuracy and pre-training data) on a battery of robustness benchmarks: ImageNet-C (corruptions), ImageNet-R (renditions), ImageNet-Sketch, ObjectNet, and adversarial attacks (PGD, AutoAttack) at multiple epsilon budgets. The hypothesis β€” motivated by ViT's global self-attention and lack of local texture bias β€” is that ViT may be less reliant on high-frequency texture cues that CNNs exploit (Geirhos et al., 2019) and therefore more robust to distribution shift. The paper's attention map visualizations (Figure 6), showing that ViT attends to semantically relevant regions, provide suggestive evidence but no quantitative comparison. A negative result (ViT and ResNets showing comparable robustness) would suggest that robustness is determined by pre-training data diversity rather than architecture. A positive result (ViT substantially more robust) would make a strong case for ViT in safety-critical applications and would motivate research into whether attention's global receptive field inherently provides robustness or whether the effect can be replicated in CNNs through architectural changes.

The role of patch size as a data-efficiency lever. The paper uses fixed patch sizes (32, 16, 14) and treats them as architectural constants, but the scaling dimension ablation in Appendix D.2 (Figure 8) shows that decreasing patch size provides "surprisingly robust improvements without introducing parameters." This suggests a novel tradeoff: smaller patches increase sequence length (quadratically more expensive self-attention) but provide finer-grained visual information that may reduce the amount of pre-training data needed. A targeted experiment would train ViT-B with patch sizes of 32, 24, 16, 12, 8, and 4 on random ImageNet subsets of 10%, 25%, 50%, and 100% of the full dataset, measuring whether smaller patch sizes shift the data-efficiency curve β€” i.e., does ViT-B/8 trained on 25% of ImageNet match ViT-B/16 trained on 100%? If so, the paper's finding that ViT requires massive datasets (JFT-300M) to beat CNNs might be partially mitigated by using smaller patches, making ViT practical for smaller data regimes. The position embedding interpolation during fine-tuning (Section 3.2) would also need to be studied as a function of patch size and resolution change, since the interpolation quality may degrade for very small patches where the positional relationships are more fine-grained.

Linear probing versus fine-tuning as a diagnostic for ViT representation quality. The paper uses both linear few-shot evaluation (Figures 4, 8, 9) and full fine-tuning (Table 2) but never systematically compares them to understand what each reveals about the learned representations. A diagnostic experiment would take a ViT pre-trained on JFT-300M and evaluate it on ImageNet with multiple protocols: linear probe on frozen patch tokens, linear probe on the class token, linear probe after global average pooling, partial fine-tuning (last N layers unfrozen), and full fine-tuning β€” all at multiple resolutions. The paper's finding that GAP and class token perform similarly with proper learning rates (Appendix D.3) suggests that the representation quality is diffuse across all tokens, not concentrated in the class token. A systematic probing study would quantify this: does the class token capture mostly high-level semantic information while patch tokens retain spatial detail, or does all information diffuse throughout the sequence? The answer determines whether ViT is suitable for tasks requiring spatial localization (detection, segmentation) without architectural modification, or whether those tasks require extracting intermediate patch token representations.

Practical Applications and Downstream Use Cases

Pre-training a single vision backbone for multi-task transfer across an organization's vision needs. The paper demonstrates that ViT pre-trained on JFT-300M transfers strongly to diverse downstream tasks β€” from fine-grained classification (Pets, Flowers) to structured reasoning (VTAB's localization and counting tasks). For an organization maintaining separate CNN models for product classification, defect detection, image search, and document understanding, ViT offers a unified backbone: pre-train one ViT-H/14 on the organization's proprietary image collection (analogous to JFT-300M in scale and diversity), then fine-tune task-specific heads for each application. The paper's efficiency numbers (2.5k TPUv3-core-days for ViT-H/14 versus 9.9k for BiT-L, Table 2) mean the pre-training cost is approximately 4Γ— lower than a comparable CNN-based unified backbone. More importantly, the paper's finding that ViT performance continues to improve with scale (Figure 5, no evidence of saturation) means that as the organization's image collection grows, the backbone can be periodically re-trained with the same architecture to capture the additional data, rather than requiring a new architecture search at each scale. The memory efficiency advantage (Figure 12, right, showing ViT fitting ~10Γ— larger batch sizes per accelerator than comparable ResNets) reduces the hardware requirements for fine-tuning on individual downstream tasks, making the multi-task workflow more practical.

Cost-efficient fine-tuning of large pre-trained models for niche domains with limited labeled data. The paper's few-shot linear evaluation protocol (Section 4.1) β€” solving a regularized least-squares regression on frozen ViT representations β€” achieves strong performance without any gradient-based fine-tuning. On the VTAB suite (Table 2), ViT-H/14 achieves 77.63% across 19 diverse tasks, each with only 1,000 training examples. This protocol is directly applicable to niche domains (medical imaging, satellite imagery, industrial inspection) where labeled data is scarce but a pre-trained ViT is available. A practitioner with 500 labeled X-ray images and a ViT-H/14 checkpoint can extract frozen patch representations, fit the closed-form linear classifier, and achieve competitive accuracy without GPU fine-tuning infrastructure. The paper's VTAB Specialized results (Figure 2, showing ViT matching BiT on medical and satellite tasks) provide direct evidence that this transfer works for domain-specific imagery. The closed-form solution eliminates the need for hyperparameter tuning (no learning rate, no optimizer, no early stopping), reducing the expertise barrier for applying computer vision to scientific and industrial domains.

Deploying high-accuracy image classifiers on edge hardware with throughput constraints. The paper's inference efficiency measurements (Figure 12, left) show that ViT-B/32 processes approximately 1,000 images per second per TPUv3 core at 224Γ—224 resolution β€” comparable to ResNet50. For an edge deployment scenario (e.g., a quality inspection camera on a factory line processing 60 frames per second), ViT-B/32 pre-trained on a large in-house dataset and fine-tuned on the specific inspection task provides state-of-the-art accuracy (84.15% on ImageNet-equivalent benchmark from Table 5, extrapolating to the specific task) while meeting the throughput requirement. The paper's finding that ViT models are "clearly more memory-efficient" (Figure 12, right) means that larger ViT variants (ViT-B/16, potentially ViT-L/32) may fit within the memory constraints of edge accelerators (e.g., Google Coral TPU, NVIDIA Jetson) that would not accommodate comparable ResNet variants, enabling higher accuracy at the same hardware tier. The position embedding interpolation during fine-tuning (Section 3.2) also enables the deployed model to handle variable input resolutions without architectural changes β€” the factory camera could switch from 224Γ—224 to 384Γ—384 if a more detailed inspection is needed, with the interpolated position embeddings preserving spatial understanding without re-deploying the model.

When to Prefer This Method

The paper articulates a clear tradeoff between ViT and CNNs based on pre-training dataset size, making a decision rule appropriate here. The crossover is empirically characterized in Figures 3 and 4:

  • Prefer ViT when: your pre-training dataset exceeds roughly 100M images (the approximate crossover where ViT begins to outperform comparable ResNets in Figure 4). This includes organizations with access to web-scale image collections (e.g., JFT-300M scale or proprietary databases of similar magnitude), or those who can use a publicly available ViT checkpoint pre-trained at scale and fine-tune it on their downstream task. ViT offers better accuracy per unit of pre-training compute (2-4Γ— advantage in Figure 5), continued performance improvement with scale (no saturation within the range tested), and simpler infrastructure through NLP-Transformer reuse.

  • Prefer CNNs (ResNets or similar) when: your pre-training dataset is below approximately 10-30M images. At ImageNet scale (1.3M images), the paper shows ViT underperforms ResNets (Figure 3, ViT-Large worse than ViT-Base, BiT ResNets dominating). At intermediate scales (14M images, ImageNet-21k), the architectures are roughly comparable (ViT-L/16 at 85.15% in Table 5), and the choice may depend on deployment constraints (CNNs may have an inference latency advantage at batch size 1) or engineering familiarity. The paper's hybrid model results (Figure 5) provide an intermediate option: at the 14M-image scale, a CNN + Transformer hybrid slightly outperforms pure ViT at the same compute budget, suggesting that practitioners at intermediate data scales might benefit from a hybrid approach rather than committing fully to either pure architecture.

  • The crossover is not a cliff but a gradient, and it depends on regularization. The paper shows that with strong regularization (weight decay, dropout, label smoothing; Section 4.3 and Table 3), ViT can be competitive on ImageNet (77.91% for ViT-B/16, Table 5), though still below ResNets. For practitioners at the 1-10M image scale who are committed to using ViT (e.g., for infrastructure consistency with an NLP team), heavy regularization plus knowledge distillation (as later shown by DeiT, Touvron et al., 2021) can partially compensate for limited data. The paper's self-supervised result (79.9% on ImageNet, Section 4.6) provides a third option: self-supervised pre-training on unlabeled images can boost ViT performance at small data scales, though a 4% gap to supervised pre-training remains.