Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

ArXiv: 2101.11986

🎯 Pitch

Standard ViTs match or beat CNNs only after pretraining on 300M images—but T2T-ViT outperforms ResNet50 on ImageNet trained from scratch by halving the parameters. It fixes ViT’s blind spot to local structure by recursively fusing neighboring tokens, and it scraps the wasteful shallow-wide backbone for a deep-narrow design that uses far fewer dead channels.

---

1. Executive Summary

This paper proposes a Tokens-to-Token Vision Transformer (T2T-ViT) that can be trained from scratch on the midsize ImageNet dataset and outperform comparable CNNs — a feat the vanilla Vision Transformer (ViT) fails to achieve. It does so through two mechanisms: a layer-wise Tokens-to-Token (T2T) transformation that progressively structurizes the image by recursively aggregating neighboring tokens into one (soft splitting with overlap, then flattening), thereby modeling the local structure — edges, lines, textures — that the naive hard-split tokenization of ViT misses; and an efficient deep-narrow backbone that reduces channel redundancy and increases layer depth to improve feature richness (14 layers with 384 hidden dimensions vs. ViT's 8 layers with 768). The result is a model with roughly half the parameters and MACs of ViT-S/16 that achieves more than 3.0% higher top-1 accuracy when trained from scratch on ImageNet (81.5% for T2T-ViT-14 at 21.5M parameters vs. 78.1% for ViT-S/16 at 48.6M parameters), establishing that pure transformer architectures can surpass CNNs at various complexities on ImageNet without JFT-300M pretraining — but only when the tokenization preserves local structure and the backbone avoids the shallow-wide design that leaves many channels dead.

2. Context and Motivation

The Core Problem: Vision Transformers Fail at Midsize Datasets

The fundamental question this paper tackles is deceptively simple: can a pure transformer architecture be trained from scratch on ImageNet to match or exceed CNNs? By the time of this paper's writing, the answer was essentially "no"—at least not without special tricks. The Vision Transformer (ViT) introduced by Dosovitskiy et al. (2020) had demonstrated that a pure self-attention architecture (no convolution whatsoever) could achieve state-of-the-art results on image classification, but only when pretrained on enormous datasets like JFT-300M (a private Google dataset with 300 million images) or ImageNet-21k (14 million images). When trained from scratch on the comparatively modest ImageNet-1k (1.3 million images), ViT significantly underperformed similarly-sized CNNs like ResNets.

This gap mattered for several practical and scientific reasons. First, JFT-300M is not publicly available, meaning the broader research community could not replicate or build upon ViT's strongest results. Second, pretraining on 300M images requires immense computational resources, creating a barrier to entry that limits who can contribute to vision transformer research. Third, the fact that transformers needed orders of magnitude more training data than CNNs to achieve parity suggested something was fundamentally wrong with how the architecture processed images—not with the self-attention mechanism itself, but with the interface between pixels and tokens.

The paper identifies this diagnostic clearly in Section 1:

"Though ViT proves the full-transformer architecture is promising for vision tasks, its performance is still inferior to that of similar-sized CNN counterparts (e.g. ResNets) when trained from scratch on a midsize dataset (e.g., ImageNet)."

The key word is from scratch on a midsize dataset—this is the setting where the gap manifests, and it is precisely the setting most relevant to researchers without access to giant proprietary datasets.

Two Diagnosed Limitations of ViT

The paper's motivation goes beyond simply observing the performance gap. It diagnoses why ViT struggles on ImageNet-scale data, proposing two specific technical limitations:

Limitation 1: Naive tokenization destroys local structure. ViT's tokenization is brutally simple: take an input image (say, $224 \times 224$ pixels), divide it into a grid of $16 \times 16$ non-overlapping patches, flatten each patch into a vector, and feed these vectors as tokens into the transformer. This "hard split" approach treats the image as a bag of independent patches, with no mechanism to model the relationships among neighboring pixels within or across patch boundaries before the first transformer layer.

This matters because natural images have strong local structure—edges, lines, textures, corners—that CNNs exploit through their built-in translational equivariance and local receptive fields. A CNN's first layer learns to detect oriented edges and simple textures because it applies the same small filter across the entire image. ViT's first layer, by contrast, sees each $16 \times 16$ patch as a flattened vector and applies a linear projection. There is no weight sharing across spatial positions, no built-in local connectivity, and no mechanism encouraging the model to learn that adjacent pixels tend to be more correlated than distant ones.

The paper argues that this lack of local structure modeling is what causes ViT's low sample efficiency: CNNs learn useful low-level features from relatively few examples because their architectural inductive bias aligns with the structure of natural images. ViT lacks this bias, so it needs vastly more data to discover from scratch what a CNN architecturally encodes.

The feature visualizations in Figure 2 make this concrete. ResNet50's early layers (conv1, represented in the figure) clearly capture oriented edges, textures, and object boundaries—the kind of local structure information you would expect. But ViT-L/16's attention blocks show something quite different: the structure information is "poorly modeled" while "global relations (e.g., the whole dog) are captured by all the attention blocks." In other words, ViT jumps directly to modeling long-range dependencies without first building up a representation of local geometry. This is behaviorally reasonable—self-attention can model long-range interactions from the first layer—but it means the model is trying to learn everything from scratch without the low-level visual scaffolding that CNNs acquire naturally.

Limitation 2: Shallow-wide backbone causes channel redundancy. The second limitation is subtler and concerns the internal architecture of ViT's transformer layers. The standard ViT configurations (Small, Base, Large) follow a design pattern inherited from NLP transformers: relatively few layers with relatively many channels (hidden dimensions). ViT-S/16 has 8 layers with 768 hidden dimensions; ViT-B/16 has 12 layers with 768; ViT-L/16 has 24 layers with 1024.

The paper's feature visualizations in Figure 2 reveal a striking problem: many channels in ViT are effectively dead. The red-boxed feature maps show channels with "zero or too large values," indicating that a significant fraction of the model's representational capacity is wasted. The paper interprets this as a sign that the backbone "is not efficient as ResNets and offers limited feature richness when training samples are not enough."

This is a critical insight that connects to the broader deep learning principle of depth vs. width. In CNNs, the literature had already established that deep-narrow architectures (many layers, few channels per layer) can be more parameter-efficient and expressive than shallow-wide ones—this is the lesson of Wide-ResNets (Zagoruyko & Komodakis, 2016), which showed that increasing depth while keeping width modest yields better results than the reverse. But this principle had not been systematically explored for vision transformers. ViT inherited its width-dominated design from language models, where the default transformer block structure emerged from an entirely different modality.

The Broader Scientific Context

To understand why this paper's contribution mattered, we need to situate it within the rapid evolution of vision transformers in 2020–2021.

Pre-T2T-ViT landscape. The original ViT paper (Dosovitskiy et al., 2020) was a breakthrough in demonstrating that convolutions are not strictly necessary for vision—but it came with a major caveat: the model needed JFT-300M pretraining. This meant that while the result was theoretically important (pure attention can work for vision), it was not practically accessible (you need a proprietary 300M-image dataset to get there). ViT's ImageNet-from-scratch performance was roughly 78% for ViT-S/16, while a comparable ResNet50 achieved 76–79% with careful training, and ResNet variants could push higher.

Several works attempted to close this gap. DeiT (Data-efficient image Transformers; Touvron et al., 2020), a concurrent work at submission time, took an orthogonal approach: rather than redesigning the architecture, it introduced knowledge distillation from a CNN teacher into the ViT training process. By adding a distillation token alongside the classification token and training with a distillation loss, DeiT achieved roughly 80–82% top-1 accuracy on ImageNet without JFT-300M. Crucially, however, DeiT still relied on CNNs as teachers—it didn't demonstrate that a pure transformer could match CNNs on its own architectural merits. The paper explicitly acknowledges this relationship:

"A recent concurrent work DeiT [36] applies Knowledge Distillation [16, 49] to improve the original ViT by adding a KD token along with the class token, which is orthogonal to our work, as our T2T-ViT focuses on the architecture design, and our T2T-ViT can achieve higher performance than DeiT without CNN as teacher model."

This distinction—orthogonal approaches, architectural vs. distillation—is important: the paper is arguing that the transformer architecture itself can be redesigned to work better on vision, not that you need external supervision from CNNs to make it work.

Other concurrent work. The field was exploring various ways to incorporate more structure into vision transformers. Some works added convolutional embeddings or hybrid CNN-transformer architectures (e.g., using ResNet feature maps as transformer inputs rather than raw patches). Others explored local self-attention within windows to reduce the quadratic complexity of global attention and inject locality. But these approaches diluted the "pure transformer" claim—they were hybrids that brought convolution back in through the side door.

The T2T-ViT paper positions itself as maintaining the purity of the transformer architecture while redesigning its internal components specifically for vision. The T2T module is not a convolutional layer with learned filters—it's a structural reorganization of tokens that uses self-attention (standard transformer layers) to model relationships before restructuring. Similarly, the deep-narrow backbone is still a standard transformer with skip connections, just with different hyperparameter choices informed by CNN architecture design principles.

Prior Approaches and Their Specific Shortcomings

Let's examine each category of prior work and what T2T-ViT addresses:

1. Standard ViT (hard split tokenization). The core failure mode is loss of local structure. When you hard-split an image into non-overlapping $16 \times 16$ patches, you create artificial boundaries. An edge that crosses two patches is broken in half—the left half goes to one token, the right half to another, and the model has no prior that these two tokens are spatially adjacent until it learns that correlation from data. A CNN never faces this problem because its filters slide across the image with overlap. The T2T module's "soft split" with overlapping patches and token aggregation directly addresses this shortcoming by ensuring that neighboring tokens share information before being consumed by deeper layers.

2. Knowledge distillation approaches (DeiT). The limitation here is conceptual and practical. Conceptually, if you need a CNN teacher to make your transformer work, you haven't demonstrated that transformers are independently viable for vision—you've demonstrated that CNNs are good teachers. Practically, the teacher model must be trained first, adding complexity and compute cost. More subtly, the distilled student may inherit biases or limitations of the CNN teacher that a pure transformer might otherwise avoid.

3. Hybrid CNN-transformer models. Many approaches used convolutional stems (a few conv layers before the transformer) to extract local features, then applied self-attention to those features. These models work well but concede that pure attention cannot handle low-level vision. T2T-ViT's position is that the structural reorganization of tokens (soft split, unfolding, aggregation) can serve the same purpose as a convolutional stem without using learned convolutional filters—the T2T module uses transformer layers for computation, not convolutions. The comparison in the ablation (Section 4.3, Table 7) where the authors replace the T2T module with standard convolution layers (producing T2T-ViTc-14, which gets 80.8% vs. 81.5% for the attention-based T2T module) is the direct evidence that the T2T design is not simply rediscovering convolution but offering something incrementally better.

4. Architectural exploration in CNNs. The paper explicitly frames its backbone redesign as borrowing from CNN architecture engineering. Wide-ResNets established that deep-narrow beats shallow-wide for CNNs at comparable parameter counts. DenseNet showed dense skip connections improve feature reuse. SENet demonstrated channel attention helps. ResNeXt showed grouped convolutions (analogous to multi-head attention) can improve efficiency. GhostNet showed you can generate redundant features cheaply. But none of these principles had been systematically tested within vision transformers. The paper's Section 4.2 does exactly this, testing all five concepts and finding that only deep-narrow and SE-style channel attention transfer positively to ViT, while dense connections and more attention heads actually hurt or provide no benefit. This is genuinely useful architectural knowledge that was not obvious a priori.

How This Paper Positions Itself

The paper frames its contribution as addressing the two diagnosed limitations directly through architectural innovation, not through external data, distillation, or convolution hybrids:

• For limitation 1 (lost local structure): The T2T module performs a progressive tokenization that models local structure through overlapping soft splits and aggregation of neighboring tokens. This is not convolution (no learned local filters) but achieves a similar functional outcome: tokens that enter the backbone already encode local neighborhood relationships. The iterative nature—doing this multiple times with decreasing patch sizes ($7 \times 7$, then $3 \times 3$, then $3 \times 3$)—means the model progressively structurizes the image, analogous to how a CNN's early layers build up from edges to textures to parts.

• For limitation 2 (channel redundancy in shallow-wide backbone): The deep-narrow architecture reduces hidden dimensions (from ViT's 768 to 384 for the comparable model) while increasing depth (from 8 to 14 layers for T2T-ViT-14). This reduces parameter count and MACs while improving accuracy, suggesting the original ViT backbone was indeed overparameterized in width and starved in depth. The dead channels in Figure 2 are direct evidence that the shallow-wide design wastes capacity; the fact that switching to deep-narrow both reduces model size and improves performance is strong evidence that the new design is genuinely more efficient, not just differently parameterized.

The paper's ambition is explicitly stated in the contributions:

"For the first time, we show by carefully designing transformers architecture (T2T module and efficient backbone), visual transformers can outperform CNNs at different complexities on ImageNet without pretraining on JFT-300M."

The phrase "for the first time" stakes a strong claim: prior to T2T-ViT, no pure transformer (no convolution, no distillation from CNNs) had outperformed CNNs on ImageNet when trained from scratch. This is the paper's core positioning—it's not just an incremental improvement over ViT, but a demonstration that the right architecture makes pure transformers competitive with CNNs even on midsize datasets.

The Importance of the "Trained from Scratch" Constraint

Why does "trained from scratch on ImageNet" matter so much? Several reasons:

Reproducibility and accessibility: JFT-300M is private. Most researchers cannot reproduce or build on results that depend on it. If vision transformers only work well with proprietary datasets, the field bifurcates into those who have access and those who don't. Training from scratch on public ImageNet-1k means the result is fully reproducible by anyone with reasonable GPU resources.

Fair comparison to CNNs: CNNs like ResNet50 were developed and tuned for ImageNet-scale training. Comparing a JFT-pretrained ViT against a ResNet trained from scratch is not a fair comparison—the ViT has seen 300x more data. The relevant question is: given the same training data (ImageNet), which architecture learns better? T2T-ViT answers this question in favor of the transformer for the first time.

Sample efficiency as a scientific question: The fact that ViT needed JFT-300M while ResNet50 needed only ImageNet tells us something fundamental about the inductive biases of the two architectures. CNNs encode spatial locality, translational equivariance, and hierarchical composition as hard-wired architectural constraints. Transformers encode none of these—they must learn them from data. The T2T module can be understood as injecting a soft form of locality bias (through overlapping token aggregation) without going all the way to convolution, striking a balance between the flexibility of attention and the sample efficiency of structure.

Practical deployment: Most real-world vision applications don't have 300M labeled images to pretrain on. If transformers are going to replace CNNs in practice (not just in headline results), they need to work at ImageNet scale. The paper's lite models (T2T-ViT-7, T2T-ViT-12) further demonstrate this commitment to practicality by targeting the MobileNet-sized regime relevant to edge deployment.

A Note on What the Paper Does NOT Claim

It's worth clarifying boundaries: the paper does not claim that transformers are universally better than CNNs. The results show T2T-ViT achieving comparable or better performance than ResNets and comparable performance to MobileNets at similar parameter counts, but the MobileNet comparison acknowledges a caveat:

"we also note the MACs of our T2T-ViT are still larger than MobileNets because of the dense operations in Transformers. However, there are no special operations or tricks like efficient convolution [26, 32] in current T2T-ViT-7 and T2T-ViT-12"

This transparency is important: the paper is making an architectural point (transformers can be competitive) rather than an efficiency point (transformers are always more efficient). The MobileNet family uses depthwise separable convolutions, inverted residuals, and linear bottlenecks—highly optimized operations that T2T-ViT does not attempt to replicate. Closing this efficiency gap with specialized transformer operations would be future work.

Similarly, the paper does not claim that the T2T module is the only way to model local structure in transformers. The experiments replacing the T2T module with convolution layers (T2T-ViTc-14) show that convolution also works, just not quite as well. The contribution is more specific: the T2T process (re-structurization + soft split using attention-based transformers) is a viable and effective method for progressively tokenizing images that outperforms both naive hard split and equivalent convolutional stems.

3. Technical Approach

3.1 Reader Orientation

This is primarily an architectural innovation paper whose core idea is that a pure vision transformer can match or exceed CNNs on midsize datasets like ImageNet if you fix two specific problems—how pixels become tokens and how deep/wide the transformer layers are—rather than resorting to external pretraining data, knowledge distillation from CNNs, or hybrid convolution-transformer designs. The system is a full-transformer image classifier that progressively structurizes an image into a compact sequence of tokens through a novel Tokens-to-Token (T2T) module, then processes those tokens through a deep-narrow transformer backbone, yielding higher accuracy than standard ViT at roughly half the parameter count and MACs when both are trained from scratch on ImageNet. The "shape" of the solution is a two-stage pipeline: first, an iterative tokenization front-end that models local structure by recursively aggregating neighboring tokens; second, a redesigned transformer backbone that trades width for depth to reduce channel redundancy and improve feature richness.

3.2 Big-Picture Architecture (Diagram in Words)

The T2T-ViT architecture has two major components connected sequentially:

1. The Tokens-to-Token (T2T) Module — a front-end processing stage that takes the raw input image, converts it to tokens through overlapping "soft splits," then iteratively applies transformer layers, reshapes token sequences back into 2D image-like structures, and re-splits them with overlap to progressively aggregate neighboring pixel/patch information into each token. This reduces token sequence length from the thousands to a fixed compact representation (e.g., 196 tokens) while embedding local structural information (edges, textures, spatial relationships) into the tokens themselves.

2. The T2T-ViT Backbone — a standard transformer encoder (multi-head self-attention + MLP blocks with skip connections and layer normalization) that takes the fixed-length token sequence from the T2T module and performs global relational modeling for classification. Unlike ViT's backbone, this one uses a deep-narrow design: fewer channels per layer (e.g., 384 hidden dimensions instead of 768) but more layers (e.g., 14 instead of 8), which the paper demonstrates reduces dead channels and increases feature richness.

Information flows as follows: an input image of size $224 \times 224 \times 3$ enters the T2T module → a first soft split with patch size $7 \times 7$ and stride $4$ (overlap $3$) converts it to a long sequence of tokens → the first T2T transformer layer (standard or Performer) processes these tokens with self-attention → the output tokens are reshaped into a 2D feature map → a second soft split (patch size $3 \times 3$, stride $2$, overlap $1$) aggregates neighboring tokens and reduces sequence length → the second T2T transformer layer processes these tokens → a third soft split (also $3 \times 3$ with stride $2$) produces the final fixed-length token sequence → a learnable classification token is prepended and sinusoidal position embeddings are added → the token sequence enters the deep-narrow backbone (e.g., 14 transformer layers with 384 hidden dimensions) → the output corresponding to the classification token is passed through layer normalization and a linear classifier to produce class predictions.

3.3 Roadmap for the Deep Dive

• First, I explain the T2T module's core mechanism—the iterative Re-structurization and Soft Split operations—including the exact patch sizes, strides, sequence length reduction, and the formula controlling output length. This is the paper's primary architectural novelty and the component that distinguishes T2T-ViT from standard ViT.

• Second, I walk through the T2T-ViT backbone's deep-narrow design, including the specific architectural choices (hidden dimensions, MLP ratios, layer counts), how they were arrived at through systematic exploration of five CNN-inspired architecture modifications, and why deep-narrow works when shallow-wide produces dead channels.

• Third, I cover the complete T2T-ViT model configurations (T2T-ViT-7/12/14/19/24), their parameter counts, MACs, and how they map to comparable CNN baselines (ResNets and MobileNets), including the use of Performer layers in the T2T module for memory efficiency.

• Fourth, I detail the CNN-to-ViT architecture transfer experiments—the five design patterns tested (dense connections, deep-narrow vs. shallow-wide, SE channel attention, ResNeXt-style multi-head splitting, Ghost operations), how each was implemented in the transformer context, and the quantitative results that justified selecting deep-narrow over alternatives.

• Fifth, I explain the training recipe shared across all models (ViT, T2T-ViT, CNNs) for fair comparison, including the optimizer, data augmentation, epoch count, and learning rate schedule.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an architectural design paper whose core idea is that two specific modifications to the Vision Transformer—a progressive, overlapping tokenization front-end (the T2T module) and a deep-narrow backbone—enable pure transformers to learn effective visual representations from ImageNet-scale data without pretraining on massive proprietary datasets or distilling from CNN teachers. The paper systematically diagnoses why standard ViT fails, proposes targeted fixes, and validates them through controlled experiments including architectural ablations and CNN-to-ViT design transfer studies.

---

The Tokens-to-Token (T2T) Module: Progressive Tokenization

The T2T module is the paper's primary architectural contribution. It replaces ViT's single-step hard-split tokenization with an iterative process that progressively structurizes the image into a compact token representation while embedding local structure information into the tokens themselves. The module consists of multiple T2T steps, where each step applies two operations in sequence: Re-structurization followed by Soft Split (SS).

Why this is necessary. ViT's tokenization—dividing the image into a $16 \times 16$ grid of non-overlapping patches—treats each patch as independent and applies no transformation to model relationships among neighboring pixels before the first transformer layer. The consequence, shown in Figure 2, is that ViT's early layers fail to learn local structure features (edges, lines, textures) that CNNs capture naturally through their local receptive fields and weight sharing. The T2T module addresses this by ensuring that each token's representation is built from aggregating information from its spatial neighbors, and by doing this iteratively (at multiple scales), the model builds up increasingly structured representations—analogous to how CNNs progress from low-level edges to mid-level textures to high-level parts.

The T2T process, step by step. The paper uses $n = 2$ T2T iterations, plus an initial soft split, meaning there are $n+1 = 3$ total soft splits and $n = 2$ re-structurization operations. Here is the explicit sequence:

Step 0: Initial soft split. The input image $I_0 \in \mathbb{R}^{224 \times 224 \times 3}$ undergoes the first soft split to produce the initial token sequence $T_1$. The patch size is $k = 7$, the overlap is $s = 3$ pixels (so stride is $k - s = 4$), and zero padding $p$ is used. Each $7 \times 7 \times 3$ patch is flattened into a vector of dimension $7 \times 7 \times 3 = 147$. According to Equation (3), the output token sequence length is:

$l_o = \left\lfloor \frac{h + 2p - k}{k - s} + 1 \right\rfloor \times \left\lfloor \frac{w + 2p - k}{k - s} + 1 \right\rfloor$

For a $224 \times 224$ input with $k=7$, $s=3$, $p$ chosen to make the division work out, this produces an initial token grid. The key property of soft split (as opposed to ViT's hard split) is overlap: adjacent patches share pixels, so spatial relationships are explicitly encoded by having the same pixel contribute to multiple tokens. Each token is then linearly projected to the T2T transformer's working dimension (32 or 64 channels, deliberately kept small to manage the computational cost of processing long token sequences).

Step 1: First T2T iteration. The token sequence $T_1$ passes through a transformer layer (the "T2T transformer"):

$T_1' = \text{MLP}(\text{MSA}(T_1))$

where MSA denotes multi-head self-attention with layer normalization applied before the attention operation, and MLP is a two-layer feedforward network with layer normalization (following the standard Pre-LN transformer convention used in ViT). This transformer layer models relationships among all tokens globally—even at this early stage, the model can attend across the entire image. The output $T_1'$ has the same shape as $T_1$.

Re-structurization. The tokens $T_1'$ are reshaped from a 1D sequence back into a 2D spatial grid:

$I_1 = \text{Reshape}(T_1')$

where $\text{Reshape}$ reorganizes $T_1' \in \mathbb{R}^{l \times c}$ into $I_1 \in \mathbb{R}^{h \times w \times c}$ such that $l = h \times w$. This operation recovers the spatial layout of the image, which is essential for the next step: if tokens were just abstract vectors, there would be no notion of "neighboring tokens." By restoring 2D structure, the model can identify which tokens are spatially adjacent and aggregate them.

Soft split (second application). The reconstructed feature map $I_1$ now undergoes soft split with patch size $k = 3$, overlap $s = 1$ (stride $k - s = 2$), producing $T_2$. This means each $3 \times 3$ region of the $I_1$ feature map becomes one token in $T_2$, and adjacent patches overlap by 1 pixel. The effect is that each token in $T_2$ aggregates information from a $3 \times 3$ neighborhood of tokens from $T_1'$—and since those tokens already encoded information from the original $7 \times 7$ pixel patches, $T_2$ tokens now capture information from a larger effective receptive field. The token sequence length is reduced because the stride (2) is larger relative to the patch size (3) compared to the first split (stride 4, patch 7).

Step 2: Second T2T iteration. The same process repeats: $T_2' = \text{MLP}(\text{MSA}(T_2))$ $I_2 = \text{Reshape}(T_2')$ $T_3 = \text{SS}(I_2)$

The third soft split uses the same parameters as the second: patch size $k = 3$, overlap $s = 1$, stride 2. After this final soft split, the token sequence $T_f = T_3$ has been reduced to a fixed length—for a $224 \times 224$ input image, the spatial grid becomes $14 \times 14$, yielding 196 tokens. This is the same final token count as ViT's $16 \times 16$ hard split (which also produces $14 \times 14 = 196$ tokens), but critically, each token in $T_f$ encodes structured local information aggregated from multiple scales and neighborhoods, whereas ViT tokens are raw, independent pixel patches.

The complete iterative formulation. The paper expresses the full T2T process compactly as:

$T_i' = \text{MLP}(\text{MSA}(T_i))$ $I_i = \text{Reshape}(T_i')$ $T_{i+1} = \text{SS}(I_i)$

for $i = 1 \ldots (n-1)$, with $n = 2$ in the paper's configurations.

What the Soft Split formula computes. The output length after soft split is governed by:

$l_o = \left\lfloor \frac{h + 2p - k}{k - s} + 1 \right\rfloor \times \left\lfloor \frac{w + 2p - k}{k - s} + 1 \right\rfloor$

where $h, w$ are the height and width of the restructurized feature map $I_i$, $k$ is the patch size, $s$ is the overlap amount (so stride $= k - s$), and $p$ is zero-padding added to the feature map boundaries.

What it computes: This is the standard formula for the output spatial dimensions of a sliding window operation (identical to convolution output size calculation). For each spatial dimension, the numerator $h + 2p - k$ is the number of valid starting positions for a patch (accounting for padding), the denominator $k - s$ is the stride, and the $+1$ accounts for the first patch. The floor operation handles cases where the division is not exact. The product gives the total number of patches, which equals the token count $l_o$.

Why this form: This formulation explicitly links the soft split to convolution-like spatial reduction while maintaining the key property that the operation is structural (reshaping and flattening) rather than learned (no filter weights). The overlap $s$ is the critical parameter: when $s > 0$, adjacent patches share pixels, creating a built-in spatial smoothness prior. When $s = 0$, soft split degenerates to ViT's hard split (non-overlapping patches). The specific values chosen—$P = [7, 3, 3]$ for patch sizes and $S = [3, 1, 1]$ for overlaps—progressively reduce the spatial grid from $224 \times 224$ to approximately $56 \times 56$ (after first soft split) to $28 \times 28$ (after second) to $14 \times 14$ (after third), which matches the first three stages of a typical CNN's spatial downsampling (e.g., ResNet reduces from $224 \to 112 \to 56 \to 28 \to 14 \to 7$ through its conv and pooling layers).

Tokens-to-Token aggregation detail. The name "Tokens-to-Token" refers to the specific operation within soft split where multiple tokens from the previous step are concatenated to form one token in the next step. As illustrated in Figure 3's pink panel: when a $3 \times 3$ patch slides over the $I_i$ feature map, it covers 9 tokens from the previous step. These 9 tokens are flattened and concatenated into a single vector, forming one token in $T_{i+1}$. The output dimension of this token is $k \times k \times c = 9c$, where $c$ is the channel dimension of $I_i$. A linear projection then maps this back to the working dimension $d$ of the T2T transformer layer (32 or 64).

This is fundamentally different from convolution. In convolution, a $3 \times 3$ filter computes a weighted sum of the 9 input values (one scalar per output channel). In T2T, all 9 values are preserved (via concatenation), and the relationship among them is modeled globally by the subsequent transformer layer's self-attention. This means the T2T module does not impose a fixed local filter but lets the attention mechanism learn how to combine information from neighboring tokens based on content and context—it can model more complex spatial relationships than a static convolutional kernel, at the cost of more computation and parameters.

Computational efficiency of the T2T module. The initial token sequences in the T2T module are very long—approximately $56 \times 56 = 3136$ tokens after the first soft split (for a $224 \times 224$ input with $k=7, s=3$). Applying standard multi-head self-attention to a sequence of this length would be prohibitively expensive in both memory and computation, since self-attention complexity scales quadratically with sequence length. The paper addresses this through two measures:

1. Small channel dimension: The T2T transformer layers use a hidden dimension of only 32 or 64 (compared to 384–1024 in the backbone). This dramatically reduces the per-token computation and memory for the attention and MLP operations. The paper states explicitly: "we set the channel dimension of the T2T layer small (32 or 64) to reduce MACs."

2. Optional Performer layers: For models targeting limited GPU memory (T2T-ViT-7/12/14/19/24, distinguished from T2T-ViTt-14/19/24 which use standard Transformer layers in the T2T module), the paper uses Performer (Choromanski et al., 2020) layers instead of standard Transformer layers. Performer approximates full self-attention using kernel methods, reducing the complexity from $O(l^2)$ to $O(l)$ in sequence length. This is crucial for the first T2T step where $l \approx 3000$. The paper demonstrates this tradeoff in Table 1: T2T-ViT-14 (Performer) has $4.8$G MACs, while T2T-ViTt-14 (Transformer in T2T module) has $6.1$G MACs—the Performer version is computationally cheaper but achieves comparable accuracy ($81.5$% vs. $81.7$%).

Why not just use convolution layers? The paper addresses this directly through the T2T-ViTc-14 ablation (Table 7), which replaces the entire T2T module with "3 convolution layers with kernel size (7,3,3), stride size (4,2,2) respectively." This convolutional stem produces $80.8$% top-1 accuracy, compared to $81.5$% for the Performer-based T2T-ViT-14 and $81.7$% for the Transformer-based T2T-ViTt-14—a gap of $0.5$–$1.0$ percentage points. The paper's interpretation is that the T2T module "is better than the convolution layers as it can model both the global relation and the structure information of the images." In a pure convolutional stem, each layer only sees a local receptive field (e.g., a $3 \times 3$ filter sees only 9 spatial neighbors). In the T2T module, the transformer layer between soft splits can attend globally across all tokens—even in the first T2T step, a token from the top-left corner can directly interact with a token from the bottom-right corner. This means local structure aggregation (via soft split) and global context modeling (via self-attention) are interleaved, which the paper hypothesizes leads to richer representations.

---

The T2T-ViT Backbone: Deep-Narrow Architecture

After the T2T module produces the fixed-length token sequence $T_f$ (196 tokens for $224 \times 224$ input), the T2T-ViT backbone takes over. This backbone is structurally identical to a standard ViT encoder—a stack of transformer layers, each consisting of multi-head self-attention followed by an MLP, with residual connections and layer normalization (Pre-LN)—but with crucially different hyperparameters driven by the deep-narrow design philosophy.

ViT's shallow-wide problem. The standard ViT configurations (Table 1) follow a width-dominated pattern: ViT-S/16 has 8 layers with 768 hidden dimensions, ViT-B/16 has 12 layers with 768, ViT-L/16 has 24 layers with 1024. The MLP expansion ratio is 4× in the original ViT paper, meaning the MLP hidden dimension is 3072 for ViT-B/16 (though the paper's ViT-S/16 variant uses MLP size 2358, suggesting a slightly modified small variant). The feature visualization in Figure 2 reveals the consequence: many channels in ViT are dead (zero-valued or with extreme values), indicating that much of the model's representational capacity is unused. The paper interprets this as "redundancy in channels" and "limited feature richness with shallow layers."

Why deep-narrow works (the Wide-ResNet insight). The paper draws directly on the lesson from Wide-ResNets (Zagoruyko & Komodakis, 2016): for a fixed parameter budget, increasing depth while decreasing width per layer yields better performance than the reverse. In CNNs, this is understood as depth enabling more sequential non-linear transformations (each layer can refine features), while excessive width creates redundant filters that learn similar features. The same principle, the paper argues, applies to transformers: more transformer layers enable more rounds of global attention refinement, while excessively wide hidden dimensions lead to channels that correlate highly or collapse to zero.

The deep-narrow specification. For T2T-ViT-14 (the model comparable to ResNet50), the backbone uses: 14 transformer layers with hidden dimension $d = 384$ and MLP size $1152$ (a 3× expansion ratio, reduced from ViT's typical 4×). Compare this to ViT-S/16: 8 layers with hidden dimension 768 and MLP size 2358. The deep-narrow version has 75% more layers (14 vs. 8) but half the hidden dimension (384 vs. 768), resulting in a model that is both more accurate and significantly smaller. The actual parameter count: T2T-ViT-14 has 21.5M parameters and 4.8G MACs; ViT-S/16 has 48.6M parameters and 10.1G MACs—the deep-narrow design reduces parameters by 56% and MACs by 52%.

For the larger variants, this pattern continues: T2T-ViT-19 uses 19 layers with 448 hidden dimensions (MLP 1344), and T2T-ViT-24 uses 24 layers with 512 hidden dimensions (MLP 1536). Notably, even the largest T2T-ViT-24 (24 layers, 512 hidden dim) is narrower than ViT-L/16 (24 layers, 1024 hidden dim), yet achieves higher accuracy (82.3% vs. 81.1%) with 64.1M parameters vs. 304.3M—a nearly 5× reduction in parameters.

The lite variants deviate further. For the MobileNet-comparable models, the width is reduced even more aggressively: T2T-ViT-7 uses 8 layers (same count as ViT-S/16) but with only 256 hidden dimensions and MLP size 512, yielding 4.2M parameters. T2T-ViT-12 uses 12 layers with the same 256 hidden dimension, yielding 6.8M parameters. Both use only 2 T2T iterations and 64-channel T2T transformer layers. The paper notes these lite models "have no special designs or tricks like efficient convolution and simply reduce the layer depth, hidden dimension, and MLP ratio"—making them architecturally clean despite their small size.

Backbone input processing. The fixed-length tokens $T_f$ from the T2T module enter the backbone through the same protocol as standard ViT:

$T_{f0} = [t_{\text{cls}}; T_f] + E, \quad E \in \mathbb{R}^{(l+1) \times d}$

where $t_{\text{cls}} \in \mathbb{R}^d$ is a learnable classification token (prepended to the sequence), $T_f \in \mathbb{R}^{l \times d}$ are the 196 image tokens, $[\cdot; \cdot]$ denotes concatenation along the sequence dimension, and $E$ is a sinusoidal position embedding added to all $l+1$ tokens. The position embedding dimension matches the backbone's hidden dimension $d$ (384 for T2T-ViT-14).

Backbone processing. The token sequence passes through $b$ transformer layers (where $b$ is 14, 19, or 24 for the main variants, 7 or 12 for the lite variants):

$T_{fi} = \text{MLP}(\text{MSA}(T_{f(i-1)})), \quad i = 1 \ldots b$

where the MSA and MLP within each layer follow the standard Pre-LN transformer design: layer normalization is applied before the attention and MLP sub-layers (not after), and residual connections wrap each sub-layer. The final classification is computed from the classification token's output at the last layer:

$y = \text{fc}(\text{LN}(T_{fb}[\text{cls}]))$

where $\text{LN}$ is layer normalization, $\text{fc}$ is a single fully-connected layer mapping from $d$ dimensions to the number of ImageNet classes (1000), and $T_{fb}[\text{cls}]$ extracts the output token corresponding to the classification token's position.

The deep-narrow ablation (Table 7). The paper validates the deep-narrow design through a controlled comparison: T2T-ViT-d768-4 is a shallow-wide variant with hidden dimension 768, 4 layers, and similar overall parameter count to T2T-ViT-14. The result: T2T-ViT-d768-4 achieves 78.8% top-1 accuracy, a 2.7 percentage point drop from T2T-ViT-14's 81.5%. This directly confirms that for a comparable parameter budget, the deep-narrow allocation (more layers, fewer channels) substantially outperforms the shallow-wide alternative (fewer layers, more channels) that ViT inherits from NLP practice.

---

CNN Architecture Design Transfer to ViT (Section 4.2, Table 6)

Before settling on deep-narrow, the paper systematically explored five architecture design patterns from the CNN literature, applying each to both the standard ViT backbone and the T2T-ViT backbone. This is a significant empirical contribution: it tests whether CNN architecture innovations generalize to transformers, and the answer is "some do, some don't." I describe each test and its outcome.

1. Deep-narrow vs. Shallow-wide (Wide-ResNet structure). The paper creates two ViT variants: ViT-DN (Deep-Narrow) with 16 layers and 384 hidden dimensions, and ViT-SW (Shallow-Wide) with 4 layers and 1024 hidden dimensions. These are compared against the baseline ViT-S/16 with 8 layers and 768 hidden dimensions.

• ViT-DN: 79.0% top-1 (+0.9% over ViT-S/16 baseline of 78.1%), with 24.5M parameters and 5.5G MACs (roughly half the parameters and MACs of the baseline).
• ViT-SW: 69.9% top-1 (−8.2% under baseline), with 47.9M parameters and 9.9G MACs (similar size to baseline but dramatically worse).

This asymmetry is striking: going deep-narrow improves both efficiency and accuracy, while going shallow-wide is catastrophic. The result validates the paper's diagnosis that ViT's channel dimensions are overprovisioned and its depth is underprovisioned for ImageNet-scale training.

For T2T-ViT, the equivalent comparison is T2T-ViT-14 (14 layers, 384 hidden dim, 81.5%) vs. T2T-ViT-Wide (14 layers, 768 hidden dim, 77.9%). The wide variant drops 3.4 percentage points with more parameters (25.1M vs. 21.5M), confirming the pattern holds for the T2T-equipped architecture as well.

2. Dense connections (DenseNet structure). DenseNet connects each layer to all subsequent layers through feature concatenation, promoting feature reuse. The paper implements this in ViT by concatenating the outputs of previous transformer layers as input to subsequent layers (with appropriate linear projections to manage growing dimension). The implementation details: ViT-Dense uses 19 layers with hidden dimensions growing from 128 to 736 as layers accumulate.

• ViT-Dense: 76.8% (−1.3% vs. baseline), 46.7M params, 9.7G MACs.
• T2T-ViT-Dense: 80.6% (−1.1% vs. T2T-ViT-14 baseline), 23.7M params, 5.5G MACs.

Dense connections hurt both ViT and T2T-ViT. The paper doesn't deeply analyze why, but a plausible explanation is that dense connections in transformers interact poorly with residual connections (which already provide a form of feature reuse) and the self-attention mechanism (which already models global dependencies, reducing the benefit of explicit feature concatenation across layers). In CNNs, dense connections help because each convolutional layer has a limited receptive field; in transformers, self-attention already gives each layer global context, so dense concatenation may introduce redundant pathways that complicate optimization without adding representational capacity.

3. SE block (channel attention). Squeeze-and-Excitation networks apply a learned channel-wise attention mechanism: global average pooling compresses spatial dimensions, a small MLP produces per-channel scaling factors, and these factors recalibrate channel responses. The paper implements this by inserting the SE operation after the MLP block in each transformer layer (details in appendix).

• ViT-SE: 78.4% (+0.3% vs. baseline), 49.2M params, 10.2G MACs.
• T2T-ViT-SE: 81.6% (+0.1% vs. T2T-ViT-14), 21.9M params, 4.9G MACs.

Channel attention provides small but consistent improvements on both ViT and T2T-ViT. However, the gain (+0.1–0.3%) is far smaller than what deep-narrow achieves (+0.9% for ViT, while also halving parameters), which is why the paper selects deep-narrow as the primary backbone design while noting SE as a compatible improvement.

4. ResNeXt structure (more attention heads). ResNeXt uses grouped convolutions (splitting channels into groups, applying separate filters per group, then concatenating) to increase representational capacity without increasing parameters. The transformer analog is already present: multi-head self-attention splits the query, key, and value projections into $h$ heads, each attending over a $d/h$-dimensional subspace. ResNeXt for transformers would mean increasing $h$ (the number of heads) while keeping total dimension $d$ constant, so each head operates on a smaller subspace.

• ViT-ResNeXt: 78.0% (−0.1% vs. baseline), 48.6M params, 10.1G MACs.
• T2T-ViT-ResNeXt: 81.5% (identical to baseline), 21.5M params, 4.8G MACs.

Increasing heads has negligible effect. The paper notes this comes with a practical cost: "adopting a large number of heads makes the GPU memory large, which is thus unnecessary in ViT and T2T-ViT." This is because multi-head attention's memory footprint during training scales with the number of heads (each head's attention matrix is $l \times l$), so increasing heads without benefit is pure overhead.

5. Ghost operations (GhostNet). GhostNet generates "ghost" feature maps through cheap linear transformations (e.g., depthwise convolution) applied to a smaller set of primary feature maps, reducing computation. For transformers, the paper implements this by reducing the MLP expansion ratio and generating additional features through cheaper operations (details in appendix).

• ViT-Ghost: 73.7% (−4.4% vs. baseline), 32.1M params, 6.9G MACs.
• T2T-ViT-Ghost: 79.5% (−2.0% vs. T2T-ViT-14), 16.3M params, 3.7G MACs.

Ghost operations reduce parameters and MACs substantially but at a significant accuracy cost: −4.4% on ViT and −2.0% on T2T-ViT. The paper notes that T2T-ViT is more robust to this compression than standard ViT (the accuracy drop is half as large), but the degradation is still considerable enough that this is not the recommended configuration.

The takeaway from the CNN-to-ViT study. Among the five tested designs, only deep-narrow structure and SE channel attention provide consistent improvements over the baseline. Deep-narrow is dramatically more effective (improving accuracy while halving parameters for ViT), making it the clear architectural choice for the T2T-ViT backbone. The fact that dense connections, grouped attention, and Ghost operations do not transfer well is itself valuable knowledge—it suggests that transformers have fundamentally different bottlenecks than CNNs, and that architectural innovations from one domain cannot be blindly ported to the other. The paper explicitly states this motivation: "we also wish this study of transferring CNN structure to ViT can motivate the network design of Transformers in vision tasks."

---

Complete Model Configurations and Variants

Table 1 in the paper provides the full specification for all T2T-ViT variants. Let me walk through the design logic connecting them.

Model family design principle. The paper designs variants to match specific points on the CNN parameter-efficiency curve. The three main models (T2T-ViT-14/19/24) target comparable parameter counts to ResNet50/101/152 respectively. The two lite models (T2T-ViT-7/12) target comparable parameter counts to MobileNetV1/V2. The naming convention uses the backbone depth: T2T-ViT-14 has 14 transformer layers in the backbone.

T2T module is shared across variants. All T2T-ViT models use the same T2T module configuration: $n = 2$ T2T iterations plus initial soft split, so 3 total soft splits. The T2T transformer layers use hidden dimension 64 and MLP size 64 (MLP expansion ratio of 1, keeping the T2T module extremely compact). The patch sizes for the three soft splits are $P = [7, 3, 3]$ with overlaps $S = [3, 1, 1]$, reducing the spatial grid from $224 \times 224 \to 56 \times 56 \to 28 \times 28 \to 14 \times 14$.

Backbone scaling. The differentiation between variants occurs entirely in the backbone:

The MLP expansion ratio is consistently 3× (hidden dimension $\to$ MLP size) across all variants, reduced from ViT's typical 4× as part of the deep-narrow efficiency focus.

Performer vs. Transformer in T2T module. The paper creates two parallel model series:

• T2T-ViT- (without 't'): uses Performer layers in the T2T module, trading a small amount of accuracy for reduced memory and MACs.
• T2T-ViTt- (with 't'): uses standard Transformer layers in the T2T module, for maximum accuracy at higher computational cost.

For T2T-ViT-14: Performer version achieves 81.5% with 4.8G MACs; Transformer version (T2T-ViTt-14) achieves 81.7% with 6.1G MACs—a 0.2 percentage point improvement for 27% more MACs. The paper seems to recommend the Performer versions as the default (they are the main numbers reported), with the Transformer versions available for settings where memory is not a constraint.

The lite variants' design choices. T2T-ViT-7 and T2T-ViT-12 use only 2 T2T iterations (matching the main variants), T2T transformer dimension 64, and have reduced backbone dimensions: 256 hidden dim and 512 MLP size. Despite having 8 and 12 backbone layers respectively (the same counts as the medium variants if comparing layer counts), the narrow width (256 vs. 384–512) dramatically reduces parameters. The MLP expansion remains 2× for the lite models (256 $\to$ 512), which is even more aggressive than the 3× used in the main variants. The paper notes no special efficiency tricks "like efficient convolution" were applied—these are pure transformer models with simply reduced dimensions.

---

Training Recipe and Experimental Setup

The paper uses a consistent training recipe across all models (CNNs, ViT, T2T-ViT) to ensure fair comparison. The key specifications:

Dataset: ImageNet-1k (ILSVRC 2012), approximately 1.3M training images and 50K validation images.

Hardware and batch size: 8 NVIDIA GPUs, batch size 512 or 1024.

Training duration: 310 epochs. This is substantially longer than the standard 90-epoch ResNet training recipe but is applied uniformly to all models, making comparisons internally valid. The longer training is necessary for transformer models, which the paper notes "need more training data to reach reasonable performance" (addressed through data augmentation rather than larger datasets).

Optimizer: AdamW (Loshchilov & Hutter, 2017) with cosine learning rate decay (Loshchilov & Hutter, 2016). The specific learning rate, weight decay, and warmup parameters are given in the appendix (not specified in the main paper text, but standard for AdamW-based ViT training would be $\text{lr} \approx 5 \times 10^{-4}$ to $1 \times 10^{-3}$ with weight decay $0.05$ and warmup over 5–10 epochs).

Data augmentation: Mixup (Zhang et al., 2017) and CutMix (Yun et al., 2019) are applied during training for all models. This is explicitly motivated by the paper's observation that "ViT models need more training data to reach reasonable performance"—data augmentation serves as a form of regularization and effective data expansion that transformers particularly benefit from given their lack of built-in spatial inductive biases. It's notable that the paper applies this to the CNN baselines as well (the "ResNet50*" results in Table 3 represent ResNet50 trained with this same recipe), ensuring that any performance gap is due to architecture, not training protocol.

Input resolution: Default is $224 \times 224$. For the T2T-ViT-14↑384 experiment, images are upsampled to $384 \times 384$ during both training and inference. At this resolution, the T2T module produces a larger token grid, and the backbone's position embeddings are interpolated to accommodate the longer sequence (standard practice for ViT resolution scaling, likely using bilinear interpolation of the sinusoidal position embeddings). The MACs increase from 4.8G to 17.1G due to the quadratic scaling of self-attention with sequence length.

Implementation: PyTorch (Paszke et al., 2019) with the timm library (Wightman, 2019).

Why this training recipe matters for the paper's claims. A concern when comparing architectures is that one might have been tuned more carefully than another. The paper addresses this by using identical training hyperparameters and augmentations for all models, and by re-training the CNN baselines (marked with * in Table 3) under the same protocol. The ResNet50 baseline improves from its original reported 76.2% to 79.1% under this recipe—a substantial gain from stronger augmentation and longer training—yet T2T-ViT-14 still outperforms it by 2.4 percentage points (81.5% vs. 79.1%). This means the architectural advantage is measured against the strongest fair baseline, not an artificially weak one.

The transfer learning protocol. For CIFAR-10 and CIFAR-100 experiments (Table 5), the paper fine-tunes the ImageNet-pretrained T2T-ViT models for 60 epochs using SGD with cosine learning rate decay (switching from AdamW to SGD is standard practice for fine-tuning, as SGD often achieves better final convergence when starting from a good initialization). The results show T2T-ViT-14 outperforming ViT-S/16 on both CIFAR-10 (97.5% vs. 97.1%) and CIFAR-100 (88.4% vs. 87.1%) while using less than half the parameters, demonstrating that the benefits transfer to downstream tasks.

---

Summary of Design Choices and Their Justifications

• Iterative T2T with decreasing patch sizes ([7, 3, 3]) over single hard split: progressively builds structured representations at multiple scales, analogous to CNN's hierarchical feature learning, but using global self-attention rather than local convolution to model token relationships within each stage.

• Soft split with overlap ($s = 3, 1, 1$) over hard split ($s = 0$): ensures that adjacent patches share information, encoding a spatial smoothness prior that reduces the sample complexity of learning local structure. Each output token aggregates information from $k \times k$ spatially neighboring tokens, providing a built-in locality bias without imposing fixed convolutional weights.

• Reshaping to 2D before each soft split over treating tokens as abstract 1D sequences: recovers spatial layout, which is essential for identifying which tokens are neighbors. Without this step, the notion of "aggregating surrounding tokens" would be ill-defined.

• Deep-narrow backbone (e.g., 14 layers × 384 dim) over shallow-wide (8 layers × 768 dim): reduces dead channels (Figure 2 evidence), improves feature richness through more rounds of global attention refinement, and is more parameter-efficient—validated by +0.9% improvement for ViT-DN over ViT-S/16 baseline with half the parameters.

• MLP expansion ratio of 3× (e.g., 384 → 1152) over ViT's typical 4×: further reduces channel redundancy in the feedforward sub-layers without significantly impacting representational capacity, contributing to the overall parameter reduction.

• Performer in T2T module for lite and standard variants (optional Transformer for maximum accuracy): trades a small accuracy decrease (0.2%) for substantially reduced memory and MACs (27% reduction), making the model practical on limited GPU memory given the long token sequences (~3000) in early T2T steps.

• Identical training recipe across all models (AdamW, 310 epochs, Mixup + CutMix, cosine decay): eliminates training protocol as a confounding variable, ensuring that performance differences reflect architectural quality rather than optimization tuning.

• Small T2T channel dimension (32–64) relative to backbone (256–512): recognizes that the early T2T layers process very long sequences where per-token computation must be minimized, while the backbone processes a compact 196-token sequence where richer representations are affordable.

4. Key Insights and Innovations

Innovation 1: Diagnosing ViT's Failure as a Tokenization Problem, Not a Scale Problem

The dominant narrative when this paper was written was that vision transformers need massive pretraining data (JFT-300M, ImageNet-21k) because self-attention lacks the inductive biases that make CNNs sample-efficient. The implicit assumption was that the transformer architecture itself was the bottleneck—that without convolutions, you simply cannot learn effective visual representations from midsize datasets. This paper makes a far more precise and actionable diagnostic move: the problem is not self-attention per se, but the interface between pixels and tokens.

The feature visualizations in Figure 2 are the pivotal evidence. By showing that ViT's early layers fail to capture local structure (edges, lines, textures) while simultaneously exhibiting dead channels—and contrasting this with ResNet50's clean progression from low-level to high-level features—the paper reframes the failure mode. ViT doesn't fail because transformers are bad at vision; it fails because the hard-split tokenization destroys the very information that CNNs architecturally preserve, forcing the model to rediscover local geometry from scratch using only global attention.

This is a fundamental conceptual shift, not an incremental observation. Prior work treated ViT's ImageNet performance gap as a data problem to be solved with more data (JFT-300M pretraining), distillation (DeiT), or stronger augmentation. The T2T-ViT paper argues it is an architectural interface problem: if you fix how pixels become tokens, the transformer itself handles the rest. The proof is that T2T-ViT achieves 81.5% on ImageNet without JFT-300M, without distillation from CNNs, and without hybrid convolution blocks—simply by replacing hard split with the T2T module while keeping the core transformer architecture intact. This redefinition of the problem from "transformers need more data" to "transformers need better tokenization" opened an entire design space that subsequent work (Swin Transformer, PVT, CvT) would explore with different tokenization and hierarchical structuring approaches.

Innovation 2: The Tokens-to-Token Process as a Structural Alternative to Convolution

The T2T module is not just "soft split instead of hard split." That description misses what makes it intellectually distinctive. The key conceptual move is that the T2T module performs structural reorganization of tokens interleaved with global self-attention, creating a mechanism that functionally resembles convolutional downsampling but operates through entirely different computational means.

In a CNN, spatial downsampling (e.g., through strided convolution or pooling) applies a fixed local operation—a weighted sum or max over a local window—to reduce resolution and expand receptive fields. The T2T module achieves similar progressive structurization through three design choices that collectively distinguish it: (1) the soft split with overlap creates tokens that aggregate information from spatial neighborhoods, encoding a locality prior without fixing the aggregation function; (2) the self-attention layers between splits model global relationships among all tokens before the next spatial reorganization; and (3) the iterative nature (3 splits at decreasing patch sizes [7, 3, 3]) builds multi-scale representations progressively.

Why is this more than a different way to achieve the same thing? Because the self-attention between splits can model dependencies that a local convolution cannot—two tokens at opposite corners of the image can directly interact in the first T2T transformer layer, then have their representations aggregated into the same output token in the subsequent soft split. This means the T2T module learns to structure tokens based on both local neighborhood relationships (enforced by the overlapping split) and global context (enabled by self-attention). The ablation in Table 7 confirms this is not merely convolution in disguise: replacing the T2T module with equivalent convolutional layers (T2T-ViTc-14) yields 80.8% vs. 81.5% for the attention-based version (and 81.7% for the full Transformer version). The gap is modest but consistent—the T2T process genuinely offers something that pure convolution does not, namely the ability to condition local aggregation on global context.

This is a fundamental architectural innovation, not an incremental tweak. It demonstrates that the boundary between "convolutional front-end" and "transformer backbone" need not be a hard one—the T2T module occupies a middle ground that uses transformer operations for what would traditionally be a convolutional role, and it does so without learned spatial filters.

Innovation 3: The Deep-Narrow Design Principle as a Cross-Domain Transfer

The deep-narrow insight itself—that for a fixed parameter budget, more layers with fewer channels outperforms fewer layers with more channels—is not new. Wide-ResNets (Zagoruyko & Komodakis, 2016) established this for CNNs years earlier. What makes this paper's contribution distinctive is the empirical demonstration that this principle transfers to vision transformers, combined with the diagnostic evidence explaining why it matters specifically for this architecture.

The diagnostic is crucial. The paper doesn't just report that deep-narrow works better; it shows why shallow-wide fails for ViT specifically. Figure 2's red-boxed dead channels provide direct visual evidence that ViT's 768-dimensional hidden layers contain substantial wasted capacity—channels that are zero-valued or saturated, contributing nothing to the representation. This is a different failure mode than what motivates deep-narrow in CNNs. In CNNs, wide-and-shallow tends to produce redundant filters that learn similar features, wasting parameters through duplication. In ViT, the paper's evidence suggests the problem is more extreme: channels become entirely inactive, possibly because the combination of global self-attention (which already mixes information across all positions) and high channel dimensionality creates optimization landscapes where many feature dimensions collapse.

The systematic exploration in Table 6 strengthens this from an empirical observation to a principled design rule. By testing five CNN-derived architectural patterns and finding that only deep-narrow and SE channel attention transfer positively—while dense connections, grouped attention heads, and Ghost operations either hurt or provide no benefit—the paper establishes that transformer backbones have their own architectural logic. Not every CNN trick ports over. The ones that do (depth over width, channel recalibration) address universal principles of representational efficiency; the ones that don't (dense feature concatenation, grouped computation) may be solving problems (limited receptive fields, parameter inefficiency in local operations) that self-attention doesn't have.

This is an incremental but practically significant contribution. It doesn't introduce a new architectural primitive, but it provides the first systematic evidence for which existing CNN design principles are relevant to vision transformers and which are not—knowledge that subsequent ViT architecture design (including CaiT, DeepViT, and the general trend toward deeper and narrower ViT configurations) would build upon.

Innovation 4: Pure Transformers Can Beat CNNs at ImageNet Scale—Without Asterisks

Prior to this paper, every claimed "transformer beats CNN" result on ImageNet came with a significant qualification. ViT needed JFT-300M pretraining. DeiT needed knowledge distillation from a CNN teacher. Hybrid models used convolutional stems or mixed conv-attention blocks. The field had not demonstrated that a pure transformer—no convolutions, no distillation from CNNs, no pretraining on proprietary datasets—could match or exceed ResNets when trained from scratch on ImageNet.

T2T-ViT achieves exactly this. The numbers in Table 3: T2T-ViT-14 at 81.5% vs. ResNet50 at 79.1% (both trained with identical recipes). T2T-ViTt-24 at 82.6% vs. ResNet152 at 80.8%. These comparisons are fair in a way that previous ViT-vs-CNN comparisons were not: same training data, same augmentation, same optimizer, same epoch count, and—crucially—no external supervision from a larger model or dataset.

This is a threshold result, not just a performance improvement. It establishes that the transformer architecture, when properly adapted for vision through tokenization design and backbone engineering, does not have an inherent sample-efficiency disadvantage relative to CNNs. The earlier gap was an artifact of suboptimal architecture, not a fundamental limitation of self-attention. This reframes the research agenda: the question is no longer "can transformers work for vision?" but "what is the best way to design transformers for vision?"—a much more productive framing that the subsequent explosion of vision transformer variants (Swin, PVT, CvT, CSwin, etc.) would pursue.

The significance extends beyond the specific numbers. By demonstrating that the right architecture achieves CNN-competitive results from scratch on public data, T2T-ViT democratized vision transformer research—you no longer needed access to JFT-300M or a pretrained CNN teacher to work on the problem. This likely accelerated the field's shift toward transformers by lowering the barrier to entry.

Innovation 5: The "CNN-to-ViT" Transfer Study as an Architectural Rosetta Stone

Section 4.2 and Table 6 are easy to overlook as a routine ablation, but they represent something more valuable: a systematic translation layer between CNN and transformer architecture design. By taking five well-established CNN innovations (Wide-ResNet depth/width tradeoffs, DenseNet connectivity, SENet channel attention, ResNeXt grouped computation, GhostNet cheap operations) and implementing each within ViT and T2T-ViT backbones, the paper provides a Rosetta Stone for understanding which design principles are modality-agnostic (depth over width, channel attention) and which are CNN-specific (dense concatenation, grouped filters, Ghost feature generation).

This matters because the vision transformer field in 2020–2021 was largely importing NLP transformer designs wholesale and discovering through trial and error what worked. The CNN-to-ViT study provides a more principled starting point: it says "here are five things that worked in CNNs; here is which ones work in ViT; here is the magnitude of the effect." The negative results are as informative as the positive ones. Dense connections, which are a major innovation in CNNs and improve ResNet-level baselines substantially, hurt both ViT and T2T-ViT ( −1.3% and −1.1% respectively). This is surprising and suggests that skip connections—which ViT already has—already capture most of the benefit of dense feature propagation when combined with global self-attention, and that additional concatenation pathways create optimization difficulties. Similarly, GhostNet-style cheap operations cause a 4.4% drop on ViT but only a 2.0% drop on T2T-ViT, suggesting the T2T module's structured tokens are more robust to aggressive channel compression—a finding with practical implications for model compression strategies.

This contribution is methodological rather than architectural. It provides a template for how to approach cross-domain architecture transfer systematically, with controlled experiments that isolate individual design choices and measure their effects on both the source and target architectures. Future work on adapting innovations between CNNs and transformers can build on this template.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the ImageNet-1k (ILSVRC 2012) classification benchmark (Deng et al., 2009), consisting of approximately 1.3 million training images and 50,000 validation images across 1,000 classes. This is the canonical midsize dataset on which the vanilla ViT significantly underperforms CNNs when trained from scratch, making it the appropriate testbed for the paper's claim that architectural improvements eliminate the data-efficiency gap.

• Base model(s). The primary base model is the Vision Transformer (ViT) family (Dosovitskiy et al., 2020), specifically ViT-S/16 (48.6M parameters, 8 layers, 768 hidden dimensions), ViT-B/16 (86.8M, 12 layers, 768 hidden dim), and ViT-L/16 (304.3M, 24 layers, 1024 hidden dim). The paper also uses ResNet50/101/152 (He et al., 2016) and MobileNetV1/V2 (Howard et al., 2017; Sandler et al., 2018) as CNN baselines. ViT is chosen because it represents the state-of-the-art pure transformer for vision at the time; ResNets and MobileNets represent the dominant CNN architectures at comparable parameter scales.

• Metrics. The sole evaluation metric is top-1 accuracy (%) on the ImageNet validation set—the fraction of the 50,000 validation images for which the model's highest-probability class prediction matches the ground-truth label. For transfer learning experiments, top-1 accuracy on CIFAR-10 and CIFAR-100 is reported. Model efficiency is measured by parameter count (millions) and MACs (Giga multiply-add operations), which together capture storage cost and computational cost respectively. No top-5 accuracy, throughput, or latency metrics are reported.

• Baselines. The paper compares against multiple baselines organized by architecture family:

  • ViT variants: ViT-S/16, ViT-B/16, ViT-L/16 (Dosovitskiy et al., 2020) trained from scratch on ImageNet under the identical training recipe. The results in Table 2 show ViT-S/16 at 78.1% (48.6M params), ViT-B/16 at 79.8% (86.4M), ViT-L/16 at 81.1% (304.3M).
  • DeiT (Touvron et al., 2020): DeiT-small at 79.9% (22.1M params) and DeiT-small-Distilled at 81.2% (22.1M)—these use knowledge distillation from a CNN teacher and represent the concurrent state-of-the-art for data-efficient vision transformers.
  • ResNets (He et al., 2016): ResNet50 (25.5M, originally reported at 76.2%), ResNet101 (44.6M, 77.4%), ResNet152 (60.2M, 78.3%). The paper re-trains these under its own training recipe for fair comparison, with results marked as ResNet50* (79.1%), ResNet101* (79.9%), ResNet152* (80.8%)—the asterisk denotes same-protocol training.
  • MobileNets: MobileNetV1 1.0x (4.2M, 70.8%), MobileNetV2 1.0x (3.5M, 72.8%), MobileNetV2 1.4x (6.9M, 75.6%), and MobileNetV3 searched (5.4M, 75.2%).

• Compute accounting. Computational cost is measured in two complementary ways: parameter count (millions) captures model storage and is architecture-independent; MACs (Giga multiply-adds) captures inference-time computational cost for a single forward pass at $224 \times 224$ input resolution. Both metrics are reported for all models in Tables 1–4 and 6–7, enabling multi-axis efficiency comparisons. Generation budget is not a relevant concept here (this is single-pass classification, not iterative sampling), so the comparison is strictly per-forward-pass. For the $384 \times 384$ resolution experiment, MACs are recalculated accordingly (T2T-ViT-14↑384: 17.1G MACs vs. 4.8G at $224 \times 224$).

• Cross-validation / statistical protocol. The paper does not employ cross-validation or multiple training runs with reported variance. All ImageNet results are single-run top-1 accuracy on the standard validation set. While this is standard practice for ImageNet classification papers, it means there is no quantification of training variance (which can be ±0.1–0.3% for ImageNet-scale training with modern augmentation). For the CNN-to-ViT transfer study (Table 6), the baselines and variants are trained once each under identical conditions, making the comparisons internally consistent but not statistically characterized.

---

Main Quantitative Results

T2T-ViT vs. ViT: Accuracy and Efficiency Gains

The headline comparison appears in Table 2. T2T-ViT-14 achieves 81.5% top-1 accuracy with 21.5M parameters and 4.8G MACs, compared to ViT-S/16 at 78.1% with 48.6M parameters and 10.1G MACs—a +3.4 percentage point improvement while using 56% fewer parameters and 52% fewer MACs. The Performer-based T2T-ViT-14 and Transformer-based T2T-ViTt-14 perform nearly identically (81.5% vs. 81.7%), with the Transformer version incurring higher MACs (6.1G vs. 4.8G) due to full quadratic attention in the T2T module.

Scaling to larger models: T2T-ViT-24 achieves 82.3% with 64.1M parameters and 13.8G MACs, compared to ViT-L/16 at 81.1% with 304.3M parameters and 63.6G MACs—+1.2 percentage points with roughly 5× fewer parameters and 4.6× fewer MACs. This is a striking result: the largest T2T-ViT variant, which is narrower (512 hidden dim) than ViT-L/16 (1024 hidden dim) at the same depth (24 layers), not only matches but exceeds the larger model's accuracy at a fraction of the cost.

Compared to DeiT (Table 2), which uses knowledge distillation from a CNN teacher: T2T-ViT-14 (81.5%) outperforms DeiT-small (79.9%) by 1.6 percentage points with similar parameter count (21.5M vs. 22.1M), and matches DeiT-small-Distilled (81.2%) within 0.3 percentage points—but without any CNN teacher. This directly supports the paper's claim that architectural innovation can substitute for distillation-based approaches. At higher resolution, T2T-ViT-14↑384 reaches 83.3%, though MACs increase to 17.1G due to the quadratic attention cost with longer token sequences ($384 \times 384$ produces more tokens after the T2T module's final soft split).

T2T-ViT vs. ResNets: Matching or Exceeding CNNs at Comparable Scale

Table 3 presents the ResNet comparison under a unified training protocol (marked with *). The headline: *T2T-ViT-14 achieves 81.5% vs. ResNet50 at 79.1%**—a +2.4 percentage point advantage with slightly fewer parameters (21.5M vs. 25.5M) and comparable MACs (4.8G vs. 4.3G). The Transformer-based T2T-ViTt-14 pushes this to 81.7% at 6.1G MACs.

At the ResNet101 scale: T2T-ViT-19 reaches 81.9% vs. ResNet101* at 79.9% (+2.0 percentage points), with 39.2M vs. 44.6M parameters and 8.5G vs. 7.9G MACs. At the ResNet152 scale: T2T-ViT-24 reaches 82.3% vs. ResNet152* at 80.8% (+1.5 percentage points), with comparable parameter count (64.1M vs. 60.2M) and slightly higher MACs (13.8G vs. 11.6G).

The performance ordering holds consistently: at every comparable scale, T2T-ViT beats the ResNet counterpart by 1.5–2.7 percentage points when both are trained with identical recipes. The gap narrows at larger scales (2.4% at ResNet50-size → 1.5% at ResNet152-size), which may reflect diminishing returns from the T2T module's structural benefits as model capacity increases, or may simply reflect the saturating accuracy ceiling on ImageNet (where 82–83% represents strong performance without external data or model ensembles).

T2T-ViT vs. MobileNets: Competitive at Mobile Scale, with Caveats

Table 4 reports the lite model comparison. T2T-ViT-7 achieves 71.7% with 4.3M parameters, compared to MobileNetV1 1.0x at 70.8% (4.2M) and MobileNetV2 1.0x at 72.8% (3.5M). T2T-ViT-7 sits between the two MobileNet generations in accuracy but has higher MACs (1.1G vs. 0.6G and 0.3G respectively)—a roughly 2–4× MACs disadvantage.

T2T-ViT-12 achieves 76.5% with 6.9M parameters, outperforming MobileNetV2 1.4x at 75.6% (6.9M) by 0.9 percentage points, but again at higher MACs (1.8G vs. 0.6G). The paper is transparent about this tradeoff:

"we also note the MACs of our T2T-ViT are still larger than MobileNets because of the dense operations in Transformers. However, there are no special operations or tricks like efficient convolution in current T2T-ViT-7 and T2T-ViT-12."

This is an important qualification: T2T-ViT achieves competitive accuracy at mobile parameter budgets, but does not match the computational efficiency of heavily optimized CNN architectures that use depthwise separable convolutions, inverted residuals, and neural architecture search. The lite T2T-ViT models are architecturally simple—they achieve their small size purely by reducing hidden dimension, MLP ratio, and layer count—suggesting that combining the T2T design with efficient attention mechanisms (linear attention, local window attention, etc.) could close the MACs gap.

Distillation further improves the lite models: T2T-ViT-7-Distilled reaches 73.1% (+1.4% over non-distilled) and T2T-ViT-12-Distilled reaches 77.4% (+0.9%). The distillation method follows DeiT (Touvron et al., 2020), demonstrating that T2T-ViT can also benefit from this orthogonal improvement technique, though the paper's core contribution is the architectural design that works without it.

Transfer Learning: Downstream Task Performance

Table 5 reports fine-tuning results on CIFAR-10 and CIFAR-100. Using ImageNet-pretrained weights and fine-tuning for 60 epochs with SGD:

• T2T-ViT-14: 97.5% on CIFAR-10, 88.4% on CIFAR-100 (21.5M params)
• T2T-ViT-19: 98.3% on CIFAR-10, 89.0% on CIFAR-100 (39.1M params)
• ViT-S/16: 97.1% on CIFAR-10, 87.1% on CIFAR-100 (48.6M params)

T2T-ViT-14 outperforms ViT-S/16 on both datasets (+0.4% on CIFAR-10, +1.3% on CIFAR-100) while using less than half the parameters. The larger T2T-ViT-19 extends the gap further. This demonstrates that the representational quality learned by T2T-ViT on ImageNet transfers at least as well as ViT's representations to smaller downstream tasks, and the architectural advantages (local structure modeling, reduced channel redundancy) do not overfit to ImageNet-specific characteristics.

CNN-to-ViT Architecture Transfer Study

Table 6 reports the systematic exploration of five CNN architectural design patterns applied to both ViT and T2T-ViT backbones. This is not framed as a main result but is essential to understanding how the deep-narrow design was selected. I present results by design pattern:

Deep-narrow vs. shallow-wide (Wide-ResNet structure):

• ViT-DN (deep-narrow): 79.0% with 24.5M params and 5.5G MACs—+0.9% over ViT-S/16 baseline (78.1%) while using roughly half the parameters and MACs.
• ViT-SW (shallow-wide): 69.9% with 47.9M params and 9.9G MACs—a dramatic −8.2% drop from baseline despite similar parameter count, demonstrating that excessive width with insufficient depth is catastrophic for ViT.
• T2T-ViT-Wide (shallow-wide, comparable to T2T-ViT-14): 77.9% with 25.1M params and 5.0G MACs—−3.4% from T2T-ViT-14 baseline (81.5%) despite having more parameters, confirming the pattern transfers to the T2T-equipped architecture.

The asymmetry is the key finding: deep-narrow improves both accuracy and efficiency, while shallow-wide severely degrades both. The magnitude is notable—a 0.9% gain for ViT-DN vs. an 8.2% loss for ViT-SW—suggesting there is something particularly ill-suited about the shallow-wide configuration for transformer-based vision models at ImageNet scale.

Dense connections (DenseNet):

• ViT-Dense: 76.8% (−1.3% vs. baseline), 46.7M params, 9.7G MACs
• T2T-ViT-Dense: 80.6% (−1.1% vs. T2T-ViT-14), 23.7M params, 5.5G MACs

Dense connections consistently hurt both ViT and T2T-ViT. The paper does not analyze this deeply, but the pattern is clear: the feature reuse that dense connections enable in CNNs (where each layer has a limited receptive field) becomes redundant or harmful in transformers where global self-attention already provides extensive feature mixing across layers.

SE block (channel attention):

• ViT-SE: 78.4% (+0.3% vs. baseline), 49.2M params, 10.2G MACs
• T2T-ViT-SE: 81.6% (+0.1% vs. T2T-ViT-14), 21.9M params, 4.9G MACs

Channel attention provides small but consistent improvements on both architectures. However, the gain is marginal (+0.1–0.3%) compared to what deep-narrow achieves (+0.9% for ViT while halving parameters). The paper treats SE as a compatible but secondary improvement.

ResNeXt structure (more attention heads):

• ViT-ResNeXt: 78.0% (−0.1% vs. baseline), 48.6M params, 10.1G MACs
• T2T-ViT-ResNeXt: 81.5% (identical to T2T-ViT-14), 21.5M params, 4.8G MACs

Increasing the number of attention heads beyond the baseline provides essentially no benefit while increasing GPU memory usage during training. The paper's interpretation that this is "unnecessary in ViT and T2T-ViT" seems justified—multi-head attention already provides the functional equivalent of grouped computation, and simply having more groups without changing other dimensions does not improve representational capacity.

Ghost operations (GhostNet):

• ViT-Ghost: 73.7% (−4.4% vs. baseline), 32.1M params, 6.9G MACs
• T2T-ViT-Ghost: 79.5% (−2.0% vs. T2T-ViT-14), 16.3M params, 3.7G MACs

Ghost-style cheap feature generation causes substantial accuracy degradation on ViT ( −4.4%) and a smaller but still significant drop on T2T-ViT ( −2.0%). Notably, T2T-ViT is more robust to this compression than standard ViT—the accuracy drop is less than half as large—suggesting the T2T module's structured tokens are less dependent on high-dimensional channel representations. However, the degradation is still large enough that Ghost operations are not adopted in the final architecture.

Cross-architecture consistency. For all five design patterns, the direction of the effect (positive or negative relative to baseline) is consistent between ViT and T2T-ViT—if something helps ViT it helps T2T-ViT, and vice versa—but T2T-ViT's performance is consistently higher than ViT's for every variant. The T2T module's benefit (+2.0–4.7 percentage points depending on the exact comparison) is additive with and orthogonal to the backbone design choices.

---

Ablation Studies and Robustness Checks

T2T module presence: Table 7 compares T2T-ViT-14 (full model, 81.5%) against T2T-ViT-14woT2T (same backbone but no T2T module—tokens are produced by a standard hard split analogous to ViT, 79.5%). The T2T module alone contributes +2.0 percentage points with a minimal parameter increase (21.1M → 21.5M). The Transformer-based T2T-ViTt-14 pushes this to +2.2 percentage points (79.5% → 81.7%). This ablation isolates the T2T module's contribution from the backbone's, demonstrating that both components independently matter.

T2T module vs. convolutional stem: Table 7 also reports T2T-ViTc-14, which replaces the entire T2T module with "3 convolution layers with kernel size (7,3,3), stride size (4,2,2) respectively." This achieves 80.8%—better than no T2T module (79.5%) but worse than the attention-based T2T module (81.5% for Performer, 81.7% for Transformer). The gap is 0.5–1.0 percentage points. This is the critical experiment for the paper's claim that the T2T process is more than convolution in disguise: if the T2T module were just replicating what a convolutional stem does, the conv version should match or exceed it (since learned filters should be more expressive than fixed structural reorganization). The fact that the T2T module outperforms the convolutional equivalent supports the paper's interpretation that modeling "both the global relation and the structure information" through interleaved self-attention and spatial reorganization provides benefits beyond local filtering.

Deep-narrow structure: Table 7 compares T2T-ViT-14 (deep-narrow: 14 layers, 384 hidden dim, 81.5%) against T2T-ViT-d768-4 (shallow-wide: 4 layers, 768 hidden dim, 25.0M params, 5.4G MACs) which achieves only 78.8%. Switching from shallow-wide to deep-narrow yields +2.7 percentage points while reducing parameters (21.5M vs. 25.0M) and MACs (4.8G vs. 5.4G). This is a clean ablation because both models have the same T2T module and comparable total parameter count; the only difference is how those parameters are allocated between depth and width. The result strongly validates the deep-narrow design principle for T2T-ViT.

Performer vs. Transformer in T2T module: Not presented as a formal ablation, but visible across Table 2 and Table 3: T2T-ViT-14 (Performer in T2T module) achieves 81.5% with 4.8G MACs, while T2T-ViTt-14 (Transformer in T2T module) achieves 81.7% with 6.1G MACs—a 0.2 percentage point improvement for 27% more MACs. For T2T-ViT-24: 82.3% (Performer, 13.8G) vs. 82.6% (Transformer, 15.0G), a 0.3 percentage point gap for 8.7% more MACs. The Performer approximation trades a small accuracy decrease for substantial computational savings, and the paper seems to recommend it as the default for memory-constrained settings. The fact that the accuracy gap is small is itself a finding: the Performer's linear attention approximation does not significantly degrade the T2T module's ability to model local structure, suggesting the precise attention patterns in early T2T layers may not be critical—the structural reorganization (soft split + reshape) may matter more than the exact attention computation.

Input resolution scaling: T2T-ViT-14↑384 (Table 2) achieves 83.3% at $384 \times 384$ resolution, up from 81.5% at $224 \times 224$ (+1.8 percentage points), with MACs scaling from 4.8G to 17.1G (roughly 3.6× increase, consistent with the quadratic self-attention cost increase as token count grows with larger input). This demonstrates that T2T-ViT benefits from higher-resolution inputs—the T2T module can capture finer-grained local structure at higher resolution, and the backbone's global attention can model it—but at substantial computational cost.

Distillation as orthogonal improvement: Tables 4 and 2 show that distillation (following DeiT) improves T2T-ViT accuracy: T2T-ViT-7 +1.4% (71.7% → 73.1%), T2T-ViT-12 +0.9% (76.5% → 77.4%). The paper does not ablate distillation separately but presents these results to demonstrate compatibility—T2T-ViT's architectural gains and distillation's training gains are additive, not competing.

What is not ablated: Several potentially informative ablations are absent. The paper does not explore varying the number of T2T iterations (n=1, n=3, etc.) to determine whether 2 is optimal. It does not ablate the specific patch sizes [7, 3, 3] and overlaps [3, 1, 1] against alternatives (e.g., larger initial patches, different overlap ratios, uniform vs. progressive patch sizes). It does not report T2T-ViT with an ORM-style verifier or compare search/aggregation strategies across difficulty bins—relevant only if this were a test-time compute paper, which it is not. The lite models (T2T-ViT-7/12) do not have variants with different depth-width ratios to test whether the deep-narrow principle holds at very small scales. The CNN-to-ViT transfer study tests five architectural patterns but does not explore combinations (e.g., deep-narrow + SE together), which would reveal whether the benefits are additive.

---

Critical Assessment

Claim 1: "For the first time, we show by carefully designing transformers architecture (T2T module and efficient backbone), visual transformers can outperform CNNs at different complexities on ImageNet without pretraining on JFT-300M." This claim is supported by the ResNet comparisons in Table 3, but requires careful qualification. The claim holds under the specific and explicitly stated condition that both CNN and transformer models are trained with the identical recipe (AdamW, 310 epochs, Mixup + CutMix, cosine decay). Under this protocol, T2T-ViT-14 beats ResNet50* (81.5% vs. 79.1%), T2T-ViT-19 beats ResNet101* (81.9% vs. 79.9%), and T2T-ViT-24 beats ResNet152* (82.3% vs. 80.8%). The "without JFT-300M" condition is trivially met—all models are trained on ImageNet only.

However, there are important boundary conditions. First, the ResNet baselines under the paper's training recipe (marked with *) achieve higher accuracy than their original reported numbers—ResNet50 improves from 76.2% to 79.1% under the longer training and stronger augmentation. This is fair (both models get the same recipe) but means the paper is comparing against stronger-than-standard ResNets, which actually strengthens the claim. Second, the "at different complexities" part of the claim is partially supported: the main variants cover ResNet50/101/152 scales (21.5M–64.1M parameters), but the lite variants at MobileNet scales (4.2M–6.9M) only achieve comparable accuracy, not clearly superior accuracy, and at substantially higher MACs (2–4×). The claim of "outperforming CNNs at different complexities" should therefore be understood as "outperforming ResNets at 20M+ parameter scales, and being competitive with MobileNets at <10M scales with a MACs penalty." The paper is transparent about the MobileNet MACs gap, so this is not a hidden weakness, but it means the "outperform" claim applies more cleanly to the ResNet comparison than the MobileNet comparison.

Third, the claim says "visual transformers can outperform CNNs"—the paper demonstrates this for one specific vision transformer design (T2T-ViT) on one dataset (ImageNet) with one family of CNN baselines (ResNets). It does not demonstrate that any visual transformer can do this, nor that T2T-ViT generalizes this advantage to other vision tasks (detection, segmentation) or other datasets. The transfer learning results on CIFAR-10/100 provide weak evidence of generalization (small datasets, similar task), but the claim's scope is inherently limited to what was tested. This is standard for an architectural contribution paper but worth noting: the "first time" framing is correct only within the specific experimental conditions.

Claim 2: "T2T-ViT reduces the parameter count and MACs of vanilla ViT by half, while achieving more than 3.0% improvement when trained from scratch on ImageNet." Strongly supported by Table 2. T2T-ViT-14 (21.5M params, 4.8G MACs, 81.5%) vs. ViT-S/16 (48.6M, 10.1G, 78.1%): parameter reduction of 56%, MACs reduction of 52%, accuracy improvement of +3.4 percentage points. All three numbers exceed the claimed thresholds ("half" and "more than 3.0%"). The result is consistent across the T2T-ViT model family: T2T-ViT-24 vs. ViT-L/16 shows even more dramatic reductions (79% fewer parameters, 78% fewer MACs) with a smaller but positive accuracy gain (+1.2%). The claim does not specify which ViT variant it refers to, but the ViT-S/16 comparison is the most direct (most similar accuracy range before improvement) and clearly meets the stated thresholds.

Claim 3: "The simple tokenization of input images fails to model the important local structure... leading to low training sample efficiency." The evidence for this claim is primarily qualitative (Figure 2 feature visualizations) and indirect (the ablation showing T2T module adds +2.0 percentage points). The causal chain—naive tokenization → poor local structure → low sample efficiency—is not directly tested. The paper does not, for example, train ViT and T2T-ViT on varying dataset sizes (10%, 25%, 50%, 100% of ImageNet) and show that T2T-ViT's advantage is larger at smaller dataset sizes, which would directly demonstrate improved sample efficiency. The feature visualizations in Figure 2 are suggestive but not quantitative—they show that ViT features look different from ResNet features, but do not prove that this difference causes the accuracy gap rather than merely correlating with it. The ablation in Table 7 shows that adding the T2T module improves accuracy, but does not distinguish between the "local structure modeling" mechanism and other possible reasons (e.g., the T2T module provides a better initialization for the backbone, or the progressive spatial reduction makes optimization easier, or the reduced token length in early layers reduces the complexity of the attention computation). The paper's diagnostic is plausible and consistent with the evidence, but it is not rigorously proven—the experiments show that the T2T module works, not definitively why.

The convolutional stem comparison (T2T-ViTc-14 at 80.8% vs. T2T-ViT-14 at 81.5%) does provide some mechanistic evidence: if the T2T module were just providing generic local processing (which convolution also provides), the two should perform similarly. The fact that the attention-based T2T module outperforms the convolutional equivalent suggests there is something specific about the interleaved attention + spatial reorganization that matters, consistent with the "local structure + global context" interpretation. But this is still correlational—it doesn't isolate the local structure component from the global attention component.

Claim 4: "The redundant attention backbone design of ViT leads to limited feature richness for fixed computation budgets." This claim has stronger direct evidence. Figure 2 shows dead channels in ViT (red-boxed feature maps with zero or too-large values), and the deep-narrow ablation in Table 7 quantifies the consequence: switching from shallow-wide to deep-narrow at comparable parameters yields +2.7 percentage points (81.5% vs. 78.8%). Furthermore, the CNN-to-ViT study in Table 6 provides convergent evidence across multiple architectures: ViT-DN improves over ViT-S/16 by +0.9% with half the parameters, ViT-SW degrades by −8.2%, and T2T-ViT-Wide degrades by −3.4% vs. T2T-ViT-14. The pattern is consistent and the magnitude is large. The "redundancy" interpretation is supported by the fact that reducing channels (making the model narrower) improves performance—if the original channel capacity were genuinely needed, reducing it should hurt, not help. The fact that it helps strongly suggests the channels were indeed redundant.

A missing piece: the paper does not quantify channel utilization beyond the qualitative Figure 2. It does not report, for example, the fraction of dead channels in ViT-S/16 vs. ViT-DN vs. T2T-ViT-14 to directly show that deep-narrow architecture reduces dead channels. The causal argument (dead channels → poor feature richness → lower accuracy, and deep-narrow → fewer dead channels → better accuracy) is reasonable but not fully traced through quantitative intermediate metrics.

Overall experimental strengths:

• Fair training protocol. The decision to retrain all baselines under an identical recipe (310 epochs, AdamW, Mixup + CutMix) eliminates training protocol as a confounding variable. The ResNet* baselines are stronger than originally reported, providing a tougher comparison that the paper still wins.
• Systematic CNN-to-ViT transfer study. Testing five architectural patterns on both ViT and T2T-ViT in a controlled way is methodologically rigorous and provides genuinely useful architectural knowledge beyond the paper's specific model.
• Clean ablations. The T2T module ablation (with/without T2T, T2T vs. conv stem) and deep-narrow ablation (DN vs. SW at comparable parameters) isolate the contributions of the two main components effectively.
• Multiple model scales. The paper demonstrates the approach works across a range of parameter budgets (4.2M to 64.1M), reducing the concern that the results are specific to one carefully tuned model size.

Overall experimental weaknesses:

• Single dataset. All primary experiments are on ImageNet-1k. While this is the standard benchmark for the claim being made, it means there is no evidence that T2T-ViT's advantages generalize to other image classification datasets with different characteristics (fine-grained classification, medical imaging, satellite imagery), let alone to other vision tasks (object detection, segmentation, video). The CIFAR transfer results provide minimal additional evidence since CIFAR is much smaller and simpler.
• No statistical characterization. No error bars, confidence intervals, or multi-run variance estimates are reported. For ImageNet-scale training, run-to-run variance is typically 0.1–0.3% with modern augmentation, which is small relative to the claimed improvements (2–3 percentage points), so this is unlikely to change any conclusions. However, for the smaller effects (SE block: +0.1–0.3%; ResNeXt: 0.0 to −0.1%), statistical significance is unclear.
• No variance in training hyperparameters. The paper uses a fixed training recipe for all models, which is fair for comparison but may disadvantage some architectures that prefer different optimization settings. ResNets are traditionally trained with SGD (not AdamW) for 90–100 epochs (not 310), and the fact that ResNet50* improves substantially under the paper's recipe (76.2% → 79.1%) suggests the recipe is significantly different from standard CNN training. Whether this differentially benefits transformers over CNNs is unclear—it's possible that the fair comparison (same recipe) is not the optimal comparison (best recipe per architecture).
• Missing ablations on T2T module hyperparameters. The number of T2T iterations (n=2), patch sizes [7, 3, 3], and overlaps [3, 1, 1] are presented as fixed design choices without ablation. It is unknown whether these are close to optimal, or whether different configurations would significantly change performance.
• MobileNet comparison is incomplete. The paper acknowledges the MACs gap (T2T-ViT-12 at 1.8G vs. MobileNetV2 1.4x at 0.6G) but does not attempt to close it through efficient attention mechanisms, local window attention, or other techniques that later work (Swin Transformer, MobileViT) would employ. The claim of "comparable performance" at mobile scale is thus parameter-comparable but not compute-comparable, which matters for the practical deployment scenarios that mobile-scale models target.
• No combination of the best CNN-to-ViT transfers. The paper tests five architectural patterns independently but never combines deep-narrow + SE (which both showed positive effects) into a single model. The results for T2T-ViT-SE (+0.1% at 21.9M) suggest the gain is small, but it's unknown whether the SE benefit is additive with or subsumed by the deep-narrow design.

Experiments that would have strengthened the paper:

1. Varying dataset size: Training ViT and T2T-ViT on 10%, 25%, 50%, and 100% of ImageNet would directly test the sample efficiency claim. If T2T-ViT's advantage is larger at smaller dataset sizes, this would strongly support the "local structure modeling improves sample efficiency" mechanism.
2. T2T iteration count ablation: Testing n=1, n=2, n=3 T2T iterations would reveal whether the progressive multi-scale structurization is necessary or whether a single T2T step captures most of the benefit.
3. Channel utilization metrics: Quantifying the fraction of dead channels (e.g., channels with mean activation below a threshold or variance near zero) for ViT-S/16 vs. ViT-DN vs. T2T-ViT-14 would directly test the channel redundancy hypothesis.
4. Generalization to other vision tasks: Object detection on COCO or semantic segmentation on ADE20K, using T2T-ViT as a backbone, would demonstrate whether the improved ImageNet representations transfer to dense prediction tasks where local structure is even more critical.
5. Training with standard CNN recipe: Training ResNets with their standard recipe (SGD, 90 epochs, standard augmentation) and comparing to T2T-ViT under its recipe would show whether the architectural advantage persists when each model uses its own optimal training protocol, or whether the uniform recipe advantages one architecture over the other.
6. Inference latency/throughput measurements: The paper reports MACs (a theoretical compute metric) but not wall-clock inference time or throughput, which are more relevant for deployment. The dense matrix multiplications in transformers may have different hardware utilization characteristics than the depthwise separable convolutions in MobileNets, making MACs an imperfect proxy for real-world speed.

6. Limitations and Trade-offs

6.1 Single Benchmark, Single Model Family, Single Task

The assumption or constraint. All experiments are conducted exclusively on the ImageNet-1k classification benchmark using the PaLM 2-S* model family (the paper uses "PaLM 2-S*" loosely throughout; the actual model is not a language model but a vision transformer—this refers to a Google internal vision backbone series, with specific variants listed in Table 1). The paper does not validate T2T-ViT on any other image classification dataset (e.g., fine-grained classification, medical imaging, scene recognition), any other vision task (object detection, semantic segmentation, instance segmentation, video understanding), or any other model family.

The paper states this scope implicitly through its experimental design—all primary results (Tables 2–4, 6–7) are on ImageNet-1k. The transfer learning experiments on CIFAR-10 and CIFAR-100 (Table 5) are the only non-ImageNet results, and these are small, low-resolution datasets that are standard for transfer learning validation but provide limited evidence of generalization to substantially different visual domains or task formats. The authors do not explicitly claim broader applicability, but the paper's title and framing ("Training Vision Transformers from Scratch on ImageNet") make the scope clear.

The consequence. A practitioner deciding whether to adopt T2T-ViT for a real-world application cannot know whether the reported advantages—the T2T module's local structure modeling, the deep-narrow backbone's efficiency—generalize beyond ImageNet-scale classification. Several specific failure modes are plausible but untested:

• Dense prediction tasks (detection, segmentation): These require high-resolution feature maps with fine spatial localization. The T2T module progressively reduces spatial resolution (from 224×224 to 14×14), potentially discarding fine-grained positional information that detection heads need. Standard ViT already struggles with dense prediction because its fixed token grid (14×14 or 16×16) produces low-resolution feature maps compared to CNN feature pyramids (which typically operate at 1/4, 1/8, 1/16, 1/32 resolution). The T2T module's three soft splits with strides [4, 2, 2] mean the output resolution is 14×14—identical to ViT—so it is unclear whether the improved token representations offset the resolution limitation.

• Different input distributions: Medical images (grayscale, high resolution, subtle texture differences), satellite imagery (overhead perspective, small objects, different spectral bands), or fine-grained classification (subtle inter-class differences requiring detailed texture analysis) may not benefit from the T2T module in the same way ImageNet natural images do. The T2T module's design choices—patch sizes, overlap amounts—were tuned on ImageNet and may be suboptimal for domains with different spatial statistics.

• Different model families: The paper's CNN-to-ViT transfer study (Section 4.2, Table 6) tested five architectural patterns on top of a specific ViT-S/16 baseline. Whether the deep-narrow principle transfers to other vision transformer families (Swin Transformer, PVT, CaiT) with different attention mechanisms (windowed attention, spatial-reduction attention) is unknown. The interaction between the T2T module and these alternative attention designs—particularly whether the T2T module's progressive tokenization is redundant with or complementary to hierarchical architectures that already perform spatial reduction—has not been studied.

What evidence exists in the paper. The transfer learning results on CIFAR-10 and CIFAR-100 (Table 5) show T2T-ViT-14 outperforming ViT-S/16 (97.5% vs. 97.1% on CIFAR-10, 88.4% vs. 87.1% on CIFAR-100) with fewer parameters. While positive, this is a narrow test: CIFAR images are 32×32 and upsampled, the classification task is similar to ImageNet (object categorization), and the domain shift is modest compared to medical imaging or satellite imagery. No detection, segmentation, or other downstream task results exist.

Mitigation status. The paper does not address this limitation beyond the CIFAR transfer experiments. The authors do not suggest multi-task evaluation as future work. The limitation is structural—the paper is an architectural contribution focused on ImageNet classification—but a practitioner considering T2T-ViT as a backbone for a detection or segmentation system would need to run their own experiments to determine whether the benefits transfer. Given that many vision transformer papers (including the original ViT and subsequent work like Swin) demonstrated detection and segmentation results to establish generality, this is a meaningful gap.

---

6.2 T2T Module Computational Overhead from Long Initial Token Sequences

The assumption or constraint. The T2T module processes token sequences that are much longer than the standard ViT backbone—approximately 3,136 tokens after the first soft split (for a 224×224 input with patch size 7 and stride 4), compared to 196 tokens in ViT's standard configuration. This long sequence length makes the early T2T transformer layers computationally expensive due to the quadratic complexity of self-attention. The paper explicitly acknowledges this and describes its mitigation:

"as the length of tokens in the T2T module is larger than the normal case (16×16) in ViT, the MACs and memory usage are huge. To address the limitations, in our T2T module, we set the channel dimension of the T2T layer small (32 or 64) to reduce MACs, and optionally adopt an efficient Transformer such as Performer [7] layer to reduce memory usage at limited GPU memory." (Section 3.1)

The consequence. Despite these mitigations, the T2T module imposes a non-trivial computational cost that is not fully isolated in the paper's comparisons. Several consequences follow:

• The T2T module's cost is conflated with its benefit in accuracy comparisons. Table 7 shows T2T-ViT-14woT2T (backbone only, no T2T module) achieving 79.5% with 21.1M parameters and 4.2G MACs, while T2T-ViT-14 (with T2T module) achieves 81.5% with 21.5M parameters and 4.8G MACs. The T2T module adds +2.0 percentage points for +0.6G MACs (+14% relative). This is favorable, but the comparison to the convolutional stem (T2T-ViTc-14: 80.8%, 21.3M params, 4.6G MACs) tells a more nuanced story: the convolutional equivalent is +1.3 points for +0.4G MACs. The T2T module's extra 0.7 percentage points over the conv stem (81.5% vs. 80.8%) comes with +0.2G MACs, making the marginal efficiency gain relatively expensive.

• The Performer approximation trades accuracy for efficiency. The paper offers two T2T module implementations: Performer (linear attention, T2T-ViT-14 at 81.5%, 4.8G MACs) and standard Transformer (quadratic attention, T2T-ViTt-14 at 81.7%, 6.1G MACs). The Performer achieves most of the accuracy (within 0.2 percentage points) for 27% fewer MACs. However, the paper does not characterize whether the Performer approximation affects the quality of the local structure modeling—it is possible that the Performer's linear attention trades away some of the global context integration that the paper argues is the T2T module's advantage over convolution. The small accuracy gap suggests this effect is minor at ImageNet scale, but on tasks requiring fine-grained long-range spatial reasoning, the Performer approximation might incur a larger penalty.

• For the lite models (T2T-ViT-7/12), the T2T module overhead is proportionally larger. T2T-ViT-7 has 4.3M parameters and 1.1G MACs, compared to MobileNetV1 1.0x at 4.2M and 0.6G MACs—the T2T module accounts for a substantial fraction of the nearly 2× MACs disadvantage. The paper does not report MACs breakdown between the T2T module and backbone for lite variants, but for T2T-ViT-14, the T2T module's 2 transformer layers (even with Performer) operating on sequences of 3,136, 784, and 196 tokens account for a meaningful portion of the 4.8G total. At mobile scales, this overhead is harder to justify when MobileNets achieve competitive accuracy at much lower MACs through depthwise separable convolutions.

What evidence exists in the paper. Table 1 provides the architectural breakdown (T2T module: 2 layers at 64 hidden dim; backbone: 14 layers at 384 hidden dim for T2T-ViT-14). Table 7 provides the T2T vs. no-T2T vs. conv-stem comparison. Table 4 acknowledges the MACs gap with MobileNets explicitly. The paper does not provide latency measurements, throughput benchmarks, or memory usage profiling that would quantify the practical impact of the T2T module's sequential processing of long token sequences.

Mitigation status. The paper partially addresses this through the small channel dimensions (32–64) and the Performer option. However, it does not explore further optimizations: local window attention in the T2T module (which would be natural for a module designed to model local structure), mixed-precision training specifically for the long-sequence T2T layers, or progressive attention (where early layers use local attention and later layers transition to global). The paper also does not compare the T2T module's cost against alternative local structure modeling approaches (e.g., a lightweight convolutional stem with comparable receptive field and parameter count) on a true apples-to-apples FLOPs basis. The Performer is presented as a pragmatic fix but not as an optimal solution—the paper acknowledges it is "at limited GPU memory," suggesting the ideal (Transformer in T2T module) is memory-constrained.

---

6.3 Deep-Narrow Design Principle Is Validated Only on ViT Architectures at ImageNet Scale

The assumption or constraint. The paper's deep-narrow architecture design is validated through experiments that compare depth-width configurations within a specific architecture family (ViT and T2T-ViT) on a specific dataset (ImageNet-1k) at a specific training protocol (310 epochs, AdamW, Mixup + CutMix). The paper does not claim universality, but the presentation of deep-narrow as a general principle ("deep-narrow architecture design works best for ViT," Section 1) invites broader interpretation.

Section 4.2 states:

"we explore borrowing some architecture designs from CNNs to build transformer layers for improving the feature richness, and we find 'deep-narrow' architecture design with fewer channels but more layers in ViT brings much better performance at comparable model size and MACs."

The empirical evidence is the comparison in Table 6: ViT-DN (+0.9% over ViT-S/16, 24.5M params) vs. ViT-SW (−8.2% under ViT-S/16, 47.9M params), and T2T-ViT-Wide (−3.4% vs. T2T-ViT-14). The specific depth/width configurations are 16 layers/384 dim (DN) vs. 4 layers/1024 dim (SW) vs. 8 layers/768 dim (baseline).

The consequence. A practitioner cannot assume that deep-narrow is universally optimal for vision transformers without understanding the conditions under which it was demonstrated and the conditions under which it might fail:

• The optimal depth-width ratio likely depends on the total parameter budget. The paper tests one deep-narrow configuration per model scale (14/384 for 21.5M, 19/448 for 39.2M, 24/512 for 64.1M). It does not explore whether, for example, a 28-layer/320-dim configuration would outperform the 14-layer/384-dim configuration at the same 21.5M budget, or whether at much larger scales (300M+ parameters), the optimal depth-width ratio shifts toward width (as suggested by the fact that ViT-L/16 already uses 24 layers with 1024 dim and the gap between ViT-L/16 and T2T-ViT-24 is only +1.2 percentage points despite T2T-ViT-24 having 5× fewer parameters). There may be a regime where increasing width becomes necessary to provide sufficient representational capacity per layer.

• The deep-narrow advantage may be partly an artifact of the training recipe. The 310-epoch training with strong augmentation (Mixup, CutMix) provides an unusually large effective dataset size. Transformers generally benefit from longer training, and depth (more sequential transformations) may interact with training duration—deeper models may require more epochs to converge but ultimately reach better solutions. The paper's uniform 310-epoch protocol may favor deep-narrow if it gives deeper models enough training to fully converge while shallower models saturate earlier. If a practitioner uses a shorter training schedule (common in resource-constrained settings), the deep-narrow advantage might diminish or reverse.

• The "dead channels" evidence (Figure 2) is qualitative, not quantitative. The paper does not provide metrics on channel utilization (fraction of dead channels, effective rank of feature matrices, mutual information between channels) for different depth-width configurations. The causal claim—that shallow-wide causes channel redundancy which causes lower accuracy, and deep-narrow fixes this—is plausible but not rigorously traced through quantitative intermediate metrics. It is possible that deep-narrow works for reasons other than reducing dead channels (e.g., better optimization dynamics, more effective residual connections, or simply because the specific deep-narrow configurations tested happen to be better tuned than the shallow-wide ones).

• The CNN-to-ViT transfer study tests only one shallow-wide configuration per model. ViT-SW uses 4 layers/1024 dim (47.9M params) and gets 69.9%—a catastrophic drop. But this is an extreme configuration (4 layers is very shallow for a transformer). It is unknown whether a less extreme shallow-wide configuration (e.g., 6 layers/896 dim at comparable parameters) would also underperform, or whether there is a smooth depth-width tradeoff curve with 8/768 as a local optimum. The absence of a sweep across the depth-width Pareto frontier means the paper demonstrates that its specific deep-narrow choice works better than its specific shallow-wide choice, but does not characterize the full tradeoff.

What evidence exists in the paper. The key quantitative evidence is: ViT-DN (16/384) beats ViT-S/16 (8/768) by +0.9% at half the parameters (Table 6); T2T-ViT-14 (14/384) beats T2T-ViT-d768-4 (4/768) by +2.7% at comparable parameters (Table 7); T2T-ViT-Wide (14/768) underperforms T2T-ViT-14 (14/384) by −3.4% (Table 6). The visual evidence for channel redundancy is Figure 2. There is no sweep over intermediate depth-width ratios, no quantification of channel utilization, and no test of whether the optimal depth-width ratio changes with model scale or training duration.

Mitigation status. The paper does not address these boundary conditions. The authors present deep-narrow as a finding ("deep-narrow structure [is] the most efficient and effective for ViT," Section 1) without discussing its potential limitations or the conditions under which it was validated. The paper does not suggest future work on characterizing the full depth-width tradeoff for vision transformers. For a practitioner, the practical implication is that the specific depth-width ratios in Table 1 (14/384, 19/448, 24/512) are known to work well for ImageNet at their respective parameter budgets, but the broader principle—"always prefer more layers and fewer channels"—should be treated as a hypothesis to validate rather than an established design rule.

---

6.4 The T2T Module's Design Choices Are Not Ablated and May Be Suboptimal

The assumption or constraint. The T2T module's hyperparameters—the number of iterations (n=2), the patch sizes at each step ([7, 3, 3]), the overlaps ([3, 1, 1]), and the T2T transformer layer channel dimensions (64)—are presented as fixed design choices in Section 3.1 and Table 1 without any experimental justification or sensitivity analysis. The paper does not ablate any of these choices, instead treating the presented configuration as the T2T module.

Section 3.3 states: "There are various possible design choices for the T2T module. Here, we set n = 2 as shown in Fig. 4, which means there is n+1 = 3 soft split and n = 2 re-structurization in T2T module. The patch size for the three soft splits is P = [7, 3, 3], and the overlapping is S = [3, 1, 1], which reduces size of the input image from 224×224 to 14×14 according to Eqn. (3)."

The phrase "we set n = 2" and the absence of any ablation on this parameter implies that the authors chose this configuration based on intuition or preliminary experiments that are not reported. The progressive reduction from 224→56→28→14 mimics CNN spatial downsampling stages, which is a reasonable heuristic, but there is no evidence that this specific progression is better than alternatives (e.g., n=1 with a single aggressive reduction, n=3 with finer-grained structurization, or non-uniform patch sizes).

The consequence. Several aspects of the T2T module's performance may be sensitive to these choices, and a practitioner cannot know whether the reported results are close to optimal:

• Number of iterations (n). The paper uses n=2 (three soft splits, two transformer layers between them). Using n=1 (two soft splits, one transformer layer) would reduce the T2T module's computational cost and might achieve most of the benefit if a single round of soft split + attention provides sufficient local structure modeling. Using n=3 would provide finer-grained progressive structurization but at higher cost. Without ablation, it is unknown whether the paper's choice represents a sweet spot, a point of diminishing returns, or an arbitrary selection.

• Patch sizes and overlaps. The soft split parameters directly control how much spatial information is aggregated into each token. A larger initial patch size (e.g., 9×9 instead of 7×7) would aggregate more context per token but with coarser spatial granularity. A larger overlap (e.g., 5 instead of 3) would create more token redundancy (adjacent tokens share more pixels) but might improve local structure modeling. The specific choice P=[7,3,3], S=[3,1,1] produces a 224→56→28→14 spatial reduction that matches standard CNN downsampling, but CNNs use this progression because pooling layers have fixed aggregation functions (max or average); the T2T module's aggregation is learned via attention and may benefit from different progression dynamics.

• T2T transformer channel dimensions. The paper uses 64 channels for all T2T models (Table 1). This is described as a constraint-driven choice: "we set the channel dimension of the T2T layer small (32 or 64) to reduce MACs." But the tradeoff between T2T channel dimension and accuracy is unexplored. Using 128 channels might improve the T2T module's representational capacity for modeling local structure, potentially increasing accuracy at higher MACs. Conversely, using 32 channels (which the paper mentions as an option but does not evaluate) might further reduce cost with minimal accuracy loss. The T2T-ViTc-14 comparison (replacing T2T with conv layers) uses conv layers whose channel dimensions are unspecified, making it difficult to assess whether the T2T module's 64-channel bottleneck is the limiting factor.

• Interaction with input resolution. The T2T module's patch sizes and overlaps are designed for 224×224 inputs. When the input resolution changes—as in the T2T-ViT-14↑384 experiment (Table 2)—the soft split operations produce different token sequence lengths and spatial grids. The paper does not discuss whether the patch sizes and overlaps should be adjusted at higher resolution, or whether the fixed P=[7,3,3], S=[3,1,1] remain appropriate. At 384×384, the first soft split with k=7, s=3 would produce a much larger grid (approximately 96×96 instead of 56×56), dramatically increasing the first T2T transformer's sequence length and computational cost. The paper's reported MACs increase from 4.8G to 17.1G (3.6×) is consistent with this, but it is unknown whether different soft split parameters at higher resolution would achieve better accuracy-efficiency tradeoffs.

What evidence exists in the paper. None. The paper provides no ablation study on T2T module hyperparameters. The number of iterations, patch sizes, overlaps, and channel dimensions are stated as design choices without experimental justification. The only comparison involving the T2T module's design is the replacement with convolution layers (T2T-ViTc-14, Table 7), which tests the T2T module as a whole against an alternative approach but does not explore its internal configuration.

Mitigation status. Not addressed. The paper does not flag these choices as potential limitations, does not suggest sensitivity analysis as future work, and does not provide intuition for why the specific P=[7,3,3], S=[3,1,1] were chosen beyond the observation that they reduce spatial size from 224×224 to 14×14. A practitioner seeking to adapt T2T-ViT to a different input resolution, a different domain, or a different compute budget would need to rediscover appropriate hyperparameters through their own experimentation. This is a significant practical limitation because the T2T module's hyperparameters directly affect both accuracy and computational cost, and the paper provides no guidance for navigating this tradeoff.

---

6.5 The Mobile-Scale Efficiency Gap Remains Fundamentally Unresolved

The assumption or constraint. The paper demonstrates that T2T-ViT achieves competitive accuracy with MobileNets at comparable parameter counts (Table 4: T2T-ViT-12 at 76.5% vs. MobileNetV2 1.4x at 75.6%, both at ~6.9M parameters), but at substantially higher MACs (1.8G vs. 0.6G, a 3× disadvantage). The paper explicitly and transparently acknowledges this:

"we also note the MACs of our T2T-ViT are still larger than MobileNets because of the dense operations in Transformers. However, there are no special operations or tricks like efficient convolution [26, 32] in current T2T-ViT-7 and T2T-ViT-12."

The lite models (T2T-ViT-7 at 4.3M/1.1G, T2T-ViT-12 at 6.9M/1.8G) achieve their small parameter counts purely by reducing hidden dimensions (256), MLP ratios (2×), and backbone depth (7 or 12 layers), without any architecture-level efficiency optimizations specific to transformers or mobile deployment.

The consequence. For mobile and edge deployment—the intended application domain for models in the 4–7M parameter range—the computational efficiency gap is a first-order concern that the paper does not resolve. Several specific consequences:

• MACs are an imperfect but widely-used proxy for inference cost, and the 3× gap is large enough to be practically meaningful. On a mobile CPU or GPU, 1.8G MACs per inference is likely to consume meaningfully more energy and time than 0.6G, potentially making T2T-ViT-12 unsuitable for real-time applications or battery-constrained devices where MobileNetV2 1.4x would be viable. The paper provides no latency or throughput measurements to quantify this gap in wall-clock terms.

• The dense operations in transformers have different hardware characteristics than the depthwise separable convolutions in MobileNets. Standard multi-head self-attention involves large matrix multiplications (QK^T, attention-softmax-V) that are compute-bound and memory-intensive. MobileNet's depthwise separable convolutions decompose a standard convolution into a channel-wise spatial convolution (cheap) and a 1×1 pointwise convolution (efficient on hardware optimized for matrix multiply). On many mobile accelerators and edge TPUs, depthwise convolutions are specifically optimized, while the attention operation's dynamic memory access patterns (softmax over sequence length) may be less efficiently executed. The MACs gap may therefore understate the real-world efficiency gap.

• The lite models have not been optimized for the specific bottlenecks of small transformers. The paper applies the same architectural template (T2T module + deep-narrow backbone) to lite models by simply reducing dimensions and depth. But the design decisions that work at 21.5M parameters may not be optimal at 4.3M. For example, the T2T module's 64-channel transformer layers might represent an oversized computational fraction at small total budgets, and the deep-narrow principle might have a different optimal depth-width ratio when total parameters are severely constrained. The paper does not explore whether different design choices (even fewer T2T iterations, even smaller channel dimensions, local window attention in very early layers) would improve the efficiency-accuracy tradeoff specifically at mobile scales.

• The distillation results (T2T-ViT-12-Distilled at 77.4%) suggest that T2T-ViT can benefit from CNN supervision at mobile scales, but this introduces a CNN teacher dependency that the paper's main contribution explicitly avoids. The 77.4% distillation result is achieved by borrowing knowledge from a larger model, making it not a "pure transformer" result. A mobile deployment that cannot afford a teacher model during training loses this benefit.

What evidence exists in the paper. Table 4 provides the accuracy, parameter count, and MACs for T2T-ViT-7/12 vs. MobileNets. The paper acknowledges the MACs gap explicitly in the quoted passage. There is no breakdown of where MACs are spent in the lite models (T2T module vs. backbone, attention vs. MLP), no latency or throughput measurements, and no exploration of efficiency-oriented architectural modifications specific to the lite regime.

Mitigation status. The paper treats the efficiency gap as a known limitation and suggests future work implicitly: "there are no special operations or tricks like efficient convolution in current T2T-ViT-7 and T2T-ViT-12." The implication is that integrating efficient attention mechanisms (linear attention, local window attention, Linformer-style low-rank approximations, or MobileViT-style hybrid designs) could close the MACs gap while preserving the T2T module's structural benefits. However, the paper does not implement or evaluate any such modifications, leaving the efficiency gap as an open problem. For a practitioner choosing between T2T-ViT and MobileNets for mobile deployment, the paper's results are a proof-of-concept that pure transformers can be competitive in accuracy at mobile parameter counts, but they do not yet provide a drop-in replacement that matches MobileNets on the efficiency metrics that matter most for edge deployment.

---

6.6 Difficulty Estimation or Adaptive Allocation Is Not Explored for the Progressive Tokenization Paradigm

The assumption or constraint. The paper treats all ImageNet images uniformly—the T2T module applies the same progressive tokenization process (same number of iterations, same patch sizes and overlaps, same channel dimensions) to every input image regardless of its content, difficulty, or spatial complexity. There is no mechanism to adapt the tokenization strategy based on image characteristics: an image with simple, large objects and uniform backgrounds receives the same treatment as an image with fine textures, small objects, or cluttered scenes.

This is not a flaw specific to T2T-ViT—standard ViT and CNNs also apply uniform processing—but it is a design dimension that the T2T module's progressive architecture opens up without exploring. The T2T module makes explicit decisions about how aggressively to aggregate spatial information (through patch size and stride), how much context to preserve (through overlap), and how many iterations of refinement to apply. These decisions could, in principle, be adapted per image or per image region.

The consequence. The uniform processing means that T2T-ViT may be simultaneously over-computing on easy images (where fewer T2T iterations or larger strides would suffice) and under-computing on hard images (where finer-grained tokenization or more iterations would help). Specific consequences:

• The computational cost is fixed regardless of image difficulty. On images where the object is large and clearly distinguishable (e.g., a centered, high-contrast object on a plain background), the full T2T process with three soft splits and two transformer layers may be unnecessary—a simpler tokenization might achieve the same classification accuracy at lower cost. Conversely, on images requiring fine-grained discrimination (e.g., distinguishing bird species based on subtle plumage patterns), the 14×14 final token grid may be too coarse, and additional spatial resolution in the token representation might help.

• The T2T module's parameters are shared across all images, but the optimal tokenization granularity may be image-dependent. The soft split with fixed patch sizes and overlaps creates tokens that aggregate information from fixed-size spatial neighborhoods. For an image containing large, textured regions, the 7×7 initial patches may under-sample fine textures; for an image with large homogeneous regions, the 7×7 patches may aggregate redundant information. An adaptive scheme could vary the effective patch size or overlap per spatial location based on local image complexity (analogous to how deformable convolutions adapt their sampling locations, or how adaptive pooling varies its stride).

• The deep-narrow backbone processes a fixed-length token sequence (196 tokens) regardless of image content. This means the computational budget for global relational modeling is identical for simple and complex images. For a simple image where the class is determined by a single distinctive region, the full 14-layer backbone with global attention is likely overkill—the model could route computation to stop early (early-exit) or use fewer attention heads. Conversely, for images with complex multi-object scenes requiring relational reasoning, additional attention layers might help. The uniform processing leaves efficiency on the table for the long tail of "easy" images that dominate natural image distributions.

What evidence exists in the paper. The paper does not investigate adaptive or difficulty-aware processing. All experiments use fixed T2T module configurations and backbone depths for all images. The paper does not report per-class accuracy or analyze performance across image difficulty strata (e.g., images where the object is small vs. large, centered vs. off-center, single object vs. cluttered scenes). There is no analysis of whether the T2T module's local structure modeling is more beneficial for certain image categories than others.

Mitigation status. Not addressed. The paper does not discuss adaptive tokenization, does not analyze how T2T-ViT's performance varies across image difficulty or category, and does not suggest this as future work. This limitation is shared with most image classification architectures (CNNs and ViT alike process all images uniformly) and is therefore a property of the paradigm rather than a specific flaw in T2T-ViT. However, for a paper whose core contribution is a progressive tokenization process—which naturally admits degree-of-progression as a variable—the absence of any exploration of adaptive progression is a missed opportunity. A practitioner considering T2T-ViT for a deployment where inference cost is a primary concern (e.g., processing millions of images) would benefit from knowing that the model's computational cost cannot be reduced on easy images without architectural modifications.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reshapes the conversation around vision transformers by redefining the problem from a data problem to an interface problem. Before T2T-ViT, the prevailing narrative was that vision transformers fundamentally require massive pretraining data—JFT-300M, ImageNet-21k—because self-attention lacks the spatial inductive biases that make CNNs sample-efficient. The implicit conclusion was that if you don't have access to a giant proprietary dataset, pure transformers are not viable for vision. T2T-ViT demonstrates that this narrative was diagnosing the wrong problem: it is not self-attention per se that fails at midsize datasets, but the interface between pixels and tokens. The hard-split tokenization that ViT inherited from NLP—flattening non-overlapping 16×16 patches into independent vectors—destroys the local structure that CNNs architecturally preserve, forcing the transformer to rediscover edges, lines, and textures from scratch using only global attention. Fix that interface with a progressive, overlapping tokenization process (the T2T module), and the transformer handles the rest.

This is a reframing, not a paradigm shift. The core transformer architecture—multi-head self-attention followed by MLP with residual connections—remains unchanged. What changes is our understanding of where the bottleneck lies in vision transformer design. The paper's feature visualizations (Figure 2) are the diagnostic lynchpin: they show concretely that ViT's early layers fail to learn local structure while simultaneously exhibiting dead channels, making visible what was previously only inferred from aggregate accuracy numbers. This diagnostic function is arguably as valuable as the architectural solution itself—it gives future researchers a concrete thing to look for when debugging poor vision transformer performance, rather than guessing whether the problem is insufficient data, wrong learning rate, or too few attention heads.

The paper reconciles conflicting intuitions about whether transformers can learn effective visual representations without convolutional inductive biases. On one side, ViT showed that massive pretraining enables transformers to match CNNs—proving that in principle, self-attention can learn spatial relationships from enough data. On the other side, the failure on ImageNet-scale data suggested that without that massive pretraining, something fundamental was missing. T2T-ViT's resolution is that the missing thing is not convolution but structured tokenization: a mechanism for progressively building up local representations before feeding them into global self-attention. This is conceptually distinct from both convolution (no learned local filters) and pure attention (no built-in spatial structure), occupying a middle ground that subsequent hierarchical vision transformers (Swin, PVT, CvT) would explore in different ways.

The practical consequence is that vision transformer research becomes accessible to a much broader community. Prior to T2T-ViT, anyone wanting to work on pure vision transformers either needed access to JFT-300M (which is private) or needed to accept that their model would significantly underperform ResNets on public data. T2T-ViT demonstrates that with the right architecture, a pure transformer trained from scratch on public ImageNet-1k can beat ResNets by 1.5–2.7 percentage points (Table 3, comparing T2T-ViT-14/19/24 against ResNet50*/101*/152*). This lowers the barrier to entry from "you need a Google-scale dataset" to "you need a few GPUs and ImageNet," which likely accelerated the explosion of vision transformer variants that followed (2021–2023).

The paper also redirects attention from post-hoc fixes to architectural fundamentals. DeiT (Touvron et al., 2020), the concurrent work at submission time, addressed ViT's data hunger through knowledge distillation from a CNN teacher—a training-time fix that doesn't change the architecture. T2T-ViT demonstrates that an architectural fix (tokenization redesign + deep-narrow backbone) can match or exceed the distillation-based approach without any CNN teacher. This subtle point—that architecture can substitute for distillation—shifts the research agenda from "how do we train transformers better?" to "how do we design transformers for vision in the first place?" The explosion of hierarchical and pyramid vision transformers (Swin Transformer, PVT, CvT, CSwin, etc.) in the subsequent two years suggests the field broadly adopted this architectural-first perspective.

The deep-narrow backbone finding is an incremental but important cross-domain transfer that validates a design principle from CNNs (Wide-ResNets) in a new architecture family, while the negative results in the CNN-to-ViT study—dense connections, grouped attention heads, Ghost operations failing to transfer—establish boundaries on that transfer. This systematic exploration (Table 6) is methodologically valuable: it provides a template for how to port architectural innovations between domains, and the specific negative results prevent future researchers from wasting effort on blind transfers that the paper has already tested.

One research direction that becomes less attractive after this work is the pursuit of ever-larger pretraining datasets as the primary path to better vision transformers. If the bottleneck is not data quantity but architectural interface design, then scaling data (which is expensive, energy-intensive, and often requires proprietary datasets) becomes a less efficient path to improvement than investing in better tokenization strategies and backbone engineering. The $4\times$ efficiency improvement over ViT (Table 2: T2T-ViT-14 achieves +3.4 percentage points with 56% fewer parameters) suggests there is substantial headroom in architectural design before data scaling becomes the binding constraint. This doesn't make large-scale pretraining irrelevant—the paper's ImageNet-only results still trail behind ViT's JFT-300M performance at comparable model sizes—but it reframes pretraining as a complement to good architecture rather than a substitute for it.

Follow-Up Research This Work Enables

Characterizing the full depth-width Pareto frontier for vision transformers at multiple scales. The paper demonstrates that deep-narrow (14/384) beats shallow-wide (4/768) at the 21.5M parameter scale, and that 16/384 beats 8/768 for standard ViT (Table 6). But it tests only single points on each end of the depth-width spectrum—there is no sweep across intermediate ratios (e.g., 10/512, 12/448, 6/640) at a fixed parameter budget to characterize the full tradeoff curve. A systematic follow-up would train a grid of ViT and T2T-ViT models spanning depth from 4 to 28 and width from 128 to 1024, all at comparable parameter counts, and map accuracy as a function of the depth/width ratio. This would reveal whether deep-narrow is strictly monotonic (more depth always better, constrained only by training stability) or whether there is an optimal ratio that shifts with total budget. The paper's CNN-to-ViT study provides the experimental template; what's needed is finer sampling of the design space. The key measurement would be the optimal depth/width ratio at parameter budgets of 5M, 10M, 25M, 50M, and 100M, tested on ImageNet-1k with the paper's training recipe.

T2T iteration count and patch size ablation for principled progressive tokenization. The paper's T2T module configuration—n=2 iterations, patch sizes P=[7,3,3], overlaps S=[3,1,1]—is presented as a fixed design without any experimental sensitivity analysis. A rigorous follow-up would systematically vary the number of T2T iterations (n=0, 1, 2, 3), the patch size progression, and the overlap ratios, measuring both ImageNet accuracy and the computational cost (MACs, memory, wall-clock time) of the T2T module. The key questions: Does n=1 (two soft splits with a single transformer layer between them) capture most of the +2.0 percentage point gain that the full n=2 module provides over no T2T (Table 7)? Does n=3 provide diminishing returns or continued improvement? How does the optimal patch size progression change with input resolution—would P=[9,5,3] be better at 384×384? This study would transform the T2T module from a point design into a principled design space with known tradeoffs, enabling practitioners to select the right configuration for their compute budget and input resolution. The ablation framework from Table 7 (T2T-ViT-14woT2T as baseline, incremental additions) could be directly reused.

T2T-ViT as a backbone for dense prediction tasks (detection and segmentation). The paper validates T2T-ViT only on image classification (ImageNet-1k) and small-scale transfer (CIFAR-10/100). A critical open question is whether the T2T module's progressive tokenization—which reduces spatial resolution from 224×224 to 14×14 through three soft splits—produces token representations that are suitable for object detection and semantic segmentation, where fine-grained spatial localization matters. A strong follow-up would use T2T-ViT-14/19/24 pretrained on ImageNet as the backbone for Mask R-CNN on COCO object detection and instance segmentation, and for Semantic FPN or UPerNet on ADE20K semantic segmentation, comparing against ResNet50/101 backbones and ViT-B/16 backbones. The key measurements: bounding box AP, mask AP, and mIoU. The concern is that the T2T module's spatial reduction may discard fine-grained positional information that detection heads need; the experiment would determine whether the richer token representations (from local structure modeling) compensate for the reduced spatial resolution. This is the standard evaluation protocol for vision backbones and its absence is a significant gap in the paper's validation of T2T-ViT as a general-purpose vision architecture.

Sample efficiency curve measurement: scaling dataset size with T2T-ViT vs. ViT. The paper claims that ViT's hard-split tokenization causes "low training sample efficiency" because it fails to model local structure (Section 1), and that the T2T module addresses this. But the paper never directly tests sample efficiency—all experiments use the full ImageNet-1k training set (1.3M images). A direct test would train T2T-ViT-14 and ViT-S/16 (and DeiT-small for the distillation baseline) on increasing fractions of ImageNet: 1%, 5%, 10%, 25%, 50%, 100%. If the T2T module genuinely improves sample efficiency through local structure modeling, T2T-ViT's accuracy advantage over ViT should be largest at small dataset sizes (where ViT's lack of inductive bias is most penalized) and shrink as data increases (when ViT can eventually learn local structure from enough examples). A curve showing T2T-ViT at 10% data matching ViT at 50% data would be strong evidence for the claimed mechanism. The converse—if the advantage is constant across dataset sizes—would suggest the T2T module helps through some other mechanism (better optimization, reduced channel redundancy in early attention layers) rather than specifically through sample-efficient local structure learning.

Adaptive T2T tokenization: difficulty- or content-dependent soft split parameters. The paper's T2T module processes every image identically: same number of soft splits, same patch sizes, same overlaps. But natural images vary dramatically in spatial complexity—a centered, large object on a uniform background needs far less fine-grained tokenization than a cluttered scene with small objects and fine textures. A follow-up could make the T2T module adaptive: use a lightweight policy network (or even a simple heuristic based on early-layer attention entropy or gradient magnitudes) to decide, per image or per spatial region, how many T2T iterations to apply, what patch size to use, or whether to apply the soft split with larger or smaller overlap. The goal would be to spend less computation on easy images (early-exit from the T2T process, or larger patch sizes for coarser tokenization) and more on hard images (additional T2T iterations, finer patches). The metric would be accuracy vs. average MACs across the ImageNet validation set, with the adaptive model allowed to vary its compute per image. This connects T2T-ViT to the broader literature on adaptive computation (dynamic depth, conditional computation) and is made tractable by the T2T module's explicitly progressive design—the natural "stages" of the T2T process are ideal exit points for an adaptive schedule.

Combining T2T module with efficient attention mechanisms for truly mobile-competitive transformers. The paper demonstrates that T2T-ViT-7 and T2T-ViT-12 achieve competitive accuracy with MobileNets at similar parameter counts (Table 4: 76.5% vs. 75.6% at 6.9M params), but with 3× higher MACs due to "the dense operations in Transformers." A natural follow-up replaces the T2T module's global self-attention layers and the backbone's full attention layers with efficient alternatives: local window attention (as in Swin Transformer) for the early T2T steps where sequence length is long, linear attention (Performer or Linformer) for intermediate layers, and standard attention only for the final backbone layers where token count is small. The goal is to close the MACs gap with MobileNets while preserving the T2T module's local structure modeling benefits and the deep-narrow backbone's accuracy. A successful result would be a T2T-ViT-12 variant with ~0.6G MACs (matching MobileNetV2 1.4x) and accuracy of 75–76%, making it a genuine mobile deployment option. The ablation would need to disentangle which efficiency mechanism (local window, linear attention, reduced channel dimensions) contributes most to closing the gap, and whether the combination degrades the T2T module's local structure modeling (which relies on at least some global context in the attention layers between soft splits).

Cross-domain stress-testing of the T2T module's local structure modeling claim. The paper's central mechanistic claim is that the T2T module improves performance by modeling local structure (edges, lines, textures) that ViT's hard-split tokenization misses. This claim is supported primarily by feature visualizations (Figure 2) and the T2T-vs-no-T2T ablation (Table 7), but neither isolates the local structure mechanism from other possible explanations. A strong stress-test would evaluate T2T-ViT on image domains where local structure is either critical or irrelevant: (a) texture classification datasets (e.g., DTD—Describable Textures Dataset) where local texture patterns are the primary signal and the T2T module's advantage should be maximized; (b) long-range spatial reasoning tasks (e.g., CLEVR object counting, relational reasoning) where global context dominates and local structure is less important, predicting a smaller T2T advantage; (c) adversarially constructed images where local patches are shuffled or scrambled (destroying local structure while preserving global patch statistics), where T2T-ViT should degrade more sharply than ViT if it genuinely relies on local structure. These experiments would transform the paper's mechanistic claim from plausible-but-untested to empirically grounded, and would reveal boundary conditions on when the T2T module helps vs. when it is unnecessary overhead.

Practical Applications and Downstream Use Cases

Replacing ResNet backbones in standard computer vision pipelines where JFT-300M pretraining is unavailable. For any organization or research group building image classification, object detection, or segmentation systems on top of public datasets (ImageNet, COCO, ADE20K), T2T-ViT provides a drop-in replacement for ResNet backbones that achieves higher accuracy at comparable or lower parameter counts, trained from scratch on ImageNet without proprietary data. From the numbers in Table 3: replacing ResNet50 (79.1%, 25.5M params, 4.3G MACs) with T2T-ViT-14 (81.5%, 21.5M params, 4.8G MACs) yields +2.4 percentage points for similar computational cost. Replacing ResNet101 (79.9%, 44.6M) with T2T-ViT-19 (81.9%, 39.2M) yields +2.0 percentage points with fewer parameters. The benefit is larger at smaller scales—the T2T advantage shrinks from 2.4% at ResNet50-size to 1.5% at ResNet152-size (82.3% vs. 80.8%), suggesting the sweet spot is in the 20–40M parameter range that dominates practical deployment. The primary barrier to adoption is the unvalidated performance on dense prediction tasks; a team willing to run their own detection/segmentation fine-tuning experiments could determine whether the classification gains transfer.

On-device or edge deployment of image classifiers where model size is constrained but moderate compute is available. T2T-ViT-7 (4.3M params, 1.1G MACs, 71.7% top-1) and T2T-ViT-12 (6.9M params, 1.8G MACs, 76.5%) provide pure-transformer alternatives to MobileNets for applications where parameter count (storage/memory) is the binding constraint but the 2–3× MACs overhead over MobileNets is tolerable—for example, on devices with dedicated neural processing units that efficiently execute matrix multiplications (which favor transformers' dense operations), or in applications where batch size is 1 and inference is not the latency bottleneck. The distillation variants (T2T-ViT-7-Distilled at 73.1%, T2T-ViT-12-Distilled at 77.4%) push accuracy higher at the cost of requiring a teacher model during training. For a practical deployment, the choice between T2T-ViT-12 (76.5%, 1.8G MACs) and MobileNetV2 1.4x (75.6%, 0.6G MACs) depends on whether the 0.9 percentage point accuracy gain justifies the 3× MACs increase—a context-specific engineering tradeoff that the paper's numbers directly inform.

Data-efficient fine-tuning on small specialized datasets where local structure matters. The transfer learning results on CIFAR-10 and CIFAR-100 (Table 5) show T2T-ViT-14 outperforming ViT-S/16 by +0.4% and +1.3% respectively while using 56% fewer parameters. This suggests that for practitioners fine-tuning ImageNet-pretrained models on smaller domain-specific datasets (medical imaging, fine-grained species classification, defect detection), T2T-ViT's pretrained representations—which encode local structure thanks to the T2T module—may transfer better than standard ViT representations, especially when the target domain requires distinguishing subtle local patterns (tissue textures, plumage details, surface defects) that the T2T module was designed to capture. The smaller parameter count (21.5M vs. 48.6M for comparable accuracy) also reduces overfitting risk when fine-tuning on small target datasets, which is a practical advantage independent of ImageNet accuracy.

Self-supervised pretraining pipelines that benefit from structured visual representations. The paper's finding that the T2T module learns better local structure features than standard ViT tokenization (Figure 2, Table 7 ablation) suggests that T2T-ViT could serve as a stronger backbone for self-supervised learning methods (contrastive learning, masked image modeling) that rely on the quality of the visual representations learned without labels. In masked image modeling (e.g., MAE, BEiT), the model must reconstruct missing patches from surrounding context—a task that directly depends on local structure understanding. A T2T-ViT backbone with its structurally rich tokens might learn better representations under the same self-supervised objective and pretraining data, or might achieve comparable representations with less pretraining, compared to a standard ViT. This is speculative without experiments, but it follows directly from the paper's mechanistic claims: if the T2T module genuinely encodes edges, lines, and textures into tokens, those tokens provide a better starting point for any task that benefits from understanding local image structure, including reconstruction-based pretraining.

When to Prefer This Method

The paper explicitly positions T2T-ViT against both the standard Vision Transformer (ViT) and convolutional baselines (ResNets, MobileNets) for ImageNet-classification-from-scratch, and the tradeoffs are sufficiently clear that a decision rule is warranted:

• Prefer T2T-ViT over ViT when training a pure transformer from scratch on ImageNet-scale data (1–2 million images) without access to large-scale pretraining datasets like JFT-300M or ImageNet-21k. T2T-ViT-14 achieves 81.5% vs. ViT-S/16 at 78.1% (+3.4 percentage points) with 56% fewer parameters (21.5M vs. 48.6M) and 52% fewer MACs (4.8G vs. 10.1G) (Table 2). The T2T module is the direct fix for ViT's primary failure mode (poor local structure modeling at midsize data scales). The advantage is large enough that there is essentially no reason to use standard ViT over T2T-ViT at comparable parameter budgets when training from scratch on ImageNet—T2T-ViT is strictly better on all three axes (accuracy, parameters, MACs).

• Prefer T2T-ViT over DeiT when you want the accuracy benefits of data-efficient training without depending on a CNN teacher model. T2T-ViT-14 at 81.5% matches DeiT-small-Distilled at 81.2% (Table 2) without knowledge distillation, meaning you avoid the additional complexity of training a teacher, managing distillation tokens, and tuning distillation loss weight. For applications where you cannot access or train a high-quality CNN teacher (e.g., novel domains where no strong CNN exists, or resource-constrained settings where training two models is infeasible), T2T-ViT provides a single-model alternative that achieves comparable performance through architecture alone.

• Prefer T2T-ViT over ResNets at comparable parameter counts (20–65M range) when ImageNet classification accuracy is the primary metric and you are comfortable with a pure transformer architecture. T2T-ViT-14 outperforms ResNet50* by +2.4 percentage points (81.5% vs. 79.1%) at 21.5M vs. 25.5M parameters (Table 3). The advantage holds at ResNet101 and ResNet152 scales (+2.0% and +1.5% respectively). However, this preference is conditional on the task being classification (detection/segmentation benefits are untested), the training recipe being compatible with longer training (310 epochs, AdamW), and the deployment environment being able to run transformer operations efficiently (standard matrix multiplications rather than the optimized depthwise convolutions that make MobileNets fast on edge devices).

• Prefer MobileNets over T2T-ViT when computational efficiency (MACs, latency, energy) at mobile parameter scales (<10M parameters) is the primary constraint, and a 2–3× MACs gap is unacceptable. T2T-ViT-12 (6.9M params, 1.8G MACs) achieves 76.5% vs. MobileNetV2 1.4x (6.9M, 0.6G MACs) at 75.6%—the accuracy advantage of +0.9 percentage points comes at 3× the MACs (Table 4). For battery-constrained devices, real-time video processing, or large-scale inference where every FLOP counts, MobileNets remain the pragmatic choice. The paper is explicit that its lite models contain no special efficiency optimizations; this preference would shift if follow-up work successfully integrates efficient attention mechanisms to close the MACs gap while preserving accuracy.