ConvNets Match Vision Transformers at Scale

ArXiv: 2310.16764

🎯 Pitch

A pure convolutional network, pre-trained on a 4-billion image dataset, matches the scaling power of Vision Transformers—a 90.4% ImageNet top‑1 result disproves the widely held belief that ConvNets lose out at web-scale.

---

1. Executive Summary

This paper empirically challenges the prevailing belief that Vision Transformers (ViTs) outperform ConvNets at web-scale pre-training by evaluating the NFNet model family—a pure convolutional architecture—on the JFT-4B dataset across compute budgets spanning 0.4k to 110k TPU-v4 core hours. The authors observe a log-log scaling law between held-out validation loss and pre-training compute, mirroring the power-law relationships established for language models, and find that scaling model depth/width and training epochs at equal rates serves as a reliable allocation heuristic. After fine-tuning on ImageNet using sharpness-aware minimization (SAM) and repeated augmentation, NFNets match the reported Top-1 accuracy of ViTs at comparable compute budgets—an NFNet-F7+ achieves 90.4% Top-1 (versus ViT-g/14 at 90.2% with ~120k TPU-v4 core hours and ViT-G/14 at 90.45% with ~280k), establishing that pre-trained ConvNets remain fully competitive with pre-trained ViTs when evaluated under fair, compute-matched conditions and that architectural choices are secondary to the scale of compute and data available.

2. Context and Motivation

The Core Problem: Did Vision Transformers Actually Win?

At the time of this paper's writing, the computer vision field had undergone a dramatic architectural shift. For nearly a decade following AlexNet (2012), convolutional neural networks—particularly ResNets and their variants—dominated every major vision benchmark. Then, starting with Dosovitskiy et al. (2020), Vision Transformers (ViTs) rapidly displaced ConvNets as the default architecture for large-scale image recognition. By 2023, the prevailing narrative in the community was that ViTs scale better than ConvNets: ConvNets might work well on smaller or moderately sized datasets, but they simply could not keep up when given access to web-scale data and massive compute.

The paper identifies this as an empirical claim that, despite wide acceptance, rests on surprisingly weak evidence. The authors state directly:

"Although most researchers in the community believe Vision Transformers show better scaling properties than ConvNets, there is surprisingly little evidence to support this claim."

This is the central gap the paper addresses: has the field actually demonstrated a fair, compute-matched comparison between pre-trained ViTs and pre-trained ConvNets at web-scale? The authors argue the answer is no, and they set out to provide that comparison.

Why This Matters: The "Bitter Lesson" and Architectural Churn

The question of ConvNet vs. ViT superiority matters far beyond a narrow architectural benchmarking exercise. It connects directly to what the authors invoke as "the bitter lesson"—the observation, articulated by Rich Sutton, that over the history of AI research, general-purpose methods that leverage increasing computation have consistently outperformed hand-designed, domain-specific methods in the long run. If ViTs genuinely do scale better than ConvNets at equivalent compute, that represents an important architectural insight and justifies the field's shift. But if ConvNets can match ViTs when given comparable compute and data, then the architectural churn across computer vision has been largely orthogonal to the factor that actually matters: scale.

The practical implications are substantial. A generation of research effort has been redirected toward Vision Transformers and their variants (Swin, ViT-22B, SoViT, etc.), often with ConvNets treated as a legacy baseline rather than a competitive architecture. If ConvNets are in fact equally capable, then investments in ConvNet-specific hardware optimization, deployment tooling, and architecture research may have been prematurely abandoned. Moreover, ConvNets have well-known practical advantages—they require no positional encoding schemes, they process images with a sliding-window locality prior that reflects natural image statistics, and they've been optimized for decades across a wide range of hardware backends. If they scale just as well as ViTs, they may actually be the better default choice for many applications.

What Created the False Impression of ViT Superiority?

The paper identifies three specific methodological problems in how the community has compared ViTs and ConvNets:

1. Weak ConvNet baselines. Many influential ViT papers compare against the original ResNet architecture from He et al. (2016a)—a 2016 design that predates years of ConvNet innovation. The authors note that "many papers studying ViTs compare to weak ConvNet baselines (typically the original ResNet architecture)." By the time ViTs emerged in 2020, the ConvNet family had already evolved substantially. The NFNet architecture (Brock et al., 2021), published concurrently with the first ViT papers, was actually "the last ConvNet to set a new SOTA on ImageNet" without extra data, achieving 86.8% Top-1 with repeated augmentation. Comparing 2020-era ViTs against 2016-era ResNets is not a meaningful architectural comparison—it compares architectures from different eras of ConvNet design.

2. Unequal compute budgets. The strongest ViT results were achieved with pre-training compute budgets that far exceeded what had been used for ConvNet pre-training experiments. For example, Zhai et al. (2022) pre-trained ViT-G/14 for over 500k TPU-v3 core hours, while strong ConvNet results typically used far less compute. This creates an asymmetry where ViTs appear to win, but the victory may be attributable to the extra compute rather than the architecture. The authors state directly: "the strongest ViT models have been pre-trained using large compute budgets beyond 500k TPU-v3 core hours, which significantly exceeds the compute used to pre-train ConvNets."

3. The metric shift in the field. Simultaneous with the rise of ViTs, the computer vision community shifted its evaluation paradigm from training randomly initialized networks on specific datasets (like ImageNet) to evaluating pre-trained networks fine-tuned from large-scale web datasets. This shift confounds the comparison: when a ViT pre-trained on JFT-3B outperforms a ResNet trained from scratch on ImageNet, it is unclear whether the ViT architecture, the pre-training data, or the pre-training compute budget is responsible for the improvement. The paper aims to isolate architectural differences by controlling for both data and compute.

Where Prior Comparisons Fall Short: A Deeper Look

The paper's critique is more specific than "no one has compared ViTs and ConvNets." Several high-profile papers have compared them, but the comparisons have structural flaws:

The Dosovitskiy et al. (2020) comparison. The original ViT paper showed that ViTs could match or outperform ResNet-based ConvNets when pre-trained on large datasets like JFT-300M. However, the ConvNet baselines were standard ResNets, not the stronger NFNet or EfficientNet architectures that existed by 2020. Additionally, the compute budgets were not systematically matched across architectures.

The MLP-Mixer comparison (Tolstikhin et al., 2021). Tolstikhin et al. famously showed that even a simple architecture consisting entirely of multi-layer perceptrons applied across spatial locations or feature channels could achieve competitive performance when pre-trained at scale. This work reinforced the "bitter lesson" message—architecture matters less than scale—but it compared MLP-Mixer against ViTs and a standard ResNet baseline, not against the strongest available ConvNets.

The scaling law literature. Work by Zhai et al. (2022) and Alabdulmohsin et al. (2023) had established careful scaling laws for Vision Transformers, mapping how ViT performance improves with model size, data, and compute. But these studies optimized within the ViT architecture family; they did not ask whether ConvNets would exhibit the same scaling behavior, or whether the ViT-specific scaling coefficients were better or worse than what ConvNets could achieve.

This last point is particularly important. By 2023, the field had scaling laws for language models (Kaplan et al., 2020; Hoffmann et al., 2022) and for Vision Transformers (Zhai et al., 2022; Alabdulmohsin et al., 2023), but no comparable scaling law had been demonstrated for ConvNets on web-scale data. This absence was widely interpreted as evidence that ConvNets don't scale—that their performance would saturate or grow sub-linearly with additional compute. The paper sets out to test this assumption directly.

How This Paper Positions Itself

The paper positions itself as a correction to a premature consensus. It does not claim that ConvNets are better than ViTs. Rather, it claims that, when evaluated under fair conditions—matching compute budgets, using the same pre-training dataset, and comparing against strong ConvNet architectures rather than legacy ResNets—ConvNets match ViTs, and both architectures obey similar scaling laws. The abstract states this clearly: "After fine-tuning on ImageNet, NFNets match the reported performance of Vision Transformers with comparable compute budgets."

The paper's approach is deliberately minimal. The authors do not modify the NFNet architecture or the training procedure beyond tuning simple hyper-parameters like learning rate and epoch budget. They state: "We do not make any changes to the model architecture or the training procedure (beyond tuning simple hyper-parameters such as the learning rate or epoch budget)." This is a deliberate choice: they want to test whether the ConvNet architecture, as it existed in 2021, scales when given ViT-level compute and data, without relying on any new innovations that could be attributed to the paper's own contributions. If the answer is yes, then the community's shift away from ConvNets was based on insufficient evidence rather than genuine architectural superiority.

The paper also connects its findings explicitly to the language modeling scaling literature. By observing "a log-log scaling law between held out loss and compute budget" and noting that "the optimal model size and the optimal epoch budget both increase in size as the compute budget increases... scaling the model size and the number of training epochs at the same rate," the authors draw a direct parallel to the Chinchilla scaling laws (Hoffmann et al., 2022). This framing positions ConvNet scaling as part of a broader, architecture-agnostic phenomenon: given sufficient compute and data, well-designed models of different architectures converge to similar performance levels. The implication is that the computer vision community should focus less on architectural innovation and more on compute and data scaling—the same "bitter lesson" that has guided language modeling research.

The Broader Intellectual Context: Architecture vs. Scale

The paper sits at the intersection of two ongoing debates in deep learning. The first is the inductive bias debate: ConvNets encode a strong prior about spatial locality and translation equivariance, while ViTs use self-attention with a much weaker inductive bias (essentially just the tokenization of images into patches). Some researchers have argued that the weaker inductive bias of ViTs allows them to learn more flexible representations when given enough data, explaining their apparent superiority at scale. This paper challenges that narrative by showing that a strong ConvNet architecture with its locality prior scales just as well as ViTs with their weaker prior—the inductive bias of convolutions does not become a liability at scale.

The second debate is the hardware efficiency debate. ViTs are often claimed to be more hardware-efficient than ConvNets on modern accelerators because their matrix-multiplication-heavy operations map well to GPU/TPU architectures optimized for large matmuls. However, the paper's compute accounting—measured in TPU-v4 core hours—directly accounts for hardware efficiency differences. If ViTs were substantially more efficient per FLOP, they would deliver better performance per core hour. The fact that NFNets match ViTs on this metric suggests that whatever hardware efficiency advantage ViTs might have is not decisive at the scale tested. (The authors do note a caveat: NFNets were optimized for TPU-v4 and "perform less well when evaluated on other devices," so the hardware efficiency question may have platform-specific answers.)

Summary of the Gap and the Paper's Response

The gap is clear: the field has accepted ViT superiority as fact without conducting the necessary controlled experiment—comparing the strongest available ConvNet architecture to the strongest available ViT architecture at matched pre-training compute budgets on the same web-scale dataset.

The paper's response is to run exactly that experiment. It takes the NFNet family—the strongest pure ConvNet architecture from the ViT era—and pre-trains a range of model sizes on JFT-4B across compute budgets up to 110k TPU-v4 core hours, then fine-tunes on ImageNet and compares against reported ViT results from Zhai et al. (2022) and Alabdulmohsin et al. (2023). The central finding—that ConvNets match ViTs—is presented not as a new architectural breakthrough but as a correction to a mistaken belief, consistent with the "bitter lesson" that compute and data, not architectural details, are the primary determinants of model performance.

3. Technical Approach

3.1 Reader Orientation

This paper conducts a controlled empirical investigation rather than proposing a new architecture or algorithm—the system under study is a pre-existing pure convolutional network (NFNet) that is pre-trained on a massive labelled image dataset (JFT-4B) and then fine-tuned on ImageNet. The paper solves the problem of whether ConvNets genuinely scale worse than Vision Transformers by running the missing controlled experiment: training strong ConvNets at ViT-comparable compute budgets on the same web-scale dataset and measuring whether their performance falls short of, matches, or exceeds the reported ViT numbers, with the "shape" of the solution being a systematic sweep over model sizes and training durations to map out the compute-performance relationship and compare against existing ViT data points.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental pipeline has five major components:

1. NFNet model family (F0 through F7+) — a series of pure convolutional architectures of increasing depth and width, serving as the "architecture variable" being tested. These are the same architectures from Brock et al. (2021) with zero modifications.

2. JFT-4B pre-training dataset — approximately 4 billion labelled images spanning 30,000 classes, with near-duplicates of ImageNet training and validation images removed to prevent contamination. This is the same dataset used to pre-train the ViT baselines.

3. Pre-training procedure — standard supervised classification on JFT-4B using SGD with Momentum and Adaptive Gradient Clipping (AGC), with each model trained at multiple epoch budgets (0.25 to 8 epochs) and a cosine decay learning rate schedule, using resolution 224×224 during training and 256×256 at evaluation.

4. Fine-tuning procedure on ImageNet — each pre-trained checkpoint is fine-tuned on ImageNet for 50 epochs using Sharpness-Aware Minimization (SAM), stochastic depth, and dropout, at resolution 384×384 (training) and 480×480 (evaluation), with optional repeated augmentation (RA) at multiplicity 4 for the strongest models.

5. Compute-matched comparison against ViTs — the pre-training compute of NFNets (measured in TPU-v4 core hours) is compared against estimated TPU-v4 costs for the ViT baselines from Zhai et al. (2022) and Alabdulmohsin et al. (2023), with ViT costs estimated using the original authors' codebase on TPU-v4 hardware.

Information flows as follows: a model size is selected from the NFNet family → the model is pre-trained on JFT-4B for a specified epoch budget at resolution 224×224 → validation loss on a 130k-image held-out set is recorded → the pre-trained checkpoint is fine-tuned on ImageNet for 50 epochs at resolution 384×384 → ImageNet Top-1 accuracy is measured → both validation loss and Top-1 accuracy are plotted against the compute budget (TPU-v4 core hours) and compared against ViT data points from the literature.

3.3 Roadmap for the Deep Dive

• First, the pre-training framework — the dataset (JFT-4B), the model family (NFNet), the training hyperparameters, and the compute budget measurement methodology — because this is the "experimental platform" on which everything else rests.
• Second, the scaling law methodology — how the authors swept model sizes and epoch budgets, how they measured validation loss, and how they identified the compute-optimal allocation between model size and training duration — because the scaling law is the paper's primary empirical contribution and the basis for comparing against ViT scaling behavior.
• Third, the fine-tuning protocol — the specific recipe for transferring pre-trained NFNets to ImageNet, including the hyperparameter choices (SAM, stochastic depth, resolution changes, repeated augmentation) — because the fine-tuning stage is where the final competitive numbers are produced and where design choices could introduce confounding factors.
• Fourth, the ViT comparison methodology — how the authors estimated TPU-v4 core hours for the ViT baselines, the specific ViT models compared against, and the caveats about hardware optimization differences — because the fairness of this comparison is the central claim of the paper.
• Fifth, the hyperparameter tuning strategy (learning rate scaling) — the empirical observation that optimal learning rate follows a predictable pattern with model size and epoch budget — because this is presented as a practical finding that reduces the cost of future scaling experiments.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical scaling paper whose core idea is that ConvNets, when evaluated under compute-matched conditions with strong architectures, exhibit the same log-log scaling behavior as Vision Transformers and achieve equivalent final performance — contradicting the widely-held belief that ViTs possess inherently superior scaling properties.

---

The NFNet Model Family: Architecture and Scaling Dimensions

The paper uses the NFNet model family exactly as described in Brock et al. (2021), without any architectural modifications. NFNet stands for "Normalizer-Free Network" — a pure convolutional architecture that achieves competitive performance without batch normalization, instead using Adaptive Gradient Clipping (AGC) to stabilize training. Understanding the NFNet design is important because it represents the state of ConvNet design circa 2021, incorporating innovations that post-date the ResNet architectures commonly used as ConvNet baselines in ViT papers.

Design principles inherited from Brock et al. (2021). NFNets are built on several architectural innovations that distinguish them from earlier ConvNets like ResNets. First, they use residual blocks with scaled activations — the output of each residual branch is multiplied by a small constant (typically $1/\sqrt{\text{depth}}$) before being added to the skip connection, which controls the variance of activations through the network without requiring batch normalization. Second, they replace batch normalization with weight standardization (reparameterizing convolutional weights to have zero mean and unit variance) combined with adaptive gradient clipping (clipping gradients whose norm exceeds a threshold relative to the weight norm). Third, they employ architectural modifications including SE (Squeeze-and-Excitation) blocks, wider stem convolutions, and a modified transition structure between stages. The key point for this paper is that NFNets were "the last ConvNet to set a new SOTA on ImageNet" without extra training data, achieving 86.8% Top-1 accuracy with repeated augmentation — establishing them as a legitimate, competitive ConvNet baseline against which ViTs should be compared.

The model size progression. The paper trains models from the NFNet family using a naming convention that encodes both depth and width. The specific models trained are F0, F1, F3, F3+, F7, and F7+. The naming convention follows a clear scaling pattern:

• F0 is the smallest model in the family.
• F1 has double the depth of F0.
• F3 has double the depth of F1 (and therefore 4× the depth of F0).
• F7 has double the depth of F3 (and therefore 8× the depth of F0), but shares the same width as F3.
• F3+ and F7+ share the same depths as F3 and F7 respectively, but have larger widths (more channels per layer).

This creates a grid that varies two scaling dimensions independently: depth (number of layers) and width (number of channels per layer). The "+" suffix indicates a width-scaled variant at a given depth. By training models across this grid, the authors can observe how the optimal depth-width tradeoff shifts as the total compute budget increases.

Why this model family over alternatives. The choice of NFNets is strategic. Alternative strong ConvNet architectures like EfficientNet (Tan and Le, 2019) also existed, but NFNets have two properties that make them particularly suitable for this study. First, they were published concurrently with the first ViT papers, meaning they represent the ConvNet design knowledge available at the time ViTs emerged — using them avoids the criticism that the ConvNet baseline is outdated. Second, their normalizer-free design makes them straightforward to train at large scale without the complications that batch normalization introduces in distributed training (where batch statistics must be synchronized across devices). Third, because Brock et al. (2021) already established a strong pre-training pipeline for NFNets on JFT, the authors can follow this pipeline exactly without introducing new methodological variables.

Acknowledged limitation. The paper notes that "NFNets were optimized for TPU-v4, and perform less well when evaluated on other devices." This is an open admission that the hardware efficiency comparison is platform-specific. The authors estimate that NFNet-F7+ would require approximately 250 TPU-v3 core hours to pre-train for 8 epochs in their codebase — roughly 2.3× the TPU-v4 cost — which implies that on TPU-v3 (where many ViT baselines were originally benchmarked), the efficiency gap might be different. This transparency is important: it means the paper's headline result of matched performance at matched TPU-v4 core hours does not automatically generalize to other hardware platforms without re-benchmarking.

---

The Pre-Training Dataset: JFT-4B

The choice of pre-training dataset is critical because it must match the datasets used for ViT pre-training to enable a fair comparison. The paper uses JFT-4B, which is described as "a large labelled dataset of images often used for training foundation models" containing "roughly 4 billion labelled images from 30k classes" (citing Sun et al., 2017).

Dataset provenance and scale. JFT (which originally stood for "JFT-300M" in its smaller version) is an internal Google dataset created by automatically labelling web images using a complex pipeline that maps images to a large ontology of approximately 30,000 classes. The JFT-4B variant used here contains approximately 4 billion images — roughly 13× the size of JFT-300M and substantially larger than ImageNet-21k (14 million images). The labels are noisy because they come from an automatic annotation process rather than human verification, which means the pre-training task is effectively a weakly-supervised classification problem. This is consistent with how ViTs are typically pre-trained: Zhai et al. (2022) used JFT-3B (a 3 billion image variant of the same dataset), while Alabdulmohsin et al. (2023) also used JFT-3B.

De-duplication against ImageNet. The authors explicitly state that they "removed near-duplicates of images in the training and validation sets of ImageNet from JFT-4B before training," following the protocol established by Kolesnikov et al. (2020). This is a crucial detail. Without de-duplication, information about ImageNet test images could leak into pre-training through near-identical images in JFT-4B, inflating the final fine-tuning accuracy and invalidating the comparison against ViTs (which also used de-duplicated pre-training data). Near-duplicate removal typically involves finding images in JFT-4B that are visually very similar (e.g., via perceptual hashing or feature similarity) to any ImageNet image and excluding them from training. This means the pre-training signal comes entirely from images that are genuinely distinct from the downstream evaluation task.

Evaluation during pre-training. Validation loss is measured on "a held out set of 130k images" from JFT-4B. This held-out set serves two purposes: it provides a signal for hyperparameter tuning (selecting the best learning rate and epoch budget) and it is the metric used to establish the log-log scaling law. The 130k figure is a standard validation split size that balances statistical reliability (enough images for a stable loss estimate) against computational cost (evaluating on the full 4B dataset would be prohibitively expensive).

Why JFT-4B over other web-scale datasets. The practical reason is continuity with the ViT literature. Both Zhai et al. (2022) and Alabdulmohsin et al. (2023) pre-trained their ViTs on variants of JFT, so using JFT-4B eliminates the dataset as a confounding variable. The theoretical reason is that JFT's scale (billions of images, tens of thousands of classes) is large enough to expose scaling behavior — on smaller datasets like ImageNet-21k, both ViTs and ConvNets might saturate before revealing differences in their scaling curves. At 4 billion images with 30k classes, the model sees each class roughly 133,000 times on average, which is far more than typical academic datasets and creates a regime where the model's capacity, not data scarcity, is the binding constraint.

---

Pre-Training Procedure: Hyperparameters and Protocol

The pre-training procedure follows the protocol established in Section 6.2 of Brock et al. (2021), with only "simple hyper-parameters such as the learning rate or epoch budget" being tuned for this study. The authors are explicit that they make no changes to the model architecture or the training procedure beyond these basic hyperparameter adjustments — this is a deliberate choice to test whether the NFNet architecture, exactly as published, scales when given more compute.

Optimizer configuration. Training uses SGD with Momentum and Adaptive Gradient Clipping (AGC) at batch size 4096. SGD with momentum is the classical optimizer for ConvNet training: it maintains a velocity term that accumulates past gradients with exponential decay, smoothing the optimization trajectory and accelerating convergence in directions with consistent gradient sign. AGC is the NFNet-specific stabilization mechanism: after computing gradients, if the norm of the gradient for any parameter exceeds a threshold proportional to the norm of that parameter (typically $\lambda \cdot ||\mathbf{W}||$ where $\mathbf{W}$ is the weight matrix and $\lambda$ is a clipping factor), the gradient is rescaled to satisfy the threshold. This prevents individual parameter updates from being disproportionately large relative to the parameter magnitude, which is the role that batch normalization plays in standard ResNets — AGC achieves this stabilization without requiring batch statistics.

The batch size of 4096 is large by academic standards but typical for web-scale pre-training on TPU pods, where it enables efficient data-parallel distributed training across many accelerator cores. Large batch sizes reduce the variance of gradient estimates (by averaging over more examples) but require correspondingly careful learning rate tuning — the paper addresses this through explicit learning rate sweeps.

Learning rate schedule. The paper uses a cosine decay learning rate schedule with a base learning rate that is "tuned separately for each epoch budget on a small logarithmic grid." Cosine decay means the learning rate follows:

$\eta(t) = \eta_{\text{base}} \cdot \frac{1}{2}\left(1 + \cos\left(\pi \frac{t}{T}\right)\right)$

where $t$ is the current training step, $T$ is the total number of training steps, and $\eta_{\text{base}}$ is the initial (peak) learning rate.

What this computes: at the start of training ($t=0$), the learning rate equals $\eta_{\text{base}}$. It then decreases smoothly following a cosine curve, reaching a value very close to zero at the final training step ($t=T$). This means the model makes large parameter updates early in training (when it is far from convergence) and progressively smaller, more careful updates as training proceeds (when it is approaching a minimum).

Why this form: cosine decay has become the standard schedule for large-scale vision pre-training because it avoids the discontinuity of step-function decay schedules (which introduce sudden changes in optimization dynamics) while providing a natural annealing behavior. The key advantage over alternatives like linear decay or constant learning rate is that cosine decay spends more time at intermediate learning rates — this tends to produce better final solutions than schedules that drop the learning rate too quickly or keep it high for too long. The fact that the base learning rate is tuned per epoch budget (rather than using a fixed value) reflects the empirical observation that longer training runs benefit from different initial learning rates, a pattern the paper explores systematically (Figure 3).

Epoch budgets. Each model is trained for a range of epoch budgets "between 0.25 and 8" epochs over the JFT-4B dataset. An epoch budget of 0.25 means the model sees 1 billion images (25% of the 4B-image dataset) during training; an epoch budget of 8 means it sees 32 billion images (passing through the dataset 8 times). In the large-data regime typical of JFT pre-training, training for fewer than one epoch is standard — the dataset is so large that the model never needs to see the same image twice to converge. The fact that the authors sweep from 0.25 to 8 epochs means they explore both the "less than one pass through the data" regime and the "multiple passes through the data" regime, which is important for understanding how the optimal epoch budget shifts with model size.

Image resolution. During pre-training, images are processed at resolution 224×224 for training, with evaluation at 256×256. The 224×224 training resolution is the standard for ImageNet-era models and is also the resolution used in the original ViT paper (Dosovitskiy et al., 2020). The higher evaluation resolution (256×256) follows the common practice of using slightly larger images at test time to improve accuracy — the convolutional network's translation equivariance means it can process larger images than it was trained on without architectural changes, though the absolute position of features may shift relative to the network's receptive field expectations. This resolution gap is modest (32 pixels in each dimension) and unlikely to introduce artifacts, but it is a departure from strict train-test resolution matching.

Stopping criterion and checkpoint selection. There is no explicit early stopping criterion mentioned; models are trained for their full epoch budget. The final validation loss is measured at the end of training. For fine-tuning, however, the authors note that "the pre-trained checkpoints achieving the lowest validation loss on JFT-4B did not always achieve the highest Top-1 accuracy on ImageNet after fine-tuning," with the fine-tuning regime "consistently favoured slightly larger models and slightly smaller epoch budgets." This means that validation loss during pre-training and downstream transfer performance are not perfectly correlated — a model that fits the pre-training data more tightly (lower validation loss) may have overfit to JFT-specific features that do not transfer well to ImageNet. The practical implication is that selecting the best checkpoint for fine-tuning requires either evaluating on the downstream task (which is expensive) or accepting that pre-training validation loss is an imperfect proxy for transfer quality.

Compute Budget Measurement and Accounting

The paper measures the computational cost of pre-training in TPU-v4 core hours. This is the primary currency for comparing across models and against ViT baselines, so understanding exactly what is being measured — and what is not — is essential.

What a TPU-v4 core hour represents. A TPU-v4 core is a single matrix-multiply unit (MXU) within a TPU-v4 chip. Each TPU-v4 chip contains two cores (called TensorCores), each capable of 275 teraflops (FP16/BF16) of peak compute — roughly double the theoretical FLOPs of a TPU-v3 core. A TPU-v4 core hour is therefore the amount of computation performed by one such core running continuously for one hour. If a model is trained on a TPU-v4 pod with 4096 cores for 10 hours, it consumes 40,960 TPU-v4 core hours. The paper notes that "TPU-v4 cores have roughly double the theoretical flops of TPU-v3 cores, however both cores have similar memory," which means a TPU-v4 core hour represents roughly 2× the raw arithmetic of a TPU-v3 core hour, but the memory capacity (which limits maximum model size per core) is comparable.

How compute budgets were estimated. The authors state that they "estimate the compute required to train each model by eye from the typical steps per second achieved by each model during training (when not pre-empted)." This is an admission that the compute measurement is approximate rather than exact. The "by eye" phrasing suggests that they observed the training throughput (steps per second) during stable training periods, multiplied by the number of training steps, and converted to core hours. The parenthetical "(when not pre-empted)" is important: on large TPU pods, training jobs can be interrupted (pre-empted) and restarted for operational reasons, and throughput can vary depending on how many devices are available after restart. The authors are effectively averaging over the "good" periods when training was running at full speed.

The pre-emption issue and training instability. The paper raises a specific concern about data loading under pre-emption that may have affected results:

"We believe this arises because our data loading pipeline did not guarantee that each training example would be sampled once per epoch if the training run was pre-empted/restarted, potentially causing some training examples to be under-sampled if a training run was restarted multiple times."

This is referenced as a possible explanation for why "the curve for NFNet-F7+ models at different pre-training budgets is not smooth" in Figure 2. In an ideal training run, the data loader cycles through the dataset deterministically, ensuring that every example is seen exactly once per epoch (for epoch budgets ≥1) or that the sampling is uniformly random (for epoch budgets <1). If pre-emption and restart cause the data loader to reset or lose its place, some examples may be seen multiple times while others are never seen, creating an effective data distribution that differs from the intended one. This introduces noise into the scaling curve — two models with the same nominal compute budget might see different effective data, and the measured validation loss might fluctuate accordingly.

This is a methodological weakness that the authors are transparent about, but it also means the scaling law in Figure 2 might be cleaner if data loading guarantees were stronger. The fact that a clear log-log trend still emerges despite this noise is evidence for the robustness of the underlying scaling relationship.

Scope of compute measurement: pre-training only. The paper separates pre-training compute and fine-tuning compute. The TPU-v4 core hour numbers on the x-axes of Figures 1 and 2 represent pre-training compute only. Fine-tuning costs are reported separately — for NFNet-F7+, fine-tuning requires "1.6k TPU-v4 core hours" (approximately 1.5% of the pre-training budget). The ViT comparison focuses on matching pre-training compute, with the implicit assumption that fine-tuning costs are similar across architectures (which is reasonable: fine-tuning runs for a fixed 50 epochs on ImageNet, which is a much smaller dataset than JFT-4B, so the bulk of the total compute is in pre-training).

Why TPU-v4 core hours over FLOP counts. The paper uses core hours rather than raw FLOP counts because core hours directly reflect the practical cost of training — they account for hardware utilization, communication overhead, and data loading bottlenecks, not just theoretical arithmetic. Two models with the same theoretical FLOP count might require different core hours if one achieves higher hardware utilization. This makes core hours a more honest metric for comparing practical training cost. However, it also means the comparison is hardware-specific: NFNets were optimized for TPU-v4, and the paper acknowledges that the ViT baselines were originally trained on TPU-v3, with their TPU-v4 costs estimated by re-running the original code on TPU-v4 hardware.

---

The Scaling Law Methodology: Sweeping Model Size and Epoch Budget

The paper's primary empirical contribution is the observation of a log-log scaling law between validation loss and pre-training compute. The methodology for establishing this law involves systematically varying two independent dimensions — model size and training duration — and measuring the resulting validation loss.

The two-dimensional sweep. For each model in the NFNet family (F0, F1, F3, F3+, F7, F7+), the authors train at multiple epoch budgets (0.25, 0.5, 1, 2, 4, 8) and measure the held-out validation loss. This creates a grid of (model, epoch budget) pairs, each with an associated compute cost (TPU-v4 core hours) and validation loss. In Figure 2, each curve represents one model swept across epoch budgets, with the x-axis being total compute and the y-axis being validation loss.

Why this two-dimensional sweep is necessary. A common mistake in scaling studies is to vary only one dimension — for example, training increasingly large models for a fixed number of epochs. This conflates model size and total compute: larger models trained for the same number of epochs receive more compute (because each training step costs more FLOPs). The two-dimensional sweep disentangles these effects, allowing the authors to ask: for a given total compute budget, what combination of model size and training duration minimizes validation loss? This is exactly the question that Hoffmann et al. (2022) asked for language models, and the paper explicitly draws this parallel.

The log-log scaling observation. When validation loss is plotted against compute on log-scaled axes (Figure 2), the data points form "a clear linear trend, consistent with a log-log scaling law." A log-log scaling law means that validation loss $L$ and compute budget $C$ are related by a power law:

$L(C) \approx a \cdot C^{-\alpha} + L_0$

where $a$ is a scaling coefficient, $\alpha$ is the scaling exponent (the slope on the log-log plot), and $L_0$ is the irreducible loss (the asymptotic minimum loss achievable with infinite compute, representing inherent noise in the data and task).

What this form means operationally: for every factor-of-10 increase in compute budget, the validation loss decreases by a fixed multiplicative factor (or, on the log-log plot, the validation loss decreases by a fixed absolute amount). The linearity of the trend in Figure 2 means this relationship holds across the entire range tested (0.4k to 110k TPU-v4 core hours, roughly a 275× increase in compute). There is no visible saturation or "knee" in the curve where additional compute stops helping — the loss continues to improve at the same log-log rate through the maximum budget tested.

Why this form matters: the existence of a clean log-log scaling law for ConvNets on web-scale data is the paper's central empirical finding. Before this work, such scaling laws had been demonstrated for language models (Kaplan et al., 2020; Hoffmann et al., 2022) and for Vision Transformers (Zhai et al., 2022; Alabdulmohsin et al., 2023), but not for ConvNets. The absence of ConvNet scaling laws had been interpreted by many as evidence that ConvNets don't scale — that their performance would eventually saturate. By showing that ConvNets obey the same functional form of scaling law as ViTs and language models, the paper undermines the claim that ViTs possess a fundamentally different (and superior) scaling regime.

The compute-optimal allocation rule. From the two-dimensional sweep, the authors extract a practical rule of thumb:

"We found that a reliable rule of thumb is to scale the model size and the number of training epochs at the same rate."

This means that if you double your compute budget, you should increase both model size (parameters) and training duration (epochs/data seen) by roughly the same factor — not double one while holding the other constant. This echoes the Chinchilla finding for language models (Hoffmann et al., 2022), which showed that compute-optimal training allocates additional compute roughly equally between more parameters and more training tokens.

The epoch budget threshold. The authors note a specific observation about when training beyond one epoch becomes optimal: "the optimal epoch budget was greater than 1 for overall compute budgets greater than roughly 5k TPU-v4 core hours." Below 5k core hours, the optimal configuration trains for less than one full pass through the 4B-image dataset — effectively, the model never sees the same image twice because the dataset is so large that fresh data provides more benefit than reusing data. Above 5k core hours, it becomes optimal to make multiple passes through the data, suggesting that at this scale, the model begins to exhaust the signal in a single pass and benefits from repeated exposure to the same examples. This threshold is specific to the JFT-4B dataset size; on a smaller dataset, the threshold would be lower (or perhaps never reached, if the dataset is large enough that one epoch is always sufficient).

Measurement of validation loss. Validation loss is measured on a held-out set of 130k images from JFT-4B. The loss function is almost certainly the standard softmax cross-entropy loss for multi-class classification with 30,000 classes (the number of labels in JFT-4B). Formally, for a single image with true class label $y$ and model-predicted logits $\mathbf{z} \in \mathbb{R}^{30000}$, the loss is:

$\mathcal{L} = -\log \frac{e^{z_y}}{\sum_{j=1}^{30000} e^{z_j}}$

where $z_y$ is the logit for the true class and the denominator sums the exponentials of all class logits.

What this computes: the negative log probability the model assigns to the correct class after the softmax normalization. A perfect model would assign probability 1.0 to the correct class, giving loss 0; a model assigning uniform probability across all 30,000 classes would give loss $\log(30000) \approx 10.3$. The observed validation losses in Figure 2 range from roughly 2.4 (best models, highest compute) to 3.0 (weakest models, lowest compute), which corresponds to the model assigning the correct class a probability in the range of roughly 5–9% — far above chance ($1/30000 \approx 0.003\%$), but far from perfect.

Why this loss function: softmax cross-entropy is the standard objective for supervised classification. It is well-calibrated for the task: the gradient drives the model to increase the logit of the true class relative to all others, with the softmax competition preventing the model from simply increasing all logits uniformly. The multi-class structure with 30,000 fine-grained labels makes the task substantially harder than ImageNet (1,000 classes), which is actually a feature for a scaling study — the difficulty prevents early saturation and allows scaling behavior to be observed across a wider compute range.

Interpretation of the scaling law slope. The paper does not report the numerical value of the scaling exponent $\alpha$ (the slope in Figure 2). This is a notable omission: the slope is the key parameter of a power-law scaling relationship, and knowing whether ConvNets and ViTs have the same or different slopes would be directly informative about whether one architecture scales faster (not just as well at a particular budget). The fact that the slope is not reported suggests either that the noisy data (due to pre-emption) made precise slope estimation unreliable, or that the authors considered the qualitative observation (log-log linearity) sufficient for their argument. From visual inspection of Figure 2, the trend spans roughly 0.6 nats of loss improvement across approximately 2.4 orders of magnitude in compute ($10^{3}$ to $10^{5.4}$), yielding an approximate slope of $-0.6 / 2.4 \approx -0.25$ in log-loss per log-compute — but this is an estimate, not a value the paper provides.

---

Learning Rate Tuning Strategy and Scaling

The paper devotes specific attention to learning rate tuning because, in large-scale training, the learning rate is the single most sensitive hyperparameter and its optimal value depends on both model size and training duration. Getting it wrong can substantially degrade performance, and grid-searching it exhaustively is expensive. The paper's findings on learning rate behavior are presented as a practical contribution that can reduce the cost of future scaling experiments.

The tuning protocol. The learning rate is tuned on a "logarithmic grid spaced by factors of 2." This means the authors test values like 0.4, 0.8, 1.6, 3.2, etc., selecting the one that minimizes validation loss for each (model, epoch budget) combination. A logarithmic grid is appropriate because learning rate sensitivity is typically multiplicative — a change from 0.1 to 0.2 matters about as much as a change from 1.0 to 2.0. The "factor of 2" spacing is relatively coarse; finer-grained tuning might squeeze out small additional improvements but at significant computational cost (each learning rate trial is a full training run).

The empirical pattern (Figure 3). The paper makes three observations about how the optimal learning rate behaves:

1. At small epoch budgets, all models share a similar optimal learning rate: "All models show similar optimal learning rates $\alpha \approx 1.6$ when the epoch budget is small."

2. The optimal learning rate falls as the epoch budget increases: as models train for more epochs (or equivalently, see more data), the best learning rate decreases. This is intuitive: with more training steps, each individual update needs to be smaller to avoid overshooting, and the model benefits from a more conservative optimization trajectory.

3. The optimal learning rate falls more quickly for larger models: "for large models the optimal learning rate falls more quickly" as epoch budget increases. This means the interaction between model size and learning rate is not simply additive — larger models are more sensitive to the choice of learning rate and require more aggressive reduction as training duration increases.

Why this matters practically. The paper claims that "one can efficiently tune the learning rate within 2 trials by assuming that the optimal learning rate falls slowly but monotonically as both the model size and the epoch budget increases." This is a concrete efficiency recommendation: rather than exhaustively grid-searching over many learning rate values, a practitioner can use the known value for a smaller model at a small epoch budget ($\alpha \approx 1.6$) as a starting point, then reduce it based on how much larger the model is and how many more epochs it will train for. The "within 2 trials" claim suggests that a simple interpolation or extrapolation followed by one verification run is sufficient.

The mechanistic explanation (what the learning rate controls). The learning rate determines the step size in parameter space for each SGD update. For a parameter vector $\theta_t$ at step $t$, the update rule with momentum is:

$v_t = \mu v_{t-1} + g_t$ $\theta_{t+1} = \theta_t - \eta \cdot v_t$

where $g_t$ is the gradient of the loss with respect to $\theta$ at step $t$, $v_t$ is the momentum accumulator (velocity), $\mu$ is the momentum coefficient (typically 0.9), and $\eta$ is the learning rate. AGC additionally clips $g_t$ (or the resulting update) if its norm exceeds a threshold.

What this computes: at each step, the parameters are moved in the direction of the (momentum-smoothed) gradient, scaled by the learning rate. Larger learning rates mean larger parameter changes per step, which can accelerate convergence but risk instability; smaller learning rates mean more cautious updates that are more likely to converge but require more steps.

Why the optimal learning rate decreases with scale: larger models have more parameters, which means the loss landscape is higher-dimensional and the gradient has more components. In higher dimensions, the squared norm of the gradient tends to be larger (because it sums over more parameters), making the raw update magnitude larger for the same learning rate. To keep the effective step size (the actual distance moved in parameter space) controlled, the learning rate must be reduced. Similarly, longer training runs mean more total parameter updates; if each update is too large, the optimization will oscillate rather than converge, so smaller updates are needed to settle into a minimum.

---

Fine-Tuning Protocol on ImageNet

The fine-tuning stage is where pre-trained NFNets are adapted to the ImageNet classification task, and it is here that the paper's final competitive numbers are produced. The fine-tuning protocol involves several specific design choices that are worth understanding in detail because they represent the state of the art in transfer learning for image classification and could interact with the pre-training regime in non-obvious ways.

Dataset and task. Fine-tuning is performed on the standard ImageNet ILSVRC-2012 dataset, which contains approximately 1.28 million training images across 1,000 classes and 50,000 validation images. This is the canonical benchmark for image classification and the same task used in the ViT comparisons (Zhai et al., 2022; Alabdulmohsin et al., 2023).

Fine-tuning duration. Each model is fine-tuned for 50 epochs on ImageNet. Fifty epochs is a relatively long fine-tuning schedule by the standards of transfer learning — many papers fine-tune for 10–30 epochs — but is consistent with the protocol used in the ViT scaling literature. The longer schedule allows the model to fully adapt its pre-trained representations to the ImageNet class distribution and image statistics (which may differ from JFT because images in JFT are web-scraped at varying resolutions and qualities, while ImageNet images are curated).

Optimizer: Sharpness-Aware Minimization (SAM). The paper uses SAM (Foret et al., 2020) for fine-tuning, which is a non-standard optimizer that explicitly seeks flat minima of the loss landscape rather than just low-loss points. SAM works by performing two gradient computations per step:

1. Compute the gradient $g$ at the current parameter $\theta$.
2. Take a step in the gradient direction to a perturbed point: $\theta' = \theta + \rho \cdot g / ||g||$, where $\rho$ is a perturbation radius.
3. Compute the gradient $g'$ at the perturbed point $\theta'$.
4. Update the original parameters using $g'$: $\theta_{t+1} = \theta_t - \eta \cdot g'$.

What this computes: SAM finds parameters where not only is the loss low, but the loss remains low even if the parameters are slightly perturbed (the loss landscape is "flat" around the solution). Flat minima have been theoretically and empirically linked to better generalization — models that converge to flat minima tend to perform better on held-out data than models in sharp minima with equivalent training loss.

Why SAM for fine-tuning: the authors do not provide an explicit justification, but the standard rationale is that fine-tuning is particularly susceptible to overfitting because the downstream dataset (ImageNet) is much smaller than the pre-training dataset (JFT-4B). A model pre-trained on 4 billion images has enormous capacity; when fine-tuned on 1.28 million images, it can easily memorize the training set rather than learning features that generalize. SAM's flat-minimum bias acts as a regularizer, discouraging the model from finding sharp minima that fit the training data perfectly but generalize poorly.

Additional regularization: stochastic depth and dropout. The paper mentions using "stochastic depth and dropout" during fine-tuning. Stochastic depth (Huang et al., 2016) randomly drops entire residual blocks during training with some probability, forcing the network to learn redundant representations and preventing co-adaptation between adjacent layers. Dropout randomly zeros out individual activations, providing a different form of redundancy. Both are standard regularization techniques for large ConvNets and ViTs, and their use here is consistent with the NFNet fine-tuning recipe from Brock et al. (2021).

Resolution changes. Fine-tuning uses resolution 384×384 for training and 480×480 for evaluation. This is a significant increase from the pre-training resolution (224×224 training, 256×256 evaluation). The resolution increase is possible because convolutional networks, unlike fixed-input transformers, can process arbitrary spatial dimensions — the convolutional filters simply slide over a larger input grid. However, the network's positional expectations (e.g., where certain features tend to appear) are calibrated for the pre-training resolution, so this adaptation is not free. Fine-tuning at the higher resolution allows the network to gradually adjust its representations to the new scale, and the even higher evaluation resolution (480×480) provides additional fine-grained spatial information at test time.

Why increase resolution for fine-tuning: higher resolution images contain more fine-grained detail (textures, object boundaries, small features) that can disambiguate between visually similar classes. On ImageNet, many classes differ only in subtle visual features (e.g., different dog breeds, different bird species), so resolution can be a significant accuracy lever. The specific choice of 384→480 follows the protocol used in state-of-the-art ImageNet submissions; Zhai et al. (2022) used similar resolution increases for their ViT fine-tuning.

Repeated augmentation (RA). For the strongest model (NFNet-F7+), the paper reports results both with and without repeated augmentation. RA (Hoffer et al., 2019; Fort et al., 2021) is a data augmentation strategy where multiple augmented views of the same image are included in the same training batch, with the loss computed as the average across views. The paper uses "augmentation multiplicity 4," meaning each image appears 4 times per batch with different random augmentations (random crops, flips, color jittering, etc.). When using RA, the authors "reduce the number of passes through the data such that the total computational cost of fine-tuning is constant" — meaning the model sees the same number of batches but each batch contains 4× fewer unique images (because each image is repeated 4 times with different augmentations).

Why RA: multiple augmented views per image provide a stronger training signal per image by exposing the model to more of the augmentation distribution. This effectively increases the diversity of the training data without requiring additional storage or I/O. The cost is that the model sees fewer unique images per unit of compute (because images are repeated within a batch), but the paper's cost-matching (reducing the number of passes to keep compute constant) controls for this.

The pre-training/checkpoint selection interaction. As noted earlier, the paper observes that "the pre-trained checkpoints achieving the lowest validation loss on JFT-4B did not always achieve the highest Top-1 accuracy on ImageNet after fine-tuning." Specifically, "under a fixed pre-training compute budget, the fine-tuning regime consistently favoured slightly larger models and slightly smaller epoch budgets." This means that the optimal model for transfer is not necessarily the one that fits the pre-training data best. There are (at least) two possible explanations:

1. Overfitting to JFT-specific features: a model that minimizes validation loss on JFT may have learned features that are highly predictive of JFT's 30,000 classes but do not transfer well to ImageNet's 1,000 classes (e.g., the model may rely on JFT-specific class distinctions that have no analogue in ImageNet).

2. Capacity for adaptation: larger models have more parameters and can more easily adapt to a new task during fine-tuning, even if they are slightly underfit on the pre-training task. The fine-tuning process can exploit this spare capacity to learn ImageNet-specific features that a smaller, more tightly-fit model cannot accommodate.

Interaction with learning rate during pre-training. The authors also note that "in some cases, slightly larger learning rates (during pre-training) also achieved better performance after fine-tuning." This is consistent with the idea that a slightly noisier, less-converged pre-training trajectory leaves the model in a broader basin that is easier to fine-tune away from — essentially, the model is less "committed" to its pre-training solution and can adapt more readily to the downstream task.

The ViT Comparison Methodology

The paper's central claim — that ConvNets match Vision Transformers at scale — requires a fair comparison methodology. The authors compare their NFNet results against two specific ViT models from the literature, plus one recent optimized ViT variant, with compute budgets estimated on the same hardware platform.

The ViT baselines. Three ViT models are used as comparison points:

1. ViT-g/14 from Zhai et al. (2022): a large Vision Transformer pre-trained on JFT-3B, achieving 90.2% ImageNet Top-1 accuracy. The paper estimates this model would require approximately 120k TPU-v4 core hours to pre-train.

2. ViT-G/14 from Zhai et al. (2022): an even larger Vision Transformer ("G" denotes the giant variant relative to "g"), also pre-trained on JFT-3B, achieving 90.45% Top-1. Estimated at 280k TPU-v4 core hours — more than 2× the cost of ViT-g/14 and roughly 2.5× the cost of the largest NFNet (F7+ at 110k core hours).

3. SoViT-400m/14 from Alabdulmohsin et al. (2023): an optimized ViT architecture that uses a compute-aware shape search to find better depth/width/MLP-dimension configurations, pre-trained on JFT-3B, achieving 90.3% Top-1. Estimated at 130k TPU-v4 core hours.

How the ViT compute estimates were obtained. The authors state: "We evaluated the pre-training speed for these models on TPU-v4 (using the original authors' codebase), and estimate that ViT-g/14 would take 120k TPU-v4 core hours to pre-train, while ViT-G/14 would take 280k TPU-v4 core hours and SoViT-400m/14 would take 130k TPU-v4 core hours."

This is a critical methodological detail. The original ViT papers reported their compute in TPU-v3 core hours (210k for ViT-g/14, >500k for ViT-G/14, 230k for SoViT-400m/14). Because TPU-v4 cores have roughly double the FLOPs of TPU-v3 cores, naively dividing by 2 would give estimates of 105k, 250k+, and 115k respectively. The paper's estimates (120k, 280k, 130k) are slightly higher than this simple FLOPs-based conversion, which likely reflects that the code does not achieve exactly 2× speedup on TPU-v4 due to I/O bottlenecks, communication overhead, or other factors that don't scale with peak FLOPs.

The hardware optimization caveat. The authors explicitly note a potential asymmetry in the comparison: "NFNets were optimized for TPU-v4, and perform less well when evaluated on other devices." They provide a specific example: "NFNet-F7+ would require 250 TPU-v3 core hours to pre-train for 8 epochs in our codebase." The factor between TPU-v4 (110k) and TPU-v3 (250k) for NFNet is approximately 2.3× — close to the theoretical 2× FLOP difference, but slightly higher, meaning NFNets achieve slightly less than the theoretical speedup. If the ViT baselines achieve a different speedup factor (e.g., closer to 2×), then comparing on TPU-v4 slightly favors NFNets. The authors do not report the speedup factors for the ViT baselines, so the magnitude of any asymmetry is unknown.

What the comparison does, and does not, control for. The paper matches pre-training compute budgets (TPU-v4 core hours) and pre-training dataset (JFT variants, with the NFNets using JFT-4B and ViTs using JFT-3B — a 1B-image difference in favor of NFNets). It does not match:

• Fine-tuning protocols: The ViT baselines may have used different fine-tuning strategies, numbers of epochs, optimizers, or regularizers. The paper does not describe or match the ViT fine-tuning recipes.
• Pre-training optimizer: NFNets use SGD with Momentum and AGC; ViTs typically use AdamW. These optimizers have different convergence properties and may interact differently with model scaling.
• Data augmentation during pre-training: The paper does not detail the augmentation strategy used during JFT-4B pre-training for NFNets, and it is not clear whether it matches what Zhai et al. used for ViTs.
• Precision and mixed-precision training: TPU-v4 training typically uses bfloat16 mixed precision, but implementation details of the precision scheme can affect both throughput and final accuracy.

Why these uncontrolled variables matter less than they might seem. The paper's argument is not that NFNets are precisely equivalent to ViTs with nanowatt-level matching of every hyperparameter. It is that the macroscopic scaling behavior — how performance improves with compute — is similar, and that the final accuracies land in the same ballpark. If ViTs had a fundamental architectural advantage, it would manifest as a large, systematic gap in final accuracy at matched compute, one that would persist even with reasonable variation in fine-tuning protocols. The fact that NFNets achieve 90.3–90.4% while ViTs achieve 90.2–90.45% at comparable compute suggests that any remaining differences are at the level of hyperparameter tuning and fine-tuning protocol details, not fundamental architectural scaling laws.

The placement of ViT data points in Figure 1. Figure 1 plots NFNet fine-tuning accuracy (Top-1 error) against pre-training compute, with three ViT reference points overlaid. The ViT-g/14 point sits near the middle of the NFNet trend line, the SoViT-400m/14 point slightly to the right (higher compute) at a similar error rate, and the ViT-G/14 point further to the right at a slightly lower error rate. The visual message is clear: the NFNet data points follow a trend that interpolates or encompasses the ViT points — meaning that at any given compute budget, an NFNet achieves roughly the same ImageNet accuracy as the best reported ViT at that budget. There is no visible systematic offset between the ConvNet and ViT scaling curves.

Training Infrastructure and Operational Details

Hardware platform. All training is conducted on TPU-v4 accelerators. The TPU-v4 is Google's fourth-generation tensor processing unit, designed specifically for machine learning workloads. Key architectural features include dedicated matrix-multiply units (MXUs) that achieve high utilization on the large matrix multiplications that dominate both ConvNet and ViT computation, high-bandwidth inter-chip interconnects (ICI) that enable efficient data-parallel and model-parallel distributed training across hundreds or thousands of chips, and support for bfloat16 mixed-precision training.

Distributed training setup. The batch size of 4096 combined with large models like NFNet-F7+ implies a substantial TPU pod configuration. At typical image throughputs, a batch size of 4096 images with a large model would require dozens to hundreds of TPU-v4 chips running in parallel, with each chip processing a subset of the batch (data parallelism) and gradients synchronized across chips via all-reduce operations. The paper does not specify the exact pod topology, but the pre-emption issues mentioned suggest training was conducted on shared, potentially pre-emptible TPU resources rather than dedicated, reservation-based allocations.

Data loading and the pre-emption problem. The authors mention that their "data loading pipeline did not guarantee that each training example would be sampled once per epoch if the training run was pre-empted/restarted." This reveals an operational detail: the data loading infrastructure was designed for best-effort throughput rather than deterministic data ordering. In an ideal system, the data loader would maintain a global state (e.g., a shuffle buffer index) that survives pre-emption and restart, ensuring that training resumes exactly where it left off. The paper's data loader apparently did not do this, meaning that after restart, the data order could shift, causing some examples to be seen multiple times and others to be missed.

Why this matters for the scaling law: if different training runs experienced different numbers of pre-emptions (which is likely, since pre-emption is stochastic), then two models trained with the same nominal epoch budget might have seen different effective data distributions. This introduces noise into the validation loss measurements, which could explain the non-smooth curve for F7+ in Figure 2. It also means the scaling law coefficients (slope, intercept) are estimated with some degree of measurement error. The fact that a clear log-log trend still emerges despite this noise strengthens the paper's conclusion — the underlying scaling signal is strong enough to survive imperfect experimental conditions.

Summary of Design Choices and Their Justifications

• NFNet architecture over ResNet: NFNets represent the 2021 state of the art in ConvNet design, incorporating normalizer-free training, architectural improvements (SE blocks, optimized stem/transition structures), and regularization techniques that make them competitive with ViTs. Using ResNets would have been comparing a 2016 architecture against 2020+ ViTs, which the paper identifies as the key flaw in prior comparisons.

• JFT-4B pre-training over training from scratch on ImageNet: web-scale pre-training on a massive dataset is the regime where ViTs were claimed to excel. Training NFNets from scratch on ImageNet would test performance in a regime (small data) where ConvNets are already known to work well, and would not address the claim that ViTs scale better when data is abundant.

• SGD with Momentum + AGC over Adam/AdamW: the standard NFNet optimizer, maintained for consistency with Brock et al. (2021), allowing the paper to claim "no changes to the training procedure." AGC is necessary to stabilize training without batch normalization. Switching to Adam would introduce a new variable (optimizer choice) that could confound the architecture comparison.

• Cosine decay over step decay or constant learning rate: provides smooth annealing to near-zero learning rate, which produces better final solutions in large-scale pre-training and is standard in both ConvNet and ViT pre-training literature.

• Two-dimensional sweep (model size × epoch budget) over one-dimensional scaling: necessary to identify the compute-optimal allocation between model capacity and training duration, and to establish that ConvNets obey the same type of scaling law as transformers (where this two-dimensional analysis was pioneered by Hoffmann et al., 2022).

• SAM for fine-tuning over standard SGD: SAM's flat-minimum bias acts as an implicit regularizer, which is particularly important when fine-tuning a massive pre-trained model on a relatively small downstream dataset (ImageNet's 1.28M images vs. JFT's 4B).

• Resolution increase for fine-tuning (384/480) over keeping pre-training resolution (224/256): higher resolution provides additional discriminative information that is particularly valuable for fine-grained ImageNet classes, and this resolution increase is standard practice in state-of-the-art ImageNet submissions.

• Repeated augmentation for the strongest model: provides additional training signal diversity without increasing the number of unique images processed, enabling the largest NFNet to squeeze out an additional 0.1% accuracy (90.3% → 90.4%).

• TPU-v4 core hours as the compute metric over raw FLOPs: accounts for real-world training efficiency including hardware utilization, communication, and I/O, making the comparison more practically relevant than theoretical FLOP counts. The re-benchmarking of ViT baselines on TPU-v4 (rather than simply converting from reported TPU-v3 hours using a fixed multiplier) attempts to make the comparison as fair as possible given hardware differences.

• Logarithmic grid (factor of 2) for learning rate tuning over finer grid or adaptive tuning: balances the cost of hyperparameter search against finding a near-optimal learning rate, with the reported regularities (optimal LR ≈ 1.6 at small budgets, decreasing with scale) intended to reduce the search cost for future work.

4. Key Insights and Innovations

Innovation 1: ConvNet Scaling Laws as a Diagnostic Refutation, Not a New Architecture

The paper's most distinctive intellectual move is not proposing a better ConvNet — it is using the existence of a clean log-log scaling law for ConvNets as a diagnostic tool to challenge a premature consensus. The field had accepted that Vision Transformers scale better than ConvNets without ever having measured ConvNet scaling behavior in the relevant regime. The paper treats this absence of evidence as a gap to be filled, and the resulting scaling law (Figure 2) functions as a refutation by counterexample: if ConvNets exhibit the same log-log relationship between validation loss and compute that transformers do (as established by Kaplan et al., 2020; Hoffmann et al., 2022 for language, and Zhai et al., 2022 for vision), then the belief that ViTs possess fundamentally superior scaling properties must be false.

Comparison to prior work. Before this paper, scaling laws had been demonstrated for transformers in two domains — language (Kaplan et al., 2020; Hoffmann et al., 2022) and vision (Zhai et al., 2022; Alabdulmohsin et al., 2023) — but notably absent for ConvNets in the web-scale pre-training regime. The field interpreted this absence as evidence of ConvNet scaling saturation: the implicit assumption was that if ConvNets could scale like ViTs, someone would have shown it. The paper reveals this as a sampling bias — no one had run the experiment because ConvNets had been prematurely abandoned as a research direction at the very moment they might have proven competitive. Tolstikhin et al. (2021) made a related point with MLP-Mixer (showing that even simple architectures could benefit from scale), but that work used a simple MLP baseline rather than a state-of-the-art ConvNet and did not systematically map the scaling curve.

Distinguishing incremental from fundamental. This is a fundamental conceptual contribution rather than an incremental refinement. The paper does not improve ConvNet accuracy by 0.5% through some architectural tweak — it establishes a new empirical regularity (ConvNets obey power-law scaling on web-scale data) that overturns a widely-held belief that shaped years of research prioritization. The practical implications are large (the field may have abandoned an entire architecture family for the wrong reasons), but the intellectual contribution is the diagnostic methodology itself: using scaling law measurement as a way to test claims about architectural superiority, rather than relying on point-comparisons between specific models at arbitrary compute budgets. The paper essentially argues that scaling behavior — the rate at which performance improves with compute, and the functional form of that improvement — is a more fundamental architectural property than any single accuracy number. And on that metric, ConvNets and ViTs are indistinguishable based on available evidence.

The specific observation that "the optimal model size and the optimal epoch budget both increase in size as the compute budget increases... scaling the model size and the number of training epochs at the same rate" (Section 3) further deepens this contribution by connecting ConvNet scaling directly to the Chinchilla framework. This is not merely "ConvNets also improve with more compute" — it is that ConvNets obey the same compute-optimal allocation rule (equal scaling of capacity and data) that Hoffmann et al. (2022) established as a general principle for transformer language models. The universality of this rule across architectures (transformers for text, transformers for images, and now ConvNets for images) suggests it may be a property of deep learning training dynamics rather than any specific architectural inductive bias — a hypothesis with implications far beyond the ViT-vs-ConvNet debate.

Evidence anchor. The linear trend in Figure 2 (log validation loss vs. log compute) across a ~275× compute range (0.4k to 110k TPU-v4 core hours) is the primary evidence, supported by the compute-optimal allocation rule described in the text.

---

Innovation 2: The "Bitter Lesson" Applied to Computer Vision Architecture Selection

The paper explicitly invokes the "bitter lesson" — the observation that general methods leveraging computation consistently outperform hand-designed, domain-specific approaches in the long run — but it does so in a way that transforms the lesson from a philosophical stance into a falsifiable empirical claim about a specific architecture comparison. The innovation is not the invocation of the bitter lesson itself (Sutton's essay is widely cited), but the paper's demonstration that the lesson applies within the modern era of deep learning architecture design: even the choice between convolutions and self-attention, which encode fundamentally different inductive biases about spatial structure, is ultimately dominated by compute and data scale rather than architectural details.

Comparison to prior work. The bitter lesson has typically been applied to contrasts between radically different paradigms — e.g., hand-crafted expert systems vs. learned neural approaches, or symbolic reasoning vs. statistical pattern recognition. Applying it to the ViT-vs-ConvNet debate is more subtle because both are learned neural architectures trained with similar optimization procedures; the debate was precisely about whether the inductive bias of convolutions (spatial locality, translation equivariance) was a help or a hindrance at scale. The ViT narrative argued that weaker inductive bias (self-attention's global receptive field with minimal spatial priors) wins at scale because the model can learn more flexible representations from massive data. The paper's evidence that ConvNets — with their strong spatial priors — scale just as well suggests that inductive bias is not the binding constraint on performance at the scales tested. Instead, compute and data availability are. This is a more specific and testable version of the bitter lesson than has typically been articulated.

Significance beyond performance. This reframing has implications for how the field allocates research effort. If the bitter lesson holds within modern architecture families, then the rapid churn of vision architectures (ResNet → ViT → Swin → ConvNeXt → etc.) may represent a misallocation of intellectual resources — researchers have been optimizing the wrong variable (architectural details) rather than the right one (scale). The paper's subtitle could be read as: "The bitter lesson, demonstrated empirically for ConvNets vs. ViTs." This is a significant reframing because it shifts the burden of proof: future claims that Architecture X scales better than Architecture Y must demonstrate a genuine difference in scaling exponents (not just point-comparisons at arbitrary budgets) and must control for the known confounds the paper identifies (weak baselines, unequal compute, dataset differences).

Distinguishing incremental from fundamental. This is a fundamental reframing of the computer vision architecture debate. The paper does not merely show that one particular ConvNet architecture is competitive — it provides evidence for a broader claim that architectural details (convolutions vs. self-attention) matter far less than scale, at least at the scales tested. The caveat is important (the paper is careful not to claim this holds for all possible architectures or all scales), but within the regime that the field has been operating in — pre-training on hundreds of millions to billions of images with TPU-scale compute — the architectural choice appears neutral. This is a negative result of sorts (neither architecture is "better"), but a productive negative result: it suggests that the field should invest more in data, compute, and training methodology, and less in architectural innovation for its own sake.

Evidence anchor. Figure 1, which shows NFNet and ViT data points interleaving along the same compute-performance trend line, is the visual centerpiece. The text explicitly cites Tolstikhin et al. (2021) as making a related point, and the Discussion section directly invokes the bitter lesson: "The most important factors determining the performance of a sensibly designed model are the compute and data available for training."

---

Innovation 3: The Pre-Training Validation Loss vs. Downstream Transfer Tradeoff as a Scaling Phenomenon

The paper reports a finding that is easy to overlook but has significant implications for transfer learning methodology: "the pre-trained checkpoints achieving the lowest validation loss on JFT-4B did not always achieve the highest Top-1 accuracy on ImageNet after fine-tuning," with the fine-tuning regime "consistently favouring slightly larger models and slightly smaller epoch budgets" (Section 3). This is not presented as a major theoretical contribution, but it constitutes a practically important diagnostic insight: the compute-optimal pre-training configuration (minimizing pre-training validation loss) is not the same as the transfer-optimal pre-training configuration (maximizing downstream accuracy after fine-tuning).

Comparison to prior work. The transfer learning literature has long known that pre-training accuracy and downstream accuracy are not perfectly correlated — this is one motivation for self-supervised pre-training objectives that prioritize general representations over task-specific accuracy. However, the paper's observation is more specific: within the same pre-training paradigm (supervised classification on JFT-4B), holding total pre-training compute fixed, the configuration that minimizes pre-training validation loss (slightly smaller model, slightly more epochs) does not maximize downstream accuracy (which prefers slightly larger model, slightly fewer epochs). This is a shift in the compute-optimal allocation rule: Hoffmann et al. (2022) optimized for minimizing language modeling loss, but if the goal is downstream transfer, the optimal parameters-change-ratio may differ.

Significance beyond performance. This finding matters for two reasons. First, it reveals a tension between pre-training optimization and downstream transfer that may be systematic rather than incidental. Larger models trained for fewer epochs may learn features that are more general and transferable (because they have more capacity and are less "committed" to the specific pre-training data distribution), while smaller models trained longer may overfit to pre-training-specific features that don't transfer. If this pattern generalizes, it would suggest that transfer-optimal pre-training requires over-parameterization relative to what is compute-optimal for the pre-training task alone — you want a model with more capacity than it "needs" for the pre-training task, because that spare capacity enables better adaptation during fine-tuning.

Second, it has immediate practical implications for model selection in transfer learning pipelines. If one is pre-training models specifically for downstream fine-tuning (which is the standard paradigm in modern computer vision), one should not select checkpoints based solely on pre-training validation loss. The paper's observation suggests either evaluating multiple checkpoints on the downstream task (expensive) or developing better proxies for transfer quality that can be measured during pre-training (a research problem the paper implicitly identifies but does not solve).

Distinguishing incremental from fundamental. This is an incremental empirical finding rather than a fundamental theoretical advance, but it has practical significance for anyone conducting large-scale transfer learning. It complicates the clean scaling-law picture from Figure 2 by revealing that the metric being optimized (pre-training validation loss) is not perfectly aligned with the metric that ultimately matters (downstream accuracy after fine-tuning). The paper does not quantify the magnitude of this misalignment, which limits how actionable the insight currently is.

Evidence anchor. The observation is reported in the text of Section 3 with the explicit statement quoted above. The fine-tuning results in Figure 1 reflect the best-performing checkpoints for each model, which implicitly uses downstream accuracy as the selection criterion rather than pre-training validation loss.

---

Innovation 4: Predictable Learning Rate Scaling as a Cost-Reduction Heuristic

The paper's empirical characterization of how the optimal learning rate varies with model size and epoch budget (Figure 3, Section 3) constitutes a practical innovation: a systematic, predictive pattern that can reduce the hyperparameter tuning cost for large-scale ConvNet pre-training by roughly a factor of 2–4. The observation that optimal learning rate is approximately 1.6 for small epoch budgets across all model sizes, and that it falls "slowly but monotonically as both the model size and the epoch budget increases," with larger models showing faster decay, provides a concrete tuning recipe — start from the known optimum for a small model at a small budget, then reduce the learning rate based on how much larger the model and how many more epochs are used, with the claim that "one can efficiently tune the learning rate within 2 trials."

Comparison to prior work. Learning rate sensitivity is a known practical challenge in large-scale training, but the typical approach is blind grid search or relying on heuristics like the linear scaling rule (learning rate ∝ batch size). The paper goes beyond these by characterizing the interaction between model size and training duration in determining the optimal learning rate, and by demonstrating that the pattern is consistent enough across different NFNet variants (F0, F3, F7+) to serve as a predictive heuristic. This is similar in spirit to the learning rate scaling rules explored in the large-batch training literature (e.g., Goyal et al., 2017), but extended to the model-size dimension and grounded in empirical measurement at web-scale rather than small-scale extrapolation.

Significance beyond performance. The practical value of this finding is straightforward: it reduces the cost of scaling experiments. At the largest budget (110k TPU-v4 core hours), each learning rate trial is extremely expensive. Being able to narrow the search to 2 trials instead of, say, 8 (a logarithmic grid of factor-of-2 spacing over a plausible range) represents a substantial resource savings for anyone attempting to reproduce or extend this work. However, it is worth noting that the paper's specific numbers (optimal LR ≈ 1.6, the shape of the decay curve) are likely specific to the NFNet architecture and the SGD + Momentum + AGC optimizer configuration; they may not transfer directly to other architectures or optimizers without re-calibration.

Distinguishing incremental from fundamental. This is an incremental practical contribution, not a theoretical advance. It does not explain why the optimal learning rate follows this pattern (the paper does not attempt a theoretical analysis), and its generality to other architectures and optimizers is untested. It falls into the category of "useful empirical rules of thumb" that make large-scale experimentation more practical but do not change the conceptual understanding of deep learning.

Evidence anchor. Figure 3 plots the optimal learning rate for three models (F0, F3, F7+) across epoch budgets from 0.25 to 8, showing the shared starting point (≈1.6) and the model-size-dependent decay pattern. The "within 2 trials" claim appears in the main text.

---

Innovation 5: Hardware-Aware Compute Comparison as a Methodological Standard

The paper's approach to comparing across architectures — re-benchmarking ViT baselines on the same hardware (TPU-v4) using the original authors' code rather than simply converting reported TPU-v3 core hours using a theoretical FLOP multiplier — establishes a methodological standard for fair architecture comparison that goes beyond what is typical in the field. The paper transparently reports the conversion factors (NFNet-F7+ costs ~250k TPU-v3 hours vs. 110k TPU-v4 hours, a ~2.3× factor rather than the theoretical 2×), explicitly notes the hardware optimization asymmetry ("NFNets were optimized for TPU-v4"), and acknowledges that the comparison is hardware-specific. This level of methodological transparency around compute measurement is uncommon in architecture comparison papers, and it raises the bar for what constitutes a fair comparison.

Comparison to prior work. Most architecture comparison papers either compare on the basis of parameter count (which ignores differences in per-parameter computational cost), theoretical FLOPs (which ignores hardware utilization), or reported training time on whatever hardware was used (which confounds hardware efficiency with architectural efficiency when comparing across papers that used different hardware generations or configurations). The paper's approach of re-running baseline code on the same hardware to get matched measurements is more rigorous, and its transparency about remaining asymmetries (TPU-v4 optimization favoring NFNets) is a model for how to handle unavoidable confounds honestly.

Significance beyond performance. This is a methodological contribution: it demonstrates a template for how the field should compare architectures when the claim is about computational efficiency (not just final accuracy). The key elements are: (1) measure compute in practical units (core hours) that account for real hardware utilization, not just theoretical FLOPs; (2) re-benchmark baselines on the same hardware whenever possible; (3) explicitly report and discuss remaining hardware-specific asymmetries rather than hiding them. If this standard were widely adopted, it would substantially reduce the number of spurious "Architecture X is more efficient than Architecture Y" claims that arise from comparing FLOP counts rather than wall-clock training time on matched hardware.

Distinguishing incremental from fundamental. This is an incremental methodological contribution. The basic principle (compare on matched hardware) is not novel, and many practitioners already do this informally. The paper's contribution is in making this practice explicit, transparent, and central to the argument — demonstrating that it can be done even when comparing against published baselines that used different hardware generations. The paper's acknowledgment that NFNets were optimized for TPU-v4 and may be less competitive on other hardware also models the kind of honest uncertainty reporting that strengthens rather than weakens a scientific claim.

Evidence anchor. The specific TPU-v4 hour estimates (120k for ViT-g/14, 280k for ViT-G/14, 130k for SoViT-400m/14) and the TPU-v3 comparison (250k for NFNet-F7+) are reported in Section 3, along with the explicit caveat about TPU-v4 optimization. Figure 1 plots all models on the same compute axis using these re-benchmarked estimates.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The pre-training dataset is JFT-4B, an internal Google dataset containing "roughly 4 billion labelled images from 30k classes" (Sun et al., 2017), with near-duplicates of ImageNet training and validation images removed following Kolesnikov et al. (2020). The downstream evaluation dataset is ImageNet ILSVRC-2012 (approximately 1.28M training images across 1,000 classes, 50,000 validation images). Validation loss during pre-training is measured on "a held out set of 130k images" from JFT-4B.

• Base model(s). The paper evaluates the NFNet model family (Brock et al., 2021), a pure convolutional architecture that uses Adaptive Gradient Clipping (AGC) and weight standardization instead of batch normalization. The specific variants trained span a depth-width scaling grid: F0 (smallest), F1 (2× depth of F0), F3 (4× depth of F0), F3+ (F3 depth with increased width), F7 (8× depth of F0, same width as F3), and F7+ (F7 depth with increased width). The authors state they "do not make any changes to the model architecture or the training procedure (beyond tuning simple hyper-parameters such as the learning rate or epoch budget)," deliberately testing whether the 2021 ConvNet state-of-the-art scales when given ViT-comparable resources.

• Metrics. The two primary metrics are: (1) JFT-4B validation loss, measured as held-out cross-entropy loss on 130k images after pre-training, used to establish the scaling law in Figure 2; and (2) ImageNet Top-1 accuracy after fine-tuning, reported as the fraction of the 50,000 ImageNet validation images correctly classified by the model's highest-confidence prediction, used for the architecture comparison in Figure 1. The paper also reports ImageNet Top-1 error (100% minus accuracy) on the y-axis of Figure 1.

• Baselines. Three Vision Transformer models from prior work serve as comparison points, with their pre-training compute costs re-estimated on TPU-v4 hardware using the original authors' code:

  • ViT-g/14 (Zhai et al., 2022): Pre-trained on JFT-3B, achieving 90.2% ImageNet Top-1 accuracy, estimated at approximately 120k TPU-v4 core hours.
  • ViT-G/14 (Zhai et al., 2022): A larger ViT variant pre-trained on JFT-3B, achieving 90.45% Top-1, estimated at approximately 280k TPU-v4 core hours.
  • SoViT-400m/14 (Alabdulmohsin et al., 2023): A compute-optimized ViT architecture pre-trained on JFT-3B, achieving 90.3% Top-1, estimated at approximately 130k TPU-v4 core hours.

  The authors also reference the original NFNet-F5 trained on ImageNet without extra data (achieving 86.8% Top-1 with repeated augmentation from Fort et al., 2021) as an implicit baseline to demonstrate the benefit of JFT-4B pre-training for ConvNets.

• Generation budget / compute accounting. Compute is measured in TPU-v4 core hours, with the authors stating they "estimate the compute required to train each model by eye from the typical steps per second achieved by each model during training (when not pre-empted)." TPU-v4 cores have "roughly double the theoretical flops of TPU-v3 cores, however both cores have similar memory." The compute numbers on the x-axes of Figures 1 and 2 represent pre-training compute only; fine-tuning costs are reported separately (1.6k TPU-v4 core hours for NFNet-F7+). The range of pre-training budgets spans approximately 0.4k to 110k TPU-v4 core hours, a roughly 275× increase. The authors acknowledge that ViT compute estimates were obtained by re-running the original code on TPU-v4 hardware, while NFNets "were optimized for TPU-v4, and perform less well when evaluated on other devices," explicitly noting this hardware optimization asymmetry.

• Cross-validation / statistical protocol. There is no formal cross-validation or statistical significance testing reported. The paper does not report confidence intervals, standard deviations, or error bars on any measured quantities. The two-fold cross-validation protocol for compute-optimal strategy selection described in the reference example does not apply, as this paper tests the scaling behavior of existing architectures rather than selecting among strategies. The only implicit validation procedure is the learning rate tuning: "the base learning rate is tuned separately for each epoch budget on a small logarithmic grid" (spaced by factors of 2), with the optimal learning rate selected as the one that minimizes validation loss on the 130k-image held-out set.

Main Quantitative Results

Pre-Training Scaling Laws (Validation Loss)

The paper's primary quantitative finding is the log-log scaling relationship between JFT-4B validation loss and pre-training compute, shown in Figure 2. The authors observe "a clear linear trend, consistent with a log-log scaling law between validation loss and pre-training compute," with the six NFNet variants (F0 through F7+) each forming a curve across epoch budgets from 0.25 to 8 epochs.

The validation loss decreases from approximately 3.0 for the smallest/cheapest configurations (F0 at low epoch budgets, near 10^3 TPU-v4 core hours) to approximately 2.4 for the largest/most expensive (F7+ at 8 epochs, near 110k TPU-v4 core hours). This represents a loss reduction of roughly 0.6 over approximately 2.4 orders of magnitude in compute. The paper does not report the numerical scaling exponent (slope), but the linearity on log-log axes implies a power-law relationship of the form $L(C) \propto C^{-\alpha}$.

A specific operational finding: "the optimal epoch budget was greater than 1 for overall compute budgets greater than roughly 5k TPU-v4 core hours." Below 5k core hours, training for less than one full pass through JFT-4B (epoch budget <1) minimizes validation loss; above this threshold, it becomes optimal to make multiple passes through the data. This threshold is specific to the JFT-4B dataset size — with 4 billion images, one epoch represents an enormous amount of data, and models at lower compute budgets have insufficient capacity to benefit from seeing the same image twice.

The compute-optimal allocation rule extracted from the two-dimensional sweep (model size × epoch budget) is: "a reliable rule of thumb is to scale the model size and the number of training epochs at the same rate." This is directly parallel to the Chinchilla finding for language models (Hoffmann et al., 2022) and generalizes that principle to convolutional architectures for vision.

A data quality caveat is explicitly noted: "the curve for NFNet-F7+ models at different pre-training budgets is not smooth," attributed to the data loading pipeline not guaranteeing uniform sampling per epoch after pre-emptions and restarts, "potentially causing some training examples to be under-sampled if a training run was restarted multiple times." This means the F7+ scaling curve in Figure 2 contains measurement noise from imperfect data ordering, though the overall log-log trend remains visually apparent.

Fine-Tuning Results: ConvNets vs. Vision Transformers on ImageNet

Figure 1 presents the paper's headline comparison: ImageNet Top-1 error (log-scaled) against pre-training compute (log-scaled) for NFNets and the three ViT baselines. The key finding is that NFNets and ViTs occupy the same compute-performance tradeoff curve with no visible architecture-specific offset.

The specific NFNet results after fine-tuning (50 epochs with SAM, stochastic depth, and dropout at resolution 384/480):

• NFNet-F7+ (110k TPU-v4 core hours pre-training, 1.6k fine-tuning): Achieves 90.3% Top-1 accuracy without repeated augmentation, and 90.4% Top-1 with repeated augmentation at multiplicity 4 (reducing the number of passes through the data to keep total fine-tuning compute constant). This is the paper's strongest reported result.

• ViT-g/14 (~120k TPU-v4 core hours): 90.2% Top-1 from Zhai et al. (2022). The NFNet-F7+ at comparable compute (110k) achieves 90.3–90.4%, slightly exceeding the ViT.

• SoViT-400m/14 (~130k TPU-v4 core hours): 90.3% Top-1 from Alabdulmohsin et al. (2023). The NFNet at slightly lower compute matches this exactly (90.3% without RA) or slightly exceeds it (90.4% with RA).

• ViT-G/14 (~280k TPU-v4 core hours): 90.45% Top-1 from Zhai et al. (2022). This ViT uses approximately 2.5× more pre-training compute than the largest NFNet. The 0.05 percentage point gap (90.45% vs. 90.4%) is negligible, but the NFNet achieves essentially equivalent accuracy at substantially lower compute. The paper does not test NFNets at the 280k core-hour budget to determine whether they would match or exceed 90.45%.

The visual arrangement in Figure 1 shows the NFNet data points forming a descending curve (decreasing error with increasing compute), with the three ViT reference points distributed along the same trend. The ViT-g/14 point sits near the middle of the NFNet trend, SoViT-400m/14 slightly to the right at similar error, and ViT-G/14 further to the right at marginally lower error. The authors interpret this as demonstrating that "the performance of pre-trained NFNets at scale is remarkably similar to the performance of pre-trained Vision Transformers."

The authors also provide a within-family baseline comparison: "the best reported Top-1 accuracy of an NFNet on ImageNet without extra data is 86.8% (Fort et al., 2021), achieved by an NFNet-F5 with repeated augmentation." The jump from 86.8% (no extra data) to 90.4% (JFT-4B pre-training) — a 3.6 percentage point absolute improvement — demonstrates "that NFNets benefit substantially from large scale pre-training."

Learning Rate Scaling Behavior

Figure 3 characterizes how the optimal learning rate varies with model size and epoch budget for three representative models (F0, F3, F7+). The key empirical observations:

1. Shared starting point: "All models show similar optimal learning rates $\alpha \approx 1.6$ when the epoch budget is small" (0.25 epochs). The optimal learning rate at minimal training duration is approximately architecture-invariant within the NFNet family.

2. Monotonic decrease with epoch budget: As the epoch budget increases from 0.25 to 8, the optimal learning rate decreases for all models. For F0 (the smallest), the optimal rate drops from roughly 1.6 to roughly 0.8 — a factor of 2 reduction. For F7+ (the largest), the drop is steeper, falling from roughly 1.6 to roughly 0.4 — a factor of 4 reduction.

3. Model-size dependence of the decay rate: "For large models the optimal learning rate falls more quickly" as epoch budget increases. The F0 and F3 curves are relatively close together across the epoch range, while F7+ separates from them substantially at higher epoch budgets (epochs ≥2), with its optimal learning rate falling below 0.8 by epoch 4 and reaching approximately 0.4 at epoch 8.

The practical prescription: "one can efficiently tune the learning rate within 2 trials by assuming that the optimal learning rate falls slowly but monotonically as both the model size and the epoch budget increases." The specific tuning strategy implied is to start from $\alpha \approx 1.6$ for small models at small epoch budgets, then reduce the learning rate by a factor that increases with both model size and epoch budget, requiring at most one or two verification runs to narrow the exact value.

Transfer Learning Dynamics: Pre-Training Loss vs. Downstream Accuracy

The paper reports a non-obvious finding about the relationship between pre-training validation loss and downstream transfer performance. The authors state that "the pre-trained checkpoints achieving the lowest validation loss on JFT-4B did not always achieve the highest Top-1 accuracy on ImageNet after fine-tuning." Specifically, "under a fixed pre-training compute budget, the fine-tuning regime consistently favoured slightly larger models and slightly smaller epoch budgets." Additionally, "in some cases, slightly larger learning rates (during pre-training) also achieved better performance after fine-tuning."

This means the compute-optimal configuration for minimizing pre-training loss (which the scaling law in Figure 2 describes) is not identical to the transfer-optimal configuration for maximizing downstream accuracy. At a fixed total pre-training compute budget, the best ImageNet accuracy comes from diverting some of the compute allocation away from training duration (epochs) and toward model capacity (parameters), relative to what would minimize validation loss. The paper does not quantify the magnitude of this shift — it does not report how much larger the model should be or how much the epoch budget should be reduced — making this a qualitative observation rather than a prescriptive rule.

Ablation Studies and Robustness Checks

This paper is notably sparse on formal ablation studies compared to typical machine learning papers, reflecting its nature as a scaling investigation rather than a methods paper proposing new components. The authors made a deliberate choice not to modify the NFNet architecture or training procedure, so traditional ablation studies (varying architectural components, loss functions, or regularizers) are absent. However, several implicit ablations and robustness checks can be identified from the experimental design and reported observations:

Model scale sweep as implicit architecture ablation: By training six NFNet variants (F0 through F7+) across a range of epoch budgets, the paper implicitly ablates the effect of model depth and width on scaling behavior. The consistent log-log trend across all variants (Figure 2) suggests that the scaling law is robust to the specific depth-width configuration — it is a property of the NFNet family as a whole rather than a particular architecture size. The smooth progression of the curves (with the F7+ irregularity attributed to data loading rather than architecture) supports this.

Epoch budget sweep as implicit data ablation: Training each model at epoch budgets from 0.25 to 8 effectively varies the total number of images seen during pre-training from 1 billion to 32 billion. The consistent improvement in validation loss across this range (Figure 2) demonstrates that ConvNet performance on JFT-4B has not saturated at 8 epochs — the models continue to benefit from seeing more data, which is consistent with the power-law scaling hypothesis (saturation would manifest as a downward curvature on the log-log plot, which is not apparent in Figure 2).

Repeated augmentation (RA) as a fine-tuning robustness check: The paper reports NFNet-F7+ results both with and without repeated augmentation during fine-tuning. Without RA: 90.3% Top-1. With RA at multiplicity 4: 90.4% Top-1. This 0.1 percentage point gain confirms that the fine-tuning result is robust to the choice of augmentation strategy — the RA improvement is small and does not fundamentally change the comparison against ViTs (which achieves 90.2–90.45%). The fact that the paper controls for fine-tuning compute when using RA ("we reduce the number of passes through the data such that the total computational cost of fine-tuning is constant") ensures that the RA gain is not simply a consequence of more fine-tuning compute.

Learning rate grid search as optimizer sensitivity check: The learning rate tuning across a logarithmic grid (factor-of-2 spacing) for each (model, epoch budget) combination implicitly measures how sensitive final performance is to the learning rate. The fact that clear patterns emerge (Figure 3) — all models share $\alpha \approx 1.6$ at small epoch budgets, and the optimal rate falls monotonically — suggests that performance is not pathologically sensitive to the learning rate (if it were, the curves would be noisy rather than smooth). However, the paper does not report the performance degradation at sub-optimal learning rates, so the quantitative sensitivity is unknown.

ViT re-benchmarking on TPU-v4 as a hardware fairness check: By re-running the ViT baselines on TPU-v4 hardware rather than simply converting reported TPU-v3 hours to TPU-v4 equivalents using a fixed multiplier, the paper implicitly ablates the effect of hardware generation on the comparison. The specific estimates (120k, 280k, 130k TPU-v4 core hours) differ from what a naive 2× speedup calculation from reported TPU-v3 hours would yield, confirming that real-world speedups are not exactly 2× and that re-benchmarking is necessary for accurate comparison.

TPU-v3 vs. TPU-v4 efficiency for NFNets as a hardware sensitivity check: The paper reports that NFNet-F7+ would require approximately 250k TPU-v3 core hours versus 110k TPU-v4 core hours — a factor of approximately 2.3× rather than the theoretical 2×. This implicitly ablates the hardware efficiency claim by demonstrating that the factor is hardware-specific and that NFNets underperform the theoretical speedup on TPU-v4 relative to TPU-v3. The practical implication is that on TPU-v3 hardware, NFNets would appear less competitive relative to ViTs (since ViTs may achieve a speedup factor closer to 2×, though this is not measured).

Resolution change during fine-tuning as a transfer protocol check: The shift from pre-training resolution (224×224 training, 256×256 evaluation) to fine-tuning resolution (384×384 training, 480×480 evaluation) represents a significant resolution increase. The fact that models fine-tuned at higher resolution still achieve competitive accuracy is an implicit robustness check: the pre-trained representations transfer across resolutions without catastrophic degradation. However, there is no ablation testing whether fine-tuning at 224×224 would achieve similar accuracy (i.e., whether the resolution increase is necessary for the top results or merely helpful).

Missing formal ablations: Several ablations that would strengthen the paper are absent. There is no comparison against a ResNet baseline pre-trained on JFT-4B with equivalent compute to validate the claim that prior ViT comparisons used "weak ConvNet baselines" — showing how much worse a ResNet would perform at matched compute would quantify the importance of using a strong ConvNet architecture. There is no ablation comparing SGD with Momentum against Adam/AdamW for NFNet pre-training to determine whether the optimizer choice affects the scaling behavior. There is no ablation of the NFNet-specific components (AGC, weight standardization, SE blocks) to determine which architectural innovations are responsible for the competitive scaling behavior versus a simpler ConvNet. There is no measurement of whether the scaling law continues beyond 110k TPU-v4 core hours (the maximum tested) — the linear trend in Figure 2 shows no sign of saturation, but this is extrapolation rather than evidence.

Critical Assessment

Does the evidence support the claim that ConvNets match Vision Transformers at scale?

The central claim — that pre-trained ConvNets match pre-trained ViTs when compared at equivalent compute budgets — is supported within the specific conditions tested, but the evidence is narrower than the claim's apparent generality and several important dimensions of "matching" are left untested.

What the evidence actually shows: Figure 1 demonstrates that NFNet-F7+ at 110k TPU-v4 core hours achieves ImageNet Top-1 accuracy (90.3–90.4%) that falls within the range of reported ViT accuracies at comparable pre-training compute (90.2% for ViT-g/14 at ~120k hours, 90.3% for SoViT-400m/14 at ~130k hours, 90.45% for ViT-G/14 at ~280k hours). The ViT data points visually lie on the same compute-performance trend as the NFNet data points. This is a comparison at the level of ImageNet fine-tuning accuracy after JFT pre-training.

What the evidence does not show: The paper does not establish that ConvNets and ViTs have the same scaling exponent. The Visual similarity of the curves in Figure 1 is suggestive but not quantified — there is no fitted slope comparison, no statistical test of whether the NFNet and ViT trends are parallel, and no demonstration that ConvNets would achieve ViT-G/14-level accuracy (90.45%) if scaled to the same 280k TPU-v4 core-hour budget. The claim of "matching" is therefore based on interpolation and visual alignment rather than a quantitative demonstration of equivalent scaling behavior across the full compute range.

The hardware optimization asymmetry is a genuine confound: The paper acknowledges that "NFNets were optimized for TPU-v4, and perform less well when evaluated on other devices," while the ViT baselines were originally developed and optimized on TPU-v3 and then re-benchmarked on TPU-v4 using the original code (which may not be TPU-v4-optimized). The NFNet TPU-v3-to-TPU-v4 speedup is ~2.3× (250k to 110k hours), which is worse than the theoretical 2×. If ViTs achieve a speedup closer to 2× on the same hardware transition, then comparing on TPU-v4 core hours gives NFNets an artificial advantage — the "matched" compute budgets would actually represent more FLOPs for the ViT models. The paper does not report the ViT speedup factors, so the magnitude of any asymmetry is unknown. This uncertainty makes the "match" claim less precise than it appears.

Only one ConvNet architecture is evaluated. All results come from the NFNet family, which the paper itself describes as "the last ConvNet to set a new SOTA on ImageNet" without extra data. This is a single data point for the hypothesis that "ConvNets match ViTs." Other strong ConvNet architectures from the same era — EfficientNetV2, ConvNeXt, RepVGG — are not tested. If NFNets are uniquely well-suited to large-scale pre-training (perhaps due to their normalizer-free design or specific regularization choices), the paper's conclusion might not generalize to ConvNets in general. The absence of even a single alternative ConvNet architecture makes the evidence architecture-specific rather than architecture-class-wide.

Pre-training data is not perfectly matched. The NFNets are pre-trained on JFT-4B (4 billion images), while the ViT baselines are pre-trained on JFT-3B (3 billion images). The 1-billion-image difference (33% more data) favors the NFNets, though the paper's scaling law analysis shows that data quantity matters less than total compute at these scales (since the epoch budget is typically <1, meaning the model never exhausts the data). The effect of this dataset size mismatch on the final comparison is likely small — both datasets are so large that the binding constraint is model capacity and training compute, not the total number of unique images available — but it is an uncontrolled variable.

Fine-tuning protocols are not matched. The paper uses a specific fine-tuning recipe for NFNets (50 epochs, SAM, stochastic depth, dropout, resolution 384/480) but does not match this recipe to what was used for the ViT baselines. Zhai et al. (2022) and Alabdulmohsin et al. (2023) may have used different fine-tuning strategies, different numbers of epochs, or different resolution schemes. The final ImageNet accuracy reflects both pre-training quality and fine-tuning effectiveness; if the ViT fine-tuning protocols were suboptimal relative to the NFNet protocol, the "match" in final accuracy could mask a difference in pre-training representation quality. The paper does not control for this.

The comparison is only at one downstream task (ImageNet). The claim that ConvNets match ViTs is based exclusively on ImageNet Top-1 accuracy after fine-tuning. ImageNet is the canonical vision benchmark, but it is a single dataset with specific properties (1,000 classes, curated images, object-centric). There is no evaluation on other downstream tasks (object detection, segmentation, video understanding) or on other classification benchmarks (CIFAR, iNaturalist, Places) that might reveal architecture-specific strengths or weaknesses. It is possible that ViTs transfer better to certain task types (e.g., those requiring global context) while ConvNets transfer better to others (e.g., those requiring fine-grained spatial localization), and the single-task evaluation misses this dimensionality.

The ViT baselines may not be compute-optimal. The ViT-g/14, ViT-G/14, and SoViT-400m/14 are specific models from the literature, but they may not represent the best possible ViT at each compute budget. Alabdulmohsin et al. (2023) showed that ViT architecture can be optimized for a given compute budget, but the ViT-g/14 and ViT-G/14 from Zhai et al. (2022) were not necessarily compute-optimal. If a compute-optimized ViT at 110k core hours would achieve 90.7% rather than 90.3%, the "match" would disappear. The paper compares against the reported ViTs, not the best possible ViTs at each budget.

Does the evidence support the scaling law claim?

The log-log scaling law in Figure 2 is the paper's most robust empirical contribution, but several methodological weaknesses temper its precision.

The compute measurement is approximate. The authors "estimate the compute required to train each model by eye from the typical steps per second achieved by each model during training (when not pre-empted)." "By eye" implies a non-systematic measurement — observing stable training periods and averaging throughput informally rather than instrumenting the training code to log exact step times and device counts. The pre-emption issue compounds this: if throughput varies after restarts (due to different device allocations), a single "typical" steps-per-second value may not accurately represent the entire training run. This makes the x-axis values in Figure 2 approximate rather than precise, and the scaling law coefficients (slope, intercept) are correspondingly noisy.

Pre-emption-induced data sampling noise affects the validation loss. The paper explicitly notes that "our data loading pipeline did not guarantee that each training example would be sampled once per epoch if the training run was pre-empted/restarted, potentially causing some training examples to be under-sampled if a training run was restarted multiple times." This means the effective data distribution varies across training runs in ways that are not captured by the nominal epoch budget. The "not smooth" F7+ curve in Figure 2 is attributed to this effect. The measured validation loss for any given (model, epoch budget) point is therefore a combination of the true scaling relationship plus noise from variable data sampling, making the scaling law less precisely measured than the clean linear trend suggests.

No scaling exponent is reported. Despite observing a log-log linear trend, the paper does not fit a power law or report the scaling exponent $\alpha$. The scaling law is presented qualitatively ("a clear linear trend") rather than quantitatively. This makes it impossible to compare the ConvNet scaling rate against the ViT scaling rate. If ViTs have $\alpha = 0.30$ and ConvNets have $\alpha = 0.25$, both would show log-log linearity, but ViTs would pull ahead at larger compute budgets. The paper cannot rule this out because it does not measure the slope.

The scaling law is only tested across a ~275× compute range. From 0.4k to 110k TPU-v4 core hours is a substantial range, but it is modest compared to the 1000×+ ranges explored in language model scaling studies (Kaplan et al., 2020; Hoffmann et al., 2022). The linear trend in Figure 2 shows no sign of saturation, but the paper cannot determine whether the power law continues beyond 110k core hours or whether ConvNets eventually saturate while ViTs do not. The absence of extrapolation makes the scaling law a description of the observed range rather than a predictive model.

Validation loss is an imperfect proxy for downstream performance. The scaling law is measured on JFT-4B validation loss, but the paper itself demonstrates that "the pre-trained checkpoints achieving the lowest validation loss on JFT-4B did not always achieve the highest Top-1 accuracy on ImageNet after fine-tuning." This means the metric being scaled (pre-training validation loss) is not perfectly aligned with the metric that matters for the architecture comparison (downstream accuracy). A ConvNet scaling law measured on validation loss might look favorable, but if ConvNets systematically require more fine-tuning compute or architecturally specific fine-tuning recipes to convert pre-training quality into downstream accuracy, the scaling law overstates practical utility.

Does the evidence support the claimed universality of the "bitter lesson" interpretation?

The paper concludes that "the most important factors determining the performance of a sensibly designed model are the compute and data available for training," invoking the bitter lesson. The evidence supports a narrower version of this claim: that for image classification, when comparing a specific strong ConvNet family (NFNets) against a specific set of ViT models trained on similar data, at the specific compute scale tested (0.4k–110k TPU-v4 core hours), architecture choice (ConvNet vs. ViT) does not produce a large, systematic accuracy difference.

What is untested: The paper does not test whether any sensibly designed architecture would match at scale. MLP-Mixer (Tolstikhin et al., 2021) was shown to be competitive at scale, but the paper does not compare it against NFNets. There is no evidence that a poorly designed architecture — one without residual connections, or with pathological gradient propagation, or with a fundamentally broken inductive bias — would also match at scale. The "sensibly designed" qualifier is doing important work here, and the paper does not characterize what design properties are necessary for an architecture to benefit from scale.

The scale ceiling is untested. The bitter lesson implies that as compute continues to increase, the advantage of general-purpose methods over domain-specific ones grows. The paper's evidence is at a particular scale (~100k TPU-v4 core hours). It does not test whether ConvNets and ViTs diverge at larger scales — 500k, 1M, or 10M core hours. If ViTs have a fundamentally different (steeper) scaling exponent, the match at 100k hours would be a transient coincidence, and the bitter lesson would actually favor ViTs (the architecture with the weaker inductive bias) in the long run.

The "sensibility" of ViT design choices is not evaluated. The paper treats ViTs as a single architecture, but the ViT literature has explored many variants — patch size, attention mechanism (standard vs. windowed vs. global), positional encoding scheme, MLP design, normalization placement. These design choices affect performance at scale (Alabdulmohsin et al., 2023). The fact that NFNets match some ViTs does not mean they match all ViTs or the best possible ViT at each compute budget. The bitter lesson claim is about architectures in general; the evidence is about one ConvNet vs. several ViTs.

Summary of missing evidence that would strengthen the paper

1. A ResNet baseline pre-trained on JFT-4B at matched compute to quantify the importance of using a strong ConvNet architecture and validate the claim that prior comparisons used weak baselines.

2. NFNets trained at the full ViT-G/14 budget (~280k core hours) to determine whether the scaling trend continues and whether ConvNets match the highest ViT accuracy.

3. At least one alternative ConvNet architecture (EfficientNet, ConvNeXt) pre-trained at comparable scale to test whether the result is NFNet-specific or generalizes across the ConvNet family.

4. Quantitative fitting of scaling law coefficients for both NFNets and ViTs, enabling comparison of scaling exponents rather than point-accuracy at specific budgets.

5. Confidence intervals or error bars on validation loss and ImageNet accuracy measurements to assess whether small differences (90.3% vs. 90.2%) are statistically meaningful.

6. Downstream evaluation beyond ImageNet (object detection, segmentation, transfer to other classification datasets) to test whether the ViT-ConvNet match holds across task types.

7. Fine-tuning protocol matching or ablation to verify that the accuracy comparison reflects pre-training quality differences rather than fine-tuning recipe differences.

8. Measurement of ViT speedup factors on TPU-v4 to quantify the hardware optimization asymmetry and determine its impact on the core-hour comparison.

9. A broader range of scale — extending to both smaller budgets (to characterize the early scaling regime) and larger budgets (to test for divergence) — to establish the robustness of the log-log relationship beyond the 275× range tested.

6. Limitations and Trade-offs

6.1 The Hardware-Specific Nature of the Comparison Limits Generalization

The assumption or constraint. The paper's central claim — that ConvNets match Vision Transformers at scale — rests on a comparison conducted on TPU-v4 hardware, and the authors explicitly acknowledge an asymmetry: "NFNets were optimized for TPU-v4, and perform less well when evaluated on other devices." The ViT baselines, by contrast, were originally developed and optimized for TPU-v3, and their TPU-v4 costs were estimated by re-running the original codebase on the newer hardware — code that was not necessarily optimized for TPU-v4. The paper reports that NFNet-F7+ would require approximately 250k TPU-v3 core hours versus 110k TPU-v4 hours (a ~2.3× speedup from v3 to v4, which exceeds the theoretical 2× FLOP increase), but does not report the corresponding speedup factors for the ViT baselines — making it impossible to know whether the 2.0–2.3× range applies symmetrically.

The consequence. If ViTs achieve a speedup factor closer to the theoretical 2× (meaning they benefit less from TPU-v4 architectural improvements than NFNets do), then the "matched" TPU-v4 core hours comparison is systematically biased in favor of NFNets. On TPU-v3 hardware — where the ViT baselines were originally developed and where many large-scale training runs still occur — the relative efficiency ordering could reverse or the gap could widen. Worse, on GPU hardware (which dominates both academic research and much of industry), neither the NFNet nor the ViT efficiency numbers are known, and the paper provides no guidance on whether the architectural comparison would hold. A practitioner running on A100 or H100 GPUs cannot infer from this paper whether a ConvNet or a ViT would deliver better accuracy per dollar at their scale.

What evidence exists in the paper. The asymmetry is acknowledged in a single sentence in Section 3: "We note however that NFNets were optimized for TPU-v4, and perform less well when evaluated on other devices. For example, we estimate that NFNet-F7+ would require 250 TPU-v3 core hours to pre-train for 8 epochs in our codebase." The ViT speedup factors on the v3-to-v4 transition are not reported. There is no GPU throughput data, no measurement of how the comparison shifts if both architectures are benchmarked on TPU-v3, and no discussion of whether NFNet's TPU-v4 optimization (which likely involves specific convolutional kernel implementations that exploit TPU-v4's matrix-multiply unit architecture) would carry over to other hardware backends.

Mitigation status. The paper does not attempt to mitigate this limitation — it acknowledges it transparently but makes no correction for it in the headline comparison (Figure 1). The authors "thank Lucas Beyer for providing training speed estimates for ViT models on TPU-v4 devices," but the specific speedup factors are not disclosed and no sensitivity analysis is performed (e.g., "if ViTs achieved a 2.0× speedup rather than 2.3×, the NFNet advantage would shrink by X TPU-v4 core hours"). The limitation is acknowledged as a caveat but is left entirely unresolved.

---

6.2 Only One ConvNet Architecture Is Evaluated, Making the Claim Architecture-Specific

The assumption or constraint. The paper's title and abstract make claims about "ConvNets" as a class: "ConvNets Match Vision Transformers at Scale." But all experiments use exactly one ConvNet architecture — the NFNet family from Brock et al. (2021). The authors specifically chose NFNets because they were "the last ConvNet to set a new SOTA on ImageNet" without extra data, and they "do not make any changes to the model architecture." This means the entirety of the evidence for the universal claim "ConvNets match ViTs" comes from a single architecture family, designed by a single team, incorporating specific innovations (normalizer-free training, AGC, weight standardization, SE blocks, specific stem and transition designs) that may or may not be representative of ConvNets in general.

The consequence. The claim that "ConvNets match ViTs" is unsubstantiated for any ConvNet architecture other than NFNets. A practitioner choosing between, say, EfficientNetV2 and a ViT for large-scale pre-training cannot conclude from this paper that their ConvNet of choice will scale competitively. The specific design choices in NFNets — particularly the normalizer-free training with AGC — may be responsible for the favorable scaling behavior, and these are not universal properties of ConvNets. For example, standard ResNets (the baseline used in most ViT papers) rely on batch normalization, which introduces complications at large batch sizes (batch statistics must be synchronized across devices) and at heterogeneous resolutions (batch norm statistics computed at one resolution may not transfer to another). NFNets sidestep these issues through their normalizer-free design. If batch-normalization-dependent ConvNets scale worse at large batch sizes or across resolution changes, then the paper's conclusion does not generalize to the most widely-used ConvNet architectures. Furthermore, the rapid evolution of ConvNet design (EfficientNet, ConvNeXt, RepVGG, MobileNet variants) means that "ConvNet" is a broad architectural class; demonstrating that one member scales competitively does not demonstrate that the class as a whole does.

What evidence exists in the paper. None. The paper does not train a single alternative ConvNet architecture. There is no ResNet, no EfficientNet, no ConvNeXt — not even as a small-scale ablation to show that NFNets outperform these alternatives at a fraction of the compute budget, which would help establish that NFNets are a reasonable proxy for "the best available ConvNet." The paper compares only against NFNets and only against ViT baselines from the literature; the within-ConvNet comparison is absent. The authors reference the original NFNet-F5 on ImageNet without extra data (86.8% Top-1) as an implicit baseline to show the benefit of pre-training, but this baseline is also an NFNet — it does not establish how other ConvNets would perform under the same pre-training protocol.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, does not discuss alternative ConvNet architectures, and does not qualify the title's universal claim. The "Discussion" section broadens the claim further: "in our view there is no strong evidence to suggest that pre-trained ViTs outperform pre-trained ConvNets when evaluated fairly" — a statement that implies generality across ConvNets, not just NFNets, but the evidence is restricted to a single architecture family. A more accurate title would be "NFNets Match Vision Transformers at Scale," but the paper consistently uses the broader term "ConvNets."

---

6.3 Downstream Evaluation Is Restricted to a Single Task (ImageNet Classification)

The assumption or constraint. All comparisons between NFNets and ViTs are based on exactly one downstream metric: ImageNet Top-1 accuracy after fine-tuning. ImageNet classification is the most canonical computer vision benchmark, but it is a single task with specific properties — 1,000 object-centric classes, curated images with objects roughly centered and at moderate resolutions, and a evaluation protocol that tests category recognition rather than spatial localization, temporal reasoning, or multi-modal understanding. The ViT literature has shown that transformers can be applied to a wide range of vision tasks (object detection, segmentation, video classification, depth estimation), and architectural advantages might manifest differently across these tasks.

The consequence. A practitioner deciding between ConvNets and ViTs for a real-world computer vision system rarely cares only about ImageNet classification accuracy. If ViTs transfer substantially better to object detection (which requires both classification and precise bounding box regression) or to video understanding (which requires temporal modeling across frames), then the ImageNet-only comparison masks architecture-specific strengths. ConvNets have a well-known inductive bias toward local spatial features, which serves them well for tasks requiring fine-grained spatial localization (like segmentation), while ViTs' global self-attention might excel at tasks requiring long-range contextual reasoning (like scene understanding or relationship detection). The paper's single-task evaluation collapses all of these dimensions into one number and cannot detect tradeoffs where each architecture wins on different task types. Additionally, the paper only evaluates transfer learning (pre-train on JFT, fine-tune on ImageNet) — it does not evaluate few-shot or zero-shot transfer, which is an important capability regime where ViTs (particularly when combined with contrastive or caption-based pre-training) have shown strengths that pure supervised pre-training on JFT may not capture.

What evidence exists in the paper. The paper's Figure 1 and all fine-tuning results are restricted to ImageNet Top-1 accuracy. The authors mention no other downstream tasks, no detection or segmentation benchmarks, and no multi-task evaluation. The ViT baselines (Zhai et al., 2022; Alabdulmohsin et al., 2023) are also evaluated primarily on ImageNet, so the comparison is internally consistent — but the scope of the claim ("ConvNets match ViTs") is broader than the scope of the evidence (ImageNet classification only).

Mitigation status. Not addressed. The paper does not acknowledge the limitation of single-task evaluation, does not discuss whether the comparison might differ on other vision tasks, and does not suggest multi-task evaluation as future work. This is a significant omission because the original ViT paper (Dosovitskiy et al., 2020) and much subsequent ViT research has emphasized transfer to multiple downstream tasks as a key strength of the architecture. Demonstrating that ConvNets match ViTs on only the most basic classification benchmark leaves open the possibility that ViTs are genuinely superior on the broader set of tasks that matter for practical deployment.

---

6.4 Pre-Training Data and Fine-Tuning Protocols Are Not Fully Matched

The assumption or constraint. The paper compares NFNets pre-trained on JFT-4B (approximately 4 billion labeled images from 30k classes) against ViTs pre-trained on JFT-3B (approximately 3 billion images). While both are variants of the same dataset, the 1-billion-image difference (~33% more data) favors the NFNets. Additionally, the fine-tuning protocols differ: NFNets are fine-tuned for 50 epochs using Sharpness-Aware Minimization (SAM), stochastic depth, and dropout, at resolution 384/480, while the ViT baselines from Zhai et al. (2022) and Alabdulmohsin et al. (2023) used their own fine-tuning recipes (which the paper does not describe, compare against, or match). The paper makes no attempt to unify these protocols or ablate their effects.

The consequence. The final ImageNet accuracy numbers reflect a composite of pre-training quality, pre-training data quantity, and fine-tuning protocol effectiveness. If the NFNet fine-tuning recipe (SAM, stochastic depth, resolution increase) is more effective than what was used for the ViT baselines — which is plausible given that SAM is a relatively recent optimizer improvement that the ViT papers may not have employed — then the "match" in final accuracy could reflect fine-tuning superiority rather than architectural parity. Conversely, if the ViT fine-tuning recipes were stronger, the NFNets might actually be better at equal fine-tuning protocols, and the headline numbers understate their advantage. The dataset size difference (4B vs. 3B) is likely a smaller confound at the epoch budgets used (typically <1 epoch, meaning both models see far fewer images than are available in either dataset), but it is an uncontrolled variable that makes the comparison slightly less precise than a perfectly matched experiment would be.

What evidence exists in the paper. The paper explicitly states the dataset sizes: "JFT-4B which contains roughly 4 billion labelled images" for NFNets, while noting the ViT baselines were "pre-training on JFT-3B" (Section 3). The 1-billion-image difference is not discussed as a potential confound. The fine-tuning protocols for ViTs are not described or compared at any point in the paper; the reader is expected to consult Zhai et al. (2022) and Alabdulmohsin et al. (2023) to determine how the ViTs were fine-tuned, and whether those protocols are as strong as the NFNet protocol.

Mitigation status. The paper does not address either the data quantity mismatch or the fine-tuning protocol mismatch. It does not run an ablation where NFNets are pre-trained on JFT-3B (the exact same data as the ViTs) to test whether the 1-billion-image advantage matters. It does not fine-tune a ViT using the same SAM-based protocol to test whether the optimizer choice affects the comparison. The paper implicitly assumes these differences are small relative to the architectural scaling effect, but this assumption is untested. The authors' decision to report NFNet-F7+ results both with and without repeated augmentation (90.3% vs. 90.4%) demonstrates awareness that fine-tuning choices affect final accuracy, but they do not extend this awareness to cross-architecture comparison.

---

6.5 The Scaling Law Is Measured on Validation Loss, Which Does Not Perfectly Predict Downstream Transfer

The assumption or constraint. The paper's scaling law (Figure 2) demonstrates a log-log relationship between JFT-4B validation loss and pre-training compute. This scaling law is presented as the primary evidence that ConvNets obey the same scaling behavior as Vision Transformers and language models. However, the paper itself demonstrates that pre-training validation loss is an imperfect predictor of downstream performance: "the pre-trained checkpoints achieving the lowest validation loss on JFT-4B did not always achieve the highest Top-1 accuracy on ImageNet after fine-tuning," with the fine-tuning regime "consistently favouring slightly larger models and slightly smaller epoch budgets" (Section 3). Additionally, "in some cases, slightly larger learning rates (during pre-training) also achieved better performance after fine-tuning."

The consequence. The clean scaling law in Figure 2 — which shows validation loss improving smoothly with compute — does not directly translate to a guarantee that downstream accuracy scales at the same rate. If the compute-optimal pre-training configuration (minimizing validation loss) and the transfer-optimal pre-training configuration (maximizing downstream accuracy) systematically diverge as compute increases, then allocating additional compute according to the validation-loss scaling law is suboptimal for the downstream task that practitioners actually care about. Worse, if the divergence between pre-training loss and downstream accuracy differs between architectures — e.g., if ViTs' validation loss translates more efficiently to downstream accuracy than ConvNets' does — then matching on validation loss scaling would not guarantee matching on downstream task performance. The paper provides no measurement of whether this divergence is architecture-dependent or how it scales with compute.

The practical implication for a practitioner is significant: the scaling law says "if you double your compute, you should double model size and epoch budget equally to minimize validation loss." But the paper's own observation suggests that for maximizing downstream accuracy, you should bias the allocation toward larger models (more capacity) and away from longer training (fewer epochs) relative to what the pre-training validation loss optimum would prescribe. The scaling law is therefore a description of pre-training dynamics that may not be the right guide for transfer-oriented pre-training decisions.

What evidence exists in the paper. The misalignment is reported qualitatively in Section 3: "the pre-trained checkpoints achieving the lowest validation loss on JFT-4B did not always achieve the highest Top-1 accuracy on ImageNet after fine-tuning." The specific direction of the bias is described ("consistently favoured slightly larger models and slightly smaller epoch budgets"), but no quantitative characterization is provided — no measurement of how much larger the model should be, how much the epoch budget should be reduced, or whether the magnitude of the shift depends on the total compute budget. The observation is accompanied by the note about learning rates ("slightly larger learning rates... also achieved better performance after fine-tuning"), suggesting the transfer-optimal configuration differs along multiple hyperparameter dimensions simultaneously. The authors do not plot downstream accuracy against pre-training validation loss to visualize the relationship or quantify the scatter.

Mitigation status. Not addressed. The paper notes the misalignment as an empirical observation but treats it as an aside rather than a limitation of the scaling law framework. The authors do not propose an alternative metric (e.g., a transfer-aware validation metric) that would better predict downstream performance. They do not discuss whether this misalignment might affect the comparison against ViTs (i.e., whether ViTs exhibit the same or a different pattern of misalignment). The practical consequence — that the scaling law's guidance for compute allocation may be suboptimal for transfer learning — is left unexplored.

---

6.6 Hard Problems (Scaling Ceilings and the Unanswered Question of Divergence)

The assumption or constraint. The scaling experiments span pre-training compute budgets from approximately 0.4k to 110k TPU-v4 core hours — a range of roughly 275×. While this is a substantial span, it covers less than a quarter of the compute used for the largest ViT (ViT-G/14 at ~280k core hours, ~2.5× beyond the largest NFNet), and it does not approach the scale of the largest vision models trained by 2023 (which could exceed 500k core hours). The paper demonstrates that NFNets and ViTs occupy the same compute-performance curve within the tested range, but it does not test whether the curves diverge beyond it. The log-log linearity of Figure 2 shows no saturation — but this is description of the observed range, not a prediction that the linear trend continues indefinitely.

The consequence. The central claim — that ConvNets "match" Vision Transformers at scale — is only substantiated up to ~110k TPU-v4 core hours. If ConvNets have a shallower scaling exponent (meaning the validation loss vs. compute slope is less steep than ViTs'), then at 280k, 500k, or 1M core hours, ViTs would pull ahead, and the "match" at 110k hours would be a transient coincidence rather than a fundamental architectural equivalence. The paper cannot rule this out because it does not fit scaling exponents, does not extrapolate beyond the observed range, and does not train NFNets at the ViT-G/14 budget to provide a direct data point at higher compute. Conversely, if ConvNets have a steeper scaling exponent, they would surpass ViTs at larger budgets — an equally important finding that the paper's limited scale range misses.

This is not merely an academic concern about extrapolation. The practical question practitioners face when choosing an architecture for large-scale pre-training is not "which architecture is better at the specific 100k core-hour scale?" but "which architecture will deliver the best accuracy for my budget, which might be 2×, 5×, or 10× larger than what this paper tested?" Without evidence on whether the scaling curves are parallel or crossing, the paper's conclusion does not generalize to budgets substantially beyond 110k core hours.

What evidence exists in the paper. Figure 2 shows no curvature or saturation in the validation loss vs. compute trend across the full 0.4k–110k range, which is suggestive of a continued power-law relationship. However, this is consistent with both a true power law that extends indefinitely and a power law with a gradual change in exponent that would only become apparent at larger scales. Figure 1 places the ViT-G/14 reference point at ~280k core hours with 90.45% accuracy, which is marginally better than NFNet-F7+ at 110k hours (90.4%). The fact that 2.5× more ViT compute yields only a 0.05 percentage point improvement is consistent with a continuing log-log relationship (diminishing returns), but it also means the NFNet curve is not extrapolated to test whether it would achieve 90.45% or higher at 280k hours.

Mitigation status. Not addressed. The paper does not fit scaling law parameters (exponent, intercept) for either architecture, does not extrapolate the NFNet trend to larger budgets, and does not discuss the possibility of divergence. The authors' decision to frame the ViT-G/14 as a comparison point at 280k hours implicitly assumes the NFNet trend would continue linearly to that budget, but this is an assumption, not an empirical finding. The paper's conclusion that "in our view there is no strong evidence to suggest that pre-trained ViTs outperform pre-trained ConvNets when evaluated fairly" is strictly true at the tested scale but may not hold at larger scales — and the paper provides no framework for determining whether it will.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper functions primarily as a corrective reframing rather than a paradigm shift — it does not introduce a new architecture or training method, but it challenges a widely-held belief that had shaped years of research prioritization and resource allocation across computer vision. The belief that Vision Transformers scale better than ConvNets was not merely an academic curiosity; it redirected funding, PhD theses, hardware optimization efforts, and industrial deployment pipelines away from convolutional architectures. By demonstrating that a strong ConvNet architecture (NFNet) matches ViTs at equivalent pre-training compute on the canonical ImageNet benchmark, the paper resets the burden of proof: the default assumption should no longer be that ViTs are the superior architecture class for large-scale vision. Instead, the null hypothesis should be that well-designed architectures of either family scale similarly, and claims of superiority require evidence of genuinely different scaling behavior — not point comparisons at arbitrary compute budgets with weak baselines.

The paper's most enduring conceptual contribution may be methodological rather than empirical: it establishes a template for how architecture comparisons should be conducted at scale. The key elements of this template are (1) comparing strong instantiations of each architecture class, not the strongest ViT against a legacy ConvNet; (2) matching pre-training compute budgets in practical units (TPU-v4 core hours) that account for real hardware utilization, not just theoretical FLOPs; (3) re-benchmarking baselines on the same hardware generation rather than converting between hardware generations with theoretical multipliers; (4) transparently reporting hardware-specific asymmetries rather than hiding them; and (5) evaluating scaling behavior (the functional relationship between compute and performance) rather than just reporting a single accuracy number at a single budget. If widely adopted, this template would substantially reduce the number of spurious "Architecture X beats Architecture Y" claims that arise from confounded comparisons — a problem that extends well beyond the ViT-vs-ConvNet debate to every area of deep learning where architectural innovation is claimed.

The paper also reconciles several tensions in the prior literature that had been interpreted as evidence of ViT superiority. The finding that ConvNets exhibit the same log-log scaling law as transformers (linear trend in Figure 2, compute-optimal allocation scaling model size and epochs equally, epoch budget >1 required above ~5k TPU-v4 core hours) suggests that the apparent ViT advantage in prior work arose from three confounds the paper identifies: weak ConvNet baselines (ResNets rather than NFNets or their equivalents), unequal compute budgets (ViTs were trained with more compute), and the simultaneous shift to web-scale pre-training (which benefited ViTs because Co)nNets were tested primarily in the from-scratch regime). The paper shows that when these confounds are removed, the architectural choice becomes neutral — an instance of the "bitter lesson" applied within the modern deep learning era, where even the fundamental inductive bias difference between convolutions (spatial locality, translation equivariance) and self-attention (global receptive field, content-dependent interactions) is dominated by the scale of compute and data available.

However, the paper's impact on research direction attractiveness is nuanced. It does not demonstrate that ConvNets are better than ViTs — only that they are not worse at the scales tested. This means the paper strengthens the case for continuing ConvNet research (architecture design, hardware optimization, efficient training methods) without weakening the case for ViT research. Both architecture families emerge as viable, and the optimal choice likely depends on factors the paper does not fully explore: hardware platform (NFNets were optimized for TPU-v4 and "perform less well when evaluated on other devices"), downstream task type (only ImageNet classification is tested), deployment constraints (ConvNets may have an edge in latency-sensitive or memory-constrained settings due to their sliding-window computation pattern), and multi-modal integration (the paper acknowledges that "ViTs may have practical advantages in specific contexts, such as the ability to use similar model components across multiple modalities"). The paper's most productive effect may be to reduce architectural dogmatism — encouraging practitioners to select architectures based on their specific constraints and empirical benchmarks rather than on a presumed scaling superiority of one family over the other.

The scale ceiling question remains the most significant unresolved tension. The paper tests budgets up to 110k TPU-v4 core hours, while the largest ViT (ViT-G/14) uses ~280k, and the field continues to push toward 500k+ hours. If ViTs have a genuinely steeper scaling exponent — which the paper cannot rule out because it fits no slopes and does not extrapolate — then the "match" at 110k hours is a transient coincidence and ViTs will pull ahead at larger budgets. The paper frames its conclusion cautiously ("in our view there is no strong evidence to suggest that pre-trained ViTs outperform pre-trained ConvNets when evaluated fairly"), but the practical message many readers will take away — "ConvNets are just as good as ViTs" — extends beyond what the evidence can support. Resolving whether the scaling curves are parallel or crossing at larger budgets is the single most important open question the paper leaves behind.

Follow-Up Research This Work Enables

Scaling law exponent measurement for ConvNets vs. ViTs on matched hardware and data. The paper observes a log-log linear trend but neither fits power-law coefficients for its NFNet data nor compares the fitted slope against ViT scaling behavior. A direct follow-up would pre-train a matched set of NFNets and ViTs (e.g., ViT-B, ViT-L, ViT-g from Zhai et al., 2022) on exactly the same JFT dataset (JFT-4B, not JFT-3B vs. JFT-4B) across at least a 500× compute range, measure both pre-training validation loss and downstream ImageNet accuracy, and fit scaling laws of the form $L(C) = a \cdot C^{-\alpha} + L_0$ for each architecture. The key output is a comparison of $\alpha_{\text{ConvNet}}$ vs. $\alpha_{\text{ViT}}$ with confidence intervals. If the exponents are statistically indistinguishable, the paper's "matching" claim is quantitatively validated and can be extrapolated to larger budgets. If ViTs have a measurably steeper slope — even by a small margin — then the paper's conclusion is bounded to the tested scale, and the field would know at what compute budget ViTs begin to pull ahead.

NFNet pre-training at ViT-G/14 scale (~280k TPU-v4 core hours) to test for divergence. The paper's strongest NFNet (F7+) uses 110k core hours and achieves 90.4% Top-1, while ViT-G/14 at ~280k hours achieves 90.45%. This leaves open whether NFNets would match or exceed ViT-G/14 if scaled to the same budget. A direct experiment would train an NFNet-F7+ (or a larger variant, e.g., F7++ with further width scaling following the paper's "scale model size and epochs at the same rate" rule) at 280k TPU-v4 core hours on JFT, fine-tune on ImageNet using the paper's protocol, and compare against the 90.45% ViT-G/14 number. Additionally, training at intermediate budgets (150k, 200k, 250k) would map the NFNet scaling curve in the range between the paper's maximum and the ViT-G/14 point, testing whether the log-log linearity continues or whether a downward curvature (saturation) emerges. This experiment directly addresses the paper's most significant evidential gap — the lack of extrapolation beyond the tested range — and would either extend the "matching" claim to larger scales or identify the budget at which ConvNet scaling begins to fall behind.

Multi-task transfer evaluation beyond ImageNet classification. The paper's comparison is restricted to a single downstream task (ImageNet Top-1 accuracy after fine-tuning), which cannot detect architecture-specific strengths that manifest on other vision tasks. A comprehensive follow-up would take the pre-trained NFNet checkpoints from this paper (F0 through F7+, across epoch budgets) and evaluate transfer performance on a standard multi-task benchmark: COCO object detection and instance segmentation (using Mask R-CNN or DETR heads), ADE20K semantic segmentation, Kinetics-400 video classification (for video-capable architectures), and several fine-grained classification datasets (iNaturalist, CUB-200, Stanford Cars). The same evaluation would be performed on equivalently pre-trained ViT checkpoints (ViT-g/14, ViT-G/14, SoViT-400m/14) fine-tuned with matched protocols. The key question is whether the "matching" observed on ImageNet holds uniformly across tasks, or whether ConvNets excel on tasks requiring fine-grained spatial localization (detection, segmentation) while ViTs excel on tasks requiring global context (scene classification, video understanding). This would transform the paper's binary "match or not" conclusion into a richer picture of architecture-task affinity.

Alternative ConvNet architecture scaling to test generality beyond NFNets. The paper's claim is about "ConvNets" as a class, but all evidence comes from the NFNet family. A rigorous test of generality would replicate the paper's scaling experiment with at least two other strong ConvNet architectures from the same era: ConvNeXt (Liu et al., 2022), which explicitly incorporates ViT-inspired design choices (patchify stem, LayerNorm, GELU activations, separate downsampling layers) into a convolutional framework, and EfficientNetV2 (Tan & Le, 2021), which uses neural architecture search and compound scaling. Both would be pre-trained on JFT-4B across a range of model sizes and epoch budgets matched to the paper's compute axis, fine-tuned on ImageNet using the paper's protocol, and compared against the NFNet and ViT scaling curves. If ConvNeXt and EfficientNetV2 also land on the same compute-performance trend, the paper's "ConvNets" claim gains strong architectural generality. If they fall below NFNets (suggesting the normalizer-free design or SE blocks in NFNets are responsible for the competitive scaling), the claim must be narrowed to specific ConvNet design families, and the field would learn which ConvNet innovations are critical for large-scale performance.

Validation loss vs. downstream transfer characterization across architectures. The paper observes that "pre-trained checkpoints achieving the lowest validation loss on JFT-4B did not always achieve the highest Top-1 accuracy on ImageNet after fine-tuning," with the fine-tuning regime favoring "slightly larger models and slightly smaller epoch budgets." This misalignment is noted but not quantified or compared across architectures. A systematic study would, for each pre-trained NFNet checkpoint across the model-size-by-epoch-budget grid, measure both JFT-4B validation loss and ImageNet Top-1 accuracy after fine-tuning, then plot one against the other to visualize the scatter and correlation. The same analysis would be performed for the ViT baselines (using available checkpoints at multiple epoch budgets). The key outputs are: (1) the correlation coefficient between pre-training loss and downstream accuracy for each architecture; (2) whether the optimal fine-tuning configuration (the point in the model-size/epoch-budget space that maximizes downstream accuracy at a fixed total pre-training compute) differs systematically between ConvNets and ViTs; (3) whether the "slightly larger model, slightly fewer epochs" bias observed for NFNets is architecture-specific or universal. This would provide practical guidance for checkpoint selection in transfer learning pipelines and reveal whether the validation-loss-based scaling law (Figure 2) is an equally faithful guide for both architectures.

Hardware-agnostic efficiency comparison across TPU, GPU, and edge devices. The paper's comparison is explicitly TPU-v4-specific, with the authors noting that "NFNets were optimized for TPU-v4, and perform less well when evaluated on other devices" and providing the example that NFNet-F7+ requires ~250k TPU-v3 hours versus 110k TPU-v4 hours. A hardware-agnostic follow-up would benchmark identical NFNet and ViT models (e.g., NFNet-F7+ and ViT-g/14) on three hardware platforms: TPU-v4 (where the paper's comparison lives), NVIDIA A100/H100 GPUs (dominating academic and much of industrial training), and a representative edge inference device (e.g., an NVIDIA Jetson or Apple Neural Engine). On each platform, the study would measure training throughput (images/second), inference throughput, memory consumption, and total wall-clock time to reach a target ImageNet accuracy. This would answer the practical deployment question the paper cannot: given a specific hardware platform and budget, which architecture delivers better accuracy per dollar and per watt? The result would likely be a matrix — NFNets might win on TPU-v4, ViTs on A100s, and neither might dominate on edge — which would transform the architecture selection problem from a universal claim about superiority to a context-dependent optimization.

Practical Applications and Downstream Use Cases

ConvNet deployment on TPU-v4 infrastructure for large-scale image classification. Organizations that operate Google Cloud TPU-v4 pods for vision workloads now have direct evidence that continuing to invest in ConvNet architectures is not architecturally disadvantaged relative to switching to Vision Transformers. The paper shows that NFNet-F7+ at 110k TPU-v4 core hours achieves 90.4% ImageNet Top-1, matching or exceeding ViT-g/14 (90.2% at ~120k hours) and SoViT-400m/14 (90.3% at ~130k hours). For a team that has existing ConvNet training pipelines, model serving infrastructure, and hardware-optimized convolutional kernels, this finding justifies staying with ConvNets rather than undertaking an expensive migration to ViTs — the migration cost would bring no accuracy benefit at matched compute. The paper's scaling rule ("scale model size and the number of training epochs at the same rate") and learning rate heuristic ("start from ~1.6 and reduce monotonically with model size and epoch budget") provide concrete guidance for how to scale up existing ConvNet training runs without extensive hyperparameter tuning.

Cost-efficient pre-training for transfer learning in multi-task vision systems. For teams building vision systems that require a single pre-trained backbone to serve multiple downstream tasks (classification, detection, segmentation), the paper's finding about the misalignment between pre-training validation loss and downstream transfer performance has immediate practical implications. The observation that transfer-optimal pre-training favors "slightly larger models and slightly smaller epoch budgets" than what minimizes pre-training validation loss means that practitioners should not select pre-training checkpoints based solely on validation loss curves. Instead, they should evaluate a small number of checkpoints (spanning the tradeoff between larger models with fewer epochs and smaller models with more epochs) on their specific downstream task suite, accepting the additional evaluation cost as necessary for optimal transfer performance. The paper provides a concrete starting point: for a fixed budget, bias toward a model ~1.5-2× larger than the validation-loss-optimal size, trained for ~0.5-0.75× the validation-loss-optimal epochs, with a slightly elevated pre-training learning rate.

Architecture selection for organizations standardizing on transformer-based multi-modal systems. The paper acknowledges that "ViTs may have practical advantages in specific contexts, such as the ability to use similar model components across multiple modalities." For organizations building multi-modal systems that process text, images, and potentially other modalities through a unified transformer backbone, the paper's evidence does not suggest switching vision components to ConvNets — even though ConvNets match ViTs on vision-only tasks. The engineering benefit of a unified architecture (shared attention mechanisms, shared optimization infrastructure, shared inference serving code) may outweigh any small accuracy or efficiency differences on individual modalities, and the paper provides no evidence that ConvNets can serve as text encoders or multi-modal fusion components. In this specific use case, ViTs remain the pragmatically correct choice despite the paper's evidence of architectural neutrality for vision-only tasks.

Budget-constrained academic and startup research on vision architectures. The paper's learning rate scaling findings (Figure 3) offer a direct cost-saving heuristic for resource-constrained groups pre-training ConvNets on large datasets. The observation that "all models show similar optimal learning rates ~1.6 when the epoch budget is small" and that "one can efficiently tune the learning rate within 2 trials" means a team pre-training an NFNet on JFT-scale data can start with a learning rate of 1.6 for initial small-epoch-budget runs, then reduce it as they scale up — avoiding the expensive grid search over 6-8 learning rate values that would otherwise be necessary. At 110k TPU-v4 core hours per training run, reducing hyperparameter tuning from 8 trials to 2 trials saves approximately 660k TPU-v4 core hours — a substantial resource savings that makes large-scale ConvNet pre-training more accessible to groups without industrial-scale compute budgets. The paper also provides a specific epoch budget threshold (~5k TPU-v4 core hours, above which training for more than 1 epoch becomes optimal) that helps practitioners decide whether to invest in multi-epoch data loading infrastructure or stick with single-pass training.