ConvNeXt: A ConvNet for the 2020s

ArXiv: 2201.03545

🎯 Pitch

A pure ConvNet can match vision Transformers by simply modernizing its design—no attention needed. The resulting ConvNeXt family outperforms Swin Transformers on COCO and ADE20K while being simpler and faster.

---

1. Executive Summary

This paper analyzes how to modernize a standard ResNet architecture by systematically incorporating design choices from hierarchical vision Transformers, producing a family of pure ConvNet models called ConvNeXt. Using the ImageNet-1K classification benchmark with ResNet-50/200 as starting points and Swin Transformers as reference targets, the authors incrementally study macro design elements — stage compute ratios, patchify stems, and inverted bottlenecks — alongside micro-level modifications including large kernel sizes (7×7 depthwise convolutions as the saturation point) and normalization/activation substitutions (LayerNorm replacing BatchNorm, GELU replacing ReLU, and reducing activation functions to one per block). The resulting ConvNeXt models achieve 87.8% ImageNet top-1 accuracy with ConvNeXt-XL while competing favorably with Swin Transformers across detection and segmentation benchmarks, demonstrating up to 49% higher inference throughput on A100 GPUs with comparable parameter counts. The modernization process establishes that the performance gap between ConvNets and Transformers is not attributable to the intrinsic superiority of self-attention, with properly modernized pure ConvNets matching or exceeding hierarchical Transformers even when pre-trained on ImageNet-22K — though the paper explicitly scopes this finding to standard vision tasks and notes that Transformers may remain preferable for multi-modal learning or tasks requiring discretized outputs.

2. Context and Motivation

The Core Problem: What Makes a Good Visual Backbone in the Post-ViT Era?

The fundamental question this paper tackles is deceptively straightforward: can a pure convolutional neural network, constructed entirely from standard ConvNet operations, match or exceed the performance of modern vision Transformers across the full spectrum of computer vision tasks? This matters because the introduction of Vision Transformers (ViTs) in 2020 abruptly ended ConvNets' decade-long dominance as the default architecture for visual recognition. The paper's framing is pointed:

"The 'Roaring 20s' of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model."

This reversal was not gradual — it was a step change. Prior to ViTs, the field had spent a decade refining ConvNet architectures (VGGNet, Inception, ResNet, ResNeXt, DenseNet, EfficientNet, RegNet), each iteration improving on the last through incremental design innovations. ViTs disrupted this trajectory by demonstrating that Transformers — originally designed for natural language processing with no image-specific inductive biases — could outperform heavily engineered ConvNets simply by scaling up model size and training data.

However, the paper identifies a critical tension: the vanilla ViT that achieved impressive ImageNet classification results is not a practical generic vision backbone. The reason lies in a fundamental design choice:

"The biggest challenge is ViT's global attention design, which has a quadratic complexity with respect to the input size. This might be acceptable for ImageNet classification, but quickly becomes intractable with higher-resolution inputs."

Computer vision is not limited to classifying 224×224 images. Object detection, semantic segmentation, and other dense prediction tasks operate on high-resolution inputs (e.g., 800×1280 for COCO detection), where global self-attention's $O(n^2)$ complexity makes vanilla ViTs computationally prohibitive.

The Hierarchical Transformer Response and Its Irony

The solution that emerged — and the paper's primary target of analysis — is hierarchical Transformers, exemplified by the Swin Transformer. These models reintroduce several design elements that were originally ConvNet innovations:

• Multi-stage architecture with progressively downsampled feature maps (identical to standard ConvNet design)
• Local window attention — essentially a sliding window strategy that restricts self-attention to neighborhoods (7×7 windows in Swin)
• Translation-equivariant processing through the local window design

The paper makes an explicit, provocative observation about this development:

"Swin Transformer's success and rapid adoption also revealed one thing: the essence of convolution is not becoming irrelevant; rather, it remains much desired and has never faded."

In other words, the very features that made Transformers practical for vision were the reintroduction of ConvNet-like inductive biases. This creates a conceptual irony that the paper exploits: if hierarchical Transformers succeed by becoming more ConvNet-like, perhaps pure ConvNets can succeed by adopting certain Transformer design choices — without ever needing self-attention.

Where Existing Comparisons Fall Short

The paper identifies a confounding variables problem in how ConvNets and Transformers were being compared. The prevailing narrative — that Transformers are inherently superior to ConvNets — was supported by system-level comparisons like "Swin Transformer vs. ResNet," but these comparisons conflate multiple differences:

"ConvNets and hierarchical vision Transformers become different and similar at the same time: they are both equipped with similar inductive biases, but differ significantly in the training procedure and macro/micro-level architecture design."

The paper enumerates the specific axes of difference:

Training procedures diverged. Vision Transformers introduced new optimization practices to the vision domain: the AdamW optimizer instead of SGD with momentum, extended training schedules (300 epochs vs. 90 for ResNets), sophisticated data augmentation (Mixup, Cutmix, RandAugment, Random Erasing), and regularization techniques (Stochastic Depth, Label Smoothing). The paper's preliminary experiment demonstrates this is not a minor factor: simply applying ViT-style training to a ResNet-50 improves it from 76.1% to 78.8% (+2.7%), implying:

"a significant portion of the performance difference between traditional ConvNets and vision Transformers may be due to the training techniques."

Macro architecture design differed. Swin Transformers use different stage compute ratios (1:1:3:1 or 1:1:9:1) compared to ResNet's (3:4:6:3), different stem cell designs (aggressive patchification vs. 7×7 conv + max pooling), and fundamentally different block structures (inverted bottleneck with depthwise-like spatial mixing vs. standard bottleneck with 3×3 conv).

Micro-level design choices differed. Transformers use GELU activations rather than ReLU, have far fewer activation functions per block (one in the MLP vs. one after every convolution), use Layer Normalization rather than Batch Normalization, and place the spatial mixing module before rather than after the 1×1 convolutions.

The critical gap is that no prior work had systematically disentangled these factors to determine whether Transformers' advantage came from self-attention itself or from these accumulated architectural and training choices. The paper writes:

"Our research is intended to bridge the gap between the pre-ViT and post-ViT eras for ConvNets, as well as to test the limits of what a pure ConvNet can achieve."

The Unresolved Status of Convolution in the Literature

The paper situates itself within an active but fragmented research landscape. Several concurrent or recent works were exploring the relationship between convolutions and attention:

• Han et al. [25] showed that local Transformer attention is mathematically equivalent to "inhomogeneous dynamic depthwise conv" — suggesting that the MSA block in Swin can be directly replaced with dynamic or regular depthwise convolution while achieving comparable performance.

• ConvMixer [4] (a concurrent anonymous submission) demonstrated that depthwise convolution alone can serve as an effective token mixing strategy in small-scale settings, though with throughput penalties due to smaller patch sizes.

• GFNet [56] adopted Fast Fourier Transform for token mixing, which is itself a form of convolution with a global kernel size and circular padding.

• Hybrid models (CoAtNet [15], ConViT [16], CvT [82], Co-Scale Conv-Attentional Transformers [88]) explicitly combined convolutions and self-attention, treating them as complementary operations.

The paper distinguishes its contribution from these efforts:

"Unlike many recent Transformer or ConvNet designs, one primary goal of our study is to provide an in-depth look at the process of modernizing a standard ResNet and achieving state-of-the-art performance."

Rather than proposing a novel hybrid operation or a new architectural primitive, the paper's approach is methodological: start from a standard ResNet, apply the Transformer training recipe, then incrementally adopt Transformer design choices — but implemented purely with convolutions — and measure the contribution of each change. This is a systematic ablation study at the architecture level, not a proposal of a new building block.

Why This Question Matters Beyond Academic Interest

The paper motivates its investigation through several practical concerns:

Simplicity. Vision Transformers, particularly hierarchical ones like Swin, require specialized operations that complicate implementation: cyclic shifting for efficient window attention, relative position biases, and careful handling of input resolutions (interpolating position embeddings when fine-tuning at different image sizes). Swin's block (Figure 4) contains two residual connections, multiple normalization layers, and specialized modules. The paper asks whether equivalent performance can be achieved with a simpler design:

"ConvNeXt maintains the efficiency of standard ConvNets, and the fully-convolutional nature for both training and testing makes it extremely simple to implement."

Efficiency at scale. The quadratic complexity of global attention is often cited as the reason hierarchical Transformers outperform vanilla ViTs on high-resolution tasks. But the paper notes an underappreciated fact:

"It is worth noting that this improved efficiency is a result of the ConvNet inductive bias, and is not directly related to the self-attention mechanism in vision Transformers."

In other words, what makes Swin efficient is local processing — a property ConvNets already possess natively. The paper's throughput measurements on A100 GPUs (Table 12) demonstrate that ConvNeXts achieve up to 49% higher inference throughput than Swin Transformers with similar FLOPs, using standard "channel last" memory layout optimizations that are well-supported for convolutions on modern hardware.

The scaling narrative. The paper explicitly addresses a widely held belief in the community:

"The widely held belief is that vision Transformers are more accurate, efficient, and scalable than ConvNets."

This belief was supported by evidence that Transformers showed more favorable scaling behavior — larger ViTs continued to improve while larger ConvNets appeared to saturate. However, this evidence came from comparisons against older ConvNet designs (ResNet) trained with older recipes. The paper asks: is it the Transformer architecture that scales better, or would a properly modernized ConvNet scale just as well?

The ImageNet-22K pre-training experiments (Table 1, lower) directly test this: ConvNeXt-XL achieves 87.8% top-1 accuracy, demonstrating that pure ConvNets can benefit from large-scale pre-training just as Transformers do.

The downstream task question. Image classification is a proxy task; the real test of a vision backbone is performance on detection and segmentation. The paper notes that vanilla ViT's difficulties with these tasks were what motivated hierarchical Transformers in the first place. The paper therefore evaluates ConvNeXt extensively on COCO detection/segmentation (Table 3) and ADE20K semantic segmentation (Table 4), making the case that a pure ConvNet can serve as a generic vision backbone — the role that Swin Transformer had claimed for Transformers.

How the Paper Positions Itself

The paper's positioning is carefully balanced. It is not arguing that convolutions are superior to attention or that Transformers are unnecessary. Rather, it is arguing that the performance gap between modern Transformers and traditional ConvNets is attributable to a collection of specific, identifiable design and training choices — not to any inherent limitation of convolution as an operation. The paper states its goal explicitly:

"We hope the new observations and discussions can challenge some common beliefs and encourage people to rethink the importance of convolutions in computer vision."

This is a significant repositioning because it reframes the narrative from "Transformers beat ConvNets" to "the field's understanding of what makes a good ConvNet was incomplete." The decade of ConvNet development (2012–2020) produced many innovations — residual connections, grouped convolutions, depthwise separable convolutions, inverted bottlenecks, Squeeze-and-Excitation — but these were developed in an era when training procedures, model scales, and the specific combination of design choices were different. The post-ViT era provides new tools (AdamW, advanced augmentation, longer training) and new reference points (Transformer block design) that enable re-examining what ConvNets can do.

The paper explicitly acknowledges that its individual design choices are not novel:

"In some ways, our observations are surprising while our ConvNeXt model itself is not completely new — many design choices have all been examined separately over the last decade, but not collectively."

The contribution is therefore integration and systematic analysis: bringing together known ConvNet techniques (depthwise convolutions, inverted bottlenecks, large kernels, GELU, LayerNorm) under a modern training paradigm and demonstrating that their combination — informed by Transformer design principles but implemented purely with convolutions — achieves state-of-the-art results without self-attention.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a systematic empirical analysis paper whose core idea is that the performance gap between traditional ConvNets and modern hierarchical vision Transformers can be bridged by incrementally adopting Transformer design principles — while implementing them entirely with standard convolutional operations. The "system" being built is not a single novel architectural module but rather a step-by-step modernization roadmap: starting from a standard ResNet-50 trained with a modern recipe, the authors identify and quantify the contribution of each architectural difference between ConvNets and Transformers, culminating in a family of pure ConvNet models (ConvNeXt) that match or exceed Swin Transformers without any attention-based components.

3.2 Big-Picture Architecture (Diagram in Words)

The modernization pipeline has five major stages, each comprising a set of related design changes, applied sequentially to a baseline ResNet model:

1. Training Recipe Modernization — Before touching the architecture, the baseline ResNet is retrained using the training procedures that vision Transformers introduced to the field (AdamW, extended epochs, Mixup/Cutmix/RandAugment, Stochastic Depth, Label Smoothing). This establishes a fair baseline by eliminating training procedure as a confounding variable.

2. Macro Design Modernization — The overall network structure is modified to match Swin Transformer conventions: the stage compute ratio is changed from ResNet's (3, 4, 6, 3) to Swin's (3, 3, 9, 3), and the stem cell is replaced with a non-overlapping 4×4 "patchify" convolution with stride 4, mirroring how ViTs and Swin Transformers process the input image.

3. ResNeXt-ification and Inverted Bottleneck — The standard bottleneck block is restructured in two steps. First, the 3×3 spatial convolution is replaced with a depthwise convolution (a grouped convolution where groups = channels), and the network width is expanded to compensate for the reduced FLOPs, following the ResNeXt principle. Second, the block is inverted: the hidden dimension is expanded by a factor of 4 (matching the MLP expansion in Transformers), creating a narrow-wide-narrow channel pattern rather than ResNet's wide-narrow-wide pattern.

4. Large Kernel Size Exploration — The position of the depthwise convolution is moved to the beginning of the block (mirroring how Transformers place the MSA before the MLP), and the kernel size is systematically swept from 3×3 up to 11×11. Performance saturates at 7×7, which becomes the default.

5. Micro Design Modernization — Layer-level design choices are adjusted to match Transformer conventions: ReLU is replaced with GELU, the number of activation functions is reduced to one per block (only between the two 1×1 convolutions in the MLP portion), the number of normalization layers is reduced to one per block (placed before the first 1×1 convolution), BatchNorm is replaced with LayerNorm, and separate 2×2 convolutional downsampling layers are introduced between stages (with additional LayerNorm layers for training stability).

Information flows through the final ConvNeXt block (Figure 4, center) as follows: input → LayerNorm → 7×7 depthwise convolution → 1×1 convolution (expand 4×) → GELU activation → 1×1 convolution (project back to original dimension) → residual addition → output. This is architecturally isomorphic to a Transformer block but uses only convolutions.

3.3 Roadmap for the Deep Dive

• First, the modernization methodology itself — how the incremental exploration is structured, how FLOPs are controlled, and why the ResNet-50/Swin-T and ResNet-200/Swin-B regimes are both studied — since this framework governs every subsequent design decision.
• Second, the training recipe baseline — the specific hyperparameters and data augmentation choices adapted from DeiT and Swin Transformer training — since all subsequent comparisons depend on this fixed recipe isolating architectural effects.
• Third, the macro design changes (stage ratio and stem) — how the overall network skeleton is reshaped — since these set the global structure within which all block-level modifications operate.
• Fourth, the ResNeXt-ification and inverted bottleneck transformations — how the internal block structure is progressively reshaped toward a Transformer-like form while remaining purely convolutional — since these represent the core architectural innovations.
• Fifth, the large kernel experiments — how kernel size is systematically varied and why 7×7 emerges as optimal — since this is the most direct analogue to the local window attention in Swin Transformers.
• Sixth, the micro design changes — activation functions, normalization layers, and downsampling strategy — since these are the fine-grained adjustments that push performance over the threshold.
• Seventh, the scaling methodology — how ConvNeXt variants (T/S/B/L/XL) are constructed by varying channel counts and block depths — since this enables the fair comparison against Swin variants.

3.4 Detailed, Sentence-Based Technical Breakdown

Modernization Methodology: Incremental, Controlled Ablation

The paper's core methodological innovation is not any single architectural change but rather the incremental modernization protocol itself. Rather than proposing a final architecture and justifying it post-hoc, the authors start from a well-known baseline (ResNet-50 or ResNet-200) and apply a sequence of independent modifications, measuring the accuracy and FLOPs impact of each step in isolation. This transforms what could have been a black-box proposal ("here is ConvNeXt, it works well") into a transparent ablation study where every design choice is justified by a measured improvement.

The procedure operates under a critical constraint: FLOPs are roughly controlled throughout the exploration. The paper states that "the FLOPs are roughly controlled over the course of the exploration, though at intermediate steps the FLOPs might be higher or lower than the reference models." This means that when a modification reduces FLOPs (e.g., switching to depthwise convolution), the network width is subsequently increased to bring FLOPs back to the target range. This prevents the common pitfall where a more expensive model naturally performs better and the improvement is misattributed to the architectural change rather than the increased capacity. The target FLOP regimes are $\sim 4.5 \times 10^9$ (ResNet-50 / Swin-T) and $\sim 15.0 \times 10^9$ (ResNet-200 / Swin-B).

The paper studies both regimes simultaneously, presenting results for the smaller regime in the main text (Figure 2, foreground bars) and the larger regime in gray background bars and Appendix C tables. The authors state that "the conclusions for higher capacity models are consistent," and any differences (e.g., kernel size saturation at 5×5 instead of 7×7 for ResNet-200) are noted explicitly. This dual-regime validation strengthens the generalizability of the findings: if a design choice only worked at one scale, it would be flagged as scale-dependent.

Each ResNet-50 regime accuracy is reported as the mean of three training runs with different random seeds, with standard deviations reported in Table 10. This provides statistical rigor that is often absent in architecture papers, where single-run results can be misleadingly noisy. For example, the "patchify stem" modification reports $79.51 \pm 0.18\%$, indicating a small but statistically significant improvement over the $79.36 \pm 0.07\%$ of the stage-ratio baseline.

The paper explicitly frames each step as answering a question: "How do design decisions in Transformers impact ConvNets' performance?" The ordering of modifications is not arbitrary — it follows a logic of moving from structural (macro) to detailed (micro) changes, with each step building on the accumulated modifications from previous steps. This means that later modifications are evaluated in the context of an already-partially-modernized ConvNet, not in isolation on the original ResNet. The paper makes this clear by stating which modifications are retained ("We will now use this stage compute ratio") or discarded (hatched bars in Figure 2).

Training Recipe Modernization: Establishing the Baseline

Before any architectural changes are made, the paper establishes that training procedure alone accounts for a substantial fraction of the perceived ConvNet-Transformer performance gap. The baseline ResNet-50 model trained with the original ResNet recipe achieves 76.1% ImageNet top-1 accuracy (from the PyTorch vision models reference). When retrained with the "enhanced recipe" — the set of training techniques associated with vision Transformer training — this jumps to 78.8% (+2.7%).

This enhanced recipe (Table 5) is explicitly modeled on the training procedures of DeiT and Swin Transformer:

Optimizer and schedule:

• AdamW optimizer with $\beta_1 = 0.9$, $\beta_2 = 0.999$, and weight decay of 0.05
• Base learning rate of $4 \times 10^{-3}$
• Training duration: 300 epochs (extended from the original 90 epochs for ResNets)
• Warmup: 20 epochs of linear warmup followed by cosine decay schedule
• Batch size: 4096

Data augmentation and regularization:

• Mixup with $\alpha = 0.8$
• Cutmix with $\alpha = 1.0$
• RandAugment with parameters (9, 0.5) — meaning 9 augmentation operations are applied with magnitude 0.5
• Random Erasing with probability 0.25
• Label Smoothing with factor 0.1 — softens one-hot targets to $(1 - 0.1)$ for the correct class and $0.1 / (K-1)$ for incorrect classes
• Stochastic Depth (also known as DropPath) — randomly drops entire residual blocks during training, with rates customized per model variant (0.1 for ConvNeXt-T, 0.4 for ConvNeXt-S, 0.5 for ConvNeXt-B/L)

Additional techniques:

• Layer Scale with initial value $1 \times 10^{-6}$ — a learnable scalar multiplier applied to the output of each residual block, initialized near zero to encourage identity-like behavior at initialization (a technique introduced by Touvron et al. for training deeper ViTs)
• Exponential Moving Average (EMA) with decay 0.9999 — maintains a moving average of model weights during training and uses this averaged model for evaluation, which the paper finds "alleviates larger models' overfitting"
• Weight initialization: truncated normal distribution with standard deviation 0.2

The paper notes a specific interaction: "EMA is disabled" for the modernization experiments in Section 2 "as we find using EMA severely hurts models with BatchNorm layers." This is a non-obvious empirical finding — EMA interacts poorly with BatchNorm's running statistics — and it means the modernization experiments in Figure 2 are run without EMA, while the final ConvNeXt models (which use LayerNorm) do benefit from EMA. This detail matters for reproducibility.

The ResNet-200 baseline similarly improves from the original 78.20% to 81.14% (+2.94%) with the enhanced recipe alone (Table 11), confirming that the training recipe effect scales to larger models.

Why this step matters beyond the accuracy gain: By standardizing the training recipe before making architectural changes, the paper ensures that all subsequent improvements can be attributed to architecture, not to optimization. This is the key methodological insight that distinguishes the paper from naive "ConvNet vs. Transformer" comparisons where both architecture and training differ simultaneously.

Macro Design: Stage Ratio and Patchify Stem

Changing stage compute ratio. The original ResNet-50 distributes its 16 residual blocks across four stages as (3, 4, 6, 3), with the heaviest computation in stage 3 ("res4"), which operates on 14×14 feature maps. This design was historically motivated by compatibility with object detection frameworks like Faster R-CNN, where the Region Proposal Network operates on the 14×14 feature map and benefits from a high-capacity representation at that resolution.

Swin Transformers use a different distribution: Swin-T uses (3, 3, 9, 3) — still heavy on stage 3, but with an even more extreme concentration (9 blocks rather than 6). Larger Swin variants use (3, 3, 27, 3) or (3, 3, 9, 3) depending on the size. The paper adopts (3, 3, 9, 3) for the ResNet-50 regime, which aligns FLOPs with Swin-T and improves accuracy from 78.8% to 79.4%.

The paper does not claim this ratio is optimal — it explicitly notes: "researchers have thoroughly investigated the distribution of computation [53, 54], and a more optimal design is likely to exist." The change is adopted not because it is provably best, but because it matches the Swin reference point, enabling a fair comparison.

For the ResNet-200 regime, the change is more dramatic: the original ResNet-200 uses (3, 24, 36, 3) (a very deep stage 2 and 3), which is transformed to Swin-B's (3, 3, 27, 3). This drastically reduces FLOPs, so the width is simultaneously increased from 64 to 84 to maintain the target FLOP range. The accuracy changes from 81.14% to 81.33% — a modest gain but at substantially different internal structure.

Changing stem to "Patchify." The stem cell is the first few layers that process the raw input image before the main network stages. In standard ResNets, the stem consists of:

• A 7×7 convolution with stride 2 (reducing spatial dimensions by 2×)
• Followed by 3×3 max pooling with stride 2 (reducing by another 2×)
• Result: 4× total downsampling, producing a 56×56 feature map from a 224×224 input

Vision Transformers use a much more aggressive initial processing: a single large-kernel convolution with non-overlapping stride. For vanilla ViT with 16×16 patches, this is equivalent to a 16×16 convolution with stride 16. Swin Transformers use a patch size of 4×4 (stride 4, no overlap) because their multi-stage design requires a higher-resolution initial feature map to accommodate subsequent downsampling stages.

The paper replaces the ResNet stem with "a patchify layer implemented using a 4×4, stride 4 convolutional layer." This is architecturally simpler (one conv layer instead of conv + pool) and matches the Swin convention. The accuracy change is minimal but non-negative: 79.4% → 79.5%. The paper concludes that "the stem cell in a ResNet may be substituted with a simpler 'patchify' layer à la ViT which will result in similar performance" — meaning the more complex stem offers no advantage when the rest of the network is properly designed.

For the ResNet-200 regime, the patchify stem improves accuracy from 81.33% to 81.59%, a slightly larger gain that suggests the simplification may be more beneficial at larger scales.

ResNeXt-ification: Depthwise Convolution and Width Expansion

This step transforms the internal structure of each residual block to incorporate grouped convolutions, following the ResNeXt design philosophy. The motivation is explicitly linked to self-attention:

"We note that depthwise convolution is similar to the weighted sum operation in self-attention, which operates on a per-channel basis, i.e., only mixing information in the spatial dimension."

The ResNeXt principle states: use grouped convolutions (where input channels are partitioned into groups, each group processed by independent filters) to reduce FLOPs, then expand the network width to recover capacity. The extreme case is depthwise convolution, where the number of groups equals the number of input channels — each channel is convolved with its own single filter, completely separating spatial processing across channels.

Implementation detail: The original ResNet bottleneck block (Figure 3a) has structure:

• 1×1 convolution: 256-d → 64-d (compress)
• 3×3 convolution: 64-d → 64-d (spatial mixing)
• 1×1 convolution: 64-d → 256-d (expand)

The paper replaces the 3×3 convolution with a 3×3 depthwise convolution. This drastically reduces FLOPs because depthwise convolution has $C \times K^2$ parameters rather than $C_{in} \times C_{out} \times K^2$ for a standard convolution. The accuracy drops as expected (80.5% → 78.3%, Table 10) due to the reduced capacity.

Width expansion: Following ResNeXt's strategy, the network width is increased from 64 to 96 channels (matching Swin-T's channel count) to compensate. This brings accuracy to 80.5% but at increased FLOPs (5.27G vs. the original 4.09G). The width expansion adds more 1×1 convolution parameters (which scale with $C_{in} \times C_{out}$), but the depthwise spatial convolutions remain cheap.

The paper frames this change through the lens of spatial vs. channel mixing separation:

"The combination of depthwise conv and 1 × 1 convs leads to a separation of spatial and channel mixing, a property shared by vision Transformers, where each operation either mixes information across spatial or channel dimension, but not both."

In a Transformer: self-attention mixes information across the spatial dimension (tokens attend to each other) while operating per-channel; the MLP (implemented as two 1×1 convolutions) mixes information across channels while operating per-token. In a depthwise-separable ConvNet: the depthwise convolution mixes information spatially (per-channel), while the 1×1 convolutions mix information across channels (per-spatial-location). The parallel is exact: the architecture enforces a clean separation of spatial and channel computation.

For the ResNet-200 regime (Table 11), the same pattern holds: switching to depthwise conv drops accuracy from 81.59% to 80.54% with FLOPs halved (7.23G), then expanding width to 128 (Swin-B's channel count) recovers to 81.85% at 16.76G FLOPs.

Inverted Bottleneck: Transformer-Style Channel Expansion

Every Transformer block contains an inverted bottleneck in its MLP sub-layer: the hidden dimension is expanded by a factor of 4 relative to the input/output dimension. For example, Swin-T's MLP maps 96-d → 384-d → 96-d. This is "inverted" relative to the standard ResNet bottleneck which compresses the channel dimension in the middle (256-d → 64-d → 256-d).

The paper draws an explicit historical connection:

"Interestingly, this Transformer design is connected to the inverted bottleneck design with an expansion ratio of 4 used in ConvNets. The idea was popularized by MobileNetV2 [61]."

The transformation from the current ResNeXt-ified block to an inverted bottleneck block is shown in Figure 3 (a) → (b). The block structure changes from:

• 1×1 conv: 96-d → 96-d (no compression in ResNeXt, maintains width)
• 3×3 depthwise conv: 96-d → 96-d
• 1×1 conv: 96-d → 384-d (expand)

to:

• 1×1 conv: 96-d → 384-d (expand by 4×)
• 3×3 depthwise conv: 384-d → 384-d
• 1×1 conv: 384-d → 96-d (compress)

Critical architectural consequence — the downsampling blocks: The FLOPs impact of this inversion is not uniform across all blocks. In ResNet, spatial downsampling occurs in the first block of each stage (except stage 1). These blocks have a shortcut connection that also requires downsampling, implemented via a 1×1 convolution with stride 2. In the original bottleneck design, this shortcut compression has large FLOPs because it operates on high channel counts. In the inverted bottleneck, the shortcut is at the narrow (96-d) channel dimension rather than the wide (384-d) dimension, dramatically reducing its cost. The paper notes:

"Despite the increased FLOPs for the depthwise convolution layer, this change reduces the whole network FLOPs to 4.6G, due to the significant FLOPs reduction in the downsampling residual blocks' shortcut 1×1 conv layer."

In other words, the depthwise convolution becomes more expensive (operating on 384 channels instead of 96), but the savings from cheaper downsampling shortcuts more than compensate. The accuracy improves slightly from 80.5% to 80.6% while FLOPs decrease.

Scale-dependent effect: For the ResNet-200 regime, this step brings a substantially larger gain: 81.85% → 82.64% (+0.79%) while reducing FLOPs from 16.76G to 15.68G. The paper notes this discrepancy explicitly ("this step brings even more gain... also with reduced FLOPs"), suggesting that the inverted bottleneck's efficiency benefits compound at larger widths.

Large Kernel Sizes: Position and Magnitude

This experiment investigates whether ConvNets benefit from larger convolutional kernels — a direct analogue to the 7×7 local window attention in Swin Transformers. The exploration proceeds in two phases: repositioning the depthwise convolution, then sweeping kernel sizes.

Phase 1: Moving up the depthwise convolution layer (Figure 3b → 3c). In Transformers, the Multi-head Self-Attention (MSA) — the "heavy" spatial mixing module — is placed before the MLP, not after. The paper mirrors this design: the depthwise convolution (the spatial mixing module) is moved from between the two 1×1 convolutions to the beginning of the block.

The new block structure becomes:

• 7×7 depthwise conv: 96-d → 96-d (spatial mixing, now first)
• 1×1 conv: 96-d → 384-d (expand)
• GELU activation
• 1×1 conv: 384-d → 96-d (compress)

The paper justifies this positioning:

"As we have an inverted bottleneck block, this is a natural design choice — the complex/inefficient modules (MSA, large-kernel conv) will have fewer channels, while the efficient, dense 1×1 layers will do the heavy lifting."

In the inverted bottleneck, the depthwise convolution at the beginning operates on the narrow 96-channel representation (before expansion), making it cheaper than if it operated on the 384-channel middle. The heavy 1×1 convolutions operate on the wider representation where they are most effective.

This intermediate repositioning reduces FLOPs from 4.64G to 4.07G (because the depthwise conv now operates on fewer channels) but causes a temporary accuracy drop from 80.6% to 79.9%. The paper treats this as an intermediate step — not an improvement on its own, but a necessary precondition for the kernel size experiments that follow.

Phase 2: Increasing kernel size. With the depthwise convolution positioned at the front of the block, the paper systematically sweeps kernel sizes: 3×3, 5×5, 7×7, 9×9, and 11×11.

"The network's performance increases from 79.9% (3×3) to 80.6% (7×7), while the network's FLOPs stay roughly the same."

The accuracy progression (Table 10) is:

• 3×3: 79.92%
• 5×5: 80.35%
• 7×7: 80.57% (peak)
• 9×9: 80.57% (saturation)
• 11×11: 80.47% (slight decline)

Key finding — saturation at 7×7: Performance improves monotonically up to 7×7, then plateaus (9×9 achieves identical accuracy) and slightly degrades at 11×11. This is significant because it establishes that the benefit of larger receptive fields has a clear optimum — unlike Swin's 7×7 window attention, going to larger kernels doesn't help and may hurt.

Scale-dependent verification: The ResNet-200 regime shows a slightly different saturation point: performance peaks at 5×5 (82.32%) rather than 7×7, with larger kernels showing slight degradation (82.30% at 7×7, 82.27% at 9×9, 82.18% at 11×11). The paper notes this but does not extensively analyze why the larger model saturates earlier — it may be that the increased depth at the ResNet-200 scale provides sufficient receptive field through layer stacking, reducing the benefit of explicit large kernels.

The paper chooses 7×7 as the default kernel size for ConvNeXt, matching Swin's window size and representing a widely supported convolution size on GPU hardware. This is a pragmatic choice: "large kernel sizes have been used in the past with ConvNets [40, 68], [but] the gold standard (popularized by VGGNet [65]) is to stack small kernel-sized (3×3) conv layers, which have efficient hardware implementations on modern GPUs [41]."

Micro Design: Activations, Normalizations, and Downsampling

These final modifications operate at the layer level, fine-tuning specific choices that collectively push ConvNeXt from approximately matching Swin-T to clearly exceeding it.

Replacing ReLU with GELU. The Gaussian Error Linear Unit is defined as:

$\text{GELU}(x) = x \cdot \Phi(x)$

where $\Phi(x)$ is the cumulative distribution function of the standard normal distribution. In practice, it is approximated as:

$\text{GELU}(x) \approx 0.5x \left(1 + \tanh\left(\sqrt{2/\pi}(x + 0.044715x^3)\right)\right)$

What it computes: GELU applies a smooth, probabilistic gating mechanism — rather than hard-thresholding at zero like ReLU ($\max(0, x)$), it multiplies the input by the probability that a standard normal random variable is less than $x$. This means GELU can output small negative values for slightly negative inputs (unlike ReLU which zeros them out), and it has a non-zero derivative everywhere (unlike ReLU which has a discontinuous derivative at zero).

Why this form: GELU provides a smoother optimization landscape than ReLU, which matters for Transformers because they lack BatchNorm's regularization effect and rely on LayerNorm instead. The smooth gradients help with training stability. However, the paper finds that "ReLU can be substituted with GELU in our ConvNet too, although the accuracy stays unchanged (80.6%)." This is notable — the choice of activation function alone does not improve performance in this context. The paper adopts GELU to match Transformer conventions, not because it provides a measurable gain.

Fewer activation functions. Transformers are sparse with activations: a standard Transformer block has exactly one activation function (in the MLP, between the two linear layers). The MSA block has no activation function between its linear projections. In contrast, standard ConvNets place a ReLU after every convolutional layer (including 1×1 convolutions).

The paper examines what happens when ConvNet blocks adopt the Transformer pattern: "we eliminate all GELU layers from the residual block except for one between two 1×1 layers, replicating the style of a Transformer block." The block structure becomes:

• 7×7 depthwise conv (no activation after)
• 1×1 conv to 384-d (no activation after)
• GELU activation (the only one)
• 1×1 conv to 96-d

This change improves accuracy from 80.6% to 81.3% (+0.7%), a substantial gain from simply removing activation functions. The paper does not provide a detailed theoretical explanation for why fewer activations help, but one plausible mechanism is that removing ReLU/GELU after the depthwise convolution allows the network to learn a wider range of spatial transformations (since activations clip negative values, they restrict the representable functions). Another possibility is that fewer non-linearities reduce optimization difficulty by making gradient flow more direct.

In the ResNet-200 regime, the gain is similar: 82.19% → 82.71% (+0.52%, Table 11).

Fewer normalization layers. Transformer blocks typically use normalization before the attention and MLP sub-layers (Pre-LN configuration), not after. The paper experiments with reducing the number of normalization layers: "we remove two BatchNorm (BN) layers, leaving only one BN layer before the conv 1×1 layers." This further improves accuracy from 81.3% to 81.4%, already surpassing Swin-T's 81.3%.

The paper empirically finds that "adding one additional BN layer at the beginning of the block does not improve performance" — meaning one normalization per block (placed before the spatial mixing module) is optimal. This makes ConvNeXt blocks have fewer normalization layers than Transformers, which have two LayerNorms per block (one before MSA, one before MLP). In the ResNet-200 regime, this step provides an even larger gain: 82.71% → 83.17% (+0.46%), suggesting that normalization efficiency matters more for larger models.

Substituting BN with LN. This is the most technically interesting micro design change because the paper is directly transplanting a Transformer-specific component into a ConvNet. Batch Normalization normalizes across the batch and spatial dimensions:

$\text{BN}(x) = \gamma \cdot \frac{x - \mu_{\text{batch}}}{\sqrt{\sigma^2_{\text{batch}} + \epsilon}} + \beta$

where $\mu_{\text{batch}}$ and $\sigma^2_{\text{batch}}$ are the mean and variance computed over the minibatch and spatial dimensions for each channel independently. Layer Normalization normalizes across the channel dimension for each spatial location independently:

$\text{LN}(x) = \gamma \cdot \frac{x - \mu_{\text{channel}}}{\sqrt{\sigma^2_{\text{channel}} + \epsilon}} + \beta$

where $\mu_{\text{channel}}$ and $\sigma^2_{\text{channel}}$ are computed over all channels at each spatial position for each sample independently.

What each computes: BN standardizes each channel's activations using statistics from other samples in the batch, introducing a dependency between samples that acts as a regularizer but can cause issues when batch statistics are noisy (small batches) or when the test distribution differs from training. LN standardizes each sample's activations independently using statistics from across its own channels, making it sample-independent and thus consistent between training and inference.

Why LN over BN in this context: The paper acknowledges that "directly substituting LN for BN in the original ResNet will result in suboptimal performance [83]." The challenge is that ConvNets with BN have co-adapted their architecture and initialization to the normalization scheme. However, "with all the modifications in network architecture and training techniques, here we revisit the impact of using LN in place of BN. We observe that our ConvNet model does not have any difficulties training with LN; in fact, the performance is slightly better, obtaining an accuracy of 81.5%."

The improvement is modest (+0.06% over BN, from 81.41% to 81.47%) but meaningful because it confirms that the modernized architecture is compatible with LayerNorm — a property the original ResNet lacked. This compatibility likely stems from the combination of design changes: the inverted bottleneck, the single normalization per block, and the depthwise-separable structure may collectively create activation statistics that are amenable to per-sample normalization rather than requiring batch-level statistics.

In the ResNet-200 regime, the gain is 83.17% → 83.35% (+0.18%), consistent with the finding.

Separate downsampling layers. In standard ResNets, spatial downsampling (reducing feature map resolution by 2×) occurs inside the first residual block of stages 2–4, using stride-2 convolutions in both the main path (3×3 conv with stride 2) and the shortcut connection (1×1 conv with stride 2). In Swin Transformers, downsampling is handled by a separate layer between stages — a 2×2 patch merging operation (equivalent to a 2×2 convolution with stride 2) that is not part of any Transformer block.

The paper adopts this pattern: "we use 2×2 conv layers with stride 2 for spatial downsampling" placed between stages. However, this "surprisingly leads to diverged training" — a critical practical finding that reveals hidden dependencies in the architecture.

The stabilization solution: The authors discover that "adding normalization layers wherever spatial resolution is changed can help stabilize training." This means inserting LayerNorm:

1. Before each downsampling layer (between stages)
2. After the stem (first LayerNorm after the patchify convolution)
3. After the final global average pooling (before the classifier head)

These additional LayerNorm instances exactly mirror Swin Transformer's use of normalization at resolution transitions. With this stabilization, the separate downsampling layers improve accuracy from 81.5% to 82.0% — a substantial +0.5% gain that pushes ConvNeXt clearly above Swin-T's 81.3%.

Why separate downsampling helps: The paper doesn't provide a mechanistic explanation, but one plausible reason is that separating the downsampling operation from the residual block allows the block itself to focus purely on feature transformation at a fixed resolution. In standard ResNets, the first block of each stage must simultaneously learn to reduce resolution (via stride-2 convolutions) and transform features — a dual objective that may complicate optimization. The normalization at resolution transitions likely helps by preventing activation scale mismatches when the spatial dimensions change abruptly.

In the ResNet-200 regime, separate downsampling yields: 83.35% → 83.60% (+0.25%), bringing ConvNeXt-B above Swin-B's 83.50%.

Scaling Methodology: Constructing ConvNeXt Variants

The final ConvNeXt family is constructed by varying two hyperparameters: the number of channels per stage ($C$) and the number of blocks per stage ($B$). The paper defines five variants:

• ConvNeXt-T: $C = (96, 192, 384, 768)$, $B = (3, 3, 9, 3)$ — derived from ResNet-50 modernization
• ConvNeXt-S: $C = (96, 192, 384, 768)$, $B = (3, 3, 27, 3)$ — deeper stage 3, same width as T
• ConvNeXt-B: $C = (128, 256, 512, 1024)$, $B = (3, 3, 27, 3)$ — wider than S, derived from ResNet-200 modernization
• ConvNeXt-L: $C = (192, 384, 768, 1536)$, $B = (3, 3, 27, 3)$ — wider still
• ConvNeXt-XL: $C = (256, 512, 1024, 2048)$, $B = (3, 3, 27, 3)$ — widest variant, tested only with ImageNet-22K pre-training

Following the convention from both ResNets and Swin Transformers, "the number of channels doubles at each new stage." The block structure is identical across all variants (the ConvNeXt block in Figure 4), only the dimensions change. This makes ConvNeXt scaling purely a matter of adjusting width and depth — there are no resolution changes, kernel size adjustments, or architectural modifications between variants. This is simpler than Swin Transformer scaling, which changes the number of attention heads and sometimes the window size.

Each variant is designed to match the FLOPs and parameter counts of the corresponding Swin variant (T/S/B/L), enabling direct comparison. For example, ConvNeXt-T has 28.6M parameters and 4.5G FLOPs vs. Swin-T's 28.3M parameters and 4.5G FLOPs (Table 1, Table 9). ConvNeXt-XL has no direct Swin counterpart; it serves as a test of whether ConvNeXt scaling continues to improve at larger sizes.

ImageNet-22K Pre-training and Fine-tuning Protocol

For the large-scale experiments, ConvNeXts are pre-trained on ImageNet-22K (14M images, 21,841 classes) and fine-tuned on ImageNet-1K. The pre-training and fine-tuning procedures have different hyperparameters (Tables 5 and 6):

Pre-training on ImageNet-22K (Table 5, right column):

• Optimizer: AdamW with base learning rate $4 \times 10^{-3}$
• Training epochs: 90 (shorter than ImageNet-1K because of the larger dataset)
• Warmup: 5 epochs (shorter warmup)
• Batch size: 4096
• Stochastic depth rates: lower than ImageNet-1K training (0.0 for T, 0.0 for S, 0.1 for B/L, 0.2 for XL) — larger datasets provide natural regularization, reducing the need for drop path
• EMA: disabled during pre-training

Fine-tuning on ImageNet-1K (Table 6):

• Base learning rate: $5 \times 10^{-5}$ (significantly lower than pre-training — 80× reduction)
• Weight decay: $1 \times 10^{-8}$ (near-zero, unlike 0.05 in pre-training)
• Training epochs: 30
• Batch size: 512 (smaller than pre-training's 4096)
• No warmup — training starts directly at the base learning rate
• Layer-wise learning rate decay: 0.7 for ImageNet-1K pre-trained models fine-tuned at 384², 0.8 for ImageNet-22K pre-trained models — each group of 3 consecutive blocks has its learning rate multiplied by this decay factor relative to the block above it
• Stochastic depth: 0.8/0.95 for ConvNeXt-B/L (when fine-tuning from ImageNet-1K pre-training), 0.0/0.1/0.2/0.3/0.4 for T/S/B/L/XL (when fine-tuning from ImageNet-22K pre-training)
• Data augmentation: RandAugment (9, 0.5) and Random Erasing (0.25) are retained; Mixup and Cutmix are removed during fine-tuning
• Head init scale: 0.001 — a small multiplier on the initial weights of the classification head
• EMA: None during fine-tuning for most models, except ConvNeXt-XL fine-tuned from 22K which uses EMA 0.9999

A crucial procedural detail: "The fine-tuning starts from the final model weights obtained in pre-training, without using the EMA weights, even if in pre-training EMA is used and EMA accuracy is reported. This is because we do not observe improvement if we fine-tune with the EMA weights (consistent with observations in [73])." The exception is ConvNeXt-L pre-trained on ImageNet-1K, where "the model accuracy is significantly lower than the EMA accuracy due to overfitting, and we select its best EMA model during pre-training as the starting point for fine-tuning."

Resolution handling: ConvNeXts are fine-tuned at both 224² and 384² resolutions. Unlike ViTs/Swin, which require interpolating position embeddings or adjusting patch sizes when changing input resolution, ConvNeXts are fully convolutional and handle arbitrary input sizes natively. The paper emphasizes this advantage: "ConvNeXts are simpler to fine-tune at different resolutions, as the network is fully-convolutional and there is no need to adjust the input patch size or interpolate absolute/relative position biases." When fine-tuning at 384², the testing crop ratio is changed from 0.875 (the standard for 224² models) to 1.0 (no cropping), following the convention in Swin Transformer and other works.

Isotropic ConvNeXt Construction (Section 3.3)

As an ablation to test whether the ConvNeXt block design is specific to hierarchical architectures, the paper constructs isotropic ConvNeXts — models with no downsampling stages, maintaining the same feature map resolution throughout all depths, matching the ViT design paradigm.

Configuration:

• Feature dimensions: 384 (ConvNeXt-S-iso), 768 (ConvNeXt-B-iso), 1024 (ConvNeXt-L-iso) — matching ViT-S/B/L respectively
• Depths: 18, 18, 36 — chosen to match the parameter counts and FLOPs of the corresponding ViT variants
• Block structure: identical to the hierarchical ConvNeXt block (Figure 4)
• Training: same settings as hierarchical models, but with "longer warmup epochs" (50 epochs instead of 20) and layer scale disabled for S/B variants; stochastic depth rates 0.1/0.2/0.5 for S/B/L

Results (Table 2): Isotropic ConvNeXts perform on par with ViTs at the same parameter/FLOP scale: ConvNeXt-S-iso achieves 79.7% vs. ViT-S 79.8%; ConvNeXt-B-iso achieves 82.0% vs. ViT-B 81.8%; ConvNeXt-L-iso achieves 82.6% vs. ViT-L 82.6%. This demonstrates that the ConvNeXt block design is not specific to hierarchical architectures — it generalizes to isotropic settings and performs competitively without any attention.

The training memory of isotropic ConvNeXts is slightly lower than ViT (e.g., 7.7GB vs. 9.1GB for the B variants at batch size 32 on V100), and the inference throughput is slightly higher (320.1 vs. 302.1 images/s for B variants). This suggests that the depthwise convolution + 1×1 convolution combination is slightly more efficient than multi-head self-attention + MLP when both operate on the same feature dimensions.

4. Key Insights and Innovations

Innovation 1: Systematically Disentangling Architecture from Training Procedure as a Scientific Methodology

The paper's most foundational contribution is not any single architectural finding but rather its methodology for controlled comparison between architectural paradigms. Before this work, the dominant practice when comparing ConvNets and vision Transformers was system-level comparison: take a ResNet trained with one recipe, take a Swin Transformer trained with a different recipe, and compare their accuracies. This conflates two entirely different sources of performance variation — the architecture itself and the optimization procedure — making it impossible to attribute differences to either one.

The paper makes this problem explicit and then solves it through a simple but powerful protocol: standardize the training procedure first, then vary the architecture incrementally. The first step of the modernization roadmap is not an architectural change at all — it is applying the Transformer training recipe (AdamW, 300 epochs, Mixup/Cutmix/RandAugment, Stochastic Depth, Label Smoothing) to a standard ResNet. This alone yields a +2.7% improvement for ResNet-50 (76.1% → 78.8%) and +2.94% for ResNet-200 (78.20% → 81.14%), demonstrating that:

"a significant portion of the performance difference between traditional ConvNets and vision Transformers may be due to the training techniques."

This insight is intellectually significant because it reframes the entire ConvNet-vs-Transformer debate. The question shifts from "are Transformers better than ConvNets?" to "how much of the observed gap is architecture, and how much is optimization?" By answering this quantitatively and then holding the training recipe fixed throughout the subsequent architecture exploration, the paper creates a controlled experimental framework where each architectural change can be evaluated on its own merits, uncontaminated by optimization differences.

This methodology has implications beyond this paper. It suggests that much of the architectural progress attributed to novel operations (self-attention, MLP-Mixers, etc.) in recent years may partly reflect improvements in training recipes that were developed concurrently. The paper implicitly argues that any fair architecture comparison must control for training procedure — a methodological standard that was not consistently applied in the pre-ViT era when ConvNets and Transformers were developed by different communities with different optimization conventions.

The incremental modernization protocol itself — 12 discrete steps, each measured for accuracy and FLOPs impact, with clear indication of which modifications are retained or discarded (Figure 2) — is a contribution to experimental design in architecture research. Rather than proposing a final model and justifying it post-hoc, the paper makes the search trajectory transparent, allowing readers to assess the contribution of each design choice independently and to understand trade-offs (e.g., the temporary accuracy drop when repositioning the depthwise convolution, which is justified as enabling subsequent kernel size gains). This transparency is rare in architecture papers and significantly increases the credibility of the findings.

Innovation 2: Convolutional Inductive Biases, Not Self-Attention, Are What Make Hierarchical Transformers Work

The paper makes a provocative conceptual move: it argues that the success of hierarchical vision Transformers is primarily attributable to their ConvNet-like structural properties — local processing, multi-stage hierarchy, spatial/channel mixing separation — rather than to self-attention per se. This is not merely a claim; it is demonstrated constructively by replacing every attention-based component in a Swin-like architecture with a standard convolutional operation and showing that performance is maintained or improved.

The key evidence is the isomorphism between the Swin Transformer block and the ConvNeXt block (Figure 4). The paper maps:

• Swin's windowed multi-head self-attention (7×7 windows) → ConvNeXt's 7×7 depthwise convolution
• Swin's MLP (1×1 "convolutions" expanding 4× then projecting) → ConvNeXt's two 1×1 convolutions with 4× expansion
• Swin's LayerNorm → ConvNeXt's LayerNorm
• Swin's residual connections → ConvNeXt's residual connections

The architectural correspondence is nearly exact. The paper observes that both designs implement the same functional decomposition: spatial mixing followed by channel mixing, with normalization and residuals. The only difference is the specific spatial mixing operation: learned attention weights conditioned on content (Swin) versus learned convolutional weights independent of content (ConvNeXt).

This finding fundamentally challenges the narrative that self-attention is a superior primitive for visual processing. If a depthwise convolution — a 1930s-era signal processing operation with no data-dependent weights, no query-key-value projections, and no attention scores — can replace multi-head self-attention with no performance loss (and in fact a gain: ConvNeXt-T achieves 82.1% vs. Swin-T's 81.3%), then self-attention is not the active ingredient in Swin's success. The active ingredients are the macro-structure (multi-stage hierarchy), the micro-structure (spatial-then-channel mixing with inverted bottleneck), and the training recipe.

The paper's concurrent works [25] and ConvMixer [4] made related observations, but the paper's contribution is more comprehensive: it demonstrates this equivalence not just in a small-scale ablation but across a full family of models (T/S/B/L/XL) on multiple benchmarks (ImageNet, COCO, ADE20K) with both ImageNet-1K and ImageNet-22K pre-training. The isotropic ConvNeXt experiment (Section 3.3, Table 2) further generalizes the finding: even in non-hierarchical ViT-style architectures, the ConvNeXt block matches or exceeds Transformer blocks, showing that the depthwise convolution approach is not dependent on the hierarchical structure.

This insight has significant implications for architecture design: it suggests that the field should focus less on inventing new attention variants and more on understanding the design principles (stage ratios, bottleneck structures, normalization placement, training recipes) that make any spatial mixing operation work well. The paper concludes pointedly that "the essence of convolution is not becoming irrelevant; rather, it remains much desired and has never faded."

Innovation 3: Architectures Have a "Modernization Gap" That Can Be Closed by Cross-Paradigm Learning

The paper introduces a conceptual framework that might be called cross-paradigm modernization: the idea that an older architectural paradigm (ConvNets) can be substantially improved by systematically adopting design principles from a newer paradigm (Transformers), without adopting the newer paradigm's core operation. This is distinct from hybrid approaches (CoAtNet, ConViT, CvT) that combine both operations — the paper deliberately uses only convolutions and asks how much of the Transformer advantage can be captured through structural mimicry alone.

The modernization roadmap (Section 2, Figure 2) makes this framework concrete. The 12-step sequence is not an arbitrary set of improvements; it is a systematic traversal of the design space that separates ConvNets from Transformers, ordered from macro to micro. At each step, a design choice from Transformers is mapped to a ConvNet equivalent and evaluated. The result is that a standard ResNet-50 is transformed from 78.8% to 82.0% — a 3.2% improvement achieved entirely through architectural changes that were developed in, or popularized by, the Transformer literature.

What makes this conceptually significant is that none of the individual modifications are novel to ConvNets. The paper explicitly acknowledges this: "many design choices have all been examined separately over the last decade, but not collectively." Depthwise convolutions appeared in MobileNet and Xception. Inverted bottlenecks appeared in MobileNetV2. GELU activations predate Transformers. Large kernels appeared in AlexNet. The innovation is the demonstration that these known techniques, when assembled in a Transformer-informed configuration and trained with a modern recipe, achieve performance that was thought to require self-attention. It is the combination and the specific architecture-level organization that matters, not any single component.

This modernization gap concept has implications beyond this specific ConvNet-Transformer comparison. It suggests a general research strategy: when a new paradigm (Transformers, MLP-Mixers, graph networks) appears to outperform an older one, the first question should be "what design and training innovations did the new paradigm introduce that can be retrofitted to the old one?" rather than "is the new paradigm's core operation superior?" The paper demonstrates that a substantial fraction of the new paradigm's advantage is portable.

The paper is careful not to overclaim: it notes that the specific modernization trajectory is guided by Swin Transformer as the reference, and different reference architectures might suggest different modernization paths. The stage compute ratio of (3, 3, 9, 3) is adopted from Swin-T but acknowledged as likely suboptimal ("a more optimal design is likely to exist"). This honesty about the contingency of the specific choices strengthens rather than weakens the conceptual contribution: the framework of systematic modernization is general, even if the specific parameter settings are not.

Innovation 4: Clarifying the Scaling Picture — ConvNets Scale Just as Well When Properly Modernized

Before this work, a widely cited advantage of vision Transformers was their superior scaling behavior: larger ViTs continued to improve while larger ConvNets appeared to saturate. The paper challenges this narrative directly by demonstrating that ConvNeXt models scale competitively across a range of model sizes (from 29M to 350M parameters) and training data scales (from 1.2M ImageNet-1K images to 14M ImageNet-22K images).

The evidence is in Table 1 and Figure 1. The ImageNet-1K trained models show consistent improvement from ConvNeXt-T (82.1%) through S (83.1%), B (83.8%), to L (84.3%) — each step increasing either depth or width. The ImageNet-22K pre-trained models show further gains: ConvNeXt-XL reaches 87.8% top-1 accuracy, which is competitive with or exceeding state-of-the-art Transformer and ConvNet models at similar scales. The bubble plot in Figure 1 places ConvNeXt variants on the same accuracy-FLOPs trade-off curve as Swin Transformers, demonstrating that the scaling trajectory is parallel — ConvNets are not saturating earlier.

The significance of this finding is that it isolates the cause of the previously observed scaling gap. Earlier comparisons that showed Transformers scaling better than ConvNets were comparing modern Transformers against older ConvNet designs (ResNet) trained with older recipes. The paper shows that when ConvNets are modernized architecturally and trained with contemporary recipes, the scaling gap disappears. This implies that architectural design and training procedure, not the choice of spatial mixing operation, determine scaling behavior. Both depthwise convolutions and windowed self-attention produce models that scale similarly when embedded in equivalent macro-architectures and trained equivalently.

The paper's downstream task results (Tables 3 and 4) extend this finding beyond classification: ConvNeXt scales as well as or better than Swin Transformer on COCO object detection/instance segmentation and ADE20K semantic segmentation. For example, ConvNeXt-XL achieves 55.2 box AP on COCO (Cascade Mask-RCNN, 3× schedule with ImageNet-22K pre-training), and 54.0 mIoU on ADE20K (UperNet with multi-scale testing). These are state-of-the-art results achieved without any attention-based components, confirming that the scaling behavior is not classification-specific.

A subtle but important aspect of this finding is what it implies about the interaction between scale and inductive bias. The original ViT paper argued that Transformers' weaker inductive bias (no built-in translation equivariance, no locality prior) becomes advantageous at large scale because the model can learn more flexible representations from data. The ConvNeXt results suggest an alternative interpretation: strong but well-designed inductive biases (locality, translation equivariance, multi-scale hierarchy) remain beneficial even at large scale, and what appeared to be inductive bias limitations were actually suboptimal architectural and training choices. ConvNeXt's inductive biases — enforced through local depthwise convolutions and hierarchical structure — do not hinder scaling; they complement it by providing efficient computation (linear complexity in input size, unlike quadratic attention) and better sample efficiency.

Innovation 5: A Practical Refutation of the "Attention Is Necessary" Assumption Through Hardware Efficiency

While the accuracy parity between ConvNeXt and Swin Transformer is the headline result, the paper makes an equally important but more practical contribution: demonstrating that the convolutional formulation is more hardware-efficient than the attention-based formulation for equivalent accuracy. This is not a theoretical claim about operations counts (FLOPs are matched by design) but an empirical claim about realized throughput and memory consumption on modern hardware.

The throughput measurements in Table 12 are striking: on A100 GPUs with TensorFloat32 and channel-last memory layout, ConvNeXt achieves throughput improvements ranging from +34% (ConvNeXt-L at 384²) to +49% (ConvNeXt-S at 224²) over equivalent Swin Transformer variants. At the small scale (ConvNeXt-T vs. Swin-T), the difference is 1943 vs. 1326 images/second. The paper also notes that training memory is lower: "training Cascade Mask-RCNN using ConvNeXt-B backbone consumes 17.4GB of peak memory with a per-GPU batch size of 2, while the reference number for Swin-B is 18.5GB."

The significance of this finding extends beyond the specific comparison. It demonstrates that the FLOPs metric alone can be misleading for comparing architectures with different computational structures. Swin Transformers and ConvNeXts have matched FLOPs by design, but the underlying operations — window partitioning, cyclic shifting, and attention computation vs. standard depthwise and 1×1 convolutions — have different hardware utilization characteristics. Convolutions benefit from decades of hardware optimization (cuDNN, Tensor Cores, Winograd algorithms, channel-last memory layouts) that attention operations, particularly the window-partitioning variants used in Swin, do not enjoy to the same degree.

The paper makes this point explicitly but without overstatement: "This preliminary study shows promising signals that ConvNeXt, employed with standard ConvNet modules and simple in design, could be practically more efficient models on modern hardwares." The word "preliminary" is appropriate — the paper only benchmarks on V100 and A100 GPUs, and the throughput advantage may vary on other hardware (TPUs, mobile processors, FPGAs). But the existence of any throughput advantage at matched accuracy is significant because it challenges the assumption that Transformers are inherently more efficient for vision tasks.

More broadly, this finding connects to a larger theme in the paper: simplicity has practical value beyond aesthetics. The ConvNeXt block (Figure 4) is a straightforward sequence of standard operations: LayerNorm → depthwise conv → 1×1 conv → GELU → 1×1 conv → residual add. It requires no window partitioning, no cyclic shifting, no relative position bias computation, no attention mask handling. This simplicity translates to easier implementation, fewer bugs, more predictable performance across hardware platforms, and simpler adaptation to different input resolutions (no position embedding interpolation needed). The paper's final sentence captures this ethos: "the architecture choice should meet the needs of the task at hand while striving for simplicity."

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary classification benchmark is ImageNet-1K, consisting of 1,000 object classes with approximately 1.2 million training images and a 50,000-image validation set on which top-1 accuracy is reported. For large-scale pre-training experiments, the authors use ImageNet-22K, a superset of ImageNet-1K containing 21,841 classes and approximately 14 million images; models pre-trained on ImageNet-22K are subsequently fine-tuned and evaluated on the ImageNet-1K validation set. Downstream task evaluations use the COCO 2017 dataset for object detection and instance segmentation, and ADE20K for semantic segmentation. Robustness evaluation (Appendix B) additionally uses ImageNet-A, ImageNet-R, ImageNet-Sketch, ImageNet-C, and ImageNet-¯C.

• Base model(s). All modernization experiments start from standard ResNet-50 and ResNet-200 architectures, using the PyTorch vision models reference implementation. For final ConvNeXt comparisons, the paper constructs five variants (ConvNeXt-T/S/B/L/XL) and compares against two Transformer families — Swin Transformer (T/S/B/L variants) and DeiT (S/B variants) — as well as two ConvNet families from neural architecture search: RegNets (RegNetY-16G) and EfficientNets (EfficientNet-B7, EfficientNetV2-L/XL). The isotropic ConvNeXt experiments compare against ViT-S/B/L using supervised training results from DeiT for ViT-S/B and MAE for ViT-L, as these employ improved training procedures over the original ViT paper.

• Metrics. ImageNet classification performance is measured by top-1 accuracy (%) on the validation set. Following Swin Transformer, the paper reports inference throughput (images/second) measured on V100 and A100 GPUs with batch size 128, using automatic mixed precision where applicable. For COCO, the standard box AP (AP^box) and mask AP (AP^mask) metrics are reported, along with AP at IoU thresholds 0.50 and 0.75. For ADE20K, mean Intersection-over-Union (mIoU) is reported, following the convention of using multi-scale testing unless otherwise noted (single-scale results appear in Appendix Table 7). Model efficiency is tracked via FLOPs (floating-point operations, measured at the specified input resolution) and parameter count.

• Baselines. The primary baseline against which ConvNeXt is compared is Swin Transformer at equivalent complexity tiers (Swin-T vs. ConvNeXt-T, Swin-S vs. ConvNeXt-S, Swin-B vs. ConvNeXt-B, Swin-L vs. ConvNeXt-L). Secondary baselines include DeiT-S/B for ImageNet-1K classification, RegNetY-16G and EfficientNet-B7/V2-L/V2-XL for ConvNet comparisons, and ResNeXt-101-32/64 and ResNet-50 for COCO detection baselines. The enhanced-recipe ResNet-50/200 serve as the starting-point baselines for the modernization experiments (Section 2). For robustness benchmarks, comparisons include RVT-S/B (robust vision Transformers).

• Generation budget / compute accounting. The paper uses FLOPs as the primary compute metric for architectural comparisons, calculated at standard resolutions (224² for ImageNet-1K classification, with higher resolutions for fine-tuned models). Throughput is measured as a practical efficiency metric on specific hardware (V100 and A100 GPUs). Training memory is measured in GB for V100 GPUs with a per-GPU batch size of 32 (classification) or 2 (Cascade Mask-RCNN). Unlike scaling law papers that count "generations," this paper's compute accounting is architectural: each modification in Section 2 reports both accuracy change and FLOPs change, with the authors explicitly stating that FLOPs are "roughly controlled over the course of the exploration."

• Cross-validation / statistical protocol. For the modernization experiments in the ResNet-50 regime, each reported accuracy is the mean of three training runs with different random seeds, with standard deviations reported in Table 10. This provides statistical error bars for the incremental modifications. The ResNet-200 regime experiments use single runs (standard for models of that scale). For COCO and ADE20K experiments, the authors conduct "a lightweight sweep" over learning rate, layer-wise learning rate decay, and stochastic depth rate, selecting the best configuration. For ImageNet-22K pre-trained Swin models, the authors re-run the fine-tuning sweep themselves using the official code and pre-trained weights to ensure fair comparison. Fine-tuning at 384² resolution uses a crop ratio of 1.0 (no cropping) during testing, matching Swin Transformer's evaluation protocol.

Main Quantitative Results

Modernization Roadmap: From ResNet-50 to ConvNeXt-T

The modernization experiments in Section 2 and Figure 2 provide the step-by-step evidence that Transformer design choices, implemented convolutionally, incrementally close the gap between ResNets and Swin Transformers. All results in this subsection come from Table 10 (ResNet-50 regime) and Table 11 (ResNet-200 regime), with the main text presenting the ResNet-50/Swin-T FLOP regime (~4.5 GFLOPs).

Training recipe alone (+2.7%): The enhanced training recipe raises ResNet-50 from 76.1% to 78.8% (+2.7 percentage points), establishing that over a quarter of the initial 5.2% gap between the original ResNet-50 and Swin-T (81.3%) is attributable to training procedure rather than architecture. ResNet-200 shows a similar training-only gain from 78.20% to 81.14% (+2.94 points).

Stage ratio adjustment (+0.5%): Changing ResNet-50's block distribution from (3, 4, 6, 3) to Swin-T's (3, 3, 9, 3) improves accuracy from 78.8% to 79.4%, with FLOPs increasing slightly from 4.09G to 4.53G. The ResNet-200 regime simultaneously changes the ratio from (3, 24, 36, 3) to (3, 3, 27, 3) and increases width from 64 to 84, resulting in a modest gain from 81.14% to 81.33%.

Patchify stem (+0.1%): Replacing the ResNet stem (7×7 conv + max pool, 4× downsampling) with a single 4×4, stride 4 "patchify" convolution yields a small improvement from 79.4% to 79.5% at reduced FLOPs (4.42G from 4.53G). The ResNet-200 regime shows a larger gain of 81.33% to 81.59%, suggesting the simplification benefits larger models more.

Depthwise convolution + width expansion (+1.1% over degraded baseline): Replacing the 3×3 spatial convolution with a 3×3 depthwise convolution drops FLOPs dramatically (4.42G to 2.35G) and accuracy falls to 78.3%. Expanding the network width from 64 to 96 channels recovers and surpasses the baseline, reaching 80.5% at 5.27G FLOPs. This two-step sequence is the largest single architectural gain in the roadmap (+1.7% from the depthwise nadir, +1.1% over the patchify stem). The ResNet-200 regime follows the same pattern: depthwise conv drops accuracy from 81.59% to 80.54% at 7.23G FLOPs, then width expansion to 128 recovers to 81.85% at 16.76G FLOPs.

Inverted bottleneck (+0.1% at reduced FLOPs): Inverting the block from wide-narrow-wide to narrow-wide-narrow (4× expansion) improves ResNet-50 regime accuracy from 80.5% to 80.6% while reducing FLOPs from 5.27G to 4.64G — a simultaneous accuracy gain and efficiency improvement driven by cheaper downsampling shortcut convolutions. In the ResNet-200 regime, this step is substantially more beneficial: 81.85% to 82.64% (+0.79%) with FLOPs dropping from 16.76G to 15.68G.

Large kernel repositioning (temporary −0.7%): Moving the depthwise convolution from between the 1×1 layers to the front of the block reduces FLOPs from 4.64G to 4.07G but drops accuracy from 80.6% to 79.9%. This is presented as an intermediate step that enables the kernel size experiments — the paper explicitly notes it is not adopted as a standalone improvement.

Kernel size sweep (+0.6% from 3×3 to 7×7): With the depthwise conv repositioned, increasing kernel size from 3×3 to 7×7 progressively improves accuracy: 3×3 (79.9%), 5×5 (80.4%), 7×7 (80.6%), with saturation at 9×9 (80.6%) and a slight decline at 11×11 (80.5%). FLOPs remain roughly constant across kernel sizes (4.10G–4.29G). The ResNet-200 regime shows a slightly different saturation point at 5×5 (82.32%), with 7×7 achieving 82.30% — the paper adopts 7×7 for consistency across both regimes.

Activation and normalization modernizations (+1.1% total): Replacing ReLU with GELU produces no change (80.6% → 80.6%). Reducing activation functions from three per block to one improves accuracy to 81.3% (+0.7%). Reducing normalization layers from three to one per block improves accuracy to 81.4% (+0.1%). Replacing BatchNorm with LayerNorm improves to 81.5% (+0.1%). The activation reduction is the single largest micro-design gain. In the ResNet-200 regime, the corresponding sequence yields: GELU (82.19%, essentially flat), fewer activations (82.71%, +0.52%), fewer norms (83.17%, +0.46%), BN→LN (83.35%, +0.18%).

Separate downsampling layers (+0.5%): Adding dedicated 2×2 stride-2 convolutional downsampling layers between stages, with additional LayerNorm at resolution transitions for training stability, brings the ResNet-50 regime model to 82.0%, exceeding Swin-T's 81.3%. This is the final step producing ConvNeXt-T. The ResNet-200 regime reaches 83.60%, exceeding Swin-B's 83.50%.

The cumulative modernization from the enhanced-recipe ResNet-50 baseline (78.8%) to ConvNeXt-T (82.0%) is +3.2%, achieved entirely through architectural modifications informed by Transformer design while using only convolutional operations. The entire process is summarized in Figure 2.

ImageNet-1K Classification: ConvNeXt vs. Transformers and ConvNets

Table 1 (upper) presents the ImageNet-1K trained results at 224² resolution, with throughput measured on V100 GPUs:

ConvNeXt-T vs. Swin-T: ConvNeXt-T achieves 82.1% top-1 accuracy vs. Swin-T's 81.3%, a +0.8 percentage point margin. Both have comparable parameters (29M vs. 28M), identical FLOPs (4.5G), and ConvNeXt-T has slightly higher throughput (774.7 vs. 757.9 images/second). ConvNeXt-T also outperforms DeiT-S (79.8%) by 2.3 points and the strong ConvNet baseline RegNetY-16G (82.9% but at 16.0G FLOPs — 3.6× more compute).

ConvNeXt-S vs. Swin-S: ConvNeXt-S achieves 83.1% vs. Swin-S's 83.0%, a narrow +0.1 point margin, with identical parameters (50M) and FLOPs (8.7G), and slightly higher throughput (447.1 vs. 436.7 images/second).

ConvNeXt-B vs. Swin-B: ConvNeXt-B achieves 83.8% vs. Swin-B's 83.5%, a +0.3 point margin, with 89M vs. 88M parameters, identical FLOPs (15.4G), and comparable throughput (292.1 vs. 286.6 images/second). At 384² resolution, ConvNeXt-B achieves 85.1% vs. Swin-B's 84.5% (+0.6 points), with 12.5% higher inference throughput (95.7 vs. 85.1 images/second) and slightly lower FLOPs (45.0G vs. 47.1G). The paper emphasizes that "the FLOPs/throughput advantage of ConvNeXt-B over Swin-B becomes larger when the resolution increases from 224² to 384²."

ConvNeXt-L: ConvNeXt-L achieves 84.3% at 224² and 85.5% at 384². No Swin-L result is reported for ImageNet-1K only training in Table 1 (upper), but the lower section provides the comparison with ImageNet-22K pre-training (see below).

Comparison with EfficientNets: The paper acknowledges that EfficientNetV2-L (85.7% at 480²) and EfficientNet-B7 (84.3% at 600²) achieve strong results with ImageNet-1K training alone. However, ConvNeXt-L at 384² (85.5%) approaches EfficientNetV2-L's performance with a simpler architecture and no architecture search.

ImageNet-22K Pre-training and Fine-tuning

Table 1 (lower) presents results when pre-training on ImageNet-22K and fine-tuning on ImageNet-1K:

At 224² resolution: ConvNeXt-T achieves 82.9% (vs. no reported Swin-T 22K result). ConvNeXt-S achieves 84.6% (vs. no reported Swin-S 22K result). ConvNeXt-B achieves 85.8% vs. Swin-B's 85.2% (+0.6 points). ConvNeXt-L achieves 86.6% vs. Swin-L's 86.3% (+0.3 points). ConvNeXt-XL achieves 87.0%.

At 384² resolution: ConvNeXt-T achieves 84.1%. ConvNeXt-S achieves 85.8%. ConvNeXt-B achieves 86.8% vs. Swin-B's 86.4% (+0.4 points). ConvNeXt-L achieves 87.5% vs. Swin-L's 87.3% (+0.2 points). ConvNeXt-XL achieves 87.8%, which is the paper's highest reported accuracy and matches or exceeds state-of-the-art models at the time. This exceeds EfficientNetV2-XL's 87.3% (at 480²), ViT-L/16's 86.8% (at 384² with AugReg training), and the Big Transfer models (ResNet-152x4: 85.4% at 480²).

Critical scaling observation: The paper emphasizes that the ImageNet-22K results are "important since a widely held view is that vision Transformers have fewer inductive biases thus can perform better than ConvNets when pre-trained on a larger scale. Our results demonstrate that properly designed ConvNets are not inferior to vision Transformers when pre-trained with large dataset." The consistent margins across model sizes (ConvNeXt outperforming Swin at T, S, B, and L scales under 22K pre-training) support the claim that ConvNets scale just as well as Transformers when properly modernized.

Isotropic ConvNeXt vs. ViT (Section 3.3, Table 2)

This ablation tests whether the ConvNeXt block design works in non-hierarchical (ViT-style) architectures that maintain constant feature resolution throughout all layers:

ConvNeXt-S-iso (22M params, 4.3G FLOPs): Achieves 79.7% vs. ViT-S (DeiT-trained, 22M, 4.6G FLOPs) at 79.8%. Essentially tied.

ConvNeXt-B-iso (87M params, 16.9G FLOPs): Achieves 82.0% vs. ViT-B (DeiT-trained, 87M, 17.6G FLOPs) at 81.8%. ConvNeXt-B-iso slightly edges out ViT-B by 0.2 points.

ConvNeXt-L-iso (306M params, 59.7G FLOPs): Achieves 82.6% vs. ViT-L (MAE-trained, 304M, 61.6G FLOPs) at 82.6%. Tied.

Efficiency comparison: Isotropic ConvNeXts have higher throughput than corresponding ViTs (ConvNeXt-S-iso: 1038.7 vs. 978.5 img/s; ConvNeXt-B-iso: 320.1 vs. 302.1 img/s; ConvNeXt-L-iso: 94.4 vs. 93.1 img/s) and lower training memory (ConvNeXt-S-iso: 4.2GB vs. 4.9GB; ConvNeXt-B-iso: 7.7GB vs. 9.1GB; ConvNeXt-L-iso: 20.4GB vs. 22.5GB at batch size 32 on V100).

The paper interprets these results as evidence that "our ConvNeXt block design is competitive when used in non-hierarchical models," showing that the block's effectiveness is not tied to the hierarchical multi-stage architecture — it also works as a drop-in replacement for Transformer blocks in isotropic designs.

Object Detection and Instance Segmentation on COCO (Table 3)

The COCO experiments evaluate ConvNeXt as a backbone for Mask R-CNN and Cascade Mask R-CNN with a 3× training schedule:

Mask R-CNN (simpler detector): ConvNeXt-T achieves 46.2 box AP and 41.7 mask AP, essentially matching Swin-T's 46.0 box AP and 41.6 mask AP, with slightly higher FPS (25.6 vs. 23.1 on A100).

Cascade Mask R-CNN at Swin-T scale: ConvNeXt-T achieves 50.4 box AP and 43.7 mask AP, matching Swin-T's 50.4 and 43.7 exactly, with 13.5 vs. 12.2 FPS.

Cascade Mask R-CNN at Swin-S scale: ConvNeXt-S achieves 51.9 box AP and 45.0 mask AP, matching Swin-S's 51.9 and 45.0, with 12.0 vs. 11.4 FPS.

Cascade Mask R-CNN at Swin-B scale: ConvNeXt-B achieves 52.7 box AP (+0.8 over Swin-B's 51.9) and 45.6 mask AP (+0.6 over Swin-B's 45.0). The paper reports that these ImageNet-1K pre-trained models show ConvNeXt-B pulling ahead of Swin-B. When both are pre-trained on ImageNet-22K (marked with ‡ in Table 3), ConvNeXt-B achieves 54.0 box AP (+1.0 over Swin-B's 53.0) and 46.9 mask AP (+1.1 over Swin-B's 45.8). This is the largest margin in the detection comparisons.

Cascade Mask R-CNN at larger scales (22K pre-trained): ConvNeXt-L achieves 54.8 box AP (+0.9 over Swin-L's 53.9) and 47.6 mask AP (+0.9 over Swin-L's 46.7). ConvNeXt-XL achieves 55.2 box AP and 47.7 mask AP — the highest results in the table, exceeding all Swin variants.

Comparison with traditional ConvNets: ResNet-50 achieves 46.3 box AP, ResNeXt-101-32 achieves 48.1, and ResNeXt-101-64 achieves 48.3. ConvNeXt-T outperforms all three by substantial margins (50.4 vs. 48.3 for the best ResNeXt), while using fewer FLOPs than ResNeXt-101-64 (741G vs. 972G). The FPS comparison shows ConvNeXt-T (13.5) is faster than ResNet-50 (16.2) despite being far more accurate, and substantially faster than ResNeXt-101-64 (12.6).

Semantic Segmentation on ADE20K (Table 4)

ADE20K semantic segmentation uses UperNet with ConvNeXt backbones, trained for 160K iterations:

ImageNet-1K pre-trained at 512² input: ConvNeXt-T achieves 46.7 mIoU vs. Swin-T's 45.8 (+0.9 points). ConvNeXt-S achieves 49.6 vs. Swin-S's 49.5 (+0.1). ConvNeXt-B achieves 49.9 vs. Swin-B's 49.7 (+0.2). The margins are smaller than for classification but consistently favor ConvNeXt.

ImageNet-22K pre-trained at 640² input: ConvNeXt-B achieves 53.1 mIoU vs. Swin-B's 51.7 (+1.4 points — a substantial margin). ConvNeXt-L achieves 53.7 vs. Swin-L's 53.5 (+0.2). ConvNeXt-XL achieves 54.0, the highest result in the table.

The paper notes that parameters and FLOPs are well-matched across ConvNeXt and Swin variants (e.g., 122M/1170G for ConvNeXt-B vs. 121M/1188G for Swin-B at 512²; 235M/2458G for ConvNeXt-L vs. 234M/2468G for Swin-L at 640²), making the comparisons fair.

Robustness Evaluation (Appendix B, Table 8)

The paper evaluates ImageNet-trained ConvNeXt models on distribution shift benchmarks without any specialized robustness modules or additional fine-tuning:

ImageNet-1K trained models (224²): ConvNeXt-T achieves 53.2 mCE on ImageNet-C (lower is better), compared to Swin-T's 62.0 — a substantial corruption robustness advantage. On ImageNet-A, ConvNeXt-T achieves 24.2% vs. Swin-T's 21.6%. On ImageNet-R, 47.2% vs. 41.3%. ConvNeXt-T also compares favorably to the specialized robust Transformer RVT-S (49.4 mCE, though RVT-S uses targeted robustness techniques). ConvNeXt-B (ImageNet-1K) achieves 46.8 mCE vs. Swin-B's 54.4, and 36.7% on ImageNet-A vs. Swin-B's 35.8%.

ImageNet-22K pre-trained models: ConvNeXt-B (22K, 384²) achieves 43.1 mCE, 62.3% on ImageNet-A, 64.9% on ImageNet-R, and 51.6% on ImageNet-Sketch. ConvNeXt-L (22K, 384²) achieves 40.2 mCE, 65.5% on A, 66.7% on R, and 52.8% on Sketch. ConvNeXt-XL achieves 38.8 mCE (best corruption robustness in the table), 69.3% on A, 68.2% on R, and 55.0% on Sketch. The paper highlights that ConvNeXt-XL "demonstrates strong domain generalization capabilities" — these robustness results come "without using any specialized modules or additional fine-tuning procedures."

Throughput Benchmarking on A100 GPUs (Appendix E, Table 12)

Beyond the V100 measurements in Table 1, the paper benchmarks inference throughput on A100 GPUs using TensorFloat32 (TF32) precision and "channel last" memory layout (PyTorch 1.10):

At 224² resolution:

• ConvNeXt-T: 1943.5 img/s vs. Swin-T: 1325.6 img/s (+47%)
• ConvNeXt-S: 1275.3 img/s vs. Swin-S: 857.3 img/s (+49%)
• ConvNeXt-B: 969.0 img/s vs. Swin-B: 662.8 img/s (+46%)
• ConvNeXt-L: 611.5 img/s vs. Swin-L: 435.9 img/s (+40%)

At 384² resolution:

• ConvNeXt-B: 336.6 img/s vs. Swin-B: 242.5 img/s (+39%)
• ConvNeXt-L: 211.4 img/s vs. Swin-L: 157.9 img/s (+34%)

The throughput advantage is substantially larger on A100 than on V100 (where ConvNeXt-T was only +2.2% faster: 774.7 vs. 757.9 img/s). The paper attributes this to the A100's TensorFloat32 tensor cores and the channel-last memory layout, both of which are better optimized for standard convolutions than for the window-partitioning and cyclic-shifting operations in Swin Transformers. ConvNeXt-XL achieves 424.4 img/s at 224² and 147.4 img/s at 384² (no Swin counterpart for comparison, as Swin-L is the largest available).

Ablation Studies and Robustness Checks

Training recipe as an ablation on architecture comparison: The first step of the modernization roadmap is itself an ablation demonstrating that training procedure accounts for +2.7% of ResNet-50 accuracy (76.1% to 78.8%). Without this step, all subsequent architectural improvements would be contaminated by conflating training effects with design effects. This is reported in Table 10 and discussed in Section 2.1.

EMA interaction with BatchNorm: The paper reveals a non-obvious empirical finding in Appendix A.1: "EMA is disabled" for the modernization experiments (Section 2) because "using EMA severely hurts models with BatchNorm layers." The final ConvNeXt models, which use LayerNorm, benefit from EMA (decay 0.9999). This interaction is not explored in depth but represents a practical finding about the compatibility of training techniques across normalization schemes.

Kernel size saturation (Tables 10, 11): The systematic kernel size sweep from 3×3 to 11×11 reveals saturation at 7×7 for the ResNet-50 regime and at 5×5 for the ResNet-200 regime. This is a genuine ablation demonstrating that larger kernels do not monotonically improve performance. The paper briefly verifies that 7×7 works well for both regimes and adopts it, but does not explore why the larger model saturates earlier — a potentially informative ablation that is left unexplored.

ReLU vs. GELU shows zero difference: Replacing ReLU with GELU in the already-modernized architecture changes accuracy from 80.6% to... 80.6% (Table 10). This is an informative negative result: the choice of activation function, often cited as a Transformer advantage, contributes nothing to performance in this context. The paper adopts GELU for consistency with Transformer design but could have kept ReLU with no accuracy penalty.

Fewer activations as the dominant micro-design factor: Reducing from three activation functions per block to one produces +0.7% (80.6% to 81.3% in Table 10), which is the largest single micro-design gain — larger than all normalization changes combined (+0.2% from fewer norms + BN→LN). This is a striking finding because activation functions are often treated as interchangeable; the paper demonstrates that their number and placement matter more than their specific form. The ResNet-200 regime confirms this (+0.52%, Table 11).

Fewer normalization layers in ResNet-200 provides larger gain: The gain from removing two of three BatchNorm layers is +0.14% in the ResNet-50 regime but +0.46% in the ResNet-200 regime (Tables 10, 11). This scale-dependent behavior is noted but not explained — it may relate to larger models being more sensitive to the regularization or optimization effects of excessive normalization.

BN to LN substitution works only after architecture modernization: The paper explicitly notes that "directly substituting LN for BN in the original ResNet will result in suboptimal performance [83]," but the post-modernization architecture handles LN without difficulty, producing +0.06% (81.41% to 81.47%, Table 10). This is an important robustness check showing that the modernization changes (inverted bottleneck, fewer norms, depthwise convolutions) collectively enable LayerNorm compatibility that the original ResNet lacked. The paper does not ablate which specific modernization step(s) enable LN compatibility.

Separate downsampling layers require additional normalization for stability: The paper reports that introducing separate 2×2 stride-2 downsampling layers "surprisingly leads to diverged training" (Section 2.6). Adding LayerNorm at all resolution transitions (before each downsampling layer, after the stem, after global average pooling) stabilizes training and enables the +0.5% gain. This is a practical finding about the interaction between resolution changes and activation statistics — without these extra normalization layers, the modification is unusable.

Isotropic ConvNeXt generalizes the block design (Table 2): This ablation tests whether the ConvNeXt block's success depends on the hierarchical multi-stage structure. Results show the block performs competitively in ViT-style isotropic architectures, matching ViT-S/B/L at equivalent scales. This is a critical robustness check: if the block only worked in hierarchical settings, its applicability would be limited. The isotropic results suggest the block design is a general-purpose primitive.

Layer-wise learning rate decay and stochastic depth rate sweeps for downstream tasks: For COCO, the authors sweep learning rate (1e-4, 2e-4), layer-wise decay (0.7, 0.8, 0.9, 0.95), and stochastic depth rate (0.3–0.8). For ADE20K, they sweep learning rate (8e-5, 1e-4), layer-wise decay (0.8, 0.9), and stochastic depth (0.3–0.5). The paper reports the best configurations but does not present the sweep results as structured ablations. For ImageNet-22K pre-trained Swin models on COCO, the authors run their own sweep to ensure fair comparison rather than relying on published numbers.

Fine-tuning without EMA weights: The paper establishes a procedural finding (Appendix A.2): fine-tuning from EMA weights does not improve results over fine-tuning from the final training weights, "consistent with observations in [73]." The exception is ConvNeXt-L pre-trained on ImageNet-1K, where overfitting makes the EMA model significantly better, so it is used as the starting point for fine-tuning.

Single-scale vs. multi-scale ADE20K testing (Table 7 vs. Table 4): Appendix Table 7 reports single-scale mIoU for ConvNeXt models, providing a reference for comparison. Single-scale results are consistently about 0.4–0.9 points lower than multi-scale (e.g., ConvNeXt-B 22K: 52.6 single-scale vs. 53.1 multi-scale), confirming that the multi-scale testing protocol (standard in the field) provides a consistent boost without changing the relative rankings.

Channel-last memory layout and TF32 precision on A100 (Table 12): This is a hardware-level ablation showing that ConvNeXt benefits more from modern GPU features than Swin Transformer. The paper does not decompose the throughput gain into contributions from TF32 vs. channel-last, but the aggregate effect (up to +49%) demonstrates that standard ConvNet operations are better optimized on current hardware than window-based attention.

Critical Assessment

Claim 1: "ConvNeXts compete favorably with Transformers in terms of accuracy and scalability"

The evidence supporting this claim is extensive. Table 1 demonstrates that ConvNeXt matches or exceeds Swin Transformer at every model size (T, S, B, L) under both ImageNet-1K and ImageNet-22K training, with margins ranging from +0.1 to +0.8 points. Figure 1 visually shows ConvNeXt and Swin variants occupying the same accuracy-FLOPs trade-off curve. Tables 3 and 4 extend this to detection and segmentation, where ConvNeXt again matches or exceeds Swin at all scales.

However, there are qualifications:

The claim is demonstrated for Swin Transformer specifically, not for "Transformers" in general. The paper's primary comparison is against Swin Transformer variants. DeiT (a vanilla ViT with distillation) is included in Table 1 but is handily outperformed by both Swin and ConvNeXt. The isotropic experiments (Table 2) compare ConvNeXt blocks against vanilla ViT blocks, showing parity but not superiority. The paper does not compare against other hierarchical Transformer designs (e.g., PVT, CvT, ViL) except in related work citations. The claim should be understood as "ConvNeXt competes favorably with Swin Transformer, the leading hierarchical vision Transformer."

The comparison is against Swin Transformers at specific scales published in the Swin paper. The Swin paper reported results at T, S, B, and L scales. ConvNeXt matches these scales. The ConvNeXt-XL scale has no direct Swin counterpart for comparison — the largest Swin variant is Swin-L. So the scalability claim for the largest model is demonstrated only against ConvNet baselines (EfficientNetV2-XL) and ViT-L, not against an equivalently scaled Swin.

The throughput advantage is hardware-dependent. The V100 throughput comparison (Table 1) shows ConvNeXt-T only 2.2% faster than Swin-T. It is the A100 comparison (Appendix E, Table 12) that shows 34–49% faster throughput. The paper is transparent about this, but the headline "49% faster" figure requires specific hardware (A100 with TF32 and channel-last layout). On different hardware (TPUs, mobile GPUs, edge devices), the relative efficiency may differ substantially. The paper acknowledges this limitation in Appendix F.

Claim 2: "The performance gap between ConvNets and Transformers is not attributable to the intrinsic superiority of self-attention"

This is the paper's central conceptual claim, and the modernization roadmap (Section 2, Figure 2) is the primary evidence. By starting from a ResNet-50, applying the Transformer training recipe, and incrementally adopting Transformer architectural choices — all implemented with convolutions — the paper demonstrates that a pure ConvNet can match the performance of a Swin Transformer that uses self-attention.

The evidence is strong: the cumulative +3.2% gain from the enhanced recipe baseline to ConvNeXt-T is achieved without any attention mechanism. The final ConvNeXt block (depthwise conv + 1×1 convs + LayerNorm + GELU) is structurally isomorphic to a Transformer block but uses convolution for spatial mixing.

However, there are important boundaries on this claim:

The claim is demonstrated for the specific case where the Transformer uses local window attention. Swin Transformer's self-attention operates within 7×7 windows — it is local, not global. The paper replaces this with 7×7 depthwise convolution, which is also a local operation. The paper does not claim that convolution can replace global self-attention or cross-attention (the isotropic experiments in Section 3.3 replace global attention with depthwise convolution of the same kernel size, but these models use patchify stems that make the feature maps small enough — 14×14 — that a 7×7 kernel covers half the feature map, approaching global receptive field). The claim should be scoped to the hierarchical, local-attention regime that dominates practical vision backbones.

The claim does not demonstrate that self-attention is unnecessary for tasks requiring long-range dependencies. The paper's tasks — ImageNet classification, COCO detection, ADE20K segmentation — are all tasks where local processing with hierarchical feature aggregation has historically worked well. Tasks that genuinely require modeling long-range dependencies (e.g., video understanding across many frames, 3D reconstruction from sparse views, multi-modal fusion) might benefit from attention in ways that convolution cannot replicate. The paper acknowledges this in Appendix F: "Transformers may be more flexible when used for tasks requiring discretized, sparse, or structured outputs."

The claim is about performance, not about the computational mechanisms. The paper demonstrates that depthwise convolution achieves equivalent or better accuracy, but it does not analyze whether the models learn similar representations or use similar computational strategies. It is possible that convolution and attention solve these tasks through different mechanisms, and the equivalence in accuracy masks differences in robustness, failure modes, or transfer learning behavior that the paper's evaluation suite does not capture.

Claim 3: "ConvNets scale just as well as Transformers when properly modernized"

The evidence for scaling behavior comes from two sources: the model family scaling (T → S → B → L → XL) showing monotonic accuracy improvements, and the ImageNet-22K pre-training results showing that ConvNeXt benefits from larger datasets.

The scaling across model sizes is clean: ConvNeXt-T (82.1%), ConvNeXt-S (83.1%), ConvNeXt-B (83.8%), ConvNeXt-L (84.3%) at 224² ImageNet-1K — each step increases depth (T→S) or width (S→B, B→L) and yields improvement. The ImageNet-22K pre-training provides further gains at each scale, and ConvNeXt-XL achieves 87.8% — showing no sign of saturation at the largest tested scale.

However, the scaling analysis has limitations:

The paper tests only two depth configurations at each width. ConvNeXt-T and ConvNeXt-S share the same width (C = 96) but differ in depth (B = (3,3,9,3) vs. (3,3,27,3)). ConvNeXt-B, L, and XL increase width while keeping depth at B = (3,3,27,3). This is a limited exploration of the depth-width Pareto frontier compared to systematic scaling studies (e.g., EfficientNet's compound scaling or the Chinchilla scaling laws). The paper does not claim to have found optimal scaling — it's demonstrating that ConvNeXt scales at all, not that it scales optimally.

The "scaling behavior" comparison against Transformers is limited to Swin's published variants. The Swin paper tested T, S, B, and L scales. The paper matches these and adds one larger scale (XL). This is 5 data points on the scaling curve. A more thorough scaling analysis would test a wider range of FLOPs budgets and would plot scaling law exponents. The paper's Figure 1 provides a visual scaling comparison (bubble plot), but no quantitative scaling law parameters are fitted.

The ImageNet-22K pre-training comparison is against a limited set of Transformer baselines. Table 1 (lower) includes ViT-B/16 and ViT-L/16 (with AugReg training), and Swin-B/Swin-L. Swin-T and Swin-S 22K results are not reported (the Swin paper did not provide them). EfficientNetV2 and BiT models serve as ConvNet baselines. A more complete picture would include other Transformer variants (DeiT-III, ViT with MAE pre-training) at matched scales.

The claim that pre-training benefits ConvNets "just as well" as Transformers is qualitative, not quantitative. The paper shows that ConvNeXt improves with 22K pre-training (e.g., ConvNeXt-B: 83.8% → 85.8%, +2.0 points) and that it matches or exceeds Swin after 22K pre-training. But it does not quantify whether the rate of improvement from additional data is identical between ConvNeXt and Swin — it only shows the final point. A full scaling law analysis would compare the data scaling exponent.

Claim 4: "ConvNeXt maintains the simplicity and efficiency of standard ConvNets"

This claim has two components — simplicity and efficiency — that need separate evaluation.

Simplicity: The paper makes a compelling case. Figure 4 shows the ConvNeXt block is structurally simpler than Swin Transformer's block: one residual connection vs. two, standard convolution operations vs. window partitioning and cyclic shifting, no relative position bias. The fully-convolutional nature means no patch embedding interpolation when changing input resolution. The architecture specification (Table 9) shows ConvNeXt-T uses only standard layers (Conv2d, LayerNorm, GELU) available in any deep learning framework.

Efficiency: The evidence is strong for GPU inference throughput. Table 12 shows substantial A100 advantages. Training memory is lower (Cascade Mask R-CNN: 17.4GB for ConvNeXt-B vs. 18.5GB for Swin-B). However, the efficiency analysis has gaps:

• Training time is not reported. The paper does not compare wall-clock training time between ConvNeXt and Swin Transformer. Depthwise convolutions have lower arithmetic intensity than dense convolutions or optimized attention, which can make them slower to train even with lower FLOPs. The throughput measurements are inference-only.

• The paper acknowledges but does not resolve the depthwise convolution efficiency problem. Section 4 notes: "Under similar FLOPs, models with depthwise convolutions are known to be slower and consume more memory than ConvNets with only dense convolutions." The paper addresses this by showing that ConvNeXt achieves comparable or better throughput than Swin, but it does not claim that ConvNeXt is faster than a hypothetical dense ConvNet of similar FLOPs — only that it is faster than Swin.

• No mobile or edge deployment benchmarks. The efficiency claims are demonstrated on V100 and A100 server GPUs. The paper explicitly cites MobileNet and EfficientNet as prior work on efficient ConvNets, but ConvNeXt models (starting at 29M parameters) are not in the mobile regime. The efficiency advantages may not translate to mobile processors where depthwise convolution optimizations differ.

Missing Experiments and Baselines

Several experiments would have strengthened the paper's claims:

A ConvNeXt trained with the original ResNet recipe as a negative control. The paper shows that enhanced training improves ResNet-50 by +2.7%. A natural follow-up would be: does ConvNeXt still outperform ResNet-50 when both use the old recipe? If ConvNeXt's advantage depends on the modern recipe, that would qualify the claim that the architecture per se is superior. The paper does not run this experiment.

Direct comparison between depthwise convolution and self-attention within the ConvNeXt block. The modernization roadmap replaces 3×3 dense convolution with 3×3 depthwise convolution, then increases the kernel to 7×7. At no point does the paper insert a self-attention module into the ConvNeXt block to directly measure the accuracy difference between convolution and attention with all other factors held constant. Such an experiment would directly quantify the contribution of the spatial mixing operation independent of architecture. The isotropic experiments (Table 2) compare whole ConvNeXt and ViT architectures, but these differ in more than just the spatial mixing operation (ConvNeXt uses depthwise conv + two 1×1 convs; ViT uses MSA + MLP with different expansion patterns).

Swin Transformer trained with the ConvNeXt training recipe from scratch. The paper uses published Swin results (which used a similar recipe) and re-sweeps hyperparameters for 22K fine-tuning. But it does not re-train Swin from scratch with exactly the ConvNeXt training setup to eliminate any remaining recipe differences.

Ablation of the LayerNorm placement for downsampling stability. The paper reports that separate downsampling layers cause training divergence, and adding LayerNorm at resolution transitions fixes it. But it does not ablate which LayerNorm placements are necessary — is it all of them (before each downsampling layer, after stem, after GAP), or would a subset suffice? This limits the actionable insight from the finding.

Sensitivity analysis of the 7×7 kernel choice. The kernel size sweep shows saturation at 7×7 for ResNet-50 regime and 5×5 for ResNet-200. The paper adopts 7×7 but does not explore whether this interacts with other design choices (e.g., would 5×5 work better with deeper models? Would larger kernels help with higher-resolution inputs?). The claim that 7×7 is optimal is demonstrated only at 224² resolution on classification.

Scaling to smaller models. The paper's smallest model is ConvNeXt-T at 29M parameters and 4.5G FLOPs. This is comparable to ResNet-50 but substantially larger than MobileNet-scale models (typically < 10M parameters, < 1G FLOPs). The design principles (inverted bottleneck, depthwise convolutions) originate from MobileNet, so it is natural to ask whether the modernization approach works at the efficient regime. The paper does not explore this, limiting its applicability to the "backbone" scale rather than the "mobile" scale.

Evaluation on video or 3D tasks. The paper explicitly scopes its evaluation to image classification, object detection, and semantic segmentation — all 2D image tasks. Computer vision includes video understanding, 3D reconstruction, point cloud processing, and multi-modal tasks. The paper's claims about ConvNeXt as a "generic vision backbone" are demonstrated only on a subset of vision tasks. The paper acknowledges this limitation in Appendix F.

Robustness evaluation is limited to ImageNet variants. The robustness results in Table 8 show ConvNeXt performing well, but robustness is evaluated only for classification models on standard distribution-shift benchmarks. Detection robustness (e.g., COCO-C) or segmentation robustness are not evaluated. The claim that ConvNeXt is robust is therefore scoped to classification.

What the Experiments Demonstrate vs. What They Do Not

What is well-demonstrated:

• A carefully modernized ResNet, using only convolutional operations, can match or slightly exceed Swin Transformer on ImageNet classification, COCO detection/segmentation, and ADE20K segmentation at matched FLOPs and model sizes.
• The Transformer training recipe accounts for approximately 2.7 percentage points of the gap between traditional ResNet-50 and modern Swin-T — roughly half of the total gap.
• Individual architectural choices (stage ratio, inverted bottleneck, large kernels, activation count, normalization scheme) each contribute small but cumulative improvements, with no single change dominating.
• ConvNeXt achieves higher inference throughput than Swin Transformer on A100 GPUs due to better hardware support for standard convolutions.
• The ConvNeXt block design works in both hierarchical and isotropic architectures, matching ViT in the latter case.

What is not demonstrated:

• That convolution is superior to attention for tasks requiring long-range or global context (since all experiments use local operations: 7×7 depthwise conv replacing 7×7 window attention).
• That ConvNeXt scales better than, or even as well as, Transformers in a rigorous scaling law sense — only that it achieves comparable accuracy at a few matched points on the scaling curve.
• That ConvNeXt is more efficient than dense ConvNets — the efficiency claims are relative to Swin Transformer, not to architectures like ResNeXt or EfficientNet that use dense convolutions.
• That the modernization approach transfers to other ConvNet architectures (ResNeXt, EfficientNet, RegNet) — only ResNet is modernized.
• That the design choices are optimal — the paper explicitly acknowledges that the stage compute ratio, kernel size, and other parameters are adopted from Swin Transformer for comparison purposes, not because they were optimized through architecture search.

6. Limitations and Trade-offs

1. The Modernization Is a Single-Trajectory Case Study on ResNet, Not a General Methodology Validated Across ConvNet Families

The assumption or constraint. The paper's core contribution is a modernization trajectory — a specific sequence of 12 design changes applied to a ResNet-50/200 baseline, guided by Swin Transformer as the reference architecture. The paper implicitly assumes that this trajectory represents generalizable design principles that would benefit other ConvNet architectures. However, the authors are transparent that the study is narrow in scope:

"All models are trained and evaluated on ImageNet-1K" (Section 2), and the starting point is exclusively ResNet architectures.

The paper does not test whether starting from a different ConvNet family (ResNeXt, EfficientNet, RegNet, DenseNet) and applying the same modernization principles would yield equivalent gains. Each architectural family has different structural priors — ResNeXt already uses grouped convolutions, EfficientNet already uses inverted bottlenecks and depthwise convolutions, RegNet already optimizes stage compute ratios — so the marginal benefit of the specific modernization steps would likely differ, and the optimal trajectory might be entirely different.

The consequence. A practitioner starting from a non-ResNet ConvNet architecture cannot directly apply the roadmap. The paper does not provide guidance on which modernization steps are universally beneficial versus which are specific to correcting ResNet's particular design limitations. For instance, the inverted bottleneck change (+0.1% at ResNet-50 scale, but +0.79% at ResNet-200 scale) might be near-zero for a MobileNetV2 that already uses inverted bottlenecks. The stage ratio change from ResNet's (3,4,6,3) to Swin's (3,3,9,3) is ResNet-specific — EfficientNet's stage ratios are already optimized through architecture search. The paper's methodology is more "here is how we made ResNet competitive" rather than "here are the universal design principles for modern ConvNets," but the framing and title ("A ConvNet for the 2020s") imply the latter.

What evidence exists in the paper. The paper only tests the modernization trajectory on ResNet-50 and ResNet-200. Table 10 and Table 11 are the exhaustive evidence. The isotropic ConvNeXt experiments (Table 2) test whether the ConvNeXt block generalizes to ViT-style architectures, but this is a forward generalization test of the final product, not a test of whether the modernization methodology generalizes to other ConvNet starting points. The paper does not modernize an EfficientNet, a RegNet, or a ResNeXt — it only compares against their published results in Table 1. This is acknowledged implicitly when the paper notes that "many design choices have all been examined separately over the last decade, but not collectively" — the paper demonstrates one specific collective configuration, not a general recipe for collecting.

Mitigation status. Not addressed. The paper does not claim to provide a general modernization methodology applicable to arbitrary ConvNet architectures; it presents a specific modernization of ResNet. However, the title and abstract framing ("A ConvNet for the 2020s," "test the limits of what a pure ConvNet can achieve") imply broader conclusions than the experimental design directly supports. A more accurate title might be "A Modernized ResNet for the 2020s." Future work would need to replicate the modernization approach with different starting architectures to establish generalizability.

---

2. Single Benchmark Suite and Single Model Family for All Core Claims

The assumption or constraint. Every modernization experiment, every architectural ablation, and all primary performance comparisons use ImageNet-1K classification with a ResNet starting point (Section 2, Figure 2). The paper then evaluates the final ConvNeXt models on COCO and ADE20K (Sections 3–4), but these downstream evaluations test the final product, not the modernization decisions. The paper implicitly assumes that design choices validated on ImageNet-1K classification at 224² resolution transfer to detection, segmentation, and other tasks. The authors acknowledge this scope limitation explicitly in Appendix F:

"While our goal is to offer a broad range of evaluation tasks, we recognize computer vision applications are even more diverse. ConvNeXt may be more suited for certain tasks, while Transformers may be more flexible for others."

The consequence. There is no guarantee that the specific design choices in the modernization roadmap — particularly the 7×7 kernel size, the (3,3,9,3) stage ratio, the GELU activation, and the single-activation-per-block design — are optimal for tasks beyond ImageNet classification. Detection and segmentation involve higher-resolution inputs, multi-scale feature usage (FPN), and different training dynamics. The paper's own evidence shows that the relative advantage of ConvNeXt over Swin Transformer varies by task: on COCO detection with Cascade Mask R-CNN (Table 3), ConvNeXt-T matches Swin-T exactly (50.4 AP), while on ImageNet-1K classification ConvNeXt-T leads by +0.8 points (82.1% vs. 81.3%). This suggests that the modernization choices optimized for classification do not necessarily translate proportionally to downstream tasks.

More subtly, tasks requiring global context or very long-range dependencies — video understanding, 3D scene reconstruction, medical imaging with large anatomical structures — might benefit from larger receptive fields than the 7×7 kernel provides. The paper's kernel size sweep (Section 2.5) tested up to 11×11 and found saturation at 7×7 for ImageNet classification at 224² resolution. This finding may not hold for tasks where discriminative features span larger spatial extents. The paper does not evaluate any task requiring global reasoning or cross-modal attention, which the authors note in Appendix F may favor Transformers.

What evidence exists in the paper. All modernization decisions (stage ratio, kernel size, activation count, normalization scheme) are validated exclusively on ImageNet-1K top-1 accuracy (Tables 10 and 11). The COCO (Table 3) and ADE20K (Table 4) evaluations use the final ConvNeXt architecture as-is; they do not test alternative kernel sizes, stage ratios, or activation configurations on these tasks. The robustness evaluation (Table 8) provides some evidence of generalization to distribution shift, but only for classification. The paper does not measure detection robustness or segmentation robustness. The A100 throughput advantage (Table 12) is measured on classification inference at 224² and 384² — detection throughput (FPS) is reported in Table 3 for Cascade Mask R-CNN, but the advantage over Swin is smaller in that setting (ConvNeXt-T: 13.5 FPS vs. Swin-T: 12.2 FPS, roughly +11% compared to the +47% at classification).

Mitigation status. Partially addressed through the COCO and ADE20K evaluations, which show that the final architecture transfers to detection and segmentation without modification. However, the paper does not test whether alternative modernization trajectories might have produced better downstream performance even at the cost of slightly lower ImageNet accuracy. This is a fundamental tradeoff in backbone design — optimizing for classification may not optimize for downstream tasks — that the paper's sequential methodology does not explore. The authors acknowledge the task diversity limitation in Appendix F but treat it as scope rather than a weakness to be addressed.

---

3. The Modernization Roadmap Confounds FLOPs Control with Design Changes, and Several Steps Change Both Accuracy and Computational Cost Simultaneously

The assumption or constraint. The paper states that "FLOPs are roughly controlled over the course of the exploration" (Section 2), but Table 10 reveals substantial variation: the depthwise convolution step drops FLOPs from 4.42G to 2.35G (a 47% reduction), and the subsequent width expansion increases them to 5.27G (a 2.2× increase over the depthwise minimum). The inverted bottleneck step then reduces FLOPs to 4.64G, and moving the depthwise conv drops them further to 4.07G. The final ConvNeXt-T at 4.49G FLOPs is roughly 10% higher than the enhanced-recipe ResNet-50 baseline at 4.09G.

The paper treats these FLOPs fluctuations as acceptable intermediate states and does not disentangle how much of each accuracy change is due to the architectural modification versus the change in computational capacity. The implicit assumption is that if FLOPs end up approximately matched at the end, the intermediate capacity changes do not distort the attribution of gains to design choices.

The consequence. The reported improvements for several steps may confound genuine architectural benefit with increased capacity. Most critically, the largest single accuracy improvement in the roadmap is the width expansion after depthwise convolution: 78.3% → 80.5% (+2.2 percentage points). But this step also increases FLOPs by 2.2× (2.35G → 5.27G). How much of that +2.2% gain is due to the ResNeXt design principle (depthwise + wider), and how much is simply having more parameters and FLOPs? The paper does not control for this by, for example, testing a wider ResNet-50 without depthwise convolutions to see whether width alone explains the gain.

Similarly, the kernel size sweep (Section 2.5) shows accuracy increasing from 79.9% at 3×3 to 80.6% at 7×7, with FLOPs "staying roughly the same" (4.07G → 4.15G — a 2% increase). While small, the 80.6% at 7×7 is achieved at slightly higher FLOPs than the 79.9% at 3×3. The paper does not test whether a slightly wider 3×3 model (matching FLOPs exactly) would close the gap, making the kernel size benefit partially attributable to the small capacity increase.

The separate downsampling layers step adds LayerNorm at resolution transitions, which slightly increases FLOPs (from ~4.46G with BN→LN to 4.49G for ConvNeXt-T). The +0.5% gain (81.5% → 82.0%) might partly reflect this added capacity rather than purely the architectural benefit of separate downsampling.

What evidence exists in the paper. Table 10 provides FLOPs and accuracy for every step, making the FLOPs variation transparent. The paper acknowledges the issue in Section 2: "at intermediate steps the FLOPs might be higher or lower than the reference models." However, the paper does not provide FLOPs-controlled ablations for the major capacity-changing steps. For example, the paper does not test: (1) ResNet-50 with width expanded to match the 5.27G FLOPs of the post-width-expansion model but without depthwise convolutions, or (2) depthwise convolution + narrower width to match the original 4.09G FLOPs, to isolate the architectural benefit from the capacity benefit. The kernel size ablation (Table 10) is the closest to a properly controlled experiment, as FLOPs vary by only ~5% across kernel sizes.

Mitigation status. Not addressed. The paper treats FLOPs matching at the endpoints (ResNet-50 ~4.1G → ConvNeXt-T ~4.5G, ResNet-200 ~15.0G → ConvNeXt-B ~15.4G) as sufficient, and does not analyze or discuss the potential confounding of intermediate capacity changes with architectural improvements. A reader cannot determine whether, for instance, the inverted bottleneck is genuinely useful (+0.1% at reduced FLOPs at ResNet-50 scale), or whether the apparent benefit of the ResNeXt-ification would survive a strict FLOPs-matched comparison. This is a methodological limitation common to incremental architecture papers, but it weakens the precision of the claimed per-step contributions.

---

4. The 7×7 Kernel Size Optimum Is Validated Only at a Single Resolution and May Not Generalize to the Variable-Resolution Inputs Typical in Detection and Segmentation

The assumption or constraint. The kernel size sweep in Section 2.5 tests kernel sizes 3×3, 5×5, 7×7, 9×9, and 11×11 exclusively on ImageNet-1K at 224² resolution. The paper finds that accuracy saturates at 7×7 and adopts this as the default kernel size for all ConvNeXt variants across all tasks and resolutions. The implicit assumption is that the optimal kernel size is resolution-independent — that a 7×7 kernel provides the right receptive field regardless of whether the input is 224², 384², 512², or 800×1280. The ResNet-200 experiment (Table 11) already shows a discrepancy: at that scale, accuracy saturates at 5×5 rather than 7×7, suggesting the kernel size optimum is not universal even for classification.

The consequence. In object detection and semantic segmentation, ConvNeXt operates on feature maps at multiple resolutions (e.g., P3–P7 in FPN, corresponding to strides 8–128 relative to the input). At higher-resolution feature maps (P3: ~100×128 for a 800×1280 COCO image), a 7×7 kernel covers a smaller fraction of the spatial extent than at the 56×56 feature map of ImageNet classification. The effective receptive field may be insufficient for detecting large objects or segmenting large regions. Conversely, at very low-resolution feature maps (P7: ~7×8), a 7×7 kernel covers nearly the entire spatial extent, making it equivalent to a fully-connected operation — potentially losing the benefits of local processing.

The paper's own evidence hints at this issue: the FLOPs/throughput advantage of ConvNeXt over Swin Transformer varies with resolution. At 224², ConvNeXt-B is +2.0% faster than Swin-B (292.1 vs. 286.6 img/s on V100). At 384², it is +12.5% faster (95.7 vs. 85.1 img/s). This suggests the computational dynamics of depthwise convolution vs. window attention change with resolution, and the kernel size choice interacts with this. A different kernel size might be optimal at typical detection resolutions (800×1280). Additionally, the paper uses a crop ratio of 1.0 (no cropping) when evaluating at 384² (Appendix A.2), which changes the distribution of object scales relative to the kernel — the 7×7 choice was validated with 0.875 cropping at 224².

What evidence exists in the paper. The kernel size sweep is conducted exclusively on ImageNet-1K at 224² resolution (Tables 10 and 11). The COCO and ADE20K evaluations (Tables 3 and 4) use the 7×7 kernel without testing alternatives. The paper does not provide detection or segmentation results with different kernel sizes, nor does it analyze how the effective receptive field at different FPN levels relates to object sizes in COCO. The ResNet-200 experiment's different saturation point (5×5 vs. 7×7) is noted but not explored — the paper simply adopts 7×7 for consistency. The paper also does not test whether the kernel size should vary by stage (larger kernels in earlier, higher-resolution stages; smaller in later stages), which is a standard design choice in many ConvNets (e.g., VGG used 3×3 throughout; AlexNet used 11×11 in the first layer and 5×5 later). Swin Transformer uses a fixed 7×7 window across all stages, and the paper matches this, but whether fixed kernel size is optimal for ConvNets remains untested.

Mitigation status. Not addressed. The paper treats the 7×7 kernel size as a fixed design decision derived from ImageNet-1K 224² classification and applies it uniformly to all tasks and resolutions. No experiments test kernel size variation across stages or resolutions, and the ResNet-200 saturation discrepancy is acknowledged but not investigated. A practitioner deploying ConvNeXt for high-resolution detection or segmentation tasks would benefit from knowing whether the 7×7 kernel remains optimal or whether a resolution-dependent or stage-dependent kernel size selection would improve performance.

---

5. The Downstream Task Efficiency Claims Are Narrowly Scoped to A100 GPU Inference Throughput, Leaving Training Cost and Deployment on Other Hardware Uncharacterized

The assumption or constraint. The paper's efficiency claims — "ConvNeXt maintains the efficiency of standard ConvNets" — are supported primarily by inference throughput measurements on V100 (Table 1) and A100 GPUs (Table 12). Training memory is measured for Cascade Mask R-CNN (Section 4, "Remarks on model efficiency") and for isotropic classification models (Table 2). The paper does not measure training wall-clock time, does not benchmark on non-GPU hardware (TPUs, mobile processors, edge accelerators), and does not characterize the memory bandwidth requirements or arithmetic intensity of ConvNeXt operations relative to alternatives. The authors partially acknowledge the hardware-specificity in Appendix A and E but treat the V100/A100 results as sufficient evidence for the efficiency claims.

The consequence. The headline throughput advantage of up to +49% over Swin Transformer (Table 12) is valid only on A100 GPUs with TF32 precision and channel-last memory layout. On V100 GPUs (Table 1), the advantage is dramatically smaller: ConvNeXt-T achieves 774.7 img/s vs. Swin-T's 757.9 img/s — only +2.2%. A practitioner using V100 or older GPU hardware would see essentially no throughput benefit. The paper does not benchmark on TPUs, where the relative performance of depthwise convolutions vs. window attention may differ substantially due to different hardware architectures (systolic arrays optimized for dense matrix multiplies vs. convolution engines).

Training efficiency is essentially uncharacterized. The paper notes that "under similar FLOPs, models with depthwise convolutions are known to be slower and consume more memory than ConvNets with only dense convolutions" (Section 4), acknowledging a known issue with depthwise convolutions' low arithmetic intensity. But the paper does not report ConvNeXt training time compared to ResNet, ResNeXt, EfficientNet, or Swin Transformer. Depthwise convolutions have lower compute-to-memory ratios than dense convolutions, meaning they can be memory-bandwidth-bound on many hardware platforms — the inference throughput advantage on A100 does not guarantee training speed advantages. Training ConvNeXt-B from scratch on ImageNet-1K for 300 epochs may take longer than training Swin-B despite equivalent FLOPs, but the paper provides no data to assess this.

For mobile or edge deployment, the efficiency picture is entirely unknown. ConvNeXt-T has 28.6M parameters — far from the mobile regime (MobileNetV3-Large has 5.4M). The paper's smallest model is larger than many deployed vision models, and the specific combination of 7×7 depthwise convolutions, 4× expansion ratios, and LayerNorm may have different efficiency characteristics on mobile GPUs, NPUs, or FPGAs than on datacenter GPUs. The paper's comparison against EfficientNet (Table 1), which was explicitly designed for mobile efficiency, shows that ConvNeXt-B (89M params, 15.4G FLOPs) achieves 83.8% vs. EfficientNet-B7 (66M params, 37.0G FLOPs) at 84.3% — EfficientNet achieves higher accuracy but with more FLOPs, while ConvNeXt has more parameters. This tradeoff is not analyzed in terms of deployment implications.

What evidence exists in the paper. Table 1 (V100 throughput) and Table 12 (A100 throughput) are the primary efficiency evidence. Training memory is mentioned for two specific configurations: Cascade Mask R-CNN with ConvNeXt-B (17.4GB, batch size 2) and isotropic ConvNeXt-B (7.7GB, batch size 32). These are point measurements, not systematic comparisons across model scales or hardware platforms. The paper does not report: training time per epoch for any model, MACs (multiply-accumulate operations) vs. FLOPs breakdown, memory bandwidth utilization, or power consumption. The A100 benchmarks (Table 12) use PyTorch 1.10 with channel-last memory layout, which the paper notes is a specific optimization — results with different framework versions or memory layouts may differ.

Mitigation status. Partially addressed through the V100 and A100 benchmarks, the training memory measurements, and the FPS reporting for detection models (Table 3). The authors acknowledge in Appendix E that the A100 throughput advantage is a "preliminary study" and frame it as "promising signals" rather than definitive efficiency claims. However, the main text (Section 4) and abstract make stronger claims: "ConvNeXt maintains the efficiency of standard ConvNets" — which implies broader efficiency than the hardware-specific inference throughput measurements actually support. The paper does not suggest specific future work on characterizing ConvNeXt's training efficiency or performance on non-GPU hardware. A practitioner would need to benchmark ConvNeXt on their target hardware and task before relying on the efficiency claims.

---

6. The Inverted Bottleneck and Single-Activation Design Trade Model Expressivity for Architectural Simplicity, Potentially Limiting Representational Capacity per Block

The assumption or constraint. The final ConvNeXt block (Figure 4) has significantly fewer non-linearities than a standard ResNet block: one GELU activation (between the two 1×1 convolutions in the expansion path) vs. three ReLU activations in a standard ResNet bottleneck (after each of the three convolutions). The paper demonstrates empirically that reducing activations improves accuracy (+0.7% for ResNet-50, Table 10), which is interpreted as evidence that fewer activations are beneficial. However, the paper assumes this accuracy improvement represents a genuine architectural advantage rather than an interaction with other design choices (depthwise convolution, inverted bottleneck, LayerNorm) that might mask a capacity-expressivity tradeoff.

The consequence. A block with a single activation function has a strictly limited representational capacity compared to a block with multiple non-linearities, assuming the same number of parameters. The depthwise convolution — which performs purely linear spatial filtering with no channel mixing — has no activation function after it in ConvNeXt, meaning the spatial mixing operation is entirely linear until the subsequent 1×1 convolution and GELU. This linear spatial filtering may be sufficient for the tasks evaluated (ImageNet classification, COCO detection, ADE20K segmentation), where features can be represented as linear combinations of local patterns. However, for tasks requiring more complex, non-linear spatial transformations — e.g., precise boundary detection, texture synthesis, or tasks involving intricate spatial relationships — the absence of non-linearity after spatial mixing could be limiting.

The paper's evidence that fewer activations help is itself ambiguous. In the ResNet-50 regime (Table 10), the step "fewer activations" removes two of three activation functions from a block that still uses BatchNorm and ReLU at that stage of the modernization. The improvement from 80.6% to 81.3% (+0.7%) occurs in a partially-modernized architecture, not in the final ConvNeXt. It is possible that the benefit of fewer activations is specific to the interaction with BatchNorm (which already introduces non-linearity through normalization and scaling) or to the particular placement of activations in the ResNet block. The paper does not ablate whether the activation reduction would still be beneficial if applied to the original ResNet without any other modernization steps, or whether different activation placements (e.g., activation after depthwise conv but not after 1×1 projections) might work better.

More fundamentally, the paper never tests whether adding an activation after the depthwise convolution in the final ConvNeXt block would help or hurt. The Transformer design principle (one activation in the MLP, none in attention) is adopted wholesale, but attention is inherently a non-linear operation (the softmax over attention scores is highly non-linear), while depthwise convolution is linear. A Transformer block without an activation after MSA still benefits from the softmax non-linearity in the attention computation; a ConvNeXt block without an activation after the depthwise convolution has no analogous non-linearity in its spatial mixing. The paper does not discuss this fundamental difference or test whether adding a non-linearity after the depthwise convolution recovers expressivity that the Transformer gets from softmax.

What evidence exists in the paper. The activation reduction is evaluated at a single point in the modernization trajectory (Table 10, step "fewer activations") and improves accuracy. The GELU vs. ReLU substitution shows zero difference (80.6% → 80.6%), suggesting that the type of activation is unimportant, but the number and placement are. However, the paper does not test alternative activation placements in the final ConvNeXt architecture — for example, adding a GELU after the depthwise convolution, or using two activations (one after depthwise conv, one between 1×1 convs), or using no activation at all. The isotropic ConvNeXt experiments (Table 2) use the same block design and match ViT performance, but ViT's MSA contains a softmax non-linearity that the depthwise convolution lacks, so the comparison does not isolate the effect of activation count.

Mitigation status. Not addressed. The paper treats the single-activation design as a settled choice derived from Transformer convention, validated by the +0.7% improvement in the modernization experiment. The authors do not discuss the potential expressivity tradeoff, do not test activation placements in the final architecture, and do not analyze whether ConvNeXt's depthwise convolution being linear while MSA's attention being non-linear (via softmax) represents a meaningful architectural difference rather than an implementation detail. A practitioner concerned about representational capacity for tasks with complex spatial reasoning might want to test whether adding a non-linearity after spatial mixing helps — the paper provides no guidance on this question.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new architectural primitive, a new training algorithm, or a new theoretical framework. It is, at its core, a systematic refutation through reconstruction: it demonstrates that the performance gap between modern vision Transformers and traditional ConvNets is not evidence for the intrinsic superiority of self-attention, but rather a reflection of accumulated design and training choices that can be retrofitted to ConvNets without adopting attention at all. The paper's methodology — incremental, controlled modernization with FLOPs tracking — transforms what could have been dismissed as "yet another ConvNet" into a diagnostic instrument that isolates which differences between architectures actually matter.

The conceptual shift this work produces is in reframing the ConvNet-vs-Transformer comparison from an architectural competition to an optimization and design-space problem. Before this work, the dominant question in the field was: "Are Transformers better than ConvNets for vision?" The paper demonstrates that this question is ill-posed because it conflates at least four independent factors: training recipe (AdamW, augmentation, epochs), macro-architecture (stage ratios, stem design), micro-architecture (activation count, normalization type, kernel size), and the spatial mixing operation itself (self-attention vs. convolution). By disentangling these factors and showing that the first three, implemented convolutionally, close the gap entirely, the paper shifts the question to: "What makes any spatial mixing architecture work well?" The answer, per this work, lies primarily in the training procedure, the block-level organization (inverted bottleneck, spatial-then-channel mixing, sparse activations), and the macro-structure (multi-stage hierarchy) — not in whether the spatial mixing uses content-dependent attention weights or learned convolutional filters.

This is not a paradigm shift on the scale of ResNets or Transformers — it does not introduce a new computational primitive or enable previously impossible capabilities. But it is a significant reframing that changes how the field should evaluate architectural proposals. The paper establishes a methodological standard: any claim that a new operation (self-attention, MLP-mixing, Fourier transform, dynamic convolution) is "better than convolution" must now control for the training recipe, the block-level design pattern, and the macro-architecture — because the paper shows that these factors alone account for the entire observed gap between a 2016 ResNet and a 2021 Swin Transformer. A new spatial mixing operation that shows gains over a ResNet-50 trained with the original 90-epoch SGD recipe has not demonstrated superiority over convolution; it has only demonstrated superiority over an outdated training and design configuration.

The paper reconciles a specific contradiction in the recent literature. Several concurrent or prior works had observed that local Transformer attention is mathematically related to depthwise convolution (Han et al., 2021), that pure MLP architectures can match Transformers (MLP-Mixer), and that convolutions can be hybridized with attention to good effect (CoAtNet, CvT, ConViT). These works collectively suggested that self-attention per se might not be the essential ingredient in Transformer success, but none provided a complete, controlled demonstration that pure convolution, properly modernized, reaches equivalent performance across classification, detection, and segmentation at matched scale. The ConvNeXt paper closes this gap by providing the full evidence chain: training recipe → macro design → ResNeXt-ification → inverted bottleneck → large kernels → micro design, with every step quantified.

The research directions this work makes more attractive include systematic design space exploration for ConvNets (the paper shows the design space is far from exhausted), hardware-aware architecture optimization (the A100 throughput advantage demonstrates that conv ops enjoy better hardware support), and self-supervised learning for modernized ConvNets (the paper only tests supervised and 22K pre-training; combining ConvNeXt with MAE-style masked image modeling is a natural extension). The research directions it makes less attractive include incremental variants of window-based attention (if depthwise convolution achieves the same effect with better throughput and simpler implementation, the burden of proof for a new attention variant increases) and claims that Transformer-specific operations like relative position bias or cyclic shifting are essential for state-of-the-art vision models.

Perhaps most significantly, the paper implicitly argues that the decade of ConvNet research (2012–2021) produced the right building blocks but assembled them suboptimally for the modern training regime. Depthwise convolutions existed since MobileNet (2017). Inverted bottlenecks existed since MobileNetV2 (2018). Large kernels existed since AlexNet (2012). GELU existed since 2016. LayerNorm existed since 2016. The contribution is not inventing these components but showing that their specific arrangement — depthwise conv at the block front, 4× expansion, one activation, one LayerNorm, 7×7 kernel — in a Swin-style macro-architecture with a ViT-style training recipe achieves what was thought to require attention. This suggests that architecture search over configurations of known primitives may be more fruitful than inventing new primitives — a lesson with implications beyond this paper.

Follow-Up Research This Work Enables

1. Self-supervised pre-training of ConvNeXt with masked image modeling (MAE or SimMIM). The paper evaluates ConvNeXt only with supervised ImageNet-1K training and supervised ImageNet-22K pre-training. The isotropic ConvNeXt experiments (Table 2) compare against ViT-L trained with MAE, but ConvNeXt itself is never pre-trained with self-supervision. This is a conspicuous gap because MAE and SimMIM have demonstrated that Transformers benefit enormously from masked image modeling, and a widely held view is that Transformers' sequence-to-sequence structure makes them particularly suited to this task. Training ConvNeXt-B with MAE-style masked image modeling and comparing against ViT-B/MAE at equivalent pre-training compute would directly test whether the reconstruction-as-pretext task is inherently better suited to self-attention or whether convolution can be equally effective. A negative result (ConvNeXt underperforms ViT under MAE) would reveal a genuine advantage of attention for dense prediction tasks; a positive result would further generalize the paper's claims beyond supervised learning. The experiment is straightforward given that ConvNeXt is fully convolutional and can process arbitrary input sizes — the masking and reconstruction pipeline would need to be adapted for the hierarchical feature maps, similar to how SimMIM was adapted for Swin.

2. Neural architecture search over the ConvNeXt design space to test whether the Swin-derived hyperparameters are locally or globally optimal. The paper explicitly acknowledges that the stage compute ratio (3, 3, 9, 3) is adopted from Swin-T for comparison purposes and that "a more optimal design is likely to exist" (Section 2.2). Similarly, the kernel size of 7×7 is adopted from a sweep that showed saturation at that value for ImageNet-1K 224² classification, but the sweep was coarse (only powers of 2 plus 3 and 11) and conducted at one resolution. A systematic architecture search over the ConvNeXt design space — varying stage ratios, kernel sizes per stage (not necessarily uniform), expansion ratios, activation placement, normalization placement, and block counts — could reveal whether the specific configuration the paper landed on is near-optimal or whether there exists a significantly better configuration that the incremental modernization missed. The search space is large but tractable since each design choice is a known ConvNet primitive. A strong follow-up would use weight-sharing or zero-cost proxies to efficiently explore the space and report whether the Swin-derived configuration sits on or off the Pareto frontier.

3. Scaling law analysis: measure the data-compute-parameters scaling exponent of ConvNeXt vs. Swin Transformer. The paper demonstrates that ConvNeXt scales competitively by showing matched accuracy at five points on the scaling curve (T/S/B/L/XL), but it does not fit scaling law parameters. A rigorous follow-up would train ConvNeXt and Swin Transformer at multiple FLOPs budgets spanning at least an order of magnitude (e.g., 1G to 100G FLOPs), measure the power-law exponent relating compute to accuracy, and test whether the two architectures have statistically distinguishable scaling exponents. This would replace the qualitative claim "ConvNeXt scales just as well" with a quantitative claim like "ConvNeXt has a scaling exponent of $E = 0.052$ vs. Swin's $E = 0.048$" (or vice versa). The paper's own evidence hints at possible differences in scaling behavior: the throughput advantage of ConvNeXt over Swin grows with resolution (V100: ~0–2% advantage at 224²; A100: 34–49% advantage; the advantage also grows from 224² to 384²), and the ResNet-200 regime shows a different kernel size saturation point than ResNet-50 (5×5 vs. 7×7), suggesting that optimal design choices may vary with scale. A scaling law analysis would quantify whether these differences compound or cancel as models grow.

4. Investigation of non-linear spatial mixing: insert a GELU or Squeeze-and-Excitation module after the depthwise convolution in ConvNeXt. The paper removes all activations after the spatial mixing operation, following the Transformer convention of having no activation between attention and the subsequent linear projection. However, as noted in the Limitations section, Transformer attention includes a softmax non-linearity in the attention computation, making the spatial mixing inherently non-linear, while depthwise convolution is purely linear. A controlled experiment would add a lightweight non-linearity after the depthwise convolution — either a simple GELU activation, or a Squeeze-and-Excitation module (a channel-wise gating mechanism that is itself non-linear) — and measure whether this recovers any representational capacity that the linear spatial mixing might lack. The experiment should test multiple tasks: classification (where the paper shows no benefit from GELU after depthwise conv, since "fewer activations" helped), detection (where precise spatial localization might benefit from non-linear spatial processing), and segmentation (where boundary delineation might require non-linear feature combinations). A positive result would refine the design principle from "fewer activations is always better" to "fewer activations helps classification but non-linear spatial mixing helps dense prediction." A negative result would strengthen the paper's claim that linear spatial mixing is genuinely sufficient.

5. Stress-test ConvNeXt on tasks requiring global context or cross-modal attention to map the boundary where self-attention becomes necessary. The paper is careful to acknowledge that "Transformers may be more flexible when used for tasks requiring discretized, sparse, or structured outputs" and that "a cross-attention module may be preferable for modeling feature interactions across many modalities" (Appendix F). However, the paper does not test any such tasks, leaving the boundary of ConvNeXt's applicability uncharted. A strong follow-up would evaluate ConvNeXt on tasks where global attention is conventionally thought to be necessary: video action recognition with long temporal horizons (e.g., Something-Something V2, which requires fine-grained temporal reasoning), 3D point cloud processing where points have no regular grid structure, visual question answering or image captioning requiring cross-modal attention between vision and language, and in-context visual reasoning tasks where the model must attend to arbitrary regions specified by a text prompt. In each case, the comparison should be against a Transformer baseline with the same macro-architecture and training recipe but with window attention replaced by global attention or cross-attention where the task requires it. The goal is not to show that ConvNeXt is universally superior, but to precisely map which task characteristics genuinely benefit from the flexibility of attention — producing a decision tree for practitioners rather than the current blanket statements.

6. Re-evaluate whether the 7×7 kernel size optimum is resolution- and task-dependent, and test stage-varying kernel sizes. The paper's kernel size sweep shows saturation at 7×7 for ImageNet-1K 224² classification, with the ResNet-200 regime saturating at 5×5. Neither experiment tests whether the optimal kernel size depends on the input resolution (which varies dramatically between ImageNet classification at 224² and COCO detection at 800×1280) or on the stage (early high-resolution stages might benefit from larger kernels to capture context; late low-resolution stages might benefit from smaller kernels to avoid over-smoothing). A concrete experiment would train ConvNeXt variants with kernel sizes {3, 5, 7, 9} at COCO detection with Cascade Mask R-CNN, both with uniform kernel sizes across all stages and with stage-varying configurations (e.g., {7, 7, 5, 3} from early to late stages). The results would reveal whether the 7×7 choice — currently justified only by ImageNet-1K 224² — is indeed optimal for the downstream tasks that motivate the hierarchical design, or whether task-specific kernel size tuning is a cheap source of additional improvement that the paper left on the table.

Practical Applications and Downstream Use Cases

Cost-efficient deployment of vision models on GPU clusters where inference throughput dominates total cost of ownership. The paper demonstrates that ConvNeXt-B achieves 95.7 images/second on V100 at 384² resolution vs. Swin-B's 85.1 images/second (+12.5%), and that on A100 with TF32 and channel-last memory layout, the advantage grows to 336.6 vs. 242.5 images/second (+39%) for the B variants at 384² (Tables 1 and 12). For an organization serving millions of inference requests per day on A100-class hardware, a 39% throughput improvement at matched accuracy directly translates to a ~28% reduction in GPU-hours and associated costs, because fewer GPU instances are needed to serve the same request volume. The improvement is largest at higher resolutions (+34–49% on A100 across variants), making ConvNeXt particularly attractive for high-resolution applications like satellite imagery analysis, medical image diagnosis (where images routinely exceed 1000² pixels), and industrial quality inspection where fine details must be resolved. These applications typically run on fixed GPU infrastructure where throughput improvements compound over months of continuous operation.

Simplified multi-resolution fine-tuning and deployment pipelines. Unlike Swin Transformer and ViT, which require interpolating position embeddings and/or adjusting patch sizes when fine-tuning at different input resolutions, ConvNeXt is fully convolutional and handles arbitrary input sizes natively with no architectural changes. The paper demonstrates this by fine-tuning ConvNeXt-T from 224² to 384² (Table 1: 82.1% → 82.9% with ImageNet-1K, 84.1% with 22K pre-training) with no modifications to the network structure. For applications that require multi-scale inference (e.g., object detection where training uses 800×1280 but deployment may use different resolutions depending on camera specs, or medical imaging where scans from different machines have different native resolutions), this eliminates the engineering complexity of managing position embeddings, window size adjustments, and patch size compatibility checks. A practitioner deploying ConvNeXt can use the same model code for training at 224², fine-tuning at 384² or 512², and inference at arbitrary test resolutions — a practical simplification that reduces bugs and accelerates iteration.

A strong, open-source pure-ConvNet baseline for research on vision architectures. ConvNeXt's public code release (https://github.com/facebookresearch/ConvNeXt) and the detailed architecture specification (Table 9) provide researchers with a modern, high-performing ConvNet baseline that can serve as a comparison point for new architectural proposals. Before ConvNeXt, researchers proposing a new attention variant or hybrid architecture would typically compare against ResNet or ResNeXt — architectures that the paper shows are suboptimal by 3–4 percentage points due to outdated training recipes and design choices. Using ResNet-50 as a baseline unfairly advantages new proposals because part of the "improvement" comes from modern training and structural design, not from the novel operation. With ConvNeXt-T achieving 82.1% ImageNet-1K top-1 (vs. ResNet-50's 76.1% original or 78.8% with enhanced recipe), new architectural proposals can be evaluated against a baseline that already incorporates modern training and Transformer-informed design, making it harder (and more scientifically meaningful) to claim improvements. The isotropic ConvNeXt variants (Table 2) provide an additional baseline in the ViT-style design space, achieving 79.7%, 82.0%, and 82.6% for S/B/L sizes — competitive with supervised ViTs.