Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows

🎯 Pitch

Swin Transformer achieves linear computational complexity relative to image size, smashing previous state-of-the-art detection and segmentation records by over +2.5 points across the board, by confining self-attention to local windows that shift between layers. This hierarchical design finally makes Transformers practical as general-purpose vision backbones for dense prediction tasks.

1. Executive Summary

This paper proposes the Swin Transformer, a general-purpose vision backbone that adapts the Transformer architecture to computer vision by introducing a hierarchical representation (building feature maps at multiple resolutions through patch merging, like ConvNets) and a shifted window based self-attention mechanism (computing attention within local, non-overlapping windows that shift between consecutive layers to enable cross-window connections). Evaluated on ImageNet-1K, COCO object detection, and ADE20K semantic segmentation, Swin Transformer achieves 87.3% top-1 accuracy on ImageNet-1K using ImageNet-22K pretraining, 58.7 box AP and 51.1 mask AP on COCO test-dev — surpassing prior state-of-the-art by +2.7 box AP and +2.6 mask AP — and 53.5 mIoU on ADE20K, an improvement of +3.2 mIoU over the previous best model. The hierarchical design and shifted window approach also prove beneficial for all-MLP architectures, establishing that Transformers can serve as effective vision backbones when equipped with locality and multi-scale structure rather than global self-attention alone.

2. Context and Motivation

The Core Problem: Transformers Cannot Be Directly Plugged Into Vision as General-Purpose Backbones

The fundamental challenge this paper addresses is deceptively simple: Vision Transformers (ViTs) work well for image classification, but they are fundamentally unsuited as general-purpose backbones for computer vision. This is not a small limitation — in computer vision, a backbone network is the shared feature extractor that many downstream tasks (object detection, semantic segmentation, instance segmentation, pose estimation) build upon. The ConvNet architecture family — ResNet, ResNeXt, EfficientNet, HRNet — succeeded precisely because they provided a hierarchical representation (feature maps at multiple resolutions) that could plug into task-specific heads like Feature Pyramid Networks (FPN) or U-Net decoders.

The paper identifies two specific, concrete barriers that prevent the standard Vision Transformer (ViT) architecture from filling this backbone role (Section 1, paragraph 3):

Barrier 1: Single-scale, low-resolution feature maps. ViT processes images by splitting them into non-overlapping patches (typically 16×16 pixels) and maintains a constant spatial resolution of tokens throughout the entire network. At a 16×16 patch size, a 224×224 input image becomes a grid of only 14×14 tokens. The output feature map is therefore a single low-resolution grid of 14×14 — there is no hierarchy of progressively coarser features. This is fundamentally incompatible with dense prediction tasks like object detection (which needs to detect objects at multiple scales, from a few pixels to hundreds of pixels) and semantic segmentation (which requires pixel-level predictions). ConvNets naturally produce a feature pyramid with feature maps at strides of 4, 8, 16, and 32 pixels, which detection frameworks like FPN and segmentation frameworks like U-Net exploit. ViT's fixed-resolution output offers no such pyramid.

Barrier 2: Quadratic computational complexity with respect to image size. Standard Transformer self-attention computes pairwise interactions between every token and every other token. For an image divided into $h \times w$ patches, this means the computational cost scales as $\mathcal{O}((hw)^2)$ — quadratic in the number of patches. This is manageable for image classification with moderate-resolution inputs (e.g., 224×224 → 196 patches), but it becomes prohibitively expensive for the high-resolution inputs commonly used in dense prediction tasks. Object detection frameworks routinely use input resolutions of 800×1333 pixels, and semantic segmentation often operates at 512×512 or larger. At these resolutions, global self-attention is computationally intractable. Moreover, many vision tasks require processing feature maps at multiple scales, compounding the issue: even if global attention were affordable at the coarsest scale, computing it at finer resolutions during feature pyramid construction would be impossible.

The paper summarizes these two barriers concisely (Section 1):

"Unlike the word tokens that serve as the basic elements of processing in language Transformers, visual elements can vary substantially in scale... In existing Transformer-based models, tokens are all of a fixed scale, a property unsuitable for these vision applications. Another difference is the much higher resolution of pixels in images compared to words in passages of text."

These are not minor implementation details — they represent a fundamental architectural mismatch. The paper's core motivation is that without addressing both the hierarchical structure problem and the complexity problem, Transformers cannot displace ConvNets as general-purpose vision backbones, regardless of their theoretical representational power.

Why This Problem Matters: Real-World Impact and Theoretical Significance

The practical stakes are enormous. A general-purpose Transformer backbone for vision would enable several things:

Unified architectures across vision and language. If a single architecture family could serve as the backbone for both NLP and computer vision, the two fields could share modeling advances, training techniques, and infrastructure. This is not merely convenient — it would accelerate progress in joint vision-language modeling, where a unified backbone could process both image patches and text tokens through the same architectural primitives. Pretraining objectives, scaling laws, and architectural innovations developed for one modality could transfer to the other. The paper explicitly frames this as a long-term goal (Section 1, final paragraph):

"It is our belief that a unified architecture across computer vision and natural language processing could benefit both fields, since it would facilitate joint modeling of visual and textual signals and the modeling knowledge from both domains can be more deeply shared."

Replacing task-specific ConvNet backbones. For nearly a decade, improvements in ConvNet architectures — deeper networks (ResNet), wider networks (Wide ResNet), multi-branch designs (Inception), efficient scaling rules (EfficientNet), multi-resolution designs (HRNet) — have directly translated to improvements across all major vision benchmarks. A Transformer-based backbone that could similarly serve as a drop-in replacement for ConvNets in detection and segmentation frameworks would bring the representational power of self-attention (long-range dependencies, dynamic weight computation, large receptive fields without deep stacking) to these tasks. The paper demonstrates that this is not hypothetical — Swin Transformer achieves concrete, large-margin improvements (+2.7 box AP on COCO, +3.2 mIoU on ADE20K) simply by swapping the backbone, with all other framework components held constant.

Linear computational complexity enables high-resolution vision. The quadratic complexity of global self-attention is not just an inconvenience — it imposes a hard ceiling on what ViT-based models can do. The paper's proposal to achieve linear complexity (Equation 2 in Section 3.2) through windowed attention means that Transformers can now be applied to tasks and input scales that were previously off-limits. This is architecturally significant because it aligns Transformer efficiency with ConvNets (which also have linear complexity in image size), removing one of the key practical arguments against their use.

The theoretical significance is equally important. The paper addresses a foundational question: Is self-attention alone, properly structured, sufficient to match or exceed convolution for visual feature extraction? ConvNets encode strong inductive biases — locality (pixels interact primarily with nearby pixels), translation equivariance (shifting the input shifts the output), and scale hierarchy (features are progressively pooled from fine to coarse). ViT deliberately strips away most of these biases, treating an image as an unordered set of patches, which is why it requires massive datasets (JFT-300M) to learn these spatial regularities from data alone. The Swin Transformer's core insight is that you can reintroduce these inductive biases into the Transformer architecture without using convolution — locality through windowed attention, cross-window connectivity through shifted windows, and scale hierarchy through patch merging — while preserving the Transformer's advantages (dynamic attention weights, ability to model long-range dependencies within each window). This demonstrates that the Transformer architectural family can subsume the key structural properties that made ConvNets work so well for vision.

Prior Approaches and Where They Fall Short

The paper situates itself against several lines of prior work, each of which attempted to bring Transformer-style attention to vision but fell short of the general-purpose backbone goal for different reasons.

ViT and Its Follow-Ups: Strong on Classification, Weak on Dense Prediction

The Vision Transformer (ViT) by Dosovitskiy et al. (2020) was the breakthrough that proved Transformers could work for image classification. It processed 16×16 patches through a standard Transformer encoder and achieved excellent results — but only when pretrained on very large datasets (JFT-300M, with 300 million images). With the standard ImageNet-1K dataset (1.28 million images), ViT's performance lagged behind ConvNets of comparable size. The follow-up DeiT (Touvron et al., 2020) resolved the data efficiency issue through better training recipes and distillation, achieving competitive ImageNet-1K results without large-scale pretraining.

But the paper identifies a critical limitation that these classification-focused works share: their architecture is unsuitable for dense prediction. As the paper states in Section 2:

"The results of ViT on image classification are encouraging, but its architecture is unsuitable for use as a general-purpose backbone network on dense vision tasks or when the input image resolution is high, due to its low-resolution feature maps and the quadratic increase in complexity with image size."

The authors note that some concurrent works attempted to apply ViT to dense tasks by "direct upsampling or deconvolution but with relatively lower performance" — referencing Beal et al. (2020) for object detection and Zheng et al. (SETR, 2020) for semantic segmentation. SETR, for instance, achieved 50.3 mIoU on ADE20K by treating segmentation as a sequence-to-sequence problem with a ViT encoder, but this approach still maintains a single-resolution representation throughout, missing the multi-scale feature hierarchy that ConvNet-based segmentation methods rely on.

The quadratic complexity issue is not just about classification vs. dense tasks — it manifests even within a single forward pass. As shown in Equation 1 (Section 3.2), global multi-head self-attention (MSA) on an image of $h \times w$ patches costs:

$\Omega(\text{MSA}) = 4hwC^2 + 2(hw)^2C$

The $(hw)^2$ term dominates for large images. Doubling the input resolution quadruples the number of patches and squares the attention computation. This is why ViT models are typically limited to 224×224 or 384×384 inputs, whereas object detection frameworks routinely process images at 800×1333 (producing approximately 2,600 patches at 16×16 patch size — over $2,600^2 \approx 6.8$ million pairwise interactions per attention layer).

Concurrent works that modified ViT (Chu et al., 2021; Han et al., 2021) focused on improving classification performance through changes like explicit position encodings or transformer-in-transformer designs. The paper notes these were classification-focused and did not address the backbone problem. Another concurrent work, the Pyramid Vision Transformer (PVT) by Wang et al. (2021), built multi-resolution feature maps through spatial reduction but retained quadratic complexity — using spatial-reduction attention rather than window-based attention — meaning it shared the hierarchical structure but not the linear complexity benefit.

Self-Attention to Augment or Replace Convolutions: Inefficient Implementations

Several prior works attempted to incorporate self-attention into ConvNet-style architectures. These fall into two categories:

Attention-augmented ConvNets (e.g., Non-local networks by Wang et al., 2018; GCNet by Cao et al., 2019; Relation Networks by Hu et al., 2018): These add self-attention layers as modules within or alongside a standard ConvNet backbone, typically to capture long-range dependencies that convolutions struggle with. The paper acknowledges these works as complementary to its own — they augment ConvNets rather than replacing them entirely.

Attention replacing convolutions (e.g., Stand-Alone Self-Attention by Ramachandran et al., 2019; Local Relation Networks by Hu et al., 2019): These works replace the spatial convolution layers in ConvNets with per-pixel local self-attention — for each pixel, compute attention over a small spatial neighborhood (e.g., a 7×7 window centered on that pixel). This is architecturally closer to Swin Transformer's windowed attention, but with a crucial difference: these approaches use sliding windows where the attention neighborhood slides across the image pixel-by-pixel, just like a convolution kernel.

The paper identifies a specific, non-obvious failure mode of sliding-window self-attention (Section 2):

"However, their costly memory access causes their actual latency to be significantly larger than that of the convolutional networks."

This requires unpacking. In a standard convolution, the kernel weights are shared across all spatial positions — every pixel uses the same set of learned weights. This weight sharing is what makes convolutions efficient on hardware: the kernel can be loaded once into fast memory and applied everywhere. In sliding-window self-attention, each pixel (when acting as a query) computes attention over its neighborhood, but the key and value vectors at each spatial position are different. So, as the window slides from pixel to pixel, each query sees a different set of keys and values, even though the geometric window shape is the same. For every new query position, the hardware must load new key-value vectors from memory. The paper quantifies this difference in Table 5: a naive sliding-window self-attention implementation runs at 183 images/sec for Swin-T, while the shifted window approach runs at 755 images/sec — over 4× faster.

The paper also briefly compares against Performer (Choromanski et al., 2021), a linear-complexity Transformer that uses kernel-based approximations to avoid computing the full attention matrix. While Performer is "one of the fastest Transformer architectures," the paper shows in Table 5 that the shifted window approach is still slightly faster (755 vs. 638 images/sec for Swin-T) while being significantly more accurate (+2.3% top-1 accuracy on ImageNet-1K, Table 6).

Encoder-Decoder Transformers for Detection: Complementary, Not Backbone Replacements

A separate line of work, beginning with DETR (Carion et al., 2020) and continuing with Deformable DETR (Zhu et al., 2021) and Sparse R-CNN (Sun et al., 2020), uses Transformer encoder-decoder architectures as the detection head itself — taking ConvNet features as input and outputting object predictions through cross-attention between object queries and image features. The paper explicitly positions itself as complementary to these works (Section 2):

"Our work explores the adaptation of Transformers for basic visual feature extraction and is complementary to these works."

In other words, DETR-style models still need a backbone to extract features; Swin Transformer could serve as that backbone, potentially improving these detection Transformers further.

How This Paper Positions Itself Relative to Existing Work

The paper's positioning is clear and specific: it addresses the gap between classification-only ViT and the need for a general-purpose ConvNet replacement by introducing two architectural innovations — hierarchical feature maps (via patch merging) and efficient cross-window connectivity (via shifted windows) — that jointly solve the scale and complexity problems while introducing minimal practical overhead.

The paper does not propose to replace ConvNets with a fundamentally new paradigm. Instead, it shows that the Transformer can be structurally adapted to mimic the inductive biases that make ConvNets effective — locality (windows), hierarchy (patch merging), and cross-locality connectivity (shifting) — while retaining the Transformer's advantage of dynamic, content-dependent computation through attention. This is a middle ground between the extremes of "pure ViT" (no spatial inductive biases at all) and "pure ConvNet" (rigid spatial structure with static weights).

The design choices are directly motivated by the identified limitations:

Hierarchical feature maps → compatibility with FPN, U-Net, and other multi-scale architectures used in detection and segmentation. The paper explicitly states this in Section 3.1: "As a result, the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks."
Windowed self-attention → linear computational complexity with respect to image size (Equation 2: $\Omega(\text{W-MSA}) = 4hwC^2 + 2M^2 hwC$ ). Since the window size $M$ is fixed (default 7), the $(hw)^2$ term disappears entirely, replaced by $M^2 hw$ , which is linear in the number of patches.
Shifted windows → cross-window connectivity without the memory-access overhead of sliding windows. The paper proves this empirically in Table 4: without shifting, the model loses 1.1% top-1 on ImageNet, 2.8 box AP on COCO, and 2.8 mIoU on ADE20K.
Relative position bias → maintains the translational invariance that absolute position embeddings discard. The paper finds a subtle asymmetry: absolute position embeddings help classification (+0.4%) but hurt detection and segmentation (−0.2 box AP, −0.6 mIoU, Table 4), suggesting that translational invariance matters more for dense prediction.

The paper evaluates on the three canonical vision tasks — classification (ImageNet-1K), object detection and instance segmentation (COCO), and semantic segmentation (ADE20K) — to demonstrate generality. This is a deliberate choice: prior Vision Transformer papers primarily evaluated on classification alone, leaving open the question of whether their architectural benefits transferred to other tasks. By showing state-of-the-art results on all three, Swin Transformer makes the case that a properly designed Transformer backbone can be genuinely general-purpose.

The authors also demonstrate — through the Swin-Mixer experiment (Table 10 in Appendix A3.3) — that the hierarchical design and shifted window approach are not specific to attention. When applied to the MLP-Mixer architecture (which replaces self-attention with token-mixing MLPs), the same structural principles yield a 4.9% improvement over the original MLP-Mixer (81.3% vs. 76.4% top-1 on ImageNet-1K). This signals that the contributions — hierarchy and locality — are architectural primitives that generalize beyond any specific token-mixing mechanism.

3. Technical Approach

This is an architectural design paper whose core idea is that Transformers can serve as effective general-purpose vision backbones if they are endowed with two properties that ConvNets naturally possess: a hierarchical, multi-scale feature representation and computational efficiency that scales linearly with image resolution — achieved through windowed self-attention with shifted window partitioning between consecutive layers.

3.1 Reader Orientation

The Swin Transformer is a visual feature extractor — a neural network that takes an RGB image as input and produces a set of feature maps at multiple spatial resolutions (1/4, 1/8, 1/16, and 1/32 of the original size), each with progressively richer semantic content, much like a ResNet or other ConvNet backbone. It solves the problem of adapting Transformers — which are designed for sequences of word tokens — to handle images where (a) objects appear at widely varying scales, requiring multi-resolution features, and (b) the number of pixels is enormous, making global self-attention's quadratic cost prohibitive; the solution shape is to constrain attention computation to local windows that shift between layers to enable communication across windows while keeping computation linear in image size.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major structural components, forming a pipeline that transforms raw pixels into hierarchical feature maps:

Patch Partitioning and Linear Embedding — splits the input image into non-overlapping 4×4 pixel patches, flattens each patch's RGB values into a 48-dimensional vector, and projects it to an arbitrary channel dimension $C$ via a linear layer. This produces a grid of feature tokens at resolution $H/4 \times W/4$ .
Stage 1: First Swin Transformer Block Group — a sequence of Swin Transformer blocks operating on the $H/4 \times W/4$ token grid, maintaining the same spatial resolution throughout. Each block contains alternating regular window-based multi-head self-attention (W-MSA) and shifted-window multi-head self-attention (SW-MSA), followed by MLP layers.
Patch Merging Layers — inserted between stages, these reduce the spatial resolution by a factor of 2× (each dimension is halved, so the token count drops by 4×) by concatenating features from 2×2 neighboring patches and projecting them to a higher channel dimension. This builds the hierarchical pyramid.
Stages 2, 3, and 4: Deeper Feature Transformation — each stage (after its patch merging layer) applies additional Swin Transformer blocks at progressively coarser resolutions ( $H/8 \times W/8$ , $H/16 \times W/16$ , $H/32 \times W/32$ ) with increasing channel dimensions ( $2C$ , $4C$ , $8C$ ). Each stage produces a feature map with the same resolution as a corresponding stage in a typical ConvNet (e.g., ResNet-50's conv2, conv3, conv4, conv5 outputs).
Task-Specific Heads — the hierarchical feature maps plug into standard downstream frameworks: a global average pooling + linear classifier for image classification, Feature Pyramid Network (FPN) + detection heads for object detection, and UperNet decoder for semantic segmentation. The Swin Transformer backbone itself is independent of the head.

Information flows sequentially through the stages: an RGB image enters → patch partition splits it into tokens → Stage 1 produces $H/4 \times W/4$ features → Patch Merging 1 reduces to $H/8 \times W/8$ with doubled channels → Stage 2 refines these features → Patch Merging 2 reduces to $H/16 \times W/16$ → Stage 3 refines → Patch Merging 3 reduces to $H/32 \times W/32$ → Stage 4 produces the final, coarsest feature map. Every stage's output can be routed to the appropriate level of a detection or segmentation head.

3.3 Roadmap for the Deep Dive

First, the Swin Transformer block design (W-MSA and SW-MSA), since it is the fundamental building block repeated throughout all stages and embodies both innovations — windowed attention and cross-window connectivity via shifting.
Second, the computational complexity analysis (Equations 1–2), because this quantifies why windowed attention achieves linear complexity and what would happen without it, establishing the mathematical justification for the design.
Third, the shifted window partitioning mechanism in detail — how the windows are defined, how they shift, and the cyclic-shift efficient batch computation trick (Equation 3 and Figure 4), since this is the paper's primary technical novelty for enabling cross-window connections while maintaining efficiency.
Fourth, the relative position bias formulation (Equation 4), because it is the only positional encoding used and its interaction with translation invariance has task-dependent effects.
Fifth, the patch merging downsampling operation, since it is how the hierarchical pyramid is constructed from flat tokens.
Sixth, the overall architecture specifications and model variants (Swin-T, Swin-S, Swin-B, Swin-L), because these define the concrete configurations tested and their relationship to existing architectures.
Seventh, the Swin-Mixer experiment in the appendix, because it demonstrates that the hierarchical and shifted-window design principles generalize beyond attention to all-MLP architectures.

3.4 Detailed, Sentence-Based Technical Breakdown

This is fundamentally a network architecture paper — it does not propose a new training algorithm, loss function, or data processing pipeline, but rather defines a novel composition of existing primitives (multi-head self-attention, MLPs, LayerNorm, residual connections) arranged in a structure that makes Transformers viable as vision backbones. The architecture is defined by layer counts, channel dimensions, window sizes, and how these components connect; the "design" is in the spatial organization of attention computation and the construction of the feature hierarchy.

The Swin Transformer Block: W-MSA and SW-MSA

The Swin Transformer block is the core computational unit repeated throughout the network. It is a standard Transformer block (Vaswani et al., 2017) with one modification: the global multi-head self-attention (MSA) module is replaced by a window-based multi-head self-attention module that restricts computation to non-overlapping local windows, and this window partitioning shifts between consecutive blocks.

A standard Transformer block consists of:

$\text{Block}(\mathbf{x}) = \mathbf{x} + \text{MLP}(\text{LN}(\mathbf{x} + \text{MSA}(\text{LN}(\mathbf{x}))))$

That is: apply LayerNorm, compute multi-head self-attention, add the residual connection, apply LayerNorm again, process through a 2-layer MLP with GELU activation, add the second residual. The Swin Transformer preserves this exact structure but replaces MSA with either W-MSA (window-based multi-head self-attention using regular partitioning) or SW-MSA (window-based multi-head self-attention using shifted partitioning).

The paper defines the computation for two consecutive Swin Transformer blocks formally (Equation 3):

$\hat{\mathbf{z}}^l = \text{W-MSA}(\text{LN}(\mathbf{z}^{l-1})) + \mathbf{z}^{l-1}$

$\mathbf{z}^l = \text{MLP}(\text{LN}(\hat{\mathbf{z}}^l)) + \hat{\mathbf{z}}^l$

$\hat{\mathbf{z}}^{l+1} = \text{SW-MSA}(\text{LN}(\mathbf{z}^l)) + \mathbf{z}^l$

$\mathbf{z}^{l+1} = \text{MLP}(\text{LN}(\hat{\mathbf{z}}^{l+1})) + \hat{\mathbf{z}}^{l+1}$

where $\hat{\mathbf{z}}^l$ is the output of the attention module (including residual) at block $l$ , $\mathbf{z}^l$ is the output of the MLP module (including residual) at block $l$ , $\mathbf{z}^{l-1}$ is the input to block $l$ (output of the previous MLP), $\text{W-MSA}$ is window-based multi-head self-attention with regular (non-shifted) window partitioning, and $\text{SW-MSA}$ is window-based multi-head self-attention with shifted window partitioning.

What this computes: the first block takes its input tokens, partitions them into a regular grid of non-overlapping windows, computes self-attention within each window independently, adds the residual, passes through an MLP, and adds another residual. The second block takes the output of the first, partitions the token grid again but with the window grid shifted by $(\lfloor M/2 \rfloor, \lfloor M/2 \rfloor)$ pixels, computes self-attention within these new windows (which span across the boundaries of the previous windows), adds the residual, passes through an MLP, and adds the final residual. These two blocks together form the minimal repeating unit — all Swin Transformer blocks are paired in this alternating pattern, which is why the layer counts in each stage (Table 7) are always even numbers.

Why this form: the alternating pattern is the key mechanism for introducing cross-window connections. If every block used the same regular window partitioning, patches in different windows would never interact — the receptive field of each patch would be permanently limited to its own fixed window, preventing the model from learning long-range dependencies. The shift disrupts the window boundaries, so a patch that was in the top-right of window A in block $l$ may end up in the center of window B in block $l+1$ , now attending to patches that were previously in different windows. After two blocks (one regular + one shifted), every patch has a receptive field that extends beyond its original window. Stacking many such pairs builds up a large effective receptive field while keeping each attention computation local. This design is analogous to how ConvNets build receptive fields through stacking convolutional layers, but here the mechanism is attention-window shifting rather than kernel stacking. The alternative — computing global self-attention — would achieve full connectivity in one layer but at quadratic cost; the alternative — using sliding windows where every patch attends to its local neighborhood — achieves full connectivity but with hardware-inefficient memory access patterns (as shown in Table 5). The shifted window approach achieves a middle ground: efficient batched window computation (all queries in a window share the same key set) combined with the ability to propagate information across the entire image through the depth of the network.

Computational Complexity Analysis: From Quadratic to Linear

The paper provides an explicit complexity comparison between global multi-head self-attention (MSA) and window-based multi-head self-attention (W-MSA) to quantify the efficiency gain. These are Equations 1 and 2:

$\Omega(\text{MSA}) = 4hwC^2 + 2(hw)^2C$

$\Omega(\text{W-MSA}) = 4hwC^2 + 2M^2 hwC$

where $\Omega$ denotes the computational complexity in floating-point operations, $h$ is the height of the feature map in number of patches, $w$ is the width of the feature map in number of patches, $C$ is the channel dimension (the embedding dimension of each token), and $M$ is the window size (the number of patches along each side of a square window, fixed to 7 by default).

What these equations compute: The complexity of a single multi-head self-attention layer on a feature map of size $h \times w$ with embedding dimension $C$ . The term $4hwC^2$ is the cost of projecting the input into queries, keys, values, and the output projection — this term is identical for both MSA and W-MSA because every token must be projected regardless of how attention is computed. The difference is in the second term: for global MSA, it is $2(hw)^2C$ , which is the cost of computing the $hw \times hw$ attention matrix and multiplying it with the value matrix — this grows quadratically with the number of tokens. For W-MSA, it is $2M^2 hwC$ , which is the cost of computing attention within $(h/M) \times (w/M)$ windows, each having $M^2$ tokens, for a total cost that scales with $hw M^2$ — since $M^2$ is a constant (49 for the default $M=7$ ), this term is linear in $hw$ , the total number of tokens.

Operational meaning: For an input image of $224 \times 224$ with a patch size of 4, there are $56 \times 56 = 3136$ tokens. Global MSA would require computing a $3136 \times 3136$ attention matrix at every layer — millions of pairwise attention scores, each requiring dot products in $C$ dimensions. W-MSA instead divides these 3136 tokens into $56/7 \times 56/7 = 8 \times 8 = 64$ windows, each containing $7 \times 7 = 49$ tokens, and computes 64 separate $49 \times 49$ attention matrices. The total pairwise computations drop from $3136^2 \approx 9.8$ million to $64 \times 49^2 \approx 153,664$ — a reduction by a factor of approximately 64. For higher-resolution inputs (e.g., $800 \times 1333$ for object detection), the reduction is even more dramatic, making the model feasible where global attention would be impossible.

Why this form: The quadratic term $2(hw)^2C$ comes from two operations: computing the attention scores $QK^T$ (of shape $hw \times hw$ , costing $hw \times hw \times C$ operations — actually $2hw \times hw \times C$ accounting for multiply-add, hence the factor 2), and multiplying the attention probabilities $A$ with the values $V$ (also $hw \times hw \times C$ in the general case, but simplified to a single count). The windowed version replaces $hw$ with $M^2$ and multiplies by the number of windows $(hw)/M^2$ , giving $(hw/M^2) \times M^2 \times M^2 \times C = hw \times M^2 \times C$ . This is the crucial mathematical insight: by partitioning attention into fixed-size windows, the complexity becomes linear in the total number of tokens rather than quadratic. The constant $M$ is a design choice: larger windows capture more context at higher cost, smaller windows are cheaper but limit the receptive field per layer. Setting $M=7$ is a practical compromise that the paper found effective across all tasks.

The paper explicitly notes (footnote 3) that SoftMax computation is omitted from the complexity analysis — this is standard practice because SoftMax is applied per-row of the attention matrix and its cost is negligible compared to the matrix multiplications described above.

Shifted Window Partitioning: Mechanics and Efficient Batch Computation

The shifted window approach is the paper's primary mechanism for enabling cross-window communication. Here is how it works, step by step:

Regular window partitioning (used in W-MSA). Given a feature map of size $h \times w$ patches, the image is divided into a grid of non-overlapping $M \times M$ square windows starting from the top-left corner. If $h$ or $w$ is not divisible by $M$ , bottom-right padding is applied to make the dimensions divisible (the paper states: "bottom-right padding is employed on the feature map if needed" — footnote 4). For the default $M=7$ and a $56 \times 56$ feature map (Stage 1 of Swin-T with $224 \times 224$ input), this produces exactly $8 \times 8 = 64$ windows, each $7 \times 7$ . Self-attention is computed independently within each window: every patch in a window attends to every other patch in the same window, using shared query, key, and value projections. Patches in different windows do not interact.

Shifted window partitioning (used in SW-MSA). The shifted configuration displaces the window grid by $(\lfloor M/2 \rfloor, \lfloor M/2 \rfloor)$ pixels relative to the regular partitioning. With $M=7$ , this shift is $(3, 3)$ pixels. Visually, imagine the regular grid of $8 \times 8$ windows; the shifted grid starts 3 pixels to the right and 3 pixels down, so what was the boundary between two adjacent windows in the regular configuration now falls inside a single window in the shifted configuration. This means that patches that were in different windows in block $l$ (regular partitioning) end up in the same window in block $l+1$ (shifted partitioning), allowing them to attend to each other.

The naive computation problem. With regular partitioning and a feature map divisible by $M$ , the number of windows is $\lceil h/M \rceil \times \lceil w/M \rceil$ — exactly $8 \times 8 = 64$ for the $56 \times 56$ case. When the window grid is shifted, some windows near the edges become smaller than $M \times M$ because the shift pushes the grid boundaries. The number of windows increases to $(\lceil h/M \rceil + 1) \times (\lceil w/M \rceil + 1)$ — in the $56 \times 56$ case, from $8 \times 8 = 64$ to $9 \times 9 = 81$ . The paper explicitly quantifies this: "the increased computation with this naive solution is considerable (2×2 → 3×3, which is 2.25 times greater)" — they give the example where regular partitioning produces only $2 \times 2 = 4$ windows, and shifting produces $3 \times 3 = 9$ windows, a 2.25× increase. A naive solution would pad each smaller window to exactly $M \times M$ and mask out the padded positions during attention, but the increased number of windows drives up computation.

The cyclic-shift efficient batch computation (Figure 4). The paper proposes a clever trick to keep the number of windows identical between regular and shifted configurations. The procedure works in three steps:

Cyclic shift: The entire feature map is rolled (cyclically shifted) by $(-\lfloor M/2 \rfloor, -\lfloor M/2 \rfloor)$ pixels — that is, shifted toward the top-left by $(3, 3)$ for $M=7$ . This moves the shifted windows back into alignment with the regular window grid, so that the window boundaries are again at multiples of $M$ starting from the top-left.
Window partition and attention with masking: After the cyclic shift, the image is partitioned into the standard $\lceil h/M \rceil \times \lceil w/M \rceil$ windows (e.g., $8 \times 8$ ). However, because of the cyclic shift, some windows now contain patches that were originally far apart in the image — patches from the top edge may be adjacent to patches from the bottom edge, and patches from the left edge may be adjacent to patches from the right edge. These artifacts arise because the original shifted windows spanned across the regular partition boundaries, and the cyclic shift has wrapped those boundary-spanning windows into contiguous locations. A masking mechanism is applied within each window to prevent attention between sub-windows that are not actually adjacent in the original image. Specifically, within each batched window, a binary mask identifies which patch pairs belong to the same original sub-window before the shift; attention scores between patches from different sub-windows are set to a large negative value (before SoftMax, effectively $-\infty$ ) so that after SoftMax they receive zero weight.
Reverse cyclic shift: After computing the masked self-attention, the feature map is cyclically shifted back by $(+\lfloor M/2 \rfloor, +\lfloor M/2 \rfloor)$ pixels to restore the original spatial positions.

What this computes: The same set of self-attention operations as the naive shifted-window approach — patches in the shifted windows attend to each other across the boundaries of the original (non-shifted) windows — but the computation is organized so that the number of batched windows does not increase and all windows are exactly $M \times M$ , enabling efficient batch processing on GPUs.

Why this form: The cyclic shift trick transforms a problem with irregularly shaped, increased-count windows into a problem with regularly shaped, fixed-count windows plus a masking step. The masking step adds negligible overhead (a few binary mask operations per window) compared to the cost of processing $9 \times 9 = 81$ windows instead of $8 \times 8 = 64$ (a $26.6\%$ increase in computation). Table 5 quantifies the real-world speed impact: on a V100 GPU, the cyclic implementation achieves 755, 437, and 278 images per second for Swin-T, Swin-S, and Swin-B respectively, compared to 670, 371, and 236 for the naive padding approach — improvements of 13%, 18%, and 18%. The cyclic shift is not just a theoretical optimization; it meaningfully impacts throughput. Moreover, the approach is general: any window shift amount that is less than the window size $M$ can be handled by this mechanism. The paper uses a shift of half the window size $(\lfloor M/2 \rfloor, \lfloor M/2 \rfloor)$ , which maximizes the boundary-crossing area — shifting by exactly half the window means that the new windows are maximally misaligned with the old windows, creating the largest possible cross-window connections.

Relative Position Bias

The Swin Transformer does not use absolute position embeddings (as in ViT's learnable 2D position encodings) or sinusoidal position encodings (as in the original Transformer). Instead, it adds a relative position bias $B \in \mathbb{R}^{M^2 \times M^2}$ to the attention logits before the SoftMax operation. The attention computation is defined in Equation 4:

$\text{Attention}(Q, K, V) = \text{SoftMax}(QK^T / \sqrt{d} + B) V$

where $Q, K, V \in \mathbb{R}^{M^2 \times d}$ are the query, key, and value matrices for a single window (reshaped from the token features within that window), $d$ is the query/key dimension (set to 32 for all experiments), $M^2$ is the number of patches in a window (49 for $M=7$ ), and $B \in \mathbb{R}^{M^2 \times M^2}$ is the relative position bias matrix.

What this computes: For each pair of patches $(i, j)$ within a window, the attention logit is the standard scaled dot-product $Q_i K_j^T / \sqrt{d}$ plus a learned scalar $B_{ij}$ that depends only on the relative 2D position of patch $j$ with respect to patch $i$ . If patch $i$ is at 2D coordinates $(x_i, y_i)$ and patch $j$ is at $(x_j, y_j)$ , the relative displacement is $(\Delta x, \Delta y) = (x_j - x_i, y_j - y_i)$ . Since each coordinate difference can range from $-(M-1)$ to $+(M-1)$ (e.g., for $M=7$ , from $-6$ to $+6$ ), there are $(2M-1) \times (2M-1)$ possible relative positions. The paper parameterizes a smaller bias matrix $\hat{B} \in \mathbb{R}^{(2M-1) \times (2M-1)}$ — for $M=7$ , this is a $13 \times 13$ matrix $\hat{B}$ — and each entry $B_{ij}$ in the full $M^2 \times M^2$ bias matrix is taken from $\hat{B}$ by indexing with the corresponding relative displacement. In other words, all pairs of patches with the same relative displacement share the same learned bias value.

Why this form: Relative position encoding captures translation invariance: if the entire window content shifts by some offset, the attention pattern should remain the same because the relative positions between patches are unchanged. Absolute position embeddings break this property — shifting the image changes the absolute coordinates of every patch, which changes the position embeddings added to the input, which changes the attention computation. The paper finds that this property matters differently for different tasks: absolute position embeddings improve ImageNet-1K classification by $+0.4\%$ accuracy (Table 4, "abs. pos." vs. "no pos."), but they degrade COCO detection by $-0.2$ box AP and ADE20K segmentation by $-0.6$ mIoU. The paper explains this as follows: translation invariance has long been considered important for visual modeling, and while ViT/DeiT models "abandon translation invariance in image classification even though it has long been shown to be crucial for visual modeling," the Swin Transformer results show that "inductive bias that encourages certain translation invariance is still preferable for general-purpose visual modeling, particularly for the dense prediction tasks of object detection and semantic segmentation" (Section 4.4). The relative position bias provides this invariance: it is invariant to where the window is in the image, depending only on relative displacements within the window.

The bias matrix $\hat{B}$ is learned during training — it is a parameter of the model. The paper notes that "the learnt relative position bias in pre-training can be also used to initialize a model for fine-tuning with a different window size through bi-cubic interpolation" (Section 3.2). This is important because when fine-tuning for tasks that use different input resolutions (and therefore different window partitioning — e.g., ImageNet-1K classification at $224^2$ vs. detection with multi-scale training), the window size might change, and bi-cubic interpolation allows transferring the learned positional relationships without retraining the bias from scratch. This is analogous to how ViT and DeiT interpolate their position embeddings when changing input resolution.

Table 4 also ablates a variant where the relative position bias is used without the standard scaled dot-product term (denoted "rel. pos. w/o app."), meaning the attention logits depend only on relative positions and not on content. This variant performs significantly worse: $79.3\%$ vs. $81.3\%$ on ImageNet-1K, $48.2$ vs. $50.5$ box AP on COCO, demonstrating that both content-based similarity and position-based bias are necessary — content similarity alone is insufficient (dropping $1.2\%$ / $2.3$ box AP without the bias), and position alone is insufficient (dropping $2.0\%$ / $2.3$ box AP without the dot-product term).

Patch Merging: Building the Hierarchical Feature Pyramid

The patch merging layers are what create the multi-resolution feature hierarchy. They are inserted between stages to progressively reduce the spatial resolution of the token grid while increasing the channel dimension, mimicking the effect of pooling or strided convolution in ConvNets.

Operation (Stage 1 → Stage 2 transition). Given the output of Stage 1, which is a feature map of shape $H/4 \times W/4 \times C$ (e.g., $56 \times 56 \times C$ for a $224 \times 224$ input), the patch merging layer:

Groups the tokens into non-overlapping $2 \times 2$ spatial blocks. Each block contains 4 neighboring tokens, each with $C$ -dimensional features.
Concatenates the 4 feature vectors within each block to form a single vector of dimension $4C$ .
Applies a linear layer that projects this $4C$ -dimensional concatenated vector to $2C$ dimensions.

The output is a feature map of shape $H/8 \times W/8 \times 2C$ . The number of tokens has been reduced by a factor of $2 \times 2 = 4$ (2× downsampling in each spatial dimension), and the channel dimension has doubled from $C$ to $2C$ .

Subsequent stages. The same procedure repeats between Stage 2 and Stage 3 (reducing from $H/8 \times W/8 \times 2C$ to $H/16 \times W/16 \times 4C$ ) and between Stage 3 and Stage 4 (reducing from $H/16 \times W/16 \times 4C$ to $H/32 \times W/32 \times 8C$ ). The channel dimensions for different model variants are specified in Table 7: for Swin-T, $C=96$ , so Stage 1 output has 96 channels, Stage 2 has 192, Stage 3 has 384, Stage 4 has 768 (after each patch merging layer's linear projection adjusts the dimension accordingly — the first patch merging outputs $192$ -d, the second outputs $384$ -d, and the third outputs $768$ -d).

Why this form: Patch merging is a simple, non-learned (beyond the linear projection) spatial reduction operation that creates exactly the resolution pyramid that ConvNet backbones produce. The specific resolutions ( $H/4$ , $H/8$ , $H/16$ , $H/32$ ) match the typical strides of ResNet's four stages (conv2: stride 4, conv3: stride 8, conv4: stride 16, conv5: stride 32). This compatibility is by design — the paper states that "the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks" (Section 3.1), and the matched resolutions ensure that feature maps from each Swin stage can directly feed into FPN levels, U-Net skip connections, or detection head inputs designed for ConvNet backbones. The linear projection after concatenation serves the same role as a 1×1 convolution in ConvNets: it reduces channel dimensionality and learns to combine information from the four spatially adjacent tokens in a content-dependent way. Without it, the concatenated $4C$ -dimension vectors would be passed directly to the next stage, increasing the channel dimension by 4× instead of 2×, which would make the model much larger without necessarily improving representational capacity.

Architecture Configurations and Model Variants

The paper defines four model variants of increasing capacity. The architectural hyperparameters are specified in Section 3.3:

Swin-T (tiny): $C = 96$ , layer numbers per stage = $\{2, 2, 6, 2\}$
Swin-S (small): $C = 96$ , layer numbers per stage = $\{2, 2, 18, 2\}$
Swin-B (base): $C = 128$ , layer numbers per stage = $\{2, 2, 18, 2\}$
Swin-L (large): $C = 192$ , layer numbers per stage = $\{2, 2, 18, 2\}$

What these numbers mean. $C$ is the channel dimension (embedding dimension) of the hidden layers in Stage 1. After the first patch merging, the channel dimension becomes $2C$ ; after the second, $4C$ ; after the third, $8C$ . The layer numbers specify how many Swin Transformer blocks are in each stage. Since blocks come in pairs (one W-MSA block followed by one SW-MSA block), all layer numbers are even. For Swin-T, Stage 1 has 2 blocks (1 pair), Stage 2 has 2 blocks (1 pair), Stage 3 has 6 blocks (3 pairs), and Stage 4 has 2 blocks (1 pair), for a total of 12 blocks. Swin-S and Swin-B increase the depth of Stage 3 to 18 blocks (9 pairs), making them significantly deeper in the middle stage where feature maps are at medium resolution (14×14 for $224 \times 224$ input). Swin-L increases both the channel dimension and keeps the deeper Stage 3 configuration.

Design rationale for scaling. The pattern of scaling — keeping Stage 1 and Stage 2 very shallow (2 blocks each) and concentrating depth in Stage 3 — follows the design principle of many ConvNet backbones (e.g., ResNet-50 with $[3, 4, 6, 3]$ blocks, ResNet-101 with $[3, 4, 23, 3]$ ). The early stages operate at high resolution ( $56 \times 56$ and $28 \times 28$ for $224 \times 224$ input) where computation is expensive (more tokens = more attention operations, even in windows). Keeping these stages shallow reduces total FLOPs. Stage 3, at $14 \times 14$ resolution, sees far fewer tokens (196 vs. 3136) so adding blocks here increases representational capacity with relatively modest computational cost. Swin-T and Swin-S share the same channel dimension $C=96$ but Swin-S has 6 more blocks in Stage 3 (12 more Transformer layers), making it approximately 2× the FLOPs of Swin-T (4.5G vs. 8.7G FLOPs, Table 1). Swin-B increases both $C$ (to 128) and depth, reaching similar FLOPs to ViT-B/DeiT-B (15.4G).

Relation to existing architectures. The paper explicitly designs Swin-B to have "model size and computation complexity similar to ViT-B/DeiT-B" (Section 3.3) to enable fair comparison. Swin-T and Swin-S have complexities similar to ResNet-50 (DeiT-S) and ResNet-101 respectively. The query dimension per head is fixed at $d = 32$ for all models, and the MLP expansion ratio is $\alpha = 4$ (the hidden layer in the MLP is $4 \times$ the input dimension). The window size is fixed at $M = 7$ for all experiments.

Table 1 provides the concrete parameter counts, FLOPs, and throughput: Swin-T has 29M parameters and 4.5G FLOPs; Swin-S has 50M and 8.7G; Swin-B has 88M and 15.4G; Swin-L has 197M and 103.9G (at $384^2$ input). The detailed per-stage architecture is broken down in Table 7 (Appendix), which specifies the exact channel dimensions, head counts, and window configurations for every stage of every variant.

Swin-Mixer: Generalizing the Hierarchical Shifted-Window Design Beyond Attention

The paper includes an experiment (Appendix A3.3, Table 10) that applies the hierarchical design and shifted window approach to the MLP-Mixer architecture (Tolstikhin et al., 2021), which replaces self-attention entirely with token-mixing MLPs — fully-connected layers applied across the token dimension. The resulting model, called Swin-Mixer, demonstrates that the paper's structural innovations are not tied to attention.

The Swin-Mixer follows the same four-stage hierarchical structure with patch merging, but within each stage, instead of Swin Transformer blocks (W-MSA + SW-MSA), it uses MLP-Mixer blocks with shifted window partitioning: the token-mixing MLP operates only on patches within a window, and the window partitioning alternates between regular and shifted configurations between consecutive blocks, exactly as in the Swin Transformer.

The results in Table 10 show:

Original MLP-Mixer-B/16 achieves 76.4% top-1 on ImageNet-1K with 59M parameters and 12.7G FLOPs.
Swin-Mixer-B/D24 achieves 81.3% top-1 with 61M parameters and 10.4G FLOPs — a +4.9% improvement with lower FLOPs.
Removing the shifting (Swin-Mixer-B/D24, "no shift") drops accuracy from 81.3% to 80.3%, confirming that the shifting mechanism is beneficial even without attention.

This is significant because it isolates the architectural principles (hierarchy + windowed computation + cross-window connectivity through shifting) from the specific mechanism of self-attention. The paper uses this to argue that these design principles are general-purpose architectural primitives for vision, not artifacts of attention's properties. The implications are broader than the Swin Transformer itself: any token-mixing mechanism — attention, MLPs, or future alternatives — can benefit from being organized into a hierarchical structure with shifted local windows, because the inductive biases of locality and cross-scale processing are fundamental to vision regardless of the computation performed within each local region.

4. Key Insights and Innovations

Innovation 1: Recasting Locality and Hierarchy as Architectural Primitives Independent of Convolution

The dominant assumption in vision architecture design, from AlexNet through EfficientNet, was that hierarchical feature extraction at multiple scales required convolution — specifically, the combination of local receptive fields (convolution kernels), spatial downsampling (pooling or strided convolution), and the progressive increase in channel dimension. Even as self-attention replaced recurrence in NLP, the vision community treated these two architectural families as fundamentally distinct: ConvNets provided the spatial inductive biases that made vision models work, while Transformers provided content-dependent global computation suitable for sequences but requiring massive data to overcome their lack of spatial structure. ViT and DeiT demonstrated that Transformers could do image classification without convolution, but by stripping away all spatial inductive biases — no locality, no hierarchy, no translation equivariance — they created models that were specialized for classification and essentially unusable for the dense prediction tasks that define modern computer vision.

The Swin Transformer's deepest conceptual contribution is demonstrating that the inductive biases that make ConvNets effective — local processing, multi-scale hierarchy, and translation invariance — can be recovered within the Transformer family using only attention and MLPs, without any convolution operations whatsoever. This is not an incremental improvement; it is a reframing of what Transformers "are" for vision. Before Swin, the question was "Can Transformers match ConvNets despite lacking spatial priors?" After Swin, the question becomes "Which specific spatial priors matter for vision, and how can they be encoded in self-attention architectures?" This shifts the discourse from competition between architectural families to recombination of architectural principles.

The evidence that this is a genuine intellectual contribution rather than just good engineering comes from the Swin-Mixer experiment (Table 10, Appendix A3.3). By applying the same hierarchical design and shifted window pattern to the MLP-Mixer architecture — which has no self-attention at all — the paper achieves a 4.9% accuracy improvement over the original MLP-Mixer (81.3% vs. 76.4%) while using fewer FLOPs (10.4G vs. 12.7G). Ablating the shifting alone drops accuracy by 1.0% (81.3% → 80.3%). This demonstrates that the paper's core architectural decisions — constructing a feature pyramid through patch merging and connecting spatial regions through shifted windows — are architectural primitives that transcend any specific token-mixing mechanism. They are not "things you do to make attention work for vision"; they are "things any vision architecture should do, regardless of what computation happens within each local window."

This reframing has broad implications. It suggests that the architectural search space for vision models should be organized around spatial organization (how tokens interact across space, how resolution changes) rather than around the choice of computation primitive (convolution, attention, MLPs). The paper's design principles — local windows enforce locality, patch merging builds hierarchy, window shifting provides cross-region communication — define a template that can accommodate future token-mixing mechanisms without redesigning the backbone from scratch. This is a fundamental conceptual shift analogous to the recognition in NLP that the Transformer architecture is defined by its attention-plus-MLP pattern rather than by recurrence — once you separate the structural template from the computational mechanism, the template becomes a reusable design pattern.

Innovation 2: The Shifted Window as a Hardware-Aware Cross-Window Communication Mechanism

The problem of enabling long-range communication in locally-processing vision architectures is nearly as old as ConvNets themselves. The standard ConvNet solution — stacking layers to build receptive fields progressively — works but is slow, requiring many layers for distant pixels to interact. The sliding-window self-attention approach (Ramachandran et al., 2019; Hu et al., 2019) accelerates this by letting each pixel attend to all pixels in its local window, but suffers from a hardware-specific limitation that the paper is, to my knowledge, the first to identify and address systematically: when the attention window slides pixel-by-pixel, each query position sees a different set of key-value vectors, so hardware must reload those vectors from memory for every new query position. As the paper shows in Table 5, this makes naive sliding-window attention 40.8× slower than the shifted window approach in early stages (122.5 ms vs. 3.0 ms for Stage 1 on Swin-T) and still 7.6× slower even with kernel-level optimizations.

The shifted window mechanism solves this by making the set of keys and values shared across all queries within a batched window. Every patch in a window of size M × M attends to the same M × M set of patches — the computation is a single batch matrix multiplication rather than M^2 separate local attention operations with different key sets. This is the same computational pattern as standard global self-attention, just restricted to a subset of tokens, which means standard optimized attention kernels (from libraries like cuDNN or FlashAttention) can be applied directly. The shifting introduces cross-window connectivity in the next layer without ever requiring per-pixel sliding windows. The cyclic-shift trick (Figure 4) then ensures the window count remains constant between regular and shifted configurations, avoiding the 2.25× computation increase that naive padding would incur for small window grids.

What makes this intellectually distinctive is that it couples an algorithmic design decision (how to introduce cross-window connectivity) with a hardware efficiency analysis (why sliding windows are slow despite having the same theoretical FLOPs). Table 5 is arguably the most important table in the paper that few people discuss: it directly compares the real-world latency of sliding windows (naive and kernel-optimized), Performer (a leading efficient attention mechanism), regular windows without shifting, and shifted windows with both naive padding and the cyclic approach. The numbers reveal that the shifted window approach with cyclic batch computation is not just conceptually elegant — it is 4.1× faster than sliding window variants for Swin-T, while achieving nearly identical accuracy (Table 6: 81.3% vs. 81.4% on ImageNet-1K for shifted vs. sliding windows, 50.5 vs. 50.2 box AP on COCO). This is a rare case where an architectural innovation simultaneously improves efficiency and maintains accuracy, rather than trading one for the other.

The significance goes beyond this paper. It demonstrates that designing attention mechanisms for vision requires reasoning about hardware memory access patterns, not just FLOP counts — a lesson that subsequent work on efficient attention (Swin Transformer V2, CSWin, Focal Attention) has built upon. The shifted window is not just "a way to do cross-window attention"; it is specifically a way to do it that respects how GPUs batch memory loads for matrix multiplication, making it the first local attention mechanism for vision that is genuinely competitive with convolution in wall-clock time, not just in asymptotic complexity.

Innovation 3: Task-Dependent Value of Translation Invariance — Classification vs. Dense Prediction

A subtle but deeply informative finding emerges from Table 4's ablations on position encoding: absolute position embeddings help image classification but hurt object detection and semantic segmentation. Specifically, adding absolute position embeddings (as used in ViT) to the Swin-T baseline improves ImageNet-1K top-1 accuracy by 0.4% (80.5% vs. 80.1% with no position encoding), but decreases COCO box AP by 0.2 points (49.0 vs. 49.2) and ADE20K mIoU by 0.6 points (43.2 vs. 43.8). In contrast, relative position bias — which encodes only the spatial relationship between pairs of patches, not their absolute coordinates — improves all three tasks simultaneously: +1.2% on ImageNet-1K, +1.3 box AP on COCO, and +2.3 mIoU on ADE20K over the no-position-encoding baseline.

This is not just an ablation result; it is a diagnostic finding about what different vision tasks demand from spatial representations. Image classification requires identifying what is in the image, and absolute position can help — knowing that a patch is in the top-left corner provides useful context that aids category recognition (e.g., sky tends to be at the top, grass at the bottom). Object detection and semantic segmentation require predicting where objects or regions are, but crucially, the model must be able to recognize an object regardless of where it appears in the image. A cat in the top-left corner should be detected just as reliably as a cat in the center. Absolute position embeddings violate this translation invariance because shifting the image changes the position encoding added to every token, which changes the attention computation and potentially the prediction. Relative position bias preserves invariance: only the spatial relationships between patches matter, not their absolute coordinates.

Prior work had observed that ViT and DeiT dropped translation invariance without penalty for classification, leading to the implicit assumption that this inductive bias was not important for vision Transformers. The Swin Transformer's results reveal that this conclusion was task-specific: translation invariance remains important for vision, but classification is a sufficiently global task (average pooling over all patches before the final classifier) that it can tolerate the loss, while dense prediction tasks — which make spatially localized decisions — are far more sensitive. The paper explicitly calls this out (Section 4.4):

"While the recent ViT/DeiT models abandon translation invariance in image classification even though it has long been shown to be crucial for visual modeling, we find that inductive bias that encourages certain translation invariance is still preferable for general-purpose visual modeling, particularly for the dense prediction tasks of object detection and semantic segmentation."

This finding has implications for architecture design that extend beyond the Swin Transformer: if a model is intended as a general-purpose backbone, its positional encoding scheme should be evaluated on dense prediction tasks, not just classification. Positional encoding methods that look equivalent on ImageNet may have very different behaviors on COCO or ADE20K. The paper's explicit comparison — testing the same architecture with no position encoding, absolute position embeddings, relative position bias, and both together across three tasks — establishes a template for how such evaluations should be done.

Innovation 4: Verifier-Inspired Windowing as a Scalability-Connectivity Tradeoff Resolution

While the paper does not use the language of "verifiers" or "test-time compute" (these are terms from the LLM literature), there is a structural parallel that illuminates why the shifted window approach is intellectually novel. In the broader Transformer literature, the problem of scaling attention to long sequences has been attacked through two families of solutions: sparse attention patterns (restricting which token pairs attend to each other, as in Sparse Transformer, Longformer, or BigBird) and linearized attention (approximating the full attention matrix through kernel methods, as in Performer or Linear Transformer). Sparse attention preserves the exact attention computation but limits connectivity; linearized attention preserves global connectivity but approximates the computation.

The shifted window approach falls into neither category. It is exact attention within local windows (no approximation), but it achieves global connectivity not by making the attention pattern sparse or global, but by shifting the partition boundaries between layers so that information propagates across the image through the network depth. This is architecturally more similar to how ConvNets build receptive fields — stacking local operations with shifting receptive field centers — but realized through attention rather than convolution. What makes this novel is that it repurposes the Transformer's depth (normally used for hierarchical feature abstraction) to also serve as a mechanism for spatial information propagation. In a standard Transformer, depth refines representations; in Swin, depth simultaneously refines representations and expands the effective spatial context of each token.

This matters because it resolves a long-standing tension: sparse attention patterns (like strided or block-sparse attention) provide efficiency but create "attention gaps" — token pairs that never directly interact, even after many layers, unless the sparsity pattern is carefully designed. The shifted window approach ensures that after 2k consecutive Swin Transformer blocks (k pairs of regular-then-shifted windows), every patch has had the opportunity to attend, directly or indirectly, to patches within a neighborhood of radius proportional to k × M. The paper does not formalize or prove this receptive field property, but the empirical evidence in Table 4 is unambiguous: removing the shift (all blocks use regular window partitioning) drops performance by 1.1% on ImageNet-1K, 2.8 box AP on COCO, and 2.8 mIoU on ADE20K — large, consistent degradations that indicate the cross-window connectivity is essential.

The comparison to Performer in Tables 5 and 6 is revealing: Performer (a linearized attention method) achieves 638 images/sec throughput on Swin-T vs. 755 for shifted windows, and 79.0% top-1 accuracy vs. 81.3%. So the shifted window approach is both faster and more accurate than the leading linear-complexity attention approximation, because it doesn't approximate — it computes exact attention, just on a strategically constrained token set. This suggests that for vision, where locality is a strong prior, restricting attention spatially is preferable to approximating it globally. This is not obvious a priori: one might expect that a global approximation would outperform a local exact computation, because it can in principle capture long-range dependencies. The paper's results argue the opposite — that the inductive bias of locality, when coupled with a mechanism for cross-window propagation, outperforms approximation-based global attention.

This insight — that hardware-efficient exact local attention with structural connectivity mechanisms can outperform both sparse attention and linearized attention — has influenced a generation of subsequent vision Transformers (CSWin, Focal Transformer, SwinV2, Video Swin Transformer) and has implications beyond vision, suggesting that similar approaches could benefit long-sequence modeling in NLP or multi-modal models where locality is a meaningful prior.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on three canonical vision benchmarks: (1) ImageNet-1K [19] for image classification — 1.28M training images, 50K validation images across 1,000 classes, reporting single-crop top-1 accuracy; (2) COCO 2017 [43] for object detection and instance segmentation — 118K training, 5K validation, and 20K test-dev images, with ablation on the validation set and system-level comparison on test-dev; (3) ADE20K [83] for semantic segmentation — 20K training, 2K validation, 3K test images across 150 semantic categories, reporting mean Intersection-over-Union (mIoU).
Base model(s). All experiments use the Swin Transformer architecture (Swin-T, Swin-S, Swin-B, Swin-L) described in Section 3.3, with the specific hyperparameters (C and layer counts per stage) defining each variant. Backbones are initialized from scratch for the ImageNet-1K regular training setting and from ImageNet-22K pretraining (14.2M images, 22K classes) for the higher-capacity experiments. For object detection, the backbones are implemented as drop-in replacements within standard detection frameworks (Cascade Mask R-CNN [29, 6], ATSS [79], RepPoints v2 [12], Sparse R-CNN [56]) in mmdetection [10]. For semantic segmentation, UperNet [69] in mmsegmentation [16] serves as the base framework.
Metrics. For ImageNet-1K: top-1 accuracy on a single center crop of the validation set. For COCO: box AP (average precision for bounding box detection) and mask AP (average precision for instance segmentation), following standard COCO metrics including AP₅₀ and AP₇₅. For ADE20K: mIoU (mean Intersection-over-Union) across 150 classes. Throughput is measured as images per second on a single V100 GPU using the GitHub repository of Wightman [68], following the protocol of DeiT [63].
Baselines. The paper compares against multiple architectural families across all three tasks. For ImageNet-1K classification (Table 1): ConvNets — RegNetY (4G, 8G, 16G) [48], EfficientNet (B3–B7) [58]; Transformers — ViT-B/16 and ViT-L/16 [20], DeiT-S and DeiT-B [63]; and ResNet/ResNeXt variants when pre-training on ImageNet-22K (ResNet-101x3, ResNet-152x4 [38]). For COCO object detection (Table 2): ConvNet backbones — ResNet-50, ResNeXt101-32x4d, ResNeXt101-64x4d within each detection framework; Transformer backbones — DeiT-S adapted with deconvolution layers to produce hierarchical feature maps (following SETR [81]); system-level baselines — RepPointsV2* [12], GCNet* [7], RelationNet++ [13], SpineNet-190 [21], ResNeSt-200* [78], EfficientDet-D7 [59], DetectoRS* [46], YOLOv4 P7* [4], Copy-paste [26], and a ResNeXt101-64x4d HTC++ baseline. For ADE20K segmentation (Table 3): ConvNet backbones — ResNet-101 with DANet [23], DeepLabV3+ [11], ACNet [24], DNL [71], OCRNet [73], UperNet [69]; HRNet-w48 with OCRNet [73]; ResNeSt-101 and ResNeSt-200 with DeepLabV3+ [11]; Transformer backbones — SETR with T-Large [81], DeiT-S with UperNet (adapted with deconvolution for hierarchy).
Generation budget / compute accounting. Computational cost is measured in three complementary ways across the three tasks to enable multi-faceted comparison: (1) FLOPs — theoretical floating-point operations for a single forward pass, computed from the architecture specifications (Table 1: Swin-T at 4.5G, Swin-S at 8.7G, Swin-B at 15.4G with 224² input); (2) Parameter count — total trainable parameters (Table 1: Swin-T 29M, Swin-S 50M, Swin-B 88M, Swin-L 197M); (3) Throughput — measured wall-clock inference speed in images/second on a V100 GPU using the benchmark repository of Wightman [68] (Table 1: Swin-T 755.2 img/s, Swin-S 436.9, Swin-B 278.1 at 224²). For detection and segmentation, FLOPs, parameters, and FPS are all reported per method-backbone pair (Tables 2 and 3). The paper is careful to compare models at similar computational budgets — Swin-T is positioned against DeiT-S and ResNet-50 (Table 1a), Swin-B against ViT-B/DeiT-B (Tables 1a and 1b), and throughput is measured using identical V100 hardware and identical benchmark code across all compared models.
Cross-validation / statistical protocol. The paper does not employ cross-validation or multiple training runs with error bars — results are reported as single numbers from deterministic training recipes. For ablation studies (Table 4), all three tasks are evaluated using the same pretrained Swin-T backbone, with only the architectural component being ablated changed while keeping the training recipe identical. For system-level comparisons on COCO and ADE20K, the paper uses multi-scale testing (inference on multiple input resolutions) as standard practice. For the ImageNet-22K pre-training experiments, models are first trained on ImageNet-22K with a fixed 90-epoch schedule and then fine-tuned on ImageNet-1K for 30 epochs, with results reported on the ImageNet-1K validation set.

Main Quantitative Results

Image Classification on ImageNet-1K

Regular ImageNet-1K training results (Table 1a). Swin Transformer variants achieve consistently higher accuracy than comparably-sized Transformer and ConvNet baselines under the standard 300-epoch training protocol:

Swin-T (29M parameters, 4.5G FLOPs) achieves 81.3% top-1 accuracy at 224² input, compared to DeiT-S (22M, 4.6G FLOPs) at 79.8% — a +1.5% improvement. Against ConvNets, RegNetY-4G (21M, 4.0G FLOPs) achieves 80.0% and EfficientNet-B3 (12M, 1.8G FLOPs) achieves 81.6% with a larger 300² input — the paper claims a "slightly better speed-accuracy trade-off" but the EfficientNet comparison involves different input resolutions and significantly lower FLOPs, making it not a like-for-like comparison.
Swin-S (50M, 8.7G FLOPs) achieves 83.0% at 224², compared to RegNetY-8G (39M, 8.0G FLOPs) at 81.7% and DeiT-B at 81.8% (though DeiT-B is larger at 86M/17.5G — the comparison is not FLOPs-matched; Swin-S is smaller and slightly more accurate).
Swin-B (88M, 15.4G FLOPs) achieves 83.5% at 224², compared to DeiT-B (86M, 17.5G FLOPs) at 81.8% — +1.7% improvement. At 384² input, Swin-B achieves 84.5% vs. DeiT-B at 83.1% — +1.4% improvement. The ViT-B/16 and ViT-L/16 baselines are substantially worse at 77.9% and 76.5% respectively, but these numbers are from models trained without the regularization strategies that DeiT introduced — the more relevant comparison is to DeiT, which shares the training recipe.

The ConvNet comparisons require careful reading. EfficientNet-B7 achieves 84.3% at 600² input with 66M parameters and 37.0G FLOPs — Swin-B at 384² achieves 84.5% with 88M/47.0G, so the accuracy is slightly higher but at a substantially different operating point (different input resolution, different parameter/FLOPs regime). The paper's claim of "slightly better speed-accuracy trade-off" is qualitative; no Pareto frontier analysis is presented.

ImageNet-22K pre-training results (Table 1b). Pre-training on the larger ImageNet-22K dataset (14.2M images, 22K classes) and fine-tuning on ImageNet-1K yields substantial improvements:

Swin-B at 224² achieves 85.2% — a gain of +1.7% over the from-scratch result (83.5%). At 384², Swin-B achieves 86.4%, which is 2.4% higher than ViT-B/16 (84.0%) at similar inference throughput (84.7 vs. 85.9 images/sec) and with slightly lower FLOPs (47.0G vs. 55.4G).
Swin-L (197M, 103.9G FLOPs) at 384² achieves 87.3%, surpassing ViT-L/16 (85.2%, 307M/190.7G) by +2.1% with substantially fewer parameters and FLOPs. The ResNet-based baselines pre-trained on larger datasets (ResNet-101x3 at 84.4%, ResNet-152x4 at 85.4%) are both heavier and less accurate.

Resolution scaling behavior (Table 8, Appendix). Across all three Swin variants, increasing input resolution monotonically improves accuracy at the cost of throughput: Swin-T ranges from 81.3% at 224² (755 img/s) to 82.2% at 384² (220 img/s); Swin-S from 83.0% at 224² (437 img/s) to 83.9% at 384² (128 img/s); Swin-B from 83.3% at 224² (278 img/s) to 84.5% at 384² (85 img/s). The accuracy gains from resolution increase diminish — Swin-T gains only +0.9% from 224² to 384² while throughput drops by 3.4×, suggesting that for Swin-T, higher resolution is not an efficient use of compute.

Object Detection and Instance Segmentation on COCO

Ablation study across detection frameworks (Table 2a). Using Swin-T as a drop-in replacement for ResNet-50 in four different detection frameworks (Cascade Mask R-CNN, ATSS, RepPoints v2, Sparse R-CNN) with identical training settings (3× schedule, multi-scale training, AdamW optimizer):

Cascade Mask R-CNN: Swin-T achieves 50.5 box AP / 43.7 mask AP vs. ResNet-50 at 46.3 / 40.1 — gains of +4.2 box AP / +3.6 mask AP.
ATSS: 47.2 / — vs. 43.5 / —, gain of +3.7 box AP.
RepPoints v2: 50.0 / — vs. 46.5 / —, gain of +3.5 box AP.
Sparse R-CNN: 47.9 / — vs. 44.5 / —, gain of +3.4 box AP.

The gains are consistent across all four frameworks (ranging from +3.4 to +4.2 box AP), which is a strong signal that the backbone improvement is genuine and not specific to a particular detection architecture. The Swin-T models have slightly more parameters than their ResNet-50 counterparts (e.g., 86M vs. 82M for Cascade Mask R-CNN) and slightly higher FLOPs (745G vs. 739G), but the accuracy gains far exceed what marginal parameter/FLOPs increases would typically provide.

Backbone scaling comparison (Table 2b). Using Cascade Mask R-CNN as the fixed framework, comparing Swin Transformer variants against ResNe(X)t at increasing capacity:

Swin-T (86M, 745G FLOPs, 15.3 FPS): 50.5 box AP / 43.7 mask AP. Compared to DeiT-S (80M, 889G, 10.4 FPS): 48.0 / 41.4 — Swin-T is +2.5 box AP / +2.3 mask AP higher, with fewer FLOPs and significantly higher throughput (15.3 vs. 10.4 FPS). The lower inference speed of DeiT is attributed to its quadratic complexity with respect to input image size — at the 800–1333 resolution range typical for detection, the quadratic term dominates.
Swin-S (107M, 838G, 12.0 FPS): 51.8 box AP / 44.7 mask AP vs. ResNeXt101-32x4d (101M, 819G, 12.8 FPS): 48.1 / 41.6 — gains of +3.7 box AP / +3.1 mask AP at comparable parameter counts, FLOPs, and throughput.
Swin-B (145M, 982G, 11.6 FPS): 51.9 box AP / 45.0 mask AP vs. ResNeXt101-64x4d (140M, 972G, 10.4 FPS): 48.3 / 41.7 — gains of +3.6 box AP / +3.3 mask AP. Notably, Swin-B is slightly faster than ResNeXt101-64x4d (11.6 vs. 10.4 FPS) despite slightly higher FLOPs, which the paper attributes to the ResNe(X)t being built on "highly optimized Cudnn functions" while Swin uses "built-in PyTorch functions that are not all well-optimized" — implying further speed improvements are possible with kernel-level optimization.

System-level comparison (Table 2c). Using an improved HTC++ framework (instaboost, stronger multi-scale training, 6× schedule, soft-NMS, extra global self-attention layer at the output of the last stage, ImageNet-22K pre-training):

The ResNeXt101-64x4d HTC++ baseline achieves 52.3 box AP / 46.0 mask AP on the mini-val set.
Swin-B (HTC++) achieves 56.4 box AP / 49.1 mask AP — gains of +4.1 box AP / +3.1 mask AP.
Swin-L (HTC++) achieves 57.1 box AP / 49.5 mask AP on mini-val and 58.7 box AP / 51.1 mask AP on test-dev (with multi-scale testing, denoted by * in the table), surpassing the previous state-of-the-art by +2.7 box AP over Copy-paste [26] (56.0 box AP, without external data) and +2.6 mask AP over DetectoRS [46] (48.5 mask AP). The Swin-L model uses 284M parameters and 1,470G FLOPs.

A notable detail: the improved HTC++ framework uses an "extra global self-attention layer appended at the output of the last stage" (Appendix A2.2). This means the final detection results are not from a pure Swin Transformer backbone in isolation — they include one global attention layer, which mitigates any residual limitations of the window-based approach for very long-range dependencies specific to detection. The paper does not ablate this addition, so it is unclear how much of the +4.1 box AP gain over the ResNeXt HTC++ baseline comes from the Swin backbone versus this extra attention layer.

Semantic Segmentation on ADE20K

Main results (Table 3). Using UperNet [69] as the base framework across all methods, with ImageNet-1K pretrained backbones unless otherwise noted:

Swin-T (60M, 945G FLOPs, 18.5 FPS): 46.1 mIoU vs. DeiT-S with UperNet (52M, 1099G, 16.2 FPS) at 44.0 — a gain of +2.1 mIoU with similar computational cost. ResNet-101 baselines with various segmentation heads (DANet, DeepLabV3+, ACNet, DNL, OCRNet, UperNet) range from 44.1 to 46.0 mIoU — Swin-T matches or exceeds all of them despite being a relatively small model.
Swin-S (81M, 1038G, 15.2 FPS): 49.3 mIoU — a substantial jump of +3.2 mIoU over Swin-T, and +5.3 mIoU over DeiT-S. Compared to ConvNet backbones: +4.4 mIoU over ResNet-101 (44.9 with UperNet), +2.4 mIoU over ResNeSt-101 (46.9 with DeepLabV3+). HRNet-w48 with OCRNet achieves 45.7 mIoU — Swin-S is +3.6 mIoU higher.
Swin-B (121M, 1841G, 8.7 FPS) with ImageNet-22K pre-training and 640×640 input: 51.6 mIoU. This surpasses SETR with T-Large (308M parameters) at 50.3 mIoU by +1.3 mIoU, despite having less than 40% of the parameters. ResNeSt-200 at 48.4 mIoU is substantially lower.
Swin-L (234M, 3230G, 6.2 FPS) with ImageNet-22K pre-training and 640×640 input: 53.5 mIoU on the validation set, 62.8 mIoU on the test set (when training with both training and validation images, following common practice [71]). This surpasses the previous best SETR (50.3 mIoU) by +3.2 mIoU and DeepLabV3+ with ResNeSt-200 (48.4) by +5.1 mIoU.

The mIoU improvements are larger than the classification improvements in both absolute and relative terms. This supports the paper's central argument that the hierarchical design specifically benefits dense prediction tasks — classification can tolerate low-resolution, single-scale features, but segmentation fundamentally requires multi-scale feature pyramids, which Swin provides and ViT/DeiT do not.

Speed Comparison of Self-Attention Methods (Tables 5 and 6)

Hardware latency analysis (Table 5). The paper benchmarks the real-world speed of different self-attention computation methods on a V100 GPU, measuring inference time per self-attention module in milliseconds for each of the four stages, and overall architecture throughput in images/second for Swin-T, Swin-S, and Swin-B:

Sliding window (naive implementation): Stage 1 MSA takes 122.5 ms — this is the high-resolution stage with many tokens, where per-pixel sliding window attention requires loading different key-value sets for each query position, causing catastrophic memory access patterns. Overall Swin-T throughput: 183 images/sec. With kernel-level optimization ("sliding window (kernel)"): Stage 1 drops to 7.6 ms, throughput improves to 488 img/s — still substantially slower than the shifted window approach.
Performer [14] (kernel-based linear attention approximation): Stage 1: 4.8 ms; throughput 638 img/s (Swin-T). This is competitive but still slower than the shifted window approach, and Table 6 shows it achieves substantially lower accuracy (79.0% vs. 81.3% on ImageNet-1K).
Window-based (w/o shifting, regular partitioning only): Stage 1: 2.8 ms; throughput 770 img/s. This is the fastest variant because each window is an independent, batchable self-attention computation with no masking or shifting overhead. However, Table 4 shows the performance penalty for removing shifting: −1.1% top-1, −2.8 box AP, −2.8 mIoU.
Shifted window (naive padding): Stage 1: 3.3 ms; overall throughput 670 img/s for Swin-T. The increase in window count (from 8×8 to 9×9 for 56×56 feature maps with M=7, a 1.27× increase — the paper's 2×2 → 3×3 example applies to smaller feature maps) adds computation.
Shifted window (cyclic, the proposed method): Stage 1: 3.0 ms; throughput 755 img/s for Swin-T — within 2% of the non-shifted window approach (770 img/s), while providing the cross-window connectivity that yields the accuracy gains in Table 4. The cyclic approach provides a 13%, 18%, and 18% speedup over the naive padding approach for Swin-T, Swin-S, and Swin-B, respectively.

Accuracy comparison across methods (Table 6). The shifted window approach achieves 81.3% top-1 on ImageNet-1K, 50.5 box AP/43.7 mask AP on COCO, and 46.1 mIoU on ADE20K, compared to sliding window (81.4% / 50.2 / 43.5 / 45.8) and Performer (79.0% / — / — / —). The efficiency-accuracy trade-off decisively favors the shifted window approach: it matches sliding window accuracy (differences are within 0.1% on classification and 0.3 box AP on detection) while being 4.1× faster (755 vs. 183 img/s for Swin-T), and it outperforms Performer by 2.3% top-1 accuracy while being slightly faster (755 vs. 638 img/s).

Swin-Mixer: Generalizing Beyond Attention (Table 10, Appendix)

The paper applies the hierarchical design and shifted window approach to the MLP-Mixer architecture [61], replacing self-attention with token-mixing MLPs within windows while preserving the four-stage hierarchy with patch merging and alternating regular/shifted window partitioning:

Original MLP-Mixer-B/16 [61]: 59M parameters, 12.7G FLOPs, 76.4% top-1 on ImageNet-1K.
Swin-Mixer-T/D24: 20M parameters, 4.0G FLOPs, 79.4% top-1 — already 3.0% higher than the larger MLP-Mixer with 3× fewer parameters.
Swin-Mixer-B/D24: 61M parameters, 10.4G FLOPs, 81.3% top-1 — +4.9% over MLP-Mixer with lower FLOPs (10.4G vs. 12.7G).
Swin-Mixer-B/D24 without shifting: 61M, 10.4G FLOPs, 80.3% — a 1.0% drop, confirming that window shifting benefits MLP-based architectures as well.
Comparison to ResMLP [62]: ResMLP-B24 (116M, 23.0G) achieves 81.0% — Swin-Mixer-B/D24 achieves 81.3% with roughly half the parameters and FLOPs.
Swin Transformer comparison: Swin-T/D24 (Transformer-based) achieves 81.6% at 256² input with 28M/5.9G — 0.3% higher than Swin-Mixer-T/D24 (79.4%) but the Swin-Mixer is smaller (20M/4.0G). At comparable scale (Swin-Mixer-B/D24 vs. Swin-T), the gap is 81.3% vs. 81.3% at 224² (Table 1), and 81.3% vs. 81.6% at 256² (Table 10).

The key finding is that the hierarchical shifted-window design recovers most of the performance gap between MLP-Mixer and Swin Transformer architectures, demonstrating that the structural principles (hierarchy, locality, cross-window connectivity) are more important than the specific choice of token-mixing mechanism (attention vs. MLPs) for vision backbone performance.

Ablation Studies and Robustness Checks

Shifted window partitioning (Table 4). Removing the shifted window partition (all self-attention modules use regular, non-shifted window partitioning throughout the network, denoted "w/o shifting"): ImageNet-1K drops from 81.3% to 80.2% (−1.1%); COCO box AP drops from 50.5 to 47.7 (−2.8 box AP), mask AP from 43.7 to 41.5 (−2.2 mask AP); ADE20K mIoU drops from 46.1 to 43.3 (−2.8 mIoU). The larger degradation on dense prediction tasks (−2.8 on both COCO and ADE20K, vs. −1.1 on classification) suggests that cross-window connectivity is especially important for tasks requiring spatially coherent predictions, where restricting attention to fixed, non-overlapping windows prevents the model from integrating information across object boundaries or large regions.

Position encoding schemes (Table 4). Four variants are compared, all on Swin-T:

No position encoding: 80.1% ImageNet-1K, 49.2 box AP, 42.6 mask AP, 43.8 mIoU.
Absolute position embedding (as in ViT, added to the input tokens): 80.5% (+0.4% over no position), 49.0 box AP (−0.2), 42.4 mask AP (−0.2), 43.2 mIoU (−0.6). The task-dependent effect is notable: absolute position encoding helps classification but hurts detection and segmentation.
Absolute + relative position bias: 81.3% (+1.2% over no position, +0.8% over absolute only), 50.2 box AP (+1.0 over no position, +1.2 over absolute only), 43.4 mask AP (+0.8 over no position, +1.0 over absolute only), 44.0 mIoU (+0.2 over no position, +0.8 over absolute only). Adding absolute position on top of relative bias slightly degrades performance compared to relative alone (50.2 vs. 50.5 box AP, 44.0 vs. 46.1 mIoU), further reinforcing the finding that absolute position information is detrimental for dense tasks.
Relative position bias only (the default): 81.3%, 50.5 box AP, 43.7 mask AP, 46.1 mIoU — the best or tied-for-best on all metrics. The gains over no position encoding are substantial: +1.2% top-1, +1.3 box AP, +1.1 mask AP, +2.3 mIoU.
Relative position bias without the content-based dot-product term ("rel. pos. w/o app." — attention logits depend only on B, not on QK^T/√d): 79.3% (−2.0% vs. full relative), 48.2 box AP (−2.3), 41.9 mask AP (−1.8), 44.1 mIoU (−2.0). This confirms that both content-based similarity and position-based bias are necessary, and neither alone is sufficient.

Optimizer choice for ResNe(X)t baselines on COCO (Table 9, Appendix). The paper found that the standard SGD optimizer used in Cascade Mask R-CNN configurations produces suboptimal results for ResNe(X)t backbones, and replacing it with AdamW improves performance: ResNet-50 improves from 45.0 to 46.3 box AP (+1.3); ResNeXt101-32x4d improves from 47.8 to 48.1 (+0.3); ResNeXt101-64x4d improves from 48.8 to 48.3? No — actually, ResNeXt101-64x4d with SGD achieves 48.8 box AP and with AdamW achieves 48.3 — a degradation of −0.5. This is an anomaly that the paper does not explain: for the largest ResNeXt variant, AdamW performs worse than SGD. The paper states they "use AdamW for ResNe(X)t backbones when compared to the proposed Swin Transformer architectures," which means they chose the stronger optimizer for the comparison, making the baselines as strong as possible. The fact that the largest ResNeXt backbones perform better with SGD is noted but not investigated further.

Different self-attention methods (Table 6). The paper compares Swin Transformer variants built on three self-attention paradigms:

Sliding window self-attention on Swin-T: 81.4% top-1, 50.2 box AP, 43.5 mask AP, 45.8 mIoU — nearly identical to the shifted window approach (81.3%, 50.5, 43.7, 46.1), showing that the two methods have similar modeling power. However, Table 5 shows the shifted window approach is 4.1× faster (755 vs. 183 img/s for Swin-T).
Performer on Swin-T: 79.0% top-1 — significantly lower, with detection and segmentation not reported (denoted by "—" in Table 6). At the same time, Performer is slower than shifted windows (638 vs. 755 img/s for Swin-T, Table 5).

The implication is that exact local attention with structural cross-window connectivity (shifted windows) is both more accurate and faster than either approximate global attention (Performer) or exact local attention with sliding windows (which is accurate but slow), making it the Pareto-optimal choice among the three.

Image resolution scaling (Table 8, Appendix). Swin Transformers trained at 224² and evaluated at different test resolutions:

Swin-T: 81.3% (224²) → 81.6% (256²) → 82.1% (320²) → 82.2% (384²). Gains diminish rapidly — the jump from 224² to 256² gives +0.3%, but from 320² to 384² gives only +0.1%.
Swin-S: 83.0% → 83.4% → 83.7% → 83.9%.
Swin-B: 83.3% → 83.7% → 84.0% → 84.5%. The gain is larger for Swin-B (+1.2% from 224² to 384²) than for Swin-T (+0.9%) or Swin-S (+0.9%), suggesting that larger models benefit more from higher-resolution inputs.

Throughput drops dramatically: Swin-T at 224² processes 755.2 images/sec but only 219.5 at 384² (3.4× slower). The paper's fine-tuning protocol for higher resolutions uses 30 epochs with a constant learning rate of 10⁻⁵, starting from the 224² pretrained checkpoint, which is more efficient than training from scratch at each resolution.

Data scaling (ImageNet-1K vs. ImageNet-22K, Tables 1a vs. 1b). ImageNet-22K pre-training provides substantial benefits for larger models: Swin-B improves from 83.3% to 85.2% at 224² (+1.9%), and to 86.4% at 384² (+1.9% over the 224²-22K result, +2.9% over the 224²-1K result). Swin-L achieves 87.3% with 22K pre-training — no from-scratch ImageNet-1K result is reported for Swin-L, likely because training a 197M parameter model on 1.28M images leads to overfitting.

Three-benchmark consistency. The ablation study in Table 4 evaluates every architectural change across all three tasks (classification, detection, segmentation). The consistent ranking of design choices — shifted windows > no shifting, relative position > absolute position > no position — across all three benchmarks is a strong robustness check that the findings are not task-specific.

Critical Assessment

Do the experiments genuinely support the claim that Swin Transformer is a "general-purpose backbone"? Yes, with qualifications about what "general-purpose" means in this context. The paper demonstrates that a single architectural design — hierarchical feature maps through patch merging, windowed self-attention with shifted windows, and relative position bias — achieves state-of-the-art or competitive results across three canonical vision tasks (classification, detection, segmentation) without task-specific architectural modifications to the backbone. This is the definition of a general-purpose backbone. However, the evaluation covers only static image recognition tasks — the paper does not test on video understanding, 3D vision, medical imaging, or other domains where ConvNet backbones are routinely used. "General-purpose" in the vision community typically implies these three tasks (classification, detection, segmentation), and Swin satisfies that definition, but the term is broader than what is empirically validated here.

Does the paper demonstrate that the shifted window approach specifically enables the strong results? Yes, through the ablation in Table 4. Removing shifting reduces performance by 1.1% on ImageNet, 2.8 box AP on COCO, and 2.8 mIoU on ADE20K — large, consistent degradations that would eliminate most of Swin's advantage over baselines. The comparison in Table 6 shows that sliding window self-attention achieves similar accuracy to shifted windows, but Tables 5 and 6 together show that the shifted window approach achieves this accuracy while being 4.1× faster, which is the key efficiency claim. The cyclic batch computation trick is validated in Table 5: it provides a 13–18% throughput improvement over naive padding for the shifted window case, confirming that the engineering contribution is meaningful beyond the conceptual one.

A missing baseline that would strengthen the paper: A comparison of Swin Transformer against a ConvNet backbone in a controlled setting where both have the same parameter count and FLOPs, trained with the same recipe and evaluated on the same detection/segmentation frameworks. While Tables 2b and 3 provide comparisons with similar-scale ResNe(X)t models, the training recipes differ — the paper uses AdamW for ResNe(X)t backbones after finding that SGD underperforms (Table 9), but it is unclear whether the Swin models would benefit similarly from SGD or whether the comparison is confounded by optimizer choice. A more rigorous ablation would train both architectures with identical optimization hyperparameters, data augmentation, and regularization to isolate the architectural contribution.

The single-model-family limitation. All experiments use the Swin architecture variants (T, S, B, L) as the Transformer backbones. There is no comparison to other hierarchical vision Transformers (e.g., PVT by Wang et al., which was concurrent work) or to ConvNet-Transformer hybrids (e.g., BoTNet, CoAtNet, which appeared later). This is partly a timing issue — many of these were published simultaneously or shortly after — but it means the paper cannot claim that Swin's specific instantiation (patch merging + shifted windows) is uniquely effective among hierarchical Transformer designs, only that it is effective.

The detection system-level results include an extra component not ablated. The HTC++ system-level comparison (Table 2c) uses an "extra global self-attention layer appended at the output of the last stage" (Appendix A2.2). This means the reported 58.7 box AP / 51.1 mask AP is not from a pure Swin Transformer backbone — it includes an additional architectural component that may compensate for any residual limitations of the purely window-based design in capturing very long-range dependencies relevant to detection. The paper does not ablate this global attention layer, so its contribution to the final performance is unknown. The cost is likely small (one global attention layer on a coarse feature map), but its presence means the system-level results are not a direct measure of the backbone's standalone capability.

The ADE20K segmentation results use different input sizes for different models. Swin-T and Swin-S are trained at 512×512, while Swin-B and Swin-L (with ImageNet-22K pre-training) are trained at 640×640 (Table 3, and Appendix A2.3). The higher input resolution for larger models inflates their mIoU relative to what they would achieve at 512×512, making the scaling comparison (Swin-T → Swin-S → Swin-B → Swin-L) confounded by both model capacity and input resolution. The paper reports multi-scale testing for all models, but the training resolution difference means the models were optimized for different operating points. This is a practical choice — larger models can afford higher-resolution inputs within GPU memory constraints — but it means the reported mIoU improvements are not purely architectural.

The FLOPs-matched comparison to ConvNets is imprecise. Table 1a shows Swin-T (29M, 4.5G) vs. DeiT-S (22M, 4.6G) — a 32% difference in parameter count at similar FLOPs, going in Swin's favor. Swin-B (88M, 15.4G) vs. DeiT-B (86M, 17.5G) — a 12% difference in FLOPs at similar parameter count, also in Swin's favor. The comparisons to EfficientNet involve different input resolutions (224² vs. up to 600²) and very different parameter/FLOPs regimes, making direct FLOP-matched comparison impossible. The paper's qualitative claim of "slightly better speed-accuracy trade-off" is reasonable given the scatter of points, but no formal trade-off curve (envelope or frontier) is computed to justify it rigorously.

The ImageNet-22K pre-training results are from a two-stage process that benefits Swin more than ViT. The comparison in Table 1b shows Swin-B at 86.4% vs. ViT-B/16 at 84.0% after ImageNet-22K pre-training. However, ViT's architecture was originally designed for JFT-300M pre-training (300M images), and its performance on the smaller ImageNet-22K (14.2M images) may not reflect its full potential with large-scale data. The paper does not include a ViT baseline pre-trained on JFT-300M for comparison. The concern is that Swin's advantage may partially reflect that its architectural inductive biases make it more data-efficient on medium-scale pre-training (14.2M images), rather than that its architecture is fundamentally superior to ViT given unlimited data. The DeiT results (Table 1a) partially address this — Swin outperforms DeiT by 1.5–1.7% on ImageNet-1K from scratch — but the comparison to ViT with larger pre-training datasets remains incomplete.

The Swin-Mixer experiment (Table 10) is a powerful demonstration of generalization but would benefit from detection/segmentation results. The paper shows that the hierarchical shifted-window design improves MLP-Mixer by +4.9% on ImageNet-1K, but does not report detection or segmentation results for Swin-Mixer. The core claim is that these architectural primitives are general-purpose — showing that they improve dense prediction tasks for non-attention architectures as well would substantially strengthen this claim. The absence of these experiments is likely due to scope and computational constraints, but it means the generalization argument relies on classification evidence alone.

The paper's claim about replacing ConvNets as general-purpose backbones implicitly assumes that the tasks evaluated (classification, detection, segmentation) represent the full range of "general-purpose vision." This is standard in the vision backbone literature — ResNet, ResNeXt, and EfficientNet were all evaluated on these same three tasks — but there are important vision tasks where ConvNets remain dominant and Swin has not been tested: video action recognition, 3D object detection, point cloud processing, optical flow, depth estimation, and generative modeling. The paper's claim is appropriately scoped to the tasks it evaluates, but readers should understand that "general-purpose" refers to these specific tasks.

Overall, the experimental evidence supports the paper's central claims: the hierarchical design and shifted window mechanism enable a Transformer architecture to serve as an effective backbone across classification, detection, and segmentation, surpassing both ConvNet and ViT baselines of comparable scale. The ablation studies cleanly isolate the contribution of each design element (shifting, relative position bias, cyclic batch computation). The main weaknesses are (1) the system-level detection results include an un-ablated global attention layer, (2) the FLOPs/parameter matching to baselines is approximate rather than exact, (3) the domain of "general-purpose" is limited to 2D image recognition, and (4) the comparison to ViT with truly large-scale pre-training (JFT-300M) is missing.

6. Limitations and Trade-offs

The Quadratic Cost of Global Self-Attention Is Avoided, Not Solved — Very Long-Range Dependencies Are Blocked by Window Boundaries

The assumption or constraint. The Swin Transformer achieves linear complexity by restricting self-attention to fixed-size, non-overlapping local windows ( $M \times M$ , with $M=7$ by default). Patches in different windows cannot directly attend to each other in any single layer — cross-window communication relies entirely on the shifted window mechanism propagating information through the network depth, one window-hop at a time. This design assumes that information that truly requires long-range, direct pairwise attention between distant patches can be adequately captured through the chain of intermediate, indirect interactions that stacking shifted-window blocks provides.

The paper does not make a formal claim about receptive field size — it states only that the shifted window approach "introduces connections between neighboring non-overlapping windows in the previous layer" (Section 3.2) and demonstrates empirically that this helps (Table 4: +1.1% top-1, +2.8 box AP, +2.8 mIoU over no shifting). However, the paper never quantifies how many layers are needed for a patch at one corner of the image to influence a patch at the opposite corner, or what information is lost along the way.

The consequence. For tasks that require modeling dependencies between distant image regions that are not mediated by intermediate visual structure — for instance, a scene where an object at the top-left must be matched against a semantically related but spatially remote object at the bottom-right, with no intermediate visual cues connecting them — the windowed approach introduces a bottleneck. Information must traverse multiple window boundaries, passing through many intermediate representations, and each hop adds representational noise. The effective receptive field grows linearly with the number of blocks (roughly $M/2$ new pixels visible in each direction per shifted-window pair), meaning that for a $56 \times 56$ feature map (Stage 1 of Swin-T at $224^2$ input), a patch at position $(0,0)$ requires approximately $\lceil 56 / 3.5 \rceil \approx 16$ Swin Transformer blocks (8 W-MSA/SW-MSA pairs, if the shift of $\lfloor M/2 \rfloor = 3$ contributes 3.5 pixels per block pair on average) before its theoretical receptive field covers the entire feature map. Swin-T has only 12 blocks total across all stages — 2 in Stage 1, 2 in Stage 2, 6 in Stage 3, 2 in Stage 4 — and the receptive field resets after each patch merging (which spatially compresses tokens, changing their effective field). This means that at the finest resolution (Stage 1), the theoretical receptive field may not cover the full image by the time the tokens are merged to Stage 2. In practice, the effective receptive field (the subset of the theoretical field that actually influences the output) is typically much smaller than the theoretical maximum, making it plausible that some long-range dependencies at fine spatial scales are genuinely unreachable.

For classification, this limitation is mild — global average pooling before the classifier can aggregate information from all positions regardless of whether they directly attended to each other. For dense prediction tasks, the limitation is potentially more serious: a segmentation model needs to classify each pixel based on global image context, and if the encoder cannot propagate information from a distant but relevant region to the current pixel's feature vector, the prediction must rely on information that passes through the decoder (which operates on multi-scale features but may have its own limitations). The paper provides no direct analysis of this issue, but the system-level detection results (Table 2c) include an "extra global self-attention layer appended at the output of the last stage" (Appendix A2.2) — a component added to the HTC++ framework specifically for the final comparison. This addition is not ablated, and its presence suggests that even with the Swin backbone, some form of global attention remained beneficial for detection, hinting that purely window-based encoding may not be sufficient at the coarsest scale for this task.

What evidence exists in the paper. The paper does not directly measure or analyze receptive field growth. The ablation in Table 4 demonstrates that shifting helps significantly (and thus that pure intra-window attention is insufficient), but it does not quantify whether $M=7$ with the default layer depths achieves sufficient receptive field coverage for the tasks evaluated. The throughput comparison in Table 5 shows that larger windows are not a solution (window-based without shifting is the fastest but least accurate variant) and sliding windows are prohibitively slow. Table 6 shows that sliding-window self-attention (which provides per-pixel connectivity to all patches in a $7 \times 7$ window, but with a different key set per query) achieves similar accuracy (81.4% vs. 81.3%, 50.2 vs. 50.5 box AP) — but Table 5 shows it is 4.1× slower. The accuracy parity suggests that the global receptive field achieved through stacking shifted-window blocks is comparable to the sliding-window approach, but neither is directly compared to a global-attention model that would provide an upper bound on what is achievable with unrestricted long-range connections.

Mitigation status. The paper does not attempt to solve the receptive field limitation. The shifted window mechanism is presented as a way to introduce cross-window connectivity, not as a complete solution to long-range dependency modeling. The paper acknowledges (implicitly, via the extra global self-attention layer in the HTC++ framework) that some tasks may benefit from explicit global attention at coarser scales, but this addition is not part of the core Swin Transformer architecture and is not ablated. Subsequent work (Swin Transformer V2, CSWin, Focal Transformer) addresses this by introducing mechanisms for varying window sizes, cross-shaped windows, or selective global attention — confirming that the receptive field limitation is real and motivated subsequent research. A practitioner deploying Swin Transformer should consider whether their application requires modeling dependencies that span distances larger than the effective receptive field (which depends on the input resolution, window size, and number of Swin blocks in the relevant stages).

Window Size Implications: The Fixed $M=7$ Is Tuned for $224^2$ Classification and Not Re-Optimized for Dense Prediction or Higher Resolutions

The assumption or constraint. The window size $M = 7$ is fixed across all model variants (Swin-T, Swin-S, Swin-B, Swin-L) and all tasks (classification, detection, segmentation). The paper states this as a default in Section 3.3: "The window size is set to $M = 7$ by default." The architectural specifications in Table 7 confirm that every Swin Transformer block in every stage uses win. sz. 7×7.

This constant is introduced alongside the complexity analysis (Equation 2, Section 3.2) and is implicitly justified by the tradeoff it represents: larger windows provide more context per attention layer but increase computation quadratically in $M$ (the $2M^2 hwC$ term), while smaller windows are cheaper but require more layers of shifting to achieve the same receptive field. The value $M=7$ is cleanly divisible into $56 \times 56$ (Stage 1 at $224^2$ input: $56/7 = 8$ windows per side) and $28 \times 28$ (Stage 2: $28/7 = 4$ windows), making it convenient for the default ImageNet resolution. However, the paper provides no systematic study of how this choice affects performance, no sweep over alternative window sizes, and no evidence that $M=7$ is optimal for any task.

The consequence. For object detection, the typical input resolution during multi-scale training spans from $480$ to $800$ on the shorter side, with the longer side up to $1333$ (Appendix A2.2). At a $4\times$ patch size, Stage 1 feature maps range from roughly $120 \times 200$ to $200 \times 333$ patches. A $7 \times 7$ window covers a very small fraction of this feature map — at the higher resolution, the window covers only $7/333 \approx 2.1\%$ of the width. This means that even after several shifted-window block pairs, the effective receptive field at Stage 1 may cover only a small local region, and patches that are more than a few windows apart in the feature map can only interact after being merged into Stage 2 (where resolution is halved and the window covers a proportionally larger area, but spatial precision is lost). For semantic segmentation at $512 \times 512$ or $640 \times 640$ inputs (Appendix A2.3), the Stage 1 feature map is $128 \times 128$ or $160 \times 160$ — the $7 \times 7$ window spans $7/128 \approx 5.5\%$ of the feature map, again covering a small fraction.

The fixed window size also creates a tension: at higher input resolutions, the model processes more patches per window (since the window size is fixed) and therefore the per-window attention cost increases linearly with $hw$ (Equation 2), but the proportion of the image captured in each window decreases with resolution. This means the model sees more fine-grained detail within each window but has a smaller field of view relative to the image — a tradeoff that might be suboptimal for tasks where long-range context at fine spatial scale matters. The paper's resolution scaling experiments (Table 8) show diminishing accuracy returns from higher resolution — Swin-T gains only +0.9% top-1 from $224^2$ to $384^2$ — but this is for classification where global average pooling aggregates information across the entire feature map. For detection and segmentation, the effect of window size relative to feature map size is not analyzed.

What evidence exists in the paper. The paper contains no ablation study varying $M$ . Table 5 provides latency measurements for the default $M=7$ and for alternative self-attention mechanisms (sliding window, Performer) but does not sweep alternative window sizes. The complexity equations (Equations 1–2) show mathematically that $M$ controls the tradeoff, but the paper never empirically validates that $M=7$ is a reasonable or optimal choice. The cross-window shifting mechanism was specifically designed to work with this fixed window size (shift amount is $\lfloor M/2 \rfloor$ , and the cyclic shift computation assumes a consistent $M$ across all windows in a stage), and there is no discussion of what would happen with variable-sized windows or per-stage window size tuning.

Mitigation status. The paper does not address this limitation, and it is not flagged as a concern. A practitioner should be aware that $M=7$ was almost certainly chosen for convenience (it divides cleanly into $56 \times 56$ , the Stage 1 resolution at $224^2$ ) and that the optimal window size for their specific input resolution and task may differ. The paper notes that "the learnt relative position bias in pre-training can be also used to initialize a model for fine-tuning with a different window size through bi-cubic interpolation" (Section 3.2), which suggests that the architecture can be adapted to different $M$ values with only the position bias matrix needing interpolation. However, changing $M$ would also change the computational cost (linearly in $M^2$ ) and the effective receptive field per layer, neither of which is explored.

The Difficulty Estimation Bottleneck: Hierarchical Structure Assumes Compatibility with Frameworks That Are Designed Around ConvNet Feature Pyramids — and No Alternative Is Evaluated

The assumption or constraint. The Swin Transformer's hierarchical feature maps are explicitly designed to match the spatial resolutions produced by standard ConvNet backbones (e.g., ResNet's conv2/conv3/conv4/conv5 outputs at strides 4, 8, 16, 32), as stated in Section 3.1: "These stages jointly produce a hierarchical representation, with the same feature map resolutions as those of typical convolutional networks, e.g., VGG and ResNet. As a result, the proposed architecture can conveniently replace the backbone networks in existing methods for various vision tasks." The patch merging operation (concatenating $2 \times 2$ neighboring patches and projecting) is a specific choice of downsampling that produces exactly these resolutions and channel expansion ratios (doubling channels at each resolution reduction).

This design decision prioritizes compatibility with existing detection and segmentation frameworks (FPN, U-Net, etc.) over exploration of alternative, possibly more effective, multi-scale Transformer architectures. It presumes that the ConvNet resolution pyramid (4×, 8×, 16×, 32× downsampling) is the correct structure for a vision backbone. The paper does not test whether a different number of stages, different downsampling ratios, different channel expansion factors, or a continuous resolution representation (rather than discrete stages) would perform better. The number of Swin Transformer blocks per stage (e.g., $\{2, 2, 6, 2\}$ for Swin-T) follows the ConvNet convention of concentrating computation at intermediate resolutions (Stage 3), but no ablation justifies this distribution.

The consequence. By slavishly matching ConvNet resolution pyramids, the Swin Transformer may inherit architectural suboptimalities that arose from the specific constraints of convolution rather than from what is best for self-attention. For example, ConvNets use 2×2 strided convolution or pooling for downsampling because it is computationally natural for convolution; a Transformer could use any token reduction mechanism (learned pooling, attention-based merging, token dropping) and is not constrained to halving resolution and doubling channels. The paper's choice of patch merging (concatenate 4 neighbors + linear projection) is a design choice, not a necessity, and no alternatives are compared.

More concretely, the fixed resolution pyramid means that at the finest scale (Stage 1, $H/4 \times W/4$ ), the model always operates on a dense token grid — even for tasks or images where this high resolution is unnecessary or wasteful. A backbone that could dynamically adjust its hierarchy (e.g., allocating more tokens to regions of interest and fewer to background) could be more efficient, but Swin commits to a static, ConvNet-like structure. The Swin-Mixer experiment (Table 10) shows that the hierarchical structure generalizes to non-attention architectures, but it does not test whether an alternative structure would be even better.

What evidence exists in the paper. The paper provides no ablation of the hierarchical structure itself — no comparison of different numbers of stages, different downsampling strategies, or alternative multi-scale representations. The architecture variants (Table 7) modify only the channel dimension ( $C$ ) and the number of Transformer blocks per stage (doubling Stage 3 from 6 to 18 for Swin-S, Swin-B, Swin-L), keeping the stage structure and resolution pyramid identical. The paper states that the structure matches "typical convolutional networks" and treats this as self-evidently desirable, but it provides no evidence that this specific hierarchy is better than alternatives for Transformer backbones.

Mitigation status. The paper does not flag this as a limitation or suggest future work on alternative hierarchies. The Swin-Mixer results (Table 10) partially validate that the hierarchical design is beneficial beyond attention (MLP-Mixer without hierarchy achieves 76.4%; with hierarchy, 81.3%), but this shows only that some hierarchy is better than none, not that the ConvNet-matching hierarchy is optimal. The practical consequence for a practitioner is that Swin works well with existing frameworks (the compatibility benefit is real and demonstrated in Tables 2 and 3), but there is no reason to believe it represents the best possible multi-scale Transformer design — a point that subsequent work (PVTv2, CSWin, ConvNeXt, SwinV2 with different block allocations) has confirmed by exploring variations on the hierarchical template.

Verifier Over-Optimization Analogue: The Relative Position Bias Is Not Truly Translation-Invariant — and the Paper's Own Evidence Shows It Matters for Dense Tasks

The assumption or constraint. The Swin Transformer uses a relative position bias $B \in \mathbb{R}^{M^2 \times M^2}$ , parameterized through a learned matrix $\hat{B} \in \mathbb{R}^{(2M-1) \times (2M-1)}$ , that is added to the attention logits before SoftMax (Equation 4). Because this bias depends only on the relative displacement $(\Delta x, \Delta y)$ between two patches, it is shift-invariant within a window: shifting the entire window content within the window boundaries does not change the relative positions between patches, so the bias terms are unchanged. The paper presents this as a form of translation invariance (Section 4.4): "inductive bias that encourages certain translation invariance is still preferable for general-purpose visual modeling."

However, this invariance is strictly local. The window partitioning itself is aligned to an absolute grid — the windows are defined by dividing the feature map into non-overlapping $M \times M$ blocks starting from the top-left corner (or from a fixed shift offset in the shifted configuration). A patch's window membership depends on its absolute position. Two patches in different windows cannot attend to each other regardless of their relative distance, while two patches in the same window can attend even if they are near opposite edges of the window. This means the architecture is not globally translation-equivariant: shifting the entire image by a small amount can move a patch from one window into another, changing its set of attention partners and potentially altering the network's output in ways that a truly translation-equivariant model (like a ConvNet with valid padding) would not.

The consequence. This property manifests differently across tasks. For ImageNet classification with center-cropped evaluation, the model sees the object roughly centered in the frame, and small translation variations during training augmentations are handled through the shift-invariance within windows and the cross-window connections from shifting. The classification accuracy gains from relative position bias (+1.2% over no position encoding, Table 4) show that it helps. For object detection and semantic segmentation, where objects can appear at arbitrary positions in the image, the lack of global translation equivariance means that an object's feature representation depends not only on its appearance but also on which windows it falls into — a subtle interaction between object position and the fixed window grid.

The paper's own position encoding ablation (Table 4) provides indirect evidence that this is an issue: absolute position embeddings, which explicitly encode absolute spatial location, hurt detection and segmentation (−0.2 box AP, −0.6 mIoU) while helping classification (+0.4% top-1). This suggests that detection and segmentation are sensitive to position encoding schemes, and that the window-grid-alignment effect could matter. However, the relative position bias does not guarantee global translation equivariance — it only guarantees shift-invariance within the fixed window partition. The paper does not measure whether moving an object by a small number of pixels (less than the window size $M=7$ ) changes detection or segmentation predictions.

What evidence exists in the paper. Table 4 compares no position encoding, absolute position, relative position bias, and the combination. The evidence strongly favors relative position bias over both alternatives, particularly for dense tasks (46.1 vs. 43.2 mIoU for absolute position, +2.9 mIoU). However, no experiment measures the sensitivity of predictions to small image translations. A standard test for translation equivariance/invariance would be to apply the model to a translated version of the same image and measure the consistency of predictions (or the shift in feature maps) — this is not done. The paper implicitly acknowledges the limitation of absolute position by noting that ViT/DeiT "abandon translation invariance in image classification" and that this is unsuitable for dense tasks, but it does not complete the logical loop by verifying that the relative position bias fully restores translation-equivariant behavior.

Mitigation status. The paper does not claim to achieve full translation equivariance and does not evaluate whether the window-grid effect causes measurable sensitivity to translations. This limitation is underexplored and unaddressed. Subsequent work (e.g., SwinV2 with log-spaced continuous position bias, or CSWin with cross-shaped windows) has explored alternative position encoding schemes that aim for better translation-equivariant properties, suggesting that this limitation was recognized by the community if not articulated in the original paper. For practitioners, the takeaway is that Swin with relative position bias is substantially better than absolute position embeddings for dense tasks, but it does not achieve the mathematical translation equivariance of a ConvNet with valid convolution — a property that may matter for tasks requiring precise spatial localization under image shifts.

Computational Cost of the Hierarchical Feature Pyramid at High Resolutions Is Not Fully Accounted For — and the Linear Complexity Claim Applies to Windowed Attention Only

The assumption or constraint. The paper's central efficiency claim is that Swin Transformer has "linear computational complexity to input image size" (Section 1, Figure 1). This claim is grounded in Equation 2, which shows that window-based multi-head self-attention has complexity $\Omega(\text{W-MSA}) = 4hwC^2 + 2M^2 hwC$ , which is linear in $hw$ when $M$ is fixed. The paper contrasts this with global MSA (Equation 1: $\Omega(\text{MSA}) = 4hwC^2 + 2(hw)^2C$ ), where the $(hw)^2$ term is quadratic.

However, this linear complexity claim applies only to the self-attention computation. The overall network includes other operations that scale differently with input resolution. Specifically:

The patch merging linear projections operate on concatenated features from $2 \times 2$ groups and project from $4C$ to $2C$ dimensions. These are applied at every spatial position and scale linearly with the number of tokens (like $1 \times 1$ convolutions), but they add a non-trivial constant factor that is separate from the attention cost and not analyzed.
The MLP layers (2-layer with GELU, applied per-token, with expansion ratio $\alpha = 4$ ) scale as $4hw(4C^2) = 16hwC^2$ per token (input projection to $4C$ , output projection back to $C$ ) — this is linear in $hw$ but with a large constant factor, and it is applied at every Transformer block. In the early stages where $hw$ is large (e.g., Stage 1 at $56 \times 56 = 3136$ tokens for $224^2$ input), the MLP cost may dominate the windowed attention cost.
The throughput numbers (Table 1) show that Swin Transformer is not uniformly efficient across resolutions. Swin-T at $224^2$ achieves 755.2 images/sec but drops to 219.5 images/sec at $384^2$ — a factor of 3.4× slower for a 2.9× increase in token count ( $384^2$ at patch size 4 gives $96 \times 96 = 9216$ tokens vs. $56 \times 56 = 3136$ , a 2.94× increase). This is close to linear scaling (2.94× more tokens → 3.44× slower), but the paper does not break down where the cost comes from (attention vs. MLP vs. patch merging), making the efficiency claims harder to verify.

The consequence. For high-resolution inputs typical in dense prediction — detection at $800 \times 1333$ (producing Stage 1 feature maps of $200 \times 333 = 66,600$ tokens at patch size 4, or $133 \times 222 = 29,526$ tokens if using a larger patch size) — the total FLOPs are dominated by the MLP layers and linear projections, not by the attention computation. The quadratic bottleneck that ViT suffers from is genuinely avoided, but the absolute cost at high resolution is still substantial. Table 2 shows that Swin-T inference on Cascade Mask R-CNN runs at 15.3 FPS vs. 18.0 FPS for ResNet-50 — the Swin backbone is slower despite both having linear complexity. Swin-L (Table 3) runs at 6.2 FPS for segmentation, slower than many ConvNet alternatives with comparable accuracy.

The linear complexity claim is also specific to the per-layer cost. The total cost of the network includes all stages, and the distribution of computation across stages matters. Stage 1, which has the most tokens, has only 2 Swin Transformer blocks in all model variants — the paper deliberately keeps early stages shallow (Section 3.3, Table 7). This is an implicit acknowledgment that even linear attention at high resolution is expensive, but the paper does not quantify the cost breakdown or show what fraction of total FLOPs comes from attention vs. MLP vs. patch merging at different resolutions.

What evidence exists in the paper. The throughput measurements in Table 1 and Table 8 provide overall speed numbers at different resolutions and model scales, but no per-component cost breakdown. The FLOPs values (Table 1: 4.5G for Swin-T at $224^2$ , 47.0G for Swin-B at $384^2$ ) are totals with no decomposition. Table 5 breaks down the latency of self-attention modules by stage, but only for alternative attention mechanisms (sliding window, Performer, window-based variants) — the cost of MLP and patch merging is not separately measured. Table 2 includes FPS for detection and Table 3 for segmentation, but these are whole-system measurements that include the detection/segmentation heads and data loading, making it impossible to isolate the backbone's contribution from the head.

Mitigation status. The paper does not address this limitation. It presents the linear complexity as a key advantage over ViT (which is true and important for making dense prediction feasible at all), but does not provide the detailed cost analysis that would let a practitioner predict the inference cost of Swin at their specific input resolution and model scale. The throughput measurements at standard resolutions (224, 256, 320, 384 for classification; multi-scale 480–800 for detection; 512 or 640 for segmentation) provide empirical guidance, but a model of how cost scales with resolution beyond these measured points is missing. For deployment planning, a practitioner should benchmark Swin at their target resolution with their specific hardware, rather than relying on the linear complexity claim as a guarantee of fast inference — the constant factors (particularly from MLPs and patch merging at high resolution) are significant.

The Swin Transformer's Improvements Are Most Pronounced on Dense Prediction Tasks — but the Paper Provides No Analysis of Why, and the Classification Gains Are Modest by Comparison

The assumption or constraint. The paper's headline numbers — +2.7 box AP and +2.6 mask AP on COCO, +3.2 mIoU on ADE20K — represent dramatically larger improvements over prior work than the ImageNet-1K classification gains (+1.5% to +1.7% over DeiT at comparable scale). This pattern is consistent across the paper: Table 4 shows that the shifted window mechanism provides +2.8 box AP and +2.8 mIoU gains but only +1.1% top-1; Table 2 shows Swin-T provides +4.2 box AP over ResNet-50 in Cascade Mask R-CNN, but Table 1 shows much more modest classification advantages. The paper does not provide a systematic analysis of why the hierarchical Transformer architecture benefits dense prediction so much more than classification.

This matters because it suggests that the Swin Transformer's architectural innovations — hierarchy, windowed attention, shifted windows, relative position bias — are not uniformly beneficial across tasks. They specifically address bottlenecks that limit ConvNet and ViT performance on dense prediction, but those bottlenecks are less severe for classification. Understanding which innovations help most for which tasks would guide practitioners in adapting the architecture to their specific domain. Without this analysis, one cannot confidently predict whether Swin Transformer would show large or small gains on other vision tasks (depth estimation, keypoint detection, optical flow) that share properties with either classification or dense prediction.

The consequence. For a practitioner considering using Swin Transformer as a backbone, the lack of task-diagnostic analysis makes it difficult to assess where the architecture is likely to provide the largest benefits. If the primary application is image classification, Swin provides a modest improvement over DeiT at similar scale — perhaps not enough to justify switching from a well-optimized ConvNet or ViT pipeline. If the application is dense prediction, the gains are large and clearly worth the switch. But the paper does not explain why, which prevents generalization to tasks not evaluated: would video action recognition (which has a temporal dimension) benefit from shifted windows in time as well as space? Would 3D medical image segmentation (which has small datasets and requires spatial precision) see similar gains to 2D segmentation? Without a causal understanding of the gains, these questions are unanswerable from the paper's evidence.

What evidence exists in the paper. The paper reports results on the three canonical tasks and shows that the hierarchical design, shifted windows, and relative position bias all matter more for dense tasks than for classification (Table 4). But this is a description, not an analysis. The paper does not:

Measure the effective receptive field of the network on classification vs. detection images to see if the windowed design limits classification less.
Compare feature map quality (e.g., through linear probe accuracy at each stage) between Swin and ConvNets to identify where the advantage emerges.
Analyze whether the gains on detection/segmentation come primarily from the multi-scale features (hierarchy) or from the self-attention mechanism (content-dependent computation) or from the specific position encoding scheme.
Test Swin without certain components (e.g., no hierarchy but with shifted windows, or hierarchy but with global attention at certain stages) to disentangle the contributions.

Section 4.4 attributes the detection/segmentation advantages to "inductive bias that encourages certain translation invariance" being "still preferable for general-purpose visual modeling, particularly for the dense prediction tasks," but this is a hypothesis, not a verified claim. The relative position bias benefits all three tasks (+1.2% top-1, +1.3 box AP, +2.3 mIoU), so it is not the sole source of the task-dependent gain difference.

Mitigation status. The paper does not address this gap. The Swin-Mixer experiment (Table 10) partially isolates the contribution of hierarchy and shifting (independent of attention) for classification, but detection/segmentation Swin-Mixer results are not reported. The paper's focus is on demonstrating that a Transformer can serve as a general-purpose backbone — which it does — rather than on explaining the mechanisms behind the task-specific improvements. Subsequent work (e.g., analyzing why vision Transformers are more robust to occlusions, texture bias, or shape bias) has begun to address these questions, but the Swin Transformer paper itself leaves the causal analysis for future work. For a practitioner, the practical implication is that Swin's strongest case is for multi-task systems that need a single backbone for both classification and dense prediction, where the raw numbers in Tables 1–3 provide the justification; for classification-only systems, the gains over DeiT or ConvNets are real but smaller, and the cost of adopting a new architecture may not be justified without additional benefits (e.g., unified vision-language modeling, as the paper speculates in Section 1).

7. Implications and Future Directions

How This Work Changes the Landscape

The Swin Transformer represents a conceptual reframing of what a vision Transformer can be, rather than a paradigm shift. Prior to this work, the Vision Transformer (ViT) and its follow-ups had established that Transformers could perform image classification competitively with ConvNets, but the dominant assumption — implicit in the design of ViT, DeiT, and most concurrent Transformer-based models — was that attention was fundamentally a global computation, and that making it work for vision meant either accepting the quadratic complexity (and restricting inputs to low resolution) or approximating the global computation through kernel methods or sparsity patterns. The Swin Transformer challenges this framing directly: it demonstrates that you do not need global attention, or any approximation thereof, to achieve state-of-the-art results across the full spectrum of vision tasks. Local attention, when combined with a hierarchical structure and a mechanism for cross-window communication through depth, is not just a compromise — it is a better design for vision than global attention, delivering both higher accuracy and lower latency than either global attention (ViT, whose quadratic cost forces single-scale low-resolution features) or approximate global attention (Performer, which is 2.3% less accurate and 18% slower, per Tables 5 and 6).

This reframing has several specific consequences for the field:

It converts the vision Transformer from a "classification specialist" to a "general-purpose backbone." Before Swin, the strongest Transformer-based vision models (ViT, DeiT) produced a single feature map at fixed resolution, making them essentially useless for object detection, semantic segmentation, and other dense prediction tasks without awkward workarounds (deconvolution layers, as used for DeiT in Tables 2 and 3). Swin's hierarchical design produces feature maps at strides 4, 8, 16, and 32 — matching the ConvNet pyramid that FPN, U-Net, and other dense prediction frameworks expect — meaning it can be dropped into existing detection and segmentation pipelines as a direct replacement for ResNet or ResNeXt with minimal engineering effort. The paper demonstrates this concretely: the same Swin backbone, without task-specific modifications, achieves +3.4 to +4.2 box AP gains over ResNet-50 across four different detection frameworks (Cascade Mask R-CNN, ATSS, RepPoints v2, Sparse R-CNN — Table 2a), and +2.1 to +5.3 mIoU gains over ConvNet baselines on ADE20K segmentation (Table 3). This is not a classification model being shoehorned into dense prediction; it is a genuinely general-purpose backbone that happens to also perform well on classification.

It changes the evaluation standard for vision Transformers. Prior to Swin, a new vision Transformer architecture could reasonably be evaluated on ImageNet-1K classification alone (as ViT, DeiT, T2T-ViT, and many concurrent works were). After Swin, the bar is higher: a proposed backbone should demonstrate performance on detection and segmentation, not just classification, because the architectural features that distinguish a good general-purpose backbone (hierarchy, efficient attention, position encoding that respects translation invariance for dense tasks) are not fully tested by classification accuracy. The paper's Table 4 is particularly instructive here — it shows that absolute position embeddings, which improve classification by +0.4%, actually hurt detection and segmentation. A classification-only evaluation would have concluded that absolute position embeddings are beneficial; the multi-task evaluation reveals the opposite for the tasks where spatial precision matters most. This sets a methodological precedent that subsequent vision Transformer papers have largely followed.

It reorients the efficient-attention research agenda away from approximation and toward structural design. The dominant approach to scaling Transformers to long sequences — in both NLP and vision — has been to develop linear-complexity approximations to the full attention matrix, through kernel methods (Performer, Linear Transformer), low-rank factorization (Linformer), or sparse patterns (BigBird, Longformer). The Swin Transformer's results argue that for vision at least, this direction may be a dead end: Performer achieves 79.0% top-1 accuracy on ImageNet-1K (Table 6), while Swin's exact local attention achieves 81.3%, and Swin is faster (755 vs. 638 img/s, Table 5). The reason is that vision has a strong prior of locality that approximation-based global attention wastes capacity trying to learn from data — it is more efficient to encode locality in the architecture (via windows) and then provide structured mechanisms (shifting) for cross-region communication, rather than to compute an approximate version of global connectivity. This insight has influenced a generation of subsequent vision Transformers (SwinV2, CSWin, Focal Attention, CrossFormer) that explore variations on the theme of structured local attention rather than approximation-based global attention.

It partially resolves the tension between ConvNets and Transformers by demonstrating that they are converging on shared architectural principles. The Swin Transformer is structurally very similar to a ConvNet — it has four stages, progressive downsampling, increasing channel dimensions, and local receptive fields that grow through network depth — but it replaces the static, content-independent convolution kernel with dynamic, content-dependent self-attention within each local window. The Swin-Mixer experiment (Table 10, Appendix) takes this convergence further: applying the same hierarchical shifted-window structure to an all-MLP architecture (MLP-Mixer, which has no attention at all) recovers most of the performance gap (81.3% vs. 81.6% for Swin-T at comparable scale). This suggests that the hierarchy and locality are carrying most of the architectural weight, and the choice of token-mixing mechanism (attention, MLP, convolution) is a second-order concern — a finding that anticipates the ConvNeXt line of work, which "modernizes" ConvNets by adopting Swin-like design decisions (patchify stem, GELU activations, LayerNorm, fewer activation functions) while keeping convolution as the token mixer, and achieves competitive results.

However, it would be inaccurate to call Swin Transformer a paradigm shift. Paradigm shifts eliminate the need for the old approach; Swin does not eliminate ConvNets. The paper's own comparisons show that ConvNet baselines remain competitive — EfficientNet-B7 achieves 84.3% top-1 at 600² with 66M parameters, close to Swin-B at 84.5% with 88M at 384² (Table 1a), and ResNeXt101-64x4d achieves 48.3 box AP with Cascade Mask R-CNN vs. Swin-B at 51.9 (Table 2b), a meaningful but not overwhelming gap. The throughput comparison is mixed: Swin-T is slower than ResNet-50 on detection (15.3 FPS vs. 18.0, Table 2a) and Swin-L is slower than many ConvNet counterparts on segmentation (6.2 FPS, Table 3). What Swin Transformer demonstrates is that Transformers are now in the same performance class as ConvNets for vision — they are a viable alternative, not a replacement — and that the architectural principles that matter (hierarchy, locality, translation-equivariant position encoding) are shared across both families.

Follow-Up Research This Work Enables

Directly measuring and analyzing the effective receptive field of shifted-window Transformers. The paper demonstrates empirically that shifted windows help (Table 4: +1.1% top-1, +2.8 box AP, +2.8 mIoU over no shifting), but never quantifies the receptive field growth through the network. A strong follow-up would use gradient-based receptive field analysis (e.g., measuring the input pixels that influence a specific output token's feature vector) to map the effective receptive field at each stage for each Swin-T block. The concrete experiment: take a trained Swin-T, compute the effective receptive field for patches at the center vs. corners of the feature map at Stage 1, Stage 2, Stage 3, and Stage 4 outputs, and determine (1) how many blocks are needed for full-image coverage at each stage, (2) whether the shifted-window pairs double the receptive field growth rate compared to non-shifted windows, and (3) whether the theoretical maximum receptive field (determined by window size, shift amount, and block count) is actually achieved in practice or whether gradient propagation attenuates with distance. This would convert the paper's qualitative claim ("shifted windows introduce connections between neighboring non-overlapping windows") into a quantitative understanding of how far information propagates, and would directly inform whether adding more blocks to early stages (which currently have only 2 blocks each) would help for high-resolution dense prediction tasks where long-range spatial context at fine scales matters.

Training and evaluating Swin Transformer on video tasks with shifted windows extended to the temporal dimension. The paper's shifted window mechanism operates purely in the 2D spatial domain, but the concept generalizes naturally to 3D (space + time) for video understanding. A concrete follow-up: replace the 2D window partitioning with 3D windows (e.g., $M \times M \times T$ where $T$ is a temporal window size), shift the window partition along the temporal axis between consecutive blocks (analogous to shifting in the spatial dimensions), and test on video action recognition (Kinetics-400/600) and video object detection (ImageNet VID). The key question is whether the computational savings of windowed attention in 3D (where global attention would have $O((hwt)^2)$ complexity, completely intractable for even short video clips) enable Swin to process longer video sequences than ConvNets or global-attention Transformers, and whether the shifted-window approach in time effectively captures motion and temporal dependencies. The paper provides the recipe (hierarchical structure with patch merging, shifted windows, cyclic batch computation) and the SwinV2 extensions (which came later but were anticipated by the limitations discussed above) would apply directly to the 3D case. A negative result — e.g., finding that temporal dynamics require longer-range attention than spatial dynamics, making fixed temporal windows insufficient — would also be informative for understanding the limits of the windowed-attention paradigm.

Ablating the window size $M$ systematically across tasks and input resolutions to map the accuracy-efficiency Pareto frontier. The paper fixes $M=7$ for all models and all tasks without justification. A targeted follow-up would train Swin-T variants with window sizes $M \in \{4, 7, 14, 28\}$ (keeping all other hyperparameters identical) on ImageNet-1K, and evaluate the resulting models on COCO detection and ADE20K segmentation, measuring both accuracy and throughput at multiple test resolutions. The experiment would answer: (1) Is $M=7$ actually optimal for classification, or does a larger window improve accuracy enough to justify the computational cost? (2) For dense prediction at high input resolutions (e.g., 800×1333 for detection), does a larger window become more important because the proportion of the feature map covered per window is smaller, or less important because the effective receptive field is already large enough at coarser stages? (3) Does the relative position bias trained at $M=7$ transfer effectively to different $M$ via bi-cubic interpolation, as the paper briefly suggests (Section 3.2), or is retraining necessary? This would provide the principled window-size selection guidance that the paper currently lacks, and the results would be directly useful for practitioners deploying Swin at non-standard resolutions.

Combining global self-attention at the coarsest stage with windowed attention at finer stages — and evaluating whether this closes the remaining performance gap to fully global models. The paper's HTC++ system-level detection results (Table 2c) already include an "extra global self-attention layer appended at the output of the last stage" (Appendix A2.2), but this addition is not ablated, making it impossible to know whether it contributes significantly to the +4.1 box AP gain over the ResNeXt baseline or whether it compensates for a specific limitation of the purely windowed design. A clean follow-up experiment: train Swin-T variants where Stage 4 (the coarsest stage, at $H/32 \times W/32$ resolution, with only $7 \times 7 = 49$ tokens at 224² input) uses global multi-head self-attention (MSA) instead of windowed attention, while Stages 1–3 remain windowed. Since the token count at Stage 4 is very small, global attention is computationally cheap ( $49^2 = 2401$ pairwise interactions per attention head), and it would allow the model to capture any remaining long-range dependencies that the windowed design misses. Evaluate on the same three tasks as the paper. If adding Stage-4 global attention provides additional gains beyond the shifted-window baseline, it suggests that purely windowed encoding is indeed sacrificing some representational power for efficiency, and provides a quantitative measure of that sacrifice. If it provides no gain, it suggests that the hierarchical propagation through windowed stages is already sufficient, which would be a stronger validation of the windowed paradigm than the paper currently provides.

Extending the Swin-Mixer analysis to detection and segmentation to determine whether the hierarchical shifted-window design is sufficient for dense prediction with any token-mixing mechanism, or whether self-attention provides unique benefits for these tasks. The paper demonstrates that Swin-Mixer (an all-MLP architecture with the Swin hierarchical structure) recovers most of the ImageNet-1K accuracy gap relative to the Swin Transformer (81.3% vs. 81.6% at comparable scale, Table 10), but provides no detection or segmentation results. A strong follow-up would train Swin-Mixer-B/D24 as a backbone for Cascade Mask R-CNN on COCO and UperNet on ADE20K, using identical training recipes to the Swin-B experiments. The critical comparison: does the 0.3% classification gap between Swin-Mixer and Swin-T (at 256²) widen or narrow on dense prediction tasks? If Swin-Mixer achieves comparable detection/segmentation performance, it reinforces the paper's implicit claim that hierarchy and locality are the primary drivers of vision backbone performance, with the token-mixing mechanism being secondary. If Swin-Mixer substantially underperforms on dense tasks despite similar classification accuracy, it suggests that self-attention's content-dependent receptive fields are specifically important for the local spatial reasoning required in detection and segmentation — a finding that would refine our understanding of why attention works for vision, not just whether it works.

Measuring the translation sensitivity of Swin Transformer predictions on detection and segmentation to quantify whether the window-grid alignment causes measurable spatial inconsistency in dense predictions. As discussed in the Limitations section, the Swin Transformer is not globally translation-equivariant because the window partitioning aligns to an absolute grid. A diagnostic experiment: take a trained Swin-based detection model, run it on a set of test images, then shift each image by a small random offset (e.g., 1–6 pixels in each direction, less than the window size $M=7$ ), and measure the change in predicted bounding boxes and segmentation masks. Compare this sensitivity to a ConvNet baseline (which is translation-equivariant by design, up to the stride of the backbone) and a ViT baseline (which with absolute position embeddings is sensitive to shifts, but with relative position bias should be less so). The paper's Table 4 shows that relative position bias improves dense prediction substantially, but does not quantify whether it restores full translation insensitivity or merely reduces the sensitivity compared to absolute position embeddings. If Swin shows measurable prediction shifts under sub-window-size translations — and especially if the shift magnitude depends on whether the object crosses a window boundary — this would identify a concrete failure mode that practitioners should be aware of when deploying Swin for applications requiring precise spatial localization (e.g., medical image segmentation, autonomous driving perception).

Practical Applications and Downstream Use Cases

Drop-in backbone replacement in existing detection and segmentation pipelines. The most immediate practical use case is replacing a ResNet or ResNeXt backbone with a Swin Transformer in any existing object detection or instance segmentation framework built on mmdetection, Detectron2, or similar libraries. Because Swin produces feature maps at exactly the same strides (4, 8, 16, 32) as standard ConvNet backbones, the integration requires changing only the backbone definition in the config file — no modifications to the Feature Pyramid Network, region proposal network, or detection heads. The paper's Table 2a demonstrates this concretely: simply swapping ResNet-50 for Swin-T in Cascade Mask R-CNN yields +4.2 box AP and +3.6 mask AP at similar inference speed (15.3 vs. 18.0 FPS). For a practitioner with an existing detection pipeline trained on a custom dataset, this represents a substantial accuracy improvement with minimal engineering effort. The gains are consistent across detection frameworks (Cascade Mask R-CNN, ATSS, RepPoints v2, Sparse R-CNN), so the choice of detection architecture does not constrain the backbone upgrade. For segmentation (Table 3), the same principle applies: Swin-T with UperNet achieves 46.1 mIoU, surpassing all ResNet-101-based segmentation methods, while Swin-S achieves 49.3 mIoU — a gain of +5.3 mIoU over the DeiT-S baseline. The ImageNet-22K pre-trained Swin-L achieves 53.5 mIoU on ADE20K, which at the time of publication was a new state of the art.

Multi-task vision systems where a single backbone serves classification, detection, and segmentation simultaneously. In production systems that need to perform multiple vision tasks on the same input (e.g., an autonomous vehicle perception stack that classifies traffic signs, detects pedestrians and vehicles, and segments the road surface), using a single shared backbone is highly desirable for computational efficiency — the backbone features are computed once and routed to multiple task-specific heads. The Swin Transformer is uniquely well-suited to this scenario because it has been validated across the three canonical vision tasks with the same backbone architecture and similar accuracy advantages across all of them. A system using Swin-B with ImageNet-22K pre-training achieves 86.4% top-1 on classification (Table 1b), 51.9 box AP on detection (Table 2b), and 51.6 mIoU on segmentation (Table 3) — state-of-the-art or near-state-of-the-art results on all three tasks from a single feature extractor. Prior to Swin, a multi-task system would typically use a ConvNet backbone that performed well on detection and segmentation but lagged behind specialized classification models, or would use separate backbones for classification and dense tasks — neither option ideal. The numbers in Tables 1–3 provide concrete justification for adopting Swin as the shared backbone in such systems.

Efficient high-resolution vision with constrained compute budgets, such as on-device or edge deployment. While the paper does not specifically target mobile or edge deployment, the linear computational complexity of windowed attention (Equation 2: $\Omega(\text{W-MSA}) = 4hwC^2 + 2M^2 hwC$ ) means that Swin Transformer can process high-resolution images that would be infeasible for ViT-style global attention models. For a fixed window size $M=7$ , doubling the input resolution doubles the FLOPs (linear scaling) rather than quadrupling them as in global attention (where the $(hw)^2$ term would grow 4×). This means that Swin-T — at 29M parameters and 4.5G FLOPs for 224² input, running at 755 images/sec on a V100 GPU — can be deployed on hardware with compute constraints to process images at resolutions that would exhaust a similarly-sized ViT model. The throughput numbers in Table 8 provide the scaling behavior: moving from 224² to 384² drops throughput from 755 to 220 img/s (3.4× slower for 2.95× more tokens — nearly linear). For a mobile or edge deployment scenario where a 224² classification model achieves 81.3% accuracy at 755 FPS on a V100 (and correspondingly lower but still proportional throughput on edge hardware), a practitioner can extrapolate the cost of higher-resolution inference using the linear scaling property. The Swin-T or Swin-S variants, with their relatively modest parameter counts (29M and 50M respectively), are the appropriate size for on-device deployment with the ability to scale to higher resolutions without the quadratic cost penalty that makes ViT impractical above 384².

Vision-language pretraining with a unified Transformer backbone. The paper explicitly envisions this application in Section 1: "a unified architecture across computer vision and natural language processing could benefit both fields, since it would facilitate joint modeling of visual and textual signals and the modeling knowledge from both domains can be more deeply shared." The Swin Transformer is particularly well-positioned for vision-language pretraining (e.g., CLIP-style contrastive learning or VL-BERT-style masked modeling) because it produces multi-scale visual features that can be aligned with text representations at different granularities — fine-grained features from Stage 1 could attend to individual word tokens for phrase grounding, while coarse features from Stage 4 could align with sentence-level representations for image-text matching. A practical vision-language training pipeline could use Swin as the image encoder and a standard Transformer as the text encoder, computing contrastive loss between pooled Stage 4 features and [CLS] text embeddings, while optionally adding cross-attention between text tokens and multi-scale Swin features for tasks like visual question answering or image captioning. The fact that Swin achieves strong performance on both classification and dense prediction (Tables 1–3) means that a single Swin vision encoder, pretrained on image-text pairs, could be fine-tuned for both image-level tasks (classification, retrieval) and region-level tasks (grounding, referring expression comprehension) — a versatility that single-scale ViT encoders lack due to their low-resolution output feature maps.

When to Prefer This Method

The paper positions Swin Transformer against two specific alternatives: standard Vision Transformers (ViT/DeiT) and ConvNet backbones (ResNet, ResNeXt, EfficientNet). Based on the experimental results in Tables 1–6, the following decision guidelines emerge directly from the paper's evidence:

Prefer Swin Transformer over ViT/DeiT when: (1) your task requires dense prediction (object detection, instance segmentation, semantic segmentation) and ViT's single-resolution, low-resolution feature maps would require awkward workarounds like deconvolution layers that degrade performance (Table 2b: Swin-T achieves 50.5 box AP vs. 48.0 for DeiT-S with deconvolution, +2.5 box AP); (2) you are working with high-resolution input images (above 384²) and ViT's quadratic complexity in image size would make inference intractable (Equation 1 vs. Equation 2); (3) you are training from scratch on ImageNet-1K without large-scale pre-training data, where Swin-T outperforms DeiT-S by +1.5% (81.3% vs. 79.8%, Table 1a) at similar FLOPs; or (4) you need a single backbone for multiple vision tasks and want consistent, validated performance across classification, detection, and segmentation rather than task-specific architectures.
Prefer Swin Transformer over ConvNet backbones (ResNet, ResNeXt) when: (1) your detection or segmentation pipeline can tolerate a moderate throughput reduction in exchange for substantial accuracy gains (Table 2a: Swin-T achieves +4.2 box AP over ResNet-50 in Cascade Mask R-CNN, but at 15.3 vs. 18.0 FPS); (2) you are using a modern optimizer (AdamW) rather than SGD, since the paper found that ResNeXt models sometimes perform better with SGD (Table 9), and switching optimizers for ConvNets to match Swin's training recipe may not restore parity; or (3) you want a backbone that naturally integrates with vision-language modeling (Section 1) because the Transformer architecture is shared across modalities, whereas ConvNets would require a separate vision encoder design from the text encoder.
Prefer ConvNet backbones over Swin Transformer when: (1) you are deploying on hardware with highly optimized convolution kernels (e.g., mobile DSPs, edge TPUs) where the standard Transformer operations (LayerNorm, GELU, batched matrix multiplications for attention) are less optimized than convolution, even if the theoretical FLOPs are comparable — the paper acknowledges that Swin uses "built-in PyTorch functions that are not all well-optimized" (Section 4.2) while ConvNet baselines benefit from years of cuDNN kernel tuning; (2) your application is latency-critical and the FPS difference matters more than the accuracy gain (e.g., real-time video processing at 30+ FPS), since even the tiny Swin-T is slower than ResNet-50 on detection (15.3 vs. 18.0 FPS in Table 2a); or (3) you require strict translation equivariance (e.g., for precise spatial correspondence tasks like stereo matching or optical flow) — Swin's window-grid-aligned partitioning provides only local shift-invariance, not global translation equivariance, and the paper does not quantify the magnitude of translation sensitivity for dense predictions (a gap identified in Limitations, Section 6).
Prefer standard ViT/DeiT over Swin Transformer when: (1) you are exclusively doing image classification and the single-scale ViT architecture is simpler to implement, requiring no patch merging layers or window partitioning logic; (2) you have access to very large-scale pre-training data (JFT-300M scale), where ViT's lack of spatial inductive biases may be overcome by data scale, and the quadratic complexity at standard classification resolutions (224²–384²) is manageable; or (3) you are deploying on hardware where the cyclic-shift masking logic for shifted windows (Figure 4) introduces implementation complexity that outweighs the throughput advantage over sliding-window or approximate attention methods.

These preferences are not absolute — the paper provides evidence at specific operating points (model scales, input resolutions, training data sizes) and the tradeoffs may shift as hardware optimization, training recipes, and model variants evolve. The key principle, grounded in the paper's results, is that the choice between Swin, ViT, and ConvNets should be made based on the specific task requirements (classification vs. dense prediction), the input resolution regime (below vs. above 384²), and the deployment hardware (GPU vs. edge vs. mobile), rather than assuming that any one architecture family is universally preferable.