Emerging Properties in Self-Supervised Vision Transformers

🎯 Pitch

Self-supervised ViTs spontaneously learn to segment objects, with attention maps directly delineating boundaries—without any labels. These same features also achieve 78.3% ImageNet top-1 accuracy using only a k-NN classifier, no fine-tuning required.

1. Executive Summary

This paper studies how self-supervised pre-training affects Vision Transformers (ViTs) compared to supervised ViTs and convolutional networks, using the DINO method—a form of self-distillation with no labels (where a student network matches the output of a momentum teacher through a standard cross-entropy loss, with only centering and sharpening to avoid collapse)—trained on ImageNet without labels. The work identifies that self-supervised ViT features exhibit two distinctive properties: they contain explicit semantic segmentation information in their self-attention maps, enabling unsupervised object discovery, and they serve as remarkably effective k-NN classifiers, with a ViT-S/8 reaching 78.3% top-1 accuracy using only nearest neighbors without any fine-tuning or data augmentation. DINO achieves 80.1% top-1 on ImageNet linear evaluation with ViT-Base with 8×8 patches, establishing that the combination of a momentum encoder, multi-crop training, and small patch sizes with ViTs yields features where segmentation and class-specific structures emerge only when these specific components are combined with Transformers under self-supervised pre-training.

2. Context and Motivation

The Core Problem: Self-Supervised Learning Has Been Studied Primarily on ConvNets, Not Transformers

The fundamental question this paper tackles is deceptively simple but has significant implications: if we train Vision Transformers with self-supervised learning instead of supervised learning, do the resulting features exhibit properties that supervised ViTs and convnets lack? This matters because the computer vision community had already observed a puzzling asymmetry: while Transformers revolutionized NLP largely thanks to self-supervised pre-training objectives (BERT's masked language modeling, GPT's autoregressive language modeling), their adoption in vision had been comparatively muted. ViTs were competitive with convnets but did not clearly outperform them—they were computationally more demanding, required more training data, and their features didn't exhibit obviously unique properties that would justify switching architectures.

This gap is significant for several reasons the authors outline (Section 1):

The BERT moment for vision has not happened yet. In NLP, self-supervised pre-training on massive unlabeled text corpora produced models that learned rich linguistic representations—syntax, semantics, and even some world knowledge—without explicit supervision. These representations then transferred effectively to downstream tasks. The vision community lacked an equivalent: no one had demonstrated that self-supervised ViTs learn fundamentally different or more useful representations than supervised ViTs.
Image-level supervision is impoverished compared to self-supervised signals. The paper argues explicitly that "image-level supervision often reduces the rich visual information contained in an image to a single concept selected from a predefined set of a few thousand categories." An image of a dog chasing a ball in a park gets reduced to the label "dog"—all spatial relationships, object boundaries, scene layout, and contextual information are discarded. Self-supervised objectives operating on the image itself (predicting relationships between different views, matching representations across augmentations) might preserve more of this rich visual structure, which could manifest as emergent properties in the learned representations.
If self-supervised ViTs unlock new capabilities, it changes the scaling equation. The paper's findings that self-supervised ViTs produce explicit segmentation masks in attention and serve as excellent k-NN classifiers without fine-tuning suggest that the architecture's potential has been underutilized. This opens the door to training ViTs on massive unlabeled image collections—potentially hundreds of millions or billions of images—without the annotation bottleneck that limits supervised approaches.

Where Existing Approaches Fall Short

The paper identifies specific limitations in prior work along several axes:

Self-supervised learning methods have been developed and optimized almost entirely for convnets. The paper's review of related work (Section 2) catalogs a rich landscape of self-supervised approaches—instance classification (Wu et al., 2018; He et al., 2020; Chen et al., 2020), clustering-based methods (Caron et al., 2018, 2020; Asano et al., 2020), and more recent approaches that avoid explicit discrimination between images entirely, such as BYOL (Grill et al., 2020), which matches representations to a momentum encoder, and methods based on whitening (Ermolov et al., 2020; Zbontar et al., 2021) or predicting bags of visual words (Gidaris et al., 2020). However, all of these methods were designed, evaluated, and tuned on ResNet architectures. The paper notes this explicitly: "many existing self-supervised methods have shown their potential on images with convnets." The implication is that the community had collectively assumed that what works for convnets works for ViTs—or at least had not systematically investigated whether ViTs exhibit qualitatively different behavior under self-supervised training.

Prior self-supervised methods carry convnet-specific baggage. The components developed to prevent collapse in self-supervised convnet training—contrastive losses requiring large batches or memory banks (SimCLR, MoCo), clustering constraints (SwAV's Sinkhorn-Knopp normalization), batch normalization dependencies in projection heads (BYOL, SimSiam)—were designed for architectures where batch normalization is standard. ViTs, by contrast, do not use batch normalization by default; they use layer normalization. The paper observes that "unlike standard convnets, ViT architectures do not use batch normalizations (BN) by default. Therefore, when applying DINO to ViT we do not use any BN also in the projection heads, making the system entirely BN-free." This is not a minor implementation detail—it means that methods relying on BN (which some works showed was critical for preventing collapse) may not transfer cleanly to ViTs, and the space of valid self-supervised training configurations for ViTs was largely unexplored.

The "BERT moment" reasoning had not been empirically validated for ViTs. While the intuitive argument that self-supervised pre-training should benefit ViTs was compelling, there was no systematic evidence demonstrating that self-supervised ViTs learn representations with properties that supervised ViTs or self-supervised convnets lack. The paper positions itself as providing exactly this evidence: "we question if self-supervised learning provides new properties to Vision Transformer that stand out compared to convolutional networks." Prior work had not made this comparison systematically.

Self-training and knowledge distillation for self-supervised learning were underexplored. The paper draws an explicit connection between self-supervised learning and knowledge distillation (Section 2), noting that prior works combining them (Fang et al., 2021; Shen et al., 2021; Noroozi et al., 2018) relied on a pre-trained fixed teacher network, using distillation as a post-processing step after self-supervised pre-training to compress the model or improve performance. The paper's insight is to cast distillation directly as the self-supervised objective itself, where the teacher is dynamically built during training through exponential moving average of the student. This eliminates the need for a pre-trained teacher and makes the framework self-contained. The paper also connects this to "codistillation" (Anil et al., 2018) where student and teacher have the same architecture, but distinguishes DINO by the use of momentum-based teacher updates rather than mutual distillation.

The Multi-Crop Problem: Why Existing Methods Transfer Unevenly to ViTs

A subtle but critical motivation that emerges from the paper's experimental work (detailed in Appendix E) is that the components that make self-supervised learning effective do not transfer uniformly across architectures and frameworks. The paper shows that multi-crop training—generating multiple views at different resolutions and enforcing local-to-global correspondences—has dramatically different effects depending on both the architecture and the underlying self-supervised loss:

With ResNet-50, DINO performs on par with SwAV and BYOL (Table 13, top panel), with linear evaluation accuracies of 74.5%, 74.1%, and 72.7% respectively—all within a narrow band.
With ViT-S, the same methods diverge sharply: DINO achieves 76.1% linear accuracy versus 71.4% (BYOL), 71.6% (MoCo-v2), and 71.8% (SwAV). The k-NN gap is even more dramatic: 72.8% for DINO versus 66.6% (BYOL), 62.0% (MoCo-v2), and 64.7% (SwAV).

This divergence is not explained by any single component. The paper's ablation in Table 14 shows that multi-crop interacts differently with each loss function: removing multi-crop hurts DINO by 3.6% (76.1% → 72.5%), MoCo-v2 by 1.9% (73.4% → 71.4%), and actually improves BYOL's linear evaluation (64.8% → 71.4%)—though with k-NN, multi-crop in BYOL degrades from 66.6% to 59.8%. This means that a component that is beneficial for one framework can be detrimental for another when applied to ViTs, and the interaction depends on the evaluation protocol.

The paper's Appendix E explicitly documents that training BYOL with multi-crop on ViT-S leads to performance that "is higher than the baseline without multi-crop for the first training epochs. However, the transfer performance growth rate is slowing down and declines after a certain amount of training." The authors tried learning rate, weight decay, and multi-crop parameter sweeps and "systematically observe the same pattern." This suggests a fundamental incompatibility that is not merely a hyperparameter issue.

This finding is important because it means the recipe for self-supervised convnets does not simply transfer to ViTs. The community needed a systematic study of which components matter, which interact synergistically with ViT architectures, and which produce novel emergent properties. DINO is presented as the synthesis of this study: a framework that works on both convnets and ViTs without architectural modifications, but whose real value is revealed specifically when applied to ViTs.

Self-Supervised NLP Provides the Conceptual Template, but the Mechanisms Differ

The paper draws explicit inspiration from NLP's self-supervised success story (Section 1), and this parallel is worth unpacking because it illuminates both the motivation and the key differences:

In NLP, the pretext task is text-specific. BERT masks words and predicts them from context; GPT predicts the next word given previous context. These objectives exploit the sequential structure of language—the fact that words in a sentence mutually constrain each other—to create a rich learning signal. Each prediction requires understanding syntax, semantics, and factual relationships.

In vision, the equivalent structures are spatial, not sequential. Images have a natural 2D spatial structure where neighboring pixels or patches constrain each other, and objects have coherent boundaries and part-whole relationships. The paper's key observation is that self-supervised ViTs appear to learn these spatial structures automatically in their self-attention maps—producing explicit object segmentations without being trained to do so (Figure 1). This is qualitatively different from what supervised ViTs produce, and from what convnets produce under either training regime.

The learning signal in DINO is "local-to-global" correspondence. Rather than masking or autoregressive prediction, DINO creates a pretext task by requiring the student network to match the teacher's output for different views of the same image—global views (large crops covering >50% of the image) and local views (small crops covering <50%). The paper describes this as encouraging "local-to-global correspondences" (Section 3.1). This forces the model to recognize that a small patch of a dog's ear and a large crop showing the entire dog in a park are views of the same underlying thing. To succeed at this task, the model must learn to aggregate information across spatial locations and identify what parts of the image belong to the same semantic entity—which naturally leads to the emergence of segmentation-like attention patterns.

This local-to-global matching objective is conceptually similar to the instance discrimination task used in many self-supervised methods (treating each image as its own class), but DINO's formulation with a dynamically updated teacher and cross-entropy loss on sharpened softmax outputs creates a different optimization dynamic that the paper argues is particularly well-suited to ViTs.

How This Paper Positions Itself

The paper positions itself at the intersection of three research threads:

1. Self-supervised learning for visual representations. The paper builds directly on the lineage of BYOL (momentum encoder, no negative pairs), SwAV (multi-crop, clustering), and MoCo (momentum encoder for contrastive learning), but synthesizes components differently. The key departure is eliminating the need for a predictor network (critical in BYOL), contrastive loss (MoCo), or iterative clustering (SwAV's Sinkhorn-Knopp), replacing all of these with a simple centering-and-sharpening operation on the teacher's output. The paper frames this as a simplification: "our method can work with only a centering and sharpening of the teacher output to avoid collapse, while other popular components such as predictor, advanced normalization or contrastive loss add little benefits in terms of stability or performance."

2. Knowledge distillation without labels. The paper explicitly interprets DINO as "a form of self-distillation with no labels" (Section 1), extending the Mean Teacher paradigm (Tarvainen and Valpola, 2017) to the case where no labeled data exists. In Mean Teacher semi-supervised learning, a student is trained to match a teacher's predictions on unlabeled data, where the teacher is an EMA of the student. DINO removes the labeled data entirely—the entire objective is the consistency between student and teacher on different views of unlabeled images. The paper's finding that the teacher consistently outperforms the student during training (Figure 6, left) supports this interpretation: the teacher serves as a better "target" that guides the student's learning, similar to how a more accurate teacher would guide a student in standard distillation.

3. Vision Transformers as a distinct architectural class. The paper treats ViTs not as a drop-in replacement for convnets but as an architecture with potentially different inductive biases and learning dynamics. The observation that "ViT architectures do not use batch normalizations by default" is highlighted as significant because it means the system trained with DINO is "entirely BN-free"—a property that distinguishes it from nearly all prior self-supervised methods. The paper also notes that ViTs process images as sequences of patches with self-attention, which means the model can potentially learn long-range spatial dependencies that convnets (with their local receptive fields) cannot easily capture. The emergence of coherent object segmentations in the self-attention maps is presented as evidence that ViTs are indeed learning something fundamentally different about spatial structure when trained with self-supervision.

The paper is not proposing a fundamentally new self-supervised algorithm. Rather, it is making two contributions: (1) a systematic empirical study showing that the interaction between self-supervised training components and ViT architectures produces features with properties not seen in supervised ViTs or convnets, and (2) a simple, unified framework (DINO) that synthesizes the best components from prior work into a form that works particularly well with ViTs. The title's framing—"Emerging Properties in Self-Supervised Vision Transformers"—reflects this: the paper is primarily about what emerges, with DINO being the vehicle for discovering and studying these properties.

3. Technical Approach

3.1 Reader Orientation

The system is a self-supervised pre-training method called DINO (which stands for "DIstillation with NO labels") that trains Vision Transformers to produce useful visual features without any annotated data. It solves the problem of learning rich image representations by having one neural network (the student) predict the output of another network (the teacher) when both see different crops of the same unlabeled image, where the teacher is an exponentially-weighted moving average of the student's own past parameters — the entire system operates as a form of self-distillation where the only learning signal comes from enforcing consistency across different views of the same image.

3.2 Big-Picture Architecture (Diagram in Words)

Imagine two copies of an identical Vision Transformer backbone topped with a projection head. You feed them different randomly-cropped views of the same unlabeled image — say, one large crop showing most of the scene and one small crop showing only a corner. The student network processes all views and produces a probability distribution over a set of $K$ dimensions (think of $K$ as a set of imaginary categories that the model invents during training). The teacher network processes only the large "global" views and produces its own probability distributions, but it does so with its own weights and with two crucial tweaks: its outputs are centered (shifted so no single dimension dominates) and sharpened (peaked so it expresses confident predictions). The student is then trained with a standard cross-entropy loss to make its predictions match the teacher's predictions, but gradients flow only through the student — the teacher's parameters are never updated by backpropagation. Instead, the teacher parameters are slowly pulled toward the student's parameters using an exponential moving average: $\theta_t \leftarrow \lambda\theta_t + (1-\lambda)\theta_s$ with $\lambda$ starting at 0.996 and following a cosine schedule to 1 during training. After pre-training, the projection head is discarded, and the ViT backbone's output serves as the feature representation for downstream tasks.

3.3 Roadmap for the Deep Dive

First, the core self-distillation objective (Equations 1–3), which defines how the student matches the teacher's predictions and why this avoids collapse without contrastive losses or predictors.
Second, the centering and sharpening mechanism on the teacher output — the paper's primary contribution to preventing collapse, which replaces all the machinery (Sinkhorn-Knopp, predictors, large batches, batch normalization) used by prior methods.
Third, the teacher network construction: the momentum encoder update rule, the rationale for using Polyak-Ruppert averaging during training rather than only at the end, and the empirical finding that the teacher consistently outperforms the student.
Fourth, the data augmentation pipeline — multi-crop training (global and local views) and the standard augmentations (color jittering, Gaussian blur, solarization) — and how the local-to-global matching objective forces the model to learn spatial correspondences.
Fifth, the network architecture specifics: ViT backbone (patch size, token sequence, class token), projection head design (MLP with $\ell_2$ bottleneck, weight-normalized final layer), and why the system is entirely batch-normalization-free when applied to ViTs.
Sixth, the implementation details: optimizer, learning rate schedule, temperature schedules, batch size, training duration, and evaluation protocols (k-NN, linear, and fine-tuning).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that a particular combination of self-supervised learning components — momentum encoder, multi-crop augmentation, cross-entropy loss, and centering-sharpening — when applied to Vision Transformers, produces features with emergent properties (explicit spatial segmentation in attention maps, exceptional k-NN classification performance) that do not appear with supervised ViTs or with convnets under any training regime. DINO is the synthesis of this component study, not a fundamentally new algorithm.

The Core Self-Distillation Objective

DINO frames self-supervised learning as knowledge distillation without labels. In standard knowledge distillation (Hinton et al., 2015), a student network is trained to match the output probability distribution of a fixed, pre-trained teacher network. DINO extends this to the self-supervised setting by making the teacher dynamically constructed from the student itself during training — there is no pre-trained teacher, no labels, and no external supervision. The student and teacher process different augmented views of the same unlabeled image, and the student learns to predict the teacher's output.

The softmax probability distribution. Both student and teacher networks output a vector of $K$ logits, which are converted to probability distributions using a temperature-scaled softmax. The student's output distribution is:

$P_s(x)^{(i)} = \frac{\exp(g_{\theta_s}(x)^{(i)} / \tau_s)}{\sum_{k=1}^K \exp(g_{\theta_s}(x)^{(k)} / \tau_s)}$

where $g_{\theta_s}$ is the student network (backbone + projection head) parameterized by $\theta_s$ , $x$ is an input image view, $i$ indexes one of $K$ output dimensions, and $\tau_s > 0$ is a temperature parameter controlling the sharpness of the student's distribution. A lower $\tau_s$ produces a peakier (more confident) distribution. The paper sets $\tau_s = 0.1$ . The teacher's distribution $P_t(x)$ uses the same formula but with its own parameters $\theta_t$ and temperature $\tau_t$ .

What this computes: for an input image crop, the network (backbone followed by projection head) produces a $K$ -dimensional vector. The softmax normalizes this vector into a probability distribution over $K$ imaginary categories. The temperature $\tau$ controls how "soft" or "sharp" the distribution is — at $\tau \to 0$ , the softmax approaches an argmax (one-hot distribution); at $\tau \to \infty$ , it approaches a uniform distribution.

Why this form: the softmax output serves as a "target" in a classification-like loss. Unlike prior methods that compare representations directly in embedding space (BYOL uses $\ell_2$ distance on normalized vectors; MoCo uses InfoNCE on embedding dot products), DINO operates in a $K$ -dimensional probability simplex. This matters because the cross-entropy loss on probability distributions naturally penalizes the student less severely when the teacher is uncertain (the gradients are scaled by the teacher's probability mass), which provides a form of adaptive learning signal. The high dimensionality $K = 65536$ (the default, with a 256-dimensional bottleneck) creates a large output space where the model can learn to assign distinct "pseudo-classes" to different visual concepts without explicit clustering.

The matching loss. Given a teacher distribution $P_t(x)$ and a student distribution $P_s(x')$ on two different views of the same image, DINO minimizes the cross-entropy:

$\min_{\theta_s} H(P_t(x), P_s(x'))$

where $H(a, b) = -a \log b$ is the cross-entropy, treating the teacher's output $P_t(x)$ as the target distribution and the student's output $P_s(x')$ as the prediction. Gradients flow only through $P_s$ (the teacher output is detached), so the optimization updates only $\theta_s$ .

What it computes: the cross-entropy between two probability distributions, measuring how many bits are needed to encode the teacher's distribution using a code optimized for the student's distribution. When the distributions match exactly, the cross-entropy equals the entropy of the teacher (minimum possible value). When they diverge, the cross-entropy increases.

Why this form: cross-entropy is the standard loss for matching probability distributions. Unlike the mean squared error used by BYOL (which operates on $\ell_2$ -normalized vectors), cross-entropy on softmax outputs naturally handles the case where the teacher expresses uncertainty by distributing probability mass across multiple dimensions — the student is penalized proportionally to the teacher's confidence. This is qualitatively different from InfoNCE (MoCo's loss), which treats all negative samples uniformly and requires large batches or memory banks to be effective. The cross-entropy form allows DINO to work with standard batch sizes (1024 or even 128) without requiring a queue or memory bank.

Multi-crop formulation. The paper extends the basic matching loss to multiple views using the multi-crop strategy introduced in SwAV (Caron et al., 2020). From each input image, DINO generates a set $V$ of different views: two global views of resolution $224 \times 224$ that cover a large area (greater than 50%) of the original image, and several local views of resolution $96 \times 96$ that cover small areas (less than 50%) of the original image. The default configuration uses 2 global views and 10 local views (denoted $2 \times 224^2 + 10 \times 96^2$ ), though the paper sweeps different numbers of local views (2, 6, 10).

The full multi-crop loss is:

$\min_{\theta_s} \sum_{x \in \{x_1^g, x_2^g\}} \sum_{\substack{x' \in V \\ x' \neq x}} H(P_t(x), P_s(x'))$

where $x_1^g$ and $x_2^g$ are the two global views. The outer sum iterates over global views (which are passed through the teacher), and the inner sum iterates over all other views (including the other global view and all local views, passed through the student). Critically, local views are only passed through the student, never through the teacher. This enforces "local-to-global" correspondences: the student must recognize that a small patch of the image (e.g., a dog's ear) belongs to the same underlying scene as the full global view, encouraging the model to learn spatial relationships and part-to-whole reasoning.

Why this form: the asymmetry — teacher sees only global views, student sees everything — is intentional. If the teacher also saw local views, the matching task would be symmetric and easier: a local view of a dog's ear would just need to match another local view of the same ear. By restricting the teacher to global views, the student is forced to map partial, local information to holistic, global representations. This is the mechanism that the paper hypothesizes leads to the emergence of spatial structure (segmentation masks) in the self-attention maps: to match a local patch to the correct global representation, the attention mechanism must learn which image regions belong together and which are distinct objects.

The loss is generalizable to any number of views. The paper notes that "this loss is general and can be used on any number of views, even only 2." When using only 2 views (both global), the loss reduces to $H(P_t(x_1^g), P_s(x_2^g)) + H(P_t(x_2^g), P_s(x_1^g))$ , which is symmetric except for the temperature and centering differences between student and teacher outputs.

Collapse Prevention: Centering and Sharpening

Self-supervised methods that use a matching loss between positive pairs risk a degenerate solution called collapse: the network outputs a constant representation regardless of the input, trivially achieving zero matching loss. Different methods prevent collapse through different mechanisms: contrastive losses push negative pairs apart (SimCLR, MoCo), clustering constraints enforce balanced assignments across samples (SwAV's Sinkhorn-Knopp), asymmetric architectures with predictors break the symmetry (BYOL, SimSiam), or batch normalization injects implicit contrastive signals through batch statistics (BYOL's reliance on BN).

DINO prevents collapse using only two operations on the teacher's output, applied before computing the cross-entropy loss: centering and sharpening. The paper shows that these two operations have complementary effects that balance each other, and together they are sufficient to avoid collapse when combined with a momentum encoder — eliminating the need for predictors, contrastive losses, clustering, or batch normalization.

Centering. The centering operation subtracts a running-mean vector $c$ from the teacher's output logits: $g_t(x) \leftarrow g_t(x) - c$ . The center $c \in \mathbb{R}^K$ is updated as an exponential moving average over the batch:

$c \leftarrow m \cdot c + (1 - m) \cdot \frac{1}{B} \sum_{i=1}^B g_{\theta_t}(x_i)$

where $m > 0$ is a rate parameter (the paper uses $m = 0.9$ by default) and $B$ is the batch size. The center is computed over the batch dimension, so it captures the average output across all samples in the current batch.

What it computes: a running estimate of the mean teacher output across all samples, which is then subtracted from each individual teacher output. This prevents any single output dimension from becoming dominant — if one dimension consistently has high activations, it will have a high mean and be subtracted more aggressively, balancing the distribution.

Why this form: the centering operation is a first-order statistic (just the mean) and does not depend on pairwise comparisons or complex normalizations. This makes it computationally cheap and batch-size independent — unlike Sinkhorn-Knopp (which requires iterative row and column normalization typically needing 3 Sinkhorn iterations) or contrastive losses (which require comparing each sample against all negatives in the batch). The paper shows in Section 5.5 that DINO works across batch sizes from 128 to 1024 with minimal accuracy degradation (k-NN top-1 of 57.9% at batch size 128 vs. 59.9% at batch size 1024 after 100 epochs), and even works at batch size 8 (reaching 35.2% after 50 epochs). This robustness is directly attributable to the simplicity of the centering operation.

Sharpening. Sharpening is achieved by using a low temperature $\tau_t$ in the teacher's softmax. The teacher's probability distribution is:

$P_t(x)^{(i)} = \frac{\exp((g_{\theta_t}(x)^{(i)} - c^{(i)}) / \tau_t)}{\sum_{k=1}^K \exp((g_{\theta_t}(x)^{(k)} - c^{(k)}) / \tau_t)}$

The paper uses a temperature schedule where $\tau_t$ is linearly warmed up from 0.04 to 0.07 during the first 30 epochs of training. The student's temperature $\tau_s$ is fixed at 0.1, so the teacher's distribution is always sharper (more peaked) than the student's — the teacher temperature is 2.5–1.4× smaller than the student's.

What it computes: a "sharpened" probability distribution where the teacher expresses higher confidence in its predictions. With $\tau_t = 0.04$ , the softmax divides logits by a very small number, making the largest logit dominate the distribution (approaching a one-hot encoding). With $\tau_t = 0.07$ , the distribution is softer but still more peaked than the student's output at $\tau_s = 0.1$ .

Why this form: sharpening has the opposite effect of centering. While centering pushes the output toward a uniform distribution (preventing any one dimension from dominating), sharpening pushes it away from uniform (encouraging the model to make confident predictions). Applying both operations simultaneously creates a balanced dynamic: centering prevents collapse to a single dimension (mode collapse), sharpening prevents collapse to the uniform distribution. The paper formalizes this by decomposing the cross-entropy:

$H(P_t, P_s) = h(P_t) + D_{KL}(P_t \| P_s)$

where $h(P_t)$ is the entropy of the teacher distribution and $D_{KL}$ is the Kullback-Leibler divergence between teacher and student.

What this decomposition reveals: the KL divergence being zero indicates collapse (constant output regardless of input). The entropy $h$ indicates which type of collapse is occurring:

If centering is removed, the entropy converges to 0 (the distribution is dominated by one dimension — mode collapse), and the KL also converges to 0 (collapse).
If sharpening is removed, the entropy converges to $-\log(1/K)$ (the uniform distribution), and the KL also converges to 0 (collapse).
With both operations, the entropy remains at an intermediate value (neither 0 nor $-\log(1/K)$ ), and the KL remains positive (no collapse).

Experimental validation of the centering-sharpening balance (Figure 7). The paper plots the entropy and KL divergence during training with and without centering and sharpening. Without centering, the teacher entropy drops to near 0 within the first few epochs (meaning all images are assigned to the same output dimension), and the KL divergence also drops to 0 (student has trivially matched the degenerate teacher). Without sharpening, the entropy rises to $-\log(1/K)$ (the uniform distribution), and KL again drops to 0. With both operations, entropy stabilizes at a mid-range value and KL remains well above 0, confirming that the training is not collapsing.

The temperature sensitivity. The paper's ablation study in Appendix D shows that the sharpening temperature $\tau_t$ has a sharp threshold:

$\tau_t = 0$ (extreme sharpening, equivalent to argmax): 43.9% k-NN top-1 — suboptimal, likely because the one-hot targets are too aggressive and discard useful uncertainty information.
$\tau_t = 0.04$ : 69.6% k-NN top-1
$\tau_t = 0.06$ : 68.7% k-NN top-1
$\tau_t = 0.08$ : 0.1% k-NN top-1 — collapse to uniform, because the teacher is now softer than the student ( $\tau_t > \tau_s = 0.1$ would make the teacher less sharp than the student with centering applied).

The warming schedule (0.04 → 0.07 over 30 epochs) achieves 69.7% k-NN top-1, slightly better than a fixed temperature. The paper hypothesizes that starting with a very sharp teacher (0.04) helps establish confident pseudo-classes early, and then relaxing to 0.07 allows the model to express richer uncertainty as training progresses.

Centering rate $m$ robustness. The smoothing parameter $m$ in the centering update controls how quickly the center adapts:

$m = 0$ : 69.1% k-NN top-1 (no smoothing — the center is just the current batch mean)
$m = 0.9$ : 69.7% (default)
$m = 0.99$ : 69.4%
$m = 0.999$ : 0.1% (collapse) — the center updates too slowly to prevent mode collapse, because the instantaneous batch mean is dominated by the exponential moving average and the correction is too small.

This robustness to a wide range of $m$ values (0 to 0.99) is important: it means the centering operation does not require careful tuning and works effectively even with an aggressively updated center.

Teacher Network Construction and Dynamics

Unlike standard knowledge distillation where the teacher is fixed and pre-trained, DINO's teacher is built dynamically from past iterations of the student. The paper studies several strategies for constructing the teacher (Figure 6, right) and finds that the momentum encoder — updating the teacher as an exponential moving average (EMA) of the student — produces the best performance, but is not the only viable option.

Momentum encoder update rule. The teacher parameters $\theta_t$ are updated as:

$\theta_t \leftarrow \lambda \theta_t + (1 - \lambda) \theta_s$

where $\lambda$ follows a cosine schedule from 0.996 to 1 during training. This means the teacher starts by incorporating 0.4% of the student's new weights at each step (so it is heavily smoothed but still tracks the student), and by the end of training, the teacher is essentially frozen (since $\lambda \to 1$ means almost no update).

What it computes: at each training iteration, the teacher's parameters are pulled slightly toward the student's current parameters. The momentum coefficient $\lambda$ controls how much the teacher "remembers" its past state versus adopting the student's new weights. With $\lambda = 0.996$ , the teacher retains 99.6% of its previous weights and adopts only 0.4% of the student's new weights — it changes very slowly, representing a temporally smoothed version of the student.

Why this form: this is identical to the momentum encoder introduced in MoCo (He et al., 2020), but the paper argues its role in DINO is fundamentally different. In MoCo, the momentum encoder provides consistent negative keys for contrastive learning, preventing the encoder from changing too rapidly and making the queue of negative samples inconsistent. In DINO, there is no queue and no contrastive loss. Instead, the paper interprets the momentum teacher as performing Polyak-Ruppert averaging (Polyak and Juditsky, 1992; Ruppert, 1988) — a standard technique for model ensembling where averaging parameters over training iterations produces a model that is better than the final iterate. Typically, Polyak-Ruppert averaging is applied after training to create the final deployed model. DINO applies it during training to create a continually improving target that guides the student.

Empirical evidence for the Polyak-Ruppert interpretation. Figure 6 (left panel) shows the k-NN evaluation accuracy of both the student and teacher throughout training (300 epochs with ViT-S/16). The key observation: the teacher consistently outperforms the student. At epoch 100, the teacher achieves approximately 68% k-NN accuracy while the student achieves approximately 64%. This gap persists throughout training, with both steadily improving but the teacher maintaining a 2–4 percentage point lead. This dynamic has not been observed in other frameworks using momentum encoders (the paper explicitly states: "This behavior has not been observed by other frameworks also using momentum, nor when the teacher is built from the previous epoch").

What this implies: the momentum teacher is not just a trick to stabilize training — it is genuinely a better model than the student at every point during training, because parameter averaging produces a more robust model (similar to how model ensembling combines multiple checkpoints). By using this better model as the target, the student receives higher-quality "supervision" than it could produce on its own at the current iteration. This creates a positive feedback loop: the student improves → the EMA teacher improves → the student gets better targets → the student improves further.

Alternative teacher strategies (Figure 6, right). The paper experiments with three alternatives:

Student copy: $\theta_t = \theta_s$ (teacher is identical to student at each iteration). Result: 0.1% k-NN top-1 — immediate collapse, because the student is trying to match itself exactly, which it can trivially do by outputting a constant (no signal to learn from).
Previous iteration: $\theta_t$ is the student's parameters from the previous training step. Result: 0.1% k-NN top-1 — also collapses, likely because the parameters change too slowly step-to-step for the difference to provide a meaningful learning signal (the student and previous-iteration student agree too closely).
Previous epoch: $\theta_t$ is the student's parameters from one full epoch ago (frozen for the entire current epoch, then updated). Result: 66.6% k-NN top-1 — competitive with BYOL (66.6%) and MoCo-v2 (62.0%), but significantly below momentum (72.8%). The previous-epoch teacher does not collapse, demonstrating that the temporal gap between student and teacher iterations matters — the teacher needs to be sufficiently "stale" to provide a non-trivial target, but not so stale that it is outdated.

Why momentum works best: the EMA teacher provides a continuously updated, smoothed target that is always slightly ahead of the student in quality (due to ensembling) but tracks the student closely enough that the learning signal remains relevant. The previous-epoch teacher introduces an artificial delay — the student is chasing a target that is one full epoch outdated, which works but is less efficient.

The cosine schedule for $\lambda$ . The paper uses a cosine schedule from 0.996 to 1, following BYOL's practice (Grill et al., 2020). This means the teacher updates more at the beginning (incorporating 0.4% of new weights at $\lambda = 0.996$ per step) and becomes progressively more frozen toward the end (at $\lambda \to 1$ , essentially static). The intuition: early in training, the student is changing rapidly and the teacher needs to track these changes to provide relevant targets; late in training, the student is converging and the teacher should be close to its final ensembled state.

The teacher's role in preventing collapse. The paper's ablation in Table 7 highlights that the momentum encoder is critical when using only centering to prevent collapse. Row 2 (no momentum, centering only) collapses to 0.1% k-NN accuracy. Row 1 (momentum, centering only) achieves 72.8%. This is because without momentum, the student and teacher are too similar — the student can trivially match the teacher's output without learning meaningful representations, and centering alone cannot prevent this. The temporal smoothing provided by momentum creates sufficient asymmetry that the student must actually learn to generalize across different views to match the teacher.

Data Augmentation Pipeline: Multi-Crop and Standard Augmentations

The data augmentation pipeline is central to DINO because it defines the "pretext task" — the student-teacher matching is only non-trivial because the student and teacher see different transformations of the same underlying image. The paper uses a combination of standard augmentations (from BYOL) and multi-crop (from SwAV).

Standard augmentations (BYOL protocol). Each view undergoes a sequence of random transformations:

Random resized crop: a random rectangular region of the image is selected, with the scale sampled uniformly from a specified range (for global views: $[s, 1]$ where $s = 0.32$ is default; for local views: $[0.05, s]$ ), and resized to the target resolution (224×224 for global, 96×96 for local).
Random horizontal flip: applied with probability 0.5.
Color jittering: random perturbations to brightness, contrast, saturation, and hue (following the standard ImageNet augmentation protocol).
Gaussian blur: applied with a random kernel size.
Solarization: a color transformation that inverts pixel values above a threshold, applied to one of the two global views (but not the other, and not to local views) to create additional diversity.

What these augmentations accomplish: they ensure that the student and teacher see dramatically different pixel-level representations of the same image — different crops, different color distributions, different blur levels. Matching representations across these transformations forces the model to learn invariances to nuisance factors (lighting, viewpoint, scale) while preserving semantic content.

Multi-crop augmentation (SwAV protocol). The paper extends the standard two-view protocol by generating multiple local views in addition to the two global views. The scale ranges are:

Global views: scale $[0.32, 1.0]$ (cover 32%–100% of the image area), resized to $224 \times 224$
Local views: scale $[0.05, 0.32]$ (cover 5%–32% of the image area), resized to $96 \times 96$

The paper uses 2 global views and a variable number of local views (default: 10, denoted $2 \times 224^2 + 10 \times 96^2$ ). The scale ranges are non-overlapping by design ( $s = 0.32$ creates the boundary), following the original SwAV implementation, though the authors note this is "arbitrarily" chosen and overlapping ranges "could definitely" work.

The scale parameter $s$ is tuned for ViTs. The default $s = 0.32$ used in DINO is significantly higher than the $s = 0.14$ used in SwAV with convnets. The paper sweeps $s$ in Appendix E and finds:

$s = 0.08$ : 65.6% k-NN top-1
$s = 0.16$ : 68.0%
$s = 0.24$ : 69.7%
$s = 0.32$ : 69.8% (near-optimal)
$s = 0.48$ : 69.5%

The higher optimal $s$ for ViTs compared to convnets suggests that ViTs benefit from larger global views (viewing more context at once) and smaller local views (seeing finer details) — this aligns with the intuition that self-attention benefits from larger spatial context to establish long-range dependencies, while smaller patches provide fine-grained information.

Multi-crop compute-performance tradeoffs (Table 8). Using ViT-S/16 on 2×8-GPU machines:

Multi-crop configuration	100-epoch top-1	100-epoch time	300-epoch top-1	300-epoch time	Peak GPU memory
$2 \times 224^2$	67.8%	15.3h	72.5%	45.9h	9.3G
$2 \times 224^2 + 2 \times 96^2$	71.5%	17.0h	74.5%	51.0h	10.5G
$2 \times 224^2 + 6 \times 96^2$	73.8%	20.3h	75.9%	60.9h	12.9G
$2 \times 224^2 + 10 \times 96^2$	74.6%	24.2h	76.1%	72.6h	15.4G

Key observations from the compute analysis:

Multi-crop improves the accuracy/time tradeoff. At 100 epochs, $2 \times 224^2 + 10 \times 96^2$ achieves 74.6% in 24.2 hours, which is better than the 72.5% achieved by $2 \times 224^2$ in 45.9 hours — a +2.1% improvement despite taking 1.9× less time. This is because the local views provide richer per-image supervision without requiring more global crops (which are the most expensive to process at 224² resolution).
Diminishing returns from additional local views. Moving from 6 to 10 local views only improves 300-epoch accuracy by +0.2% (75.9% → 76.1%) but increases time by 19% (60.9h → 72.6h).
Longer training cannot compensate for multi-crop. The $2 \times 224^2$ configuration at 300 epochs (72.5%) is worse than $2 \times 224^2 + 10 \times 96^2$ at 100 epochs (74.6%), despite taking 1.9× longer. This shows that the local-to-global correspondence objective provides a qualitatively better learning signal than more epochs of global-only matching.

Bicubic interpolation for position embeddings. ViTs use learned position embeddings that correspond to specific patch locations in a fixed-resolution grid. When processing images at different resolutions (224² vs. 96²), the number of patches changes and the position embeddings must be resized. The paper uses bicubic interpolation to adapt the position embeddings to the new scale, following the practice established in DeiT (Touvron et al., 2020) and the original ViT paper (Dosovitskiy et al., 2020).

Network Architecture: ViT Backbone and Projection Head

The network $g$ is composed of a backbone $f$ followed by a projection head $h$ : $g = h \circ f$ . After pre-training, the projection head is discarded, and only the backbone output is used for downstream tasks.

Vision Transformer (ViT) backbone. The paper uses standard ViT architectures following the DeiT implementation (Touvron et al., 2020), with the configurations summarized in Table 1:

Model	Blocks	Dim	Heads	#Tokens (224²)	#Params	Im/sec
ViT-S/16	12	384	6	197	21M	1007
ViT-S/8	12	384	6	785	21M	180
ViT-B/16	12	768	12	197	85M	312
ViT-B/8	12	768	12	785	85M	63
ResNet-50	—	2048	—	—	23M	1237

Token sequence formation. The ViT takes as input an image divided into a grid of non-overlapping contiguous patches of resolution $N \times N$ (typically $N = 16$ for "/16" models or $N = 8$ for "/8" models). Each patch is flattened and passed through a linear projection to form a $d$ -dimensional embedding (where $d$ is the "dim" in the table). The number of patches for a 224×224 image is $(224/N)^2$ , so:

ViT-S/16: $14 \times 14 = 196$ patch tokens
ViT-S/8: $28 \times 28 = 784$ patch tokens

In addition to the patch tokens, a learnable class token [CLS] is prepended to the sequence, following BERT (Devlin et al., 2018) and the original ViT. The [CLS] token is a vector of the same dimension $d$ that is initialized randomly and learned during training. After processing through all Transformer blocks, the output of the [CLS] token is used as the image representation — the projection head $h$ is attached to the [CLS] output. The total token sequence length is $(224/N)^2 + 1$ (196+1=197 for /16, 784+1=785 for /8).

The [CLS] token's role in self-supervised training. In supervised ViTs, the [CLS] token aggregates information from all patches and is fed into a classification head. In DINO, the [CLS] token is not attached to any label or supervision — it learns to aggregate information useful for the self-distillation task. The paper notes this explicitly: "We refer to this token as the class token [CLS] for consistency with previous works, even though it is not attached to any label nor supervision in our case." This is significant because the [CLS] token's self-attention maps (which patches it attends to) reveal what the model has learned to focus on — and these maps show emergent object segmentation (Figure 1, Figure 3, Figure 10).

Transformer block architecture. Each Transformer block consists of:

Pre-norm layer normalization: normalization is applied before the sub-layer, not after ("pre-norm" variant from Chen et al., 2018 and Klein et al., 2017). This is the standard in modern Transformer implementations and differs from the original Transformer which used post-norm.
Multi-head self-attention: the token representations are updated by attending to all other tokens in the sequence using the standard scaled dot-product attention mechanism.
Feed-forward network: a two-layer MLP with GELU activations, applied position-wise (same weights for each token).

Both sub-layers have residual skip connections.

Projection head $h$ . The projection head follows the design from SwAV's "prototype layer" but with modifications for ViTs. It consists of:

A 3-layer MLP: hidden dimension 2048, using GELU activations (matching the ViT backbone's activation function), with the first 2 layers followed by GELU and the third layer without activation.
$\ell_2$ normalization bottleneck: the MLP output is $\ell_2$ -normalized, constraining the representation to lie on the unit hypersphere. The bottleneck dimension is 256 by default.
Weight-normalized fully connected layer (Salimans and Kingma, 2016): this final layer projects from the 256-dimensional bottleneck to $K = 65536$ dimensions, with weight normalization applied (the weight vectors are reparameterized as $w = g \cdot v / \|v\|_2$ where $g$ is a learnable scalar and $v$ is a learnable vector).

The $\ell_2$ bottleneck is critical for training stability. Appendix C shows that without the $\ell_2$ bottleneck, deep projection heads cause DINO training to fail:

With $\ell_2$ bottleneck: 4-layer head achieves 69.3% k-NN top-1 (100 epochs)
Without $\ell_2$ bottleneck: 4-layer head achieves 0.1% k-NN top-1 (collapse)

The $\ell_2$ normalization constrains the representations before the final weight-normalized layer, preventing the logits from growing unboundedly and stabilizing the softmax computation. It also enables using a high-dimensional output space ( $K = 65536$ ) with a moderate increase in parameters (the bottleneck is only 256-dimensional, so the weight-normalized layer has $256 \times 65536 \approx 16.8$ M parameters — manageable but large enough to provide rich expressivity).

Output dimensionality $K$ . The paper sweeps $K$ in Appendix C and finds that larger dimensionality improves performance up to a point:

$K = 1024$ : 67.8% k-NN top-1
$K = 4096$ : 69.3%
$K = 16384$ : 69.2%
$K = 65536$ : 69.7% (default)
$K = 262144$ : 69.1%

The default $K = 65536$ is motivated as providing a large output space where the model can learn many distinct "pseudo-classes" without explicit clustering. The large $K$ allows the self-distillation objective to act as a soft clustering mechanism — different images or image regions can be assigned to different output dimensions, creating an emergent taxonomy of visual concepts without any labels.

Batch normalization is entirely absent in ViT + DINO. The paper states: "unlike standard convnets, ViT architectures do not use batch normalizations (BN) by default. Therefore, when applying DINO to ViT we do not use any BN also in the projection heads, making the system entirely BN-free." This is emphasized as "a great advantage" because synchronized BN across distributed processes slows down training considerably, and many prior self-supervised methods (BYOL, MoCo-v2, SwAV) relied on BN in their projection heads. The fact that DINO works without any BN also demonstrates that collapse is being prevented by the centering-sharpening mechanism and momentum encoder, not by implicit contrastive signals from batch statistics.

Implementation Details: Optimization, Schedules, and Training Configuration

The paper provides a complete specification of the training recipe, which is essential for reproducibility and understanding the computational characteristics.

Optimizer. AdamW (Loshchilov and Hutter, 2018) is used with:

Base learning rate determined by linear scaling: $lr = 0.0005 \times \text{batchsize} / 256$ . For the default batch size of 1024, this gives $lr = 0.0005 \times 4 = 0.002$ .
Weight decay follows a cosine schedule from 0.04 to 0.4 (increasing over time, which is unusual — most schedules decrease weight decay or keep it constant).
AdamW $\beta$ parameters are not explicitly stated, but the paper uses the standard defaults from the PyTorch implementation (likely $\beta_1 = 0.9, \beta_2 = 0.999$ ).

Learning rate schedule.

Linear warmup for the first 10 epochs: the learning rate starts at 0 and linearly increases to the base value.
Cosine decay after warmup: the learning rate follows a half-cosine curve from the base value to 0 over the remaining epochs (Loshchilov and Hutter, 2016).

Batch size. Default is 1024, distributed over 16 GPUs (64 per GPU for ViT-S/16). The paper shows robustness to batch size in Section 5.5: with batch size 128 (single GPU), 100 epochs without multi-crop achieves 57.9% k-NN top-1 vs. 59.9% with batch size 1024 — a small degradation. With batch size 8 (an extreme setting barely fitting one image per GPU), the model reaches 35.2% after 50 epochs, showing that the method can scale down to limited hardware.

Training duration. Default is 300 epochs for most experiments. The paper also reports 800-epoch results for ViT-S variants (72.8% → 74.5% k-NN with ViT-S/16). The 800-epoch training produces the best reported k-NN results for DINO + ViT-S/16.

Temperature schedules:

Student temperature $\tau_s$ : fixed at 0.1 throughout training.
Teacher temperature $\tau_t$ : linearly warmed up from 0.04 to 0.07 during the first 30 epochs, then held at 0.07 for the remaining epochs.

Momentum schedule for teacher:

$\lambda$ follows a cosine schedule from 0.996 to 1 during training. This means the teacher update rate $1 - \lambda$ starts at 0.004 and decreases to 0.

Centering momentum $m$ :

Default value 0.9 (used in all experiments unless otherwise specified). This means the center updates as $c \leftarrow 0.9 \cdot c + 0.1 \cdot \text{batch\_mean}$ .

Data augmentations:

Following BYOL protocol exactly: random resized crop, horizontal flip, color jittering, Gaussian blur, and solarization (applied to one global view only).
Multi-crop: 2 global views at $224^2$ (scale $[0.32, 1.0]$ ) + 10 local views at $96^2$ (scale $[0.05, 0.32]$ ), unless otherwise specified.
Bicubic interpolation for resizing position embeddings when input resolutions differ from the training resolution.

Distributed training. Experiments use 2×8-GPU machines (16 GPUs total) for the default ViT-S/16 configuration. Training takes 3 days (72.6 hours) for 300 epochs with the $2 \times 224^2 + 10 \times 96^2$ crop setting.

The Algorithm in Pseudocode (Algorithm 1, Formalized in Prose)

The paper provides a PyTorch-style pseudocode for DINO without multi-crop. The multi-crop extension is described in text. Here is the complete algorithm flow:

Initialization:

Initialize student network $g_{\theta_s}$ randomly (ViT backbone + projection head).
Copy student parameters to teacher: $\theta_t \leftarrow \theta_s$ .
Initialize center $c \in \mathbb{R}^K$ to zeros.

For each training iteration (batch of $B$ images, each with $V$ views total: 2 global + $L$ local):

Generate views: For each of the $B$ images in the batch, randomly sample 2 global crops ( $224^2$ , scale $[0.32, 1.0]$ ) and $L$ local crops ( $96^2$ , scale $[0.05, 0.32]$ ). Apply all standard augmentations (color jittering, Gaussian blur, horizontal flip, solarization on one global view).
Forward pass — teacher: Feed only the global views through the teacher network, producing logits $g_{\theta_t}(x)$ for each global view. Center these logits: $g_{\theta_t}(x) - c$ . Apply sharpened softmax with temperature $\tau_t$ to get teacher probability distributions $P_t$ for each global view.
Forward pass — student: Feed all views (global + local) through the student network, producing logits $g_{\theta_s}(x)$ for each view. Apply softmax with temperature $\tau_s$ to get student probability distributions $P_s$ .
Compute loss: For each global view, compute cross-entropy between its teacher distribution and the student distributions of all other views (both the other global view and all local views): $\mathcal{L} = \sum_{x \in \text{global}} \sum_{x' \in \text{all}, x' \neq x} -P_t(x) \log P_s(x')$ Average this sum over the batch.
Backward pass: Compute gradients of $\mathcal{L}$ with respect to $\theta_s$ only (teacher outputs are detached). Update student parameters with AdamW.
Update teacher: $\theta_t \leftarrow \lambda \theta_t + (1 - \lambda) \theta_s$ where $\lambda$ follows cosine schedule 0.996 → 1.
Update center: $c \leftarrow m \cdot c + (1 - m) \cdot \text{mean}(g_{\theta_t}(x) \text{ over batch})$ where $m = 0.9$ .

After pre-training (evaluation):

Discard the projection head $h$ .
Use the backbone $f$ to extract features: the output of the [CLS] token (and optionally, concatenated features from the last 4 layers for linear evaluation with ViT-S, or pooled patch tokens for ViT-B).
Apply k-NN classification or train a linear classifier on frozen features.

Evaluation Protocols

The paper uses three evaluation protocols, which are important to understand because the observed properties (k-NN friendliness, emergent segmentation) depend on the evaluation method used.

1. k-NN classification (Section 3.2, Appendix F.1). This is the simplest and most revealing evaluation:

Freeze the pre-trained model and compute the feature representation of every image in the downstream training set (e.g., ImageNet training set). The representation is the output [CLS] token ( $d = 384$ for ViT-S, $d = 768$ for ViT-B), with no data augmentation applied.
Store all training features in a database.
For each test image, compute its feature, find the k nearest neighbors in the stored training features using cosine similarity, and predict the label by weighted voting: class $c$ gets weight $\sum_{i \in \mathcal{N}_k} \alpha_i \cdot \mathbb{1}_{c_i = c}$ where $\alpha_i = \exp(\text{sim}(x, x_i) / \tau)$ with $\tau = 0.07$ (not tuned).
$k = 20$ is used consistently across runs. The paper sweeps $k$ and finds 20 works best.
This protocol requires no hyperparameter tuning (beyond $k$ , which is swept once and fixed), no data augmentation, and only one pass over the dataset.

Why this matters: the k-NN evaluation directly probes the structure of the feature space — if features form coherent clusters by class without any task-specific adaptation, k-NN accuracy will be high. The paper's finding that DINO ViT features achieve k-NN accuracy close to linear probing accuracy (74.5% vs. 77.0% for ViT-S/16, 78.3% vs. 79.7% for ViT-S/8) is evidence that the features inherently organize by semantic class, without requiring a learned linear separator to disentangle them.

2. Linear classification (Appendix F.2). The standard self-supervised evaluation protocol:

Remove the projection head; freeze the backbone.
Train a linear classifier (single fully-connected layer) on top of frozen features using SGD with batch size 1024, no weight decay, for 100 epochs on ImageNet.
Only random resized crops and horizontal flips as data augmentation during training. Report central-crop top-1 accuracy.
For ViT-S: concatenate [CLS] tokens from the last 4 layers (following BERT's feature-based evaluation practice; Devlin et al., 2018). This gives a $4 \times 384 = 1536$ -dimensional representation. The paper finds $l = 4$ layers to be optimal, improving accuracy from 76.1% (1 layer) to 77.0% (4 layers).
For ViT-B: use only the final layer [CLS] token, concatenated with average-pooled output patch tokens (following the "global average pooling" practice in convnet evaluations). This gives a $768 + 768 = 1536$ -dimensional representation. The paper finds that concatenating multiple [CLS] layers does not help ViT-B.
Learning rate is swept for each model.

3. Fine-tuning (Section 4.2.3, Table 6). The full model (backbone + task-specific head) is initialized with pre-trained weights and all parameters are updated during training on downstream tasks. The paper follows the DeiT fine-tuning protocol (Touvron et al., 2020).

4. Video object segmentation (Section 4.2.2, Table 5). Uses a nearest-neighbor matching protocol between consecutive video frames (following Jabri et al., 2020), directly on frozen patch tokens without any fine-tuning or task-specific training. This tests whether the features retain spatial information useful for dense correspondence tasks.

Summary of Key Design Choices and Their Justifications

Cross-entropy loss on softmax outputs over InfoNCE or MSE: The cross-entropy on probability distributions allows DINO to work without large batches or memory banks (which contrastive losses require), and provides a richer learning signal than MSE on $\ell_2$ -normalized vectors (which treats all dimensions uniformly). The sharpened teacher distributions provide confident targets when the teacher is certain and diffuse targets when uncertain, creating an adaptive learning signal.

Momentum encoder teacher over fixed teacher or no teacher: The momentum teacher provides a slowly-evolving, ensemble-quality target that consistently outperforms the student (Figure 6). This is essential both for preventing collapse (centering alone fails without momentum; Table 15, row 4 → 0.1%) and for providing high-quality learning targets.

Centering + sharpening over other collapse prevention mechanisms: Prior methods use contrastive losses (MoCo, SimCLR), clustering constraints (SwAV), predictors (BYOL, SimSiam), or batch normalization (BYOL). DINO replaces all of these with just two first-order operations on the teacher output. This makes the method simpler, batch-size independent, and entirely BN-free when applied to ViTs.

Small patches (8×8) over standard patches (16×16): Reducing patch size from 16 to 8 quadruples the number of tokens (197 → 785) and the compute cost (1007 → 180 im/sec), but significantly improves feature quality (+3.6% k-NN for ViT-S, from 72.8% to 76.4% at 800 epochs; Figure 5). The paper hypothesizes this is because smaller patches provide finer spatial resolution, enabling the self-attention to learn more precise spatial relationships (supporting the emergent segmentation property).

$\ell_2$ normalization bottleneck in projection head: The bottleneck constrains the representation before the high-dimensional weight-normalized layer, preventing logit explosion and enabling stable training with deep projection heads. Without it, heads with more than 2 layers collapse (Appendix C).

Two-fold cross-validation-style evaluation: While not a training design choice, the paper uses a standard evaluation protocol that sweeps hyperparameters (learning rate for linear evaluation, number of nearest neighbors for k-NN) on the validation set, which is important for reproducing the reported numbers. The k-NN evaluation eliminates this tuning step entirely, making it the most reproducible and fair comparison protocol.

4. Key Insights and Innovations

Innovation 1: Self-Supervised ViTs Exhibit Emergent Spatial Understanding That Supervised ViTs and ConvNets Lack

The paper’s most striking conceptual contribution is not a new algorithm but a diagnostic discovery: when you train a Vision Transformer with self-supervision instead of supervision, its internal representations spontaneously organize around object structure in ways that neither supervised ViTs nor convnets (under any training regime) achieve. This is a property-level finding, not a methodological advance, and it reframes the question of what self-supervised learning is actually doing inside Transformers.

What makes this distinctive at the idea level: Prior work had already established that self-supervised convnets learn useful features, and that supervised ViTs can match or exceed convnet performance. The dominant assumption was that self-supervised pre-training’s primary benefit was simply providing more data-efficient representations — better transfer, better few-shot performance, but fundamentally similar kinds of features. The paper challenges this directly by showing that the self-attention maps of the final Transformer block, when queried with the [CLS] token, produce explicit, coherent object segmentation masks (Figure 1). These masks emerge without any segmentation-specific objective, without any pixel-level supervision, and without any architectural modifications — the model was trained only to match different views of unlabeled images. The fact that this property appears with self-supervised ViTs but not with supervised ViTs (Figure 4) suggests that the training objective, not the architecture alone, is responsible.

The diagnostic move that makes this a contribution: The paper doesn't just claim "self-supervised features are better" — it probes where the spatial information lives and how it differs across training paradigms. Figure 4 quantifies this by thresholding self-attention maps (keeping 60% of the probability mass) and computing Jaccard similarity against ground-truth segmentation masks on PASCAL VOC:

Supervised ViT-S/16: 27.3 Jaccard
DINO ViT-S/16: 45.9 Jaccard
Random ViT-S/16: 22.0

The supervised model barely outperforms random weights for this property, while DINO nearly doubles the similarity. This is not simply a "better features" story — the supervised ViT achieves comparable or better ImageNet classification accuracy (79.8% vs. 77.0% linear), yet its attention maps contain almost no explicit segmentation information. The spatial layout is present in self-supervised features but suppressed or absent in supervised ones, even though both models use identical architectures and see the same images during training.

Comparison to prior work: Before this paper, the closest evidence for emergent spatial structure in self-supervised models came from work on convnets (Gur et al., 2020), which showed that segmentation-like information could be extracted from convnet weights using dedicated factorization methods. However, that information was latent — it required specialized post-hoc techniques to reveal. DINO's contribution is showing that with ViTs, the spatial structure is directly accessible in the self-attention maps with no post-processing beyond visualizing attention weights. This is a qualitative shift: from "spatial information is buried in the weights and can be excavated" to "spatial information is explicit in the forward pass." The Appendix G results (Table 5) further confirm this is not limited to single-image segmentation — the frozen patch tokens work for video object segmentation via nearest-neighbor matching between frames, achieving 69.9 (J&F)m with ViT-S/8 without any fine-tuning, competitive with methods specifically designed for the task (MAST at 65.5, STC at 67.6).

Significance beyond performance: This finding matters because it suggests self-supervised pre-training could be the key to unlocking a "BERT moment" for vision — where a model pre-trained on massive unlabeled image collections learns not just classification-ready features, but rich, structured representations of visual scenes that include object localization, part-whole relationships, and scene layout without any explicit spatial supervision. The paper positions this as evidence that "self-supervised learning could be the key to developing a BERT-like model based on ViT" (Section 6), and the emergent segmentation provides concrete evidence for this vision. It also explains why DINO features excel at dense tasks like video segmentation and retrieval: the spatial structure is already encoded in the representation, so downstream tasks don't need to learn it from scratch.

Innovation 2: Self-Distillation with No Labels as a Unifying Framework for Self-Supervised Learning

DINO's formulation — training a student to match a momentum teacher's sharpened, centered outputs using cross-entropy — is not itself algorithmically novel in its individual components (momentum encoder from MoCo, multi-crop from SwAV, cross-entropy matching from knowledge distillation). What is novel is the conceptual synthesis: casting the entire self-supervised pre-training objective as self-distillation with no labels, where the teacher is dynamically constructed from the student during training rather than being a pre-trained fixed model.

Why this framing matters: Prior self-supervised methods were conceptualized through specific mechanisms designed to prevent collapse: contrastive learning (pushing negatives apart), clustering (enforcing balanced assignments), or architectural asymmetry (predictor networks breaking the symmetry between student and teacher). Each of these framings carried its own conceptual baggage and practical constraints — contrastive methods required large batches or memory banks, clustering methods required iterative Sinkhorn-Knopp normalization, and asymmetric methods required careful design of the predictor architecture. DINO's distillation framing unifies these under a single interpretation: the goal is simply to match probability distributions between a student and a better (momentum-averaged) teacher, and collapse is prevented by balancing entropy through centering and sharpening. No negatives, no clustering, no architectural asymmetry needed.

The empirical evidence that validates this framing: Figure 6 (left) is the critical diagnostic. It shows that throughout training, the momentum teacher consistently achieves higher k-NN accuracy than the student — at epoch 100, roughly 68% vs. 64%; at epoch 300, roughly 73% vs. 71%. The paper states explicitly: "This behavior has not been observed by other frameworks also using momentum, nor when the teacher is built from the previous epoch." This is the hallmark of genuine distillation: the teacher is a better model, and the student improves by trying to match it. The momentum encoder, originally designed for maintaining consistent negative keys in contrastive learning, is reinterpreted as performing Polyak-Ruppert averaging during training — constructing an ensemble-quality model that serves as a superior target.

Comparison to prior work: Mean Teacher (Tarvainen and Valpola, 2017) used EMA teachers for semi-supervised learning, but required labeled data for the supervised loss component. BYOL (Grill et al., 2020) used an EMA teacher with MSE loss on $\ell_2$ -normalized vectors, but required a predictor network to prevent collapse and relied on batch normalization (Chen and He, 2020 showed BYOL fails without BN). DINO removes both the predictor and the BN dependence, showing that the distillation framing works with just centering and sharpening — two simple, first-order operations on the teacher output. This is a simplification, not a complication, and it makes the framework both architecture-agnostic (works on convnets and ViTs without modification) and batch-size independent (works at batch size 8, Table 9).

Significance beyond performance: The self-distillation framing is significant because it suggests a mechanism for why self-supervised learning works, not just a recipe. If the teacher is consistently better than the student (due to parameter averaging), the learning process becomes a virtuous cycle: the student learns to match the teacher → the student improves → the EMA teacher improves further → the student gets even better targets. This is a clean conceptual model that can guide future research — for example, predicting that any technique which improves the teacher's quality relative to the student (better averaging, longer momentum horizons, ensemble of multiple past checkpoints) should accelerate learning. The paper's finding that a previous-epoch teacher (Figure 6, right) achieves 66.6% k-NN (competitive with BYOL's 66.6% and better than MoCo-v2's 62.0%) but not as good as the momentum teacher (72.8%) directly supports this: the previous-epoch teacher is also a form of parameter averaging (just coarser), so it provides a weaker but still functional distillation target. This continuum — from cop

y (collapse, 0.1%) to previous-iteration (collapse, 0.1%) to previous-epoch (66.6%) to momentum (72.8%) — maps cleanly onto the quality of the teacher as an ensemble.

Innovation 3: The Interaction Between Architecture and Self-Supervised Components Is Non-Trivial and Architecture-Specific

A substantial portion of the paper's value comes from what is essentially a well-controlled negative result: the components that work well for self-supervised convnets do not transfer straightforwardly to ViTs, and the interactions are complex enough that naive transfer will leave substantial performance on the table or even cause training to fail entirely. This is not a new algorithm — it’s a diagnostic contribution that reframes how the community should think about self-supervised learning research.

The core finding: Table 13 shows that when switching from ResNet-50 to ViT-S, the ranking of self-supervised methods changes dramatically. On ResNet-50, SwAV (74.1% linear, 65.4% k-NN), BYOL (72.7%, 65.4%), and DINO (74.5%, 65.6%) are all comparable — within a ~2% band. On ViT-S, the methods diverge sharply: DINO achieves 76.1% linear and 72.8% k-NN, while BYOL drops to 71.4%/66.6%, SwAV to 71.8%/64.7%, and MoCo-v2 to 71.6%/62.0%. The k-NN gap is particularly revealing: DINO’s 72.8% vs. the next-best BYOL at 66.6% represents a 6.2 percentage point gap — far larger than anything seen with convnets.

The mechanism behind the divergence: Table 14 provides the systematic ablation that explains this. Multi-crop training, which SwAV introduced and which the community had largely treated as a universal performance booster, interacts differently with each loss function and architecture:

Removing multi-crop from DINO hurts by 3.6% (76.1% → 72.5%).
Removing multi-crop from MoCo-v2 hurts by 1.9% (73.4% → 71.4%).
Multi-crop with BYOL on ViT-S degrades linear evaluation from 71.4% (without) to 64.8% (with), though k-NN drops even further from 66.6% to 59.8%.

This is not a hyperparameter sensitivity issue — Appendix E documents that "we have performed learning rate, weight decay, multi-crop parameters sweeps for this setting and systematically observe the same pattern" with BYOL on ViT-S. The finding that a component which is unequivocally beneficial for one architecture-framework combination can be actively harmful for another combination challenges the common practice of treating self-supervised components as interchangeable modules that can be mixed and matched.

Why this matters conceptually: The field of self-supervised learning has developed through an accumulation of components — momentum encoders, multi-crop, projection heads, predictor networks, specific loss functions — that were almost exclusively designed, tuned, and validated on ResNet architectures. The implicit assumption was that these components solved general problems (preventing collapse, providing rich learning signals) in architecture-independent ways. DINO’s ablation study demonstrates that this assumption is false: the components interact with architectural inductive biases (convolutional locality vs. self-attention globality, batch normalization dependence, sequence processing) in ways that are not separable. This implies that self-supervised learning research cannot safely treat architecture as an afterthought — new architectures require re-examining the entire component stack, not just swapping the backbone.

Significance beyond performance: This insight has methodological implications for how self-supervised learning research should be conducted. A paper that proposes a new component and validates it only on ResNet-50 may be overclaiming its generality. Conversely, a component that seems lackluster on convnets might shine on ViTs (as multi-crop does for DINO). The paper’s approach — systematically ablating components across both architectures and loss functions, and reporting both linear and k-NN evaluations (which can rank methods differently, as Table 13 shows) — provides a template for more rigorous empirical methodology in the field. The finding that k-NN evaluation sometimes reveals gaps that linear evaluation obscures (e.g., DINO ViT-S at 77.0% linear vs. 74.5% k-NN, compared to BYOL’s 71.4% vs. 66.6% — the k-NN gap of 7.9 points far exceeds the linear gap of 5.6 points) suggests that evaluation protocol choice can mask or reveal properties of the learned representations, and future work should report both.

Innovation 4: Collapse Prevention Can Be Achieved Through Dual Entropy Control Rather Than Architectural or Loss-Based Mechanisms

Prior to DINO, the self-supervised learning literature had converged on three strategies for preventing representational collapse: contrastive losses that explicitly push apart negative pairs (SimCLR, MoCo), architectural asymmetries that break the symmetry between twin networks (BYOL's predictor, SimSiam's stop-gradient plus predictor), and clustering-based constraints that enforce balanced assignment across samples (SwAV's Sinkhorn-Knopp, DeepCluster's pseudo-label balancing). Each of these strategies carried significant costs — contrastive methods required large batches or memory banks, asymmetric methods were fragile to architectural choices (BYOL collapsed without batch normalization), and clustering methods required expensive iterative normalizations.

DINO demonstrates that an entirely different mechanism — what the paper calls the balance between centering and sharpening — can prevent collapse with nothing more than two first-order operations on the teacher's output distribution. This is a conceptual contribution about the degrees of freedom in collapse prevention. The paper decomposes the cross-entropy loss into $H(P_t) + D_{KL}(P_t \| P_s)$ and shows that the two operations control different collapse modes: centering prevents the entropy from going to zero (mode collapse to a single dimension), while sharpening prevents the entropy from going to its maximum $\log K$ (collapse to the uniform distribution). Together, they keep the entropy at a mid-range value and the KL divergence away from zero — which is both necessary and sufficient for stable training.

The diagnostic evidence (Figure 7): Without centering, the teacher entropy collapses to near-zero and KL drops to zero. Without sharpening, the entropy rises to $\log K$ and KL drops to zero. With both, the entropy stabilizes at intermediate values and KL remains positive. This is a clean, mechanistic demonstration that collapse is fundamentally an entropy control problem, and that it can be solved without negative samples, without architectural asymmetry, and without iterative clustering — just by carefully balancing the entropy of the target distribution.

Why this is fundamental rather than incremental: Previous collapse prevention strategies were indirect — they prevented collapse as a side effect of their primary mechanism (contrasting negatives, clustering, architectural asymmetry). DINO's centering-sharpening mechanism is direct — it explicitly controls the entropy of the teacher distribution to keep it in a regime where the student can learn without collapsing. This reframes the collapse problem from "what mechanism can we design that happens to avoid collapse?" to "what are the sufficient conditions on the teacher distribution that prevent collapse, and how minimally can we enforce them?" The answer — just two first-order operations — suggests that many of the complex mechanisms in prior self-supervised methods may have been doing more than just preventing collapse (e.g., providing additional learning signals, regularizing the representation), and that disentangling these effects could lead to simpler, more robust methods.

Empirical evidence that this mechanism is robust: The paper demonstrates that the centering-sharpening balance works across batch sizes from 8 to 1024 (Table 9, Appendix D), with the centering rate $m$ robust from 0 to 0.99 (only collapsing at 0.999), and with sharpening temperature $\tau_t$ having a sharp but predictable threshold (collapse only when $\tau_t > 0.08$ , at which point the teacher becomes softer than the student). This robustness stands in contrast to the fragility of, say, BYOL without batch normalization or SimSiam without a carefully tuned predictor — both of which can collapse entirely under small hyperparameter changes.

Significance beyond this paper: The entropy-control perspective opens up new directions for designing self-supervised methods. Instead of engineering complex loss functions or architectural modifications, future work could focus on directly controlling the entropy and concentration of the teacher distribution through other means — learned priors, adversarial training to maintain a target entropy, or adaptive temperature schedules that respond to the current state of collapse risk. The paper's finding that the teacher consistently outperforms the student (Figure 6, left) further suggests that the quality of the target distribution matters not just for preventing collapse but for the efficiency of learning — a teacher with well-calibrated uncertainty (entropy neither too high nor too low) provides the most informative learning signal.

5. Experimental Analysis

Evaluation Methodology

Dataset. All pre-training uses the ImageNet dataset (Russakovsky et al., 2015) without labels — approximately 1.28 million training images across 1,000 classes. The standard ImageNet validation set (50,000 images) is used for all classification evaluations. Additional datasets used for downstream evaluation include: Places205 (Zhou et al., 2014), PASCAL VOC 2007 (Everingham et al., 2010), Oxford-102 Flowers (Nilsback and Zisserman, 2008), CIFAR-10, CIFAR-100, iNaturalist 2018/2019, and the DAVIS-2017 video object segmentation benchmark (Pont-Tuset et al., 2017). For image retrieval, the paper uses revisited Oxford and Paris datasets (Radenović et al., 2018) and Google Landmarks v2 (Weyand et al., 2020). Copy detection evaluation uses the INRIA Copydays "strong" subset (Douze et al., 2009) with 10k distractors from YFCC100M (Thomee et al., 2015). For the transfer learning experiments in Table 6, models are fine-tuned on CIFAR-10, CIFAR-100, iNaturalist 2018, iNaturalist 2019, Oxford-102 Flowers, Stanford Cars, and ImageNet itself.
Base model(s). The paper primarily uses Vision Transformer architectures from the DeiT family (Touvron et al., 2020): ViT-Small (12 blocks, 384 dimensions, 6 heads, 21M parameters) and ViT-Base (12 blocks, 768 dimensions, 12 heads, 85M parameters), both with patch sizes of either 16×16 ("/16") or 8×8 ("/8"). A ResNet-50 (He et al., 2016) with 23M parameters is used for architecture-comparison experiments. The choice of ViT-S is motivated by its similarity to ResNet-50 in parameter count (21M vs. 23M), throughput (1007 vs. 1237 im/sec), and supervised ImageNet performance (79.8% vs. 79.3%), enabling fair architecture-to-architecture comparisons. The ViT-B variants and larger ResNet configurations (ResNet-50w2, ResNet-50w4, ResNet-50w5, ResNet-152w3+SK, ResNet-200w2) are explored for across-architecture comparisons to establish the performance ceiling.
Metrics. The primary metric is top-1 classification accuracy on the ImageNet validation set, evaluated using three distinct protocols: (1) k-NN classification — a weighted 20-nearest-neighbor classifier with cosine similarity and temperature τ = 0.07, requiring no hyperparameter tuning or data augmentation, with accuracy computed in a single pass over the stored training features; (2) linear classification — a linear classifier trained with SGD on frozen features for 100 epochs using only random resized crops and horizontal flips, with learning rate swept per model and central-crop accuracy reported; (3) fine-tuning — the full model initialized with pre-trained weights and all parameters updated on the downstream task. For retrieval tasks, Mean Average Precision (mAP) is used on the Medium and Hard splits of revisited Oxford and Paris. For video object segmentation on DAVIS-2017, the metrics are mean region similarity J_m and mean contour-based accuracy F_m, along with their average (J&F)_m. For copy detection, mAP is reported on the "strong" subset of Copydays. For the self-attention segmentation quality probe, Jaccard similarity (intersection over union) between thresholded self-attention maps and ground-truth PASCAL VOC 2012 segmentation masks is used.
Baselines. The paper compares DINO against several self-supervised methods: MoCo-v2 (Chen et al., 2020), using an InfoNCE contrastive loss with a momentum encoder and queue; BYOL (Grill et al., 2020), using MSE loss between a student with predictor and a momentum teacher; SwAV (Caron et al., 2020), using swapped cluster assignment prediction with Sinkhorn-Knopp normalization and multi-crop; SimCLR (Chen et al., 2020), using an InfoNCE loss with large batches; Barlow Twins (Zbontar et al., 2021), using redundancy reduction; InfoMin (Tian et al., 2020); OBoW (Gidaris et al., 2020); and DeepCluster-v2 (Caron et al., 2020). For across-architecture comparisons, the paper includes SimCLR-v2 (Chen et al., 2020) with various ResNet scales. Supervised baselines include a standard supervised ResNet-50, supervised ViT models trained with the DeiT procedure (Touvron et al., 2020), and supervised ViT models pretrained on JFT-300M from the original ViT paper (Dosovitskiy et al., 2020). For the copy detection task, the MultiGrain model (Berman et al., 2019), which was specifically designed for particular object retrieval, serves as an upper-bound reference. For video object segmentation, baselines include supervised ViT-S/8 trained on ImageNet, STM (Oh et al., 2019), CT (Wang et al., 2019), MAST (Lai et al., 2020), and STC (Jabri et al., 2020).
Generation budget / compute accounting. The paper does not measure a "generation budget" in the sense of LLM sampling — this is a self-supervised pre-training paper where compute is measured by training epochs, GPU hours, throughput (images per second at inference), and memory usage (peak GPU memory per device). The primary controlled compute axis is training duration (100, 300, or 800 epochs). For comparing compute efficiency, the paper reports throughput in images/second on a single NVIDIA V100 GPU with 128 samples per forward pass (Table 1), and total training time on 2×8-GPU machines for various crop configurations (Table 8). The FLOPs of different architectures are not computed directly; instead, the paper uses parameter count and throughput as efficiency proxies.
Cross-validation / statistical protocol. For linear evaluation, the learning rate is swept for each model independently, and the best learning rate is selected based on validation accuracy — this is standard practice but means the reported numbers are not strictly comparable across methods unless all methods undergo the same sweep. The k-NN evaluation eliminates this tuning: k = 20 is used consistently across all models and runs, and the temperature τ = 0.07 is fixed without tuning. For the self-attention segmentation probe on PASCAL VOC 2012, the threshold is fixed at 60% of the attention mass across all models, with no per-model tuning. No cross-validation across multiple training runs is reported — the paper does not provide error bars, confidence intervals, or standard deviations for any result, which is a notable omission given the often-small gaps between methods (e.g., the 0.2% difference between DINO and SwAV on ResNet-50 in Table 13). The paper does not describe any statistical significance testing.

Main Quantitative Results

Comparing DINO with SSL Frameworks on ImageNet (Table 2, Table 13)

Same-architecture comparison (ResNet-50 and ViT-S). On ResNet-50 after 300 epochs of pre-training, DINO achieves 75.3% linear top-1 and 67.5% k-NN top-1, placing it on par with SwAV (75.3% linear, 65.7% k-NN) and slightly ahead of BYOL (74.4% linear, 64.8% k-NN) and DeepCluster-v2 (75.2% linear, 67.1% k-NN). The performance band across the top methods is narrow — roughly 2 percentage points — consistent with the field's state where multiple methods achieve similar convnet performance. On ViT-S, the landscape shifts dramatically. DINO achieves 77.0% linear top-1 and 74.5% k-NN top-1, outperforming BYOL (71.4% linear, 66.6% k-NN) by +5.6 points in linear evaluation and +7.9 points in k-NN, MoCo-v2 (72.7% linear, 64.4% k-NN) by +4.3 and +10.1 points, and SwAV (73.5% linear, 66.3% k-NN) by +3.5 and +8.2 points. The k-NN gaps are substantially larger than the linear gaps for every method, indicating that DINO ViT features are qualitatively more amenable to simple nearest-neighbor retrieval than features from other methods — the gap is not just about linear separability. Notably, DINO ViT-S achieves 74.5% k-NN accuracy, which is only 2.5 points below its linear accuracy of 77.0%, while BYOL ViT-S shows a much larger gap of 4.8 points (71.4% − 66.6%). Supervised ViT-S achieves 79.8% in both evaluations (linear and k-NN are identical since a supervised model's features are already linearly separated by class).

Across-architecture comparison (bottom panel of Table 2). Moving to larger models and smaller patches, DINO ViT-S/8 (21M parameters, 180 im/sec) achieves 79.7% linear and 78.3% k-NN top-1. DINO ViT-B/16 (85M parameters, 312 im/sec) reaches 78.2% linear and 76.1% k-NN. DINO ViT-B/8 (85M parameters, 63 im/sec) achieves the best results: 80.1% linear and 77.4% k-NN. For context, the previous state of the art, SimCLR-v2 with ResNet-152w3+SK (794M parameters, 46 im/sec), achieved 79.8% linear and 73.1% k-NN. DINO ViT-B/8 reaches higher linear accuracy (80.1% vs. 79.8%) with 9.3× fewer parameters (85M vs. 794M) and 1.4× faster inference (63 vs. 46 im/sec). The k-NN gap is even more pronounced: 77.4% for DINO ViT-B/8 vs. 73.1% for SimCLR-v2 — a 4.3-point advantage despite the vast parameter difference.

Table 13 provides a controlled architecture swap experiment. When trained with ResNet-50, the same codebase and hyperparameter tuning procedure yield: MoCo-v2 at 71.1% linear / 62.9% k-NN, BYOL at 72.7% / 65.4%, SwAV at 74.1% / 65.4%, and DINO at 74.5% / 65.6% — all clustered within a 3.4-point linear range. On ViT-S with identical training recipes, the spread widens: MoCo-v2 drops to 71.6% / 62.0%, BYOL to 71.4% / 66.6%, SwAV to 71.8% / 64.7%, and DINO rises to 76.1% / 72.8%. DINO gains +1.6 points in linear accuracy when switching from ResNet-50 to ViT-S, while BYOL loses −1.3 points and SwAV loses −2.3 points. The k-NN behavior is even more asymmetric: DINO gains +7.2 k-NN points from the architecture switch, while BYOL gains only +1.2 and MoCo-v2 loses −0.9. This demonstrates that the benefit is not simply "ViTs are better architectures" — if that were true, all methods would improve proportionally. Instead, DINO's specific combination of components (momentum encoder with centering-sharpening, cross-entropy loss, multi-crop) extracts disproportionately more value from the ViT architecture than other self-supervised frameworks do.

Properties of ViT Trained with SSL

Nearest neighbor retrieval performance (Tables 3 and 4). On image retrieval, DINO ViT-S/16 trained on ImageNet achieves 41.8 mAP on the Medium split and 13.7 mAP on the Hard split of revisited Oxford, and 63.1/34.4 on revisited Paris — substantially outperforming supervised ViT-S/16 (33.5/8.9 and 63.0/37.2) and DINO ResNet-50 (35.4/11.1 and 55.9/27.5). The supervised ViT-S/16 was trained on the same ImageNet dataset with the DeiT procedure, so the comparison controls for architecture and data — the only difference is training objective (supervised classification vs. DINO). The DINO features outperform supervised features despite the supervised model having access to the exact class labels that define semantic similarity. When DINO ViT-S/16 is trained on the unlabeled Google Landmarks v2 (GLDv2) dataset — which contains landmark images specifically relevant to the retrieval task — performance jumps to 51.5/24.3 on Oxford and 75.3/51.6 on Paris, surpassing all off-the-shelf methods. This demonstrates one of self-supervised learning's practical advantages: the ability to pre-train on domain-specific unlabeled data without annotation cost. For copy detection on the Copydays "strong" subset (Table 4), DINO ViT-B/16 at 224² resolution achieves 81.7 mAP, outperforming both supervised ViT-B/16 (76.4) and the specialized MultiGrain model (75.1) designed for particular object retrieval. At 320² resolution, DINO ViT-B/8 reaches 85.5 mAP, compared to MultiGrain's 82.5 at resolution 800 — achieving better performance with less than half the maximum image dimension.

Semantic layout discovery (Tables 4 and 5, Figure 4). On DAVIS-2017 video object segmentation using frozen features with a simple nearest-neighbor matching protocol between consecutive frames (no fine-tuning, no task-specific training), DINO ViT-S/8 achieves 69.9 (J&F)_m, with J_m = 66.6 and F_m = 73.1 (Table 5). This surpasses methods specifically designed for video object segmentation: MAST (65.5), STC (67.6), and CT (48.7), and approaches the performance of STM (81.8) which uses supervised training on multiple datasets including DAVIS and YouTube-VOS. The supervised ViT-S/8 achieves only 66.0 (J&F)_m despite having identical architecture — DINO provides a +3.9 point improvement purely through its pre-training objective. The small-patch variants dramatically outperform standard-patch models: ViT-B/8 achieves 71.4 (J&F)_m versus ViT-B/16 at 62.3, a +9.1 point gap, despite having identical parameter counts. This demonstrates that the spatial granularity of the learned features scales with the patch size, supporting the interpretation that the emergent segmentation information depends on the model's spatial resolution. Figure 4 quantifies the segmentation quality in self-attention maps on PASCAL VOC 2012: thresholding the self-attention to keep 60% of the mass yields a Jaccard similarity of 45.9 for DINO ViT-S/16 versus 27.3 for supervised ViT-S/16 and 22.0 for random weights. The table in Figure 4 extends this comparison across self-supervised methods: DINO achieves 45.9, MoCo-v2 46.3, BYOL 47.8, and SwAV 46.8 — all within a narrow band of ~2 points, while supervised languishes at 27.3. This demonstrates that the emergence of segmentation in self-attention is a property of self-supervised training generally, not unique to DINO. DINO without multi-crop achieves 45.1, suggesting that the segmentation emergence is driven by the self-supervised objective itself rather than the multi-crop component specifically.

Transfer learning by fine-tuning (Table 6). When pre-trained models are fine-tuned on downstream datasets, DINO ViT-S/16 outperforms supervised ViT-S/16 on 5 out of 6 tasks: +2.0 on iNaturalist 2018 (72.0 vs. 70.7), +1.6 on iNaturalist 2019 (78.2 vs. 76.6), +0.3 on Flowers (98.5 vs. 98.2), +0.9 on Cars (93.0 vs. 92.1), and +1.6 on ImageNet itself (81.5 vs. 79.9). On CIFAR-10, both achieve 99.0. For ViT-B/16, DINO outperforms supervised on 4 of 6 tasks, with the largest gain on ImageNet (+1.0, 82.8 vs. 81.8) and no difference on CIFAR-10. The consistent improvement on ImageNet despite supervised initialization being matched to the exact task labels suggests that DINO pre-training provides better weight initialization than supervised pre-training — a finding consistent with prior observations on convnets (He et al., 2020; Caron et al., 2020) but now validated for ViT architectures. The pattern of self-supervised pre-training improving transfer while supervised pre-training specializing to the pre-training task labels is well-established, and these results confirm it extends to ViTs.

Low-shot learning with frozen features (Table 12). On ImageNet with 1% of labels (approximately 12.8 labeled images per class), a logistic regression trained on frozen DINO ViT-S/16 features achieves 64.5% top-1 accuracy. With 10% of labels, it reaches 72.2%. These numbers are notable because they use only frozen features with no fine-tuning and no data augmentation — compare to semi-supervised methods that fine-tune the entire network: FixMatch achieves 71.5% at 10% labels and MPL achieves 73.9%, both with ResNet-50. SimCLRv2 with knowledge distillation on a ResNet-152w3+SK (794M parameters) achieves 76.6% at 1% labels and 80.9% at 10% labels, but with fine-tuning, semi-supervised training, and 9.3× more parameters. DINO ViT-S/16's frozen features at 1% labels (64.5%) outperform BYOL ResNet-50 with fine-tuning (53.2%) by +11.3 points, and approach SwAV ResNet-50 with fine-tuning (53.9%). The k-NN evaluation in Table 10 further demonstrates the label-efficiency: on ImageNet with 1% labels, ViT-S achieves 61.3% k-NN top-1 versus 47.2% for ResNet-50 — a +14.1 point gap. This pattern — ViT features being significantly more k-NN-friendly than ResNet features — holds across datasets: +9.8 points on ImageNet 10%, +7.0 on ImageNet 100%, +3.1 on PASCAL VOC, and +1.2 on Flowers.

Self-supervised pre-training for supervised ViT training (Table 11). When training a supervised ViT-B/16 from scratch on ImageNet with the DeiT procedure, random initialization achieves 81.8% top-1. Supervised pre-training on ImageNet followed by supervised fine-tuning on ImageNet (a redundant pipeline that serves as a control) achieves 81.9% — essentially no improvement. DINO pre-training on ImageNet followed by supervised fine-tuning achieves 82.8%, a +1.0 point improvement over random initialization. This demonstrates that self-supervised pre-training provides a better initialization than either random weights or supervised pre-training on the same dataset. The gap to models that use external data or model distillation is informative: DeiT with distillation from a supervised RegNetY teacher achieves 83.4%, supervised pre-training on JFT-300M achieves 84.2%, and masked patch prediction on JFT-300M followed by supervised fine-tuning at 384² resolution achieves 79.9%. DINO's 82.8% using only ImageNet (no external data, no distillation from a convnet) is competitive with DeiT's distillation pipeline and substantially closes the gap to methods requiring massive external datasets.

Ablations on ViT-S: Component Importance, Teacher Choice, Collapse Prevention, and Compute Scaling

Multi-crop scaling and compute-accuracy tradeoffs (Table 8, Figure 5). Training ViT-S/16 for 100 epochs with 2 global views only (2×224²) achieves 67.8% linear top-1 in 15.3 hours and 9.3 GB GPU memory. Adding 2 local views raises accuracy to 71.5% (+3.7 points) with only a 1.7-hour increase in training time (11% more time). Adding 10 local views reaches 74.6% (+6.8 points over baseline) in 24.2 hours (58% more time). The non-linear benefit of multi-crop is demonstrated by the fact that training for 300 epochs with 2 global views only achieves 72.5% — which is below the 100-epoch result with 10 local views (74.6%) despite taking 1.9× longer (45.9h vs. 24.2h). For 300-epoch training: 2×224² → 72.5%, +2 local → 74.5% (+2.0), +6 local → 75.9% (+3.4), +10 local → 76.1% (+3.6). The diminishing returns from additional local views are evident: +6 vs. +10 local views improves only +0.2 points at 300 epochs while costing 19% more training time.

Patch size effect (Figure 5). Increasing spatial resolution by reducing patch size improves k-NN accuracy substantially at the cost of throughput. ViT-S with 16×16 patches achieves 72.8% k-NN at 1007 im/sec (300 epochs). With 8×8 patches, accuracy jumps to approximately 76.4% k-NN, but throughput drops to 180 im/sec — a 5.6× slowdown. With 5×5 patches, accuracy reaches approximately 78% k-NN at 44 im/sec — a 22.9× slowdown from the /16 baseline. ViT-B shows similar scaling: /16 achieves approximately 76% k-NN at 312 im/sec, while /8 reaches approximately 77.5% at 63 im/sec. The performance gain from smaller patches does not add parameters — ViT-S/16 and ViT-S/8 both have 21M parameters — but the quadratic increase in token count (197 → 785 → 1681 for /16, /8, /5 respectively) dramatically increases the self-attention computation. This establishes a clear compute-quality Pareto frontier where the practitioner trades inference speed for feature quality.

Teacher network construction strategy (Figure 6, right). Four teacher construction strategies are evaluated on ViT-S/16 at 300 epochs, measured by k-NN top-1:

Student copy (θ_t = θ_s): 0.1% — immediate collapse, as the student trivially matches itself.
Previous iteration: 0.1% — also collapses, as parameters change too slowly between iterations for the difference to provide a meaningful learning signal.
Previous epoch (teacher frozen for one full epoch): 66.6% — does not collapse, achieving competitive performance with BYOL (66.6%) and substantially better than MoCo-v2 (62.0%).
Momentum encoder: 72.8% — the best performance, with a +6.2 point margin over previous-epoch.

The transition from collapse (0.1%) to functional (66.6%) to strong (72.8%) as the teacher becomes more temporally smoothed but still tracks the student suggests that the effective "staleness" of the teacher matters: too recent (copy, previous iteration) provides no useful signal, coarsely stale (previous epoch) provides a functional but noisy target, and smoothly tracked (momentum) provides the best balance.

Training dynamics of student vs. teacher (Figure 6, left). Throughout 300 epochs of ViT-S/16 training, the momentum teacher consistently achieves higher k-NN accuracy than the student. At epoch 100: teacher ~68%, student ~64% (gap ~4 points). At epoch 300: teacher ~73%, student ~71% (gap ~2 points). The gap narrows over time but never closes. This dynamic — teacher consistently outperforming student — is presented as evidence for the self-distillation interpretation and is noted as not having been observed in prior momentum-encoder frameworks. The paper reports that the same behavior occurs with ResNet-50 (Appendix D), confirming it is not ViT-specific.

Collapse prevention via centering and sharpening (Figure 7, Appendix D). Figure 7 tracks the teacher's output entropy h(P_t) and the KL divergence between teacher and student throughout training under three conditions:

No centering: entropy drops to near 0 within ~10 epochs (mode collapse — all probability mass on a single dimension), and KL divergence also approaches 0 (complete collapse).
No sharpening: entropy rises to −log(1/K) = log(K) (the uniform distribution maximum), and KL divergence approaches 0 (collapse to uniform output).
Both centering and sharpening: entropy stabilizes at an intermediate value (neither 0 nor maximum), and KL divergence remains well above 0 throughout training.

The temperature sensitivity study in Appendix D shows: τ_t = 0 yields 43.9% k-NN (suboptimal — one-hot targets too aggressive), τ_t = 0.02 yields 66.7%, τ_t = 0.04 yields 69.6%, τ_t = 0.06 yields 68.7%, and τ_t = 0.08 yields 0.1% (collapse — teacher softer than student with centering pushes toward uniform). The warmup schedule (0.04 → 0.07 over 30 epochs) achieves 69.7%, slightly better than any fixed temperature. The centering momentum m is robust from 0 to 0.99 (all achieving 69.1–69.7% k-NN) and only collapses at m = 0.999 (0.1%), where the center adapts too slowly to prevent mode collapse.

Batch size robustness (Table 9). On ViT-S/16 trained for 100 epochs without multi-crop: batch size 128 achieves 57.9% k-NN, 256 achieves 59.1%, 512 achieves 59.6%, 1024 achieves 59.9%. The degradation from reducing batch size by 8× (1024 → 128) is only 2.0 percentage points, demonstrating that DINO does not require large batches for stability. The paper also reports reaching 35.2% k-NN after 50 epochs with batch size 8, showing the method can scale to extremely limited hardware.

Importance of specific components (Table 7). This table quantifies the contribution of individual components on ViT-S/16 trained for 300 epochs:

Full DINO (momentum ✓, SK ✗, multi-crop ✓, CE loss ✓, predictor ✗): 76.1% linear / 72.8% k-NN.
Removing momentum (row 2): collapses to 0.1% / 0.1% — momentum is essential for stability with only centering.
Adding Sinkhorn-Knopp (row 3): 76.0% / 72.2% — essentially identical to DINO, showing that the more complex SK normalization adds no benefit when momentum and centering are present.
Removing multi-crop (row 4): 72.5% / 67.9% — a 3.6-point linear drop and 4.9-point k-NN drop.
Switching from cross-entropy to MSE loss (row 5): 62.4% / 52.6% — a 13.7-point linear drop and 20.2-point k-NN drop, demonstrating that the loss function choice is critical.
Adding a predictor (row 6): 75.6% / 71.8% — small degradation, suggesting the predictor is unnecessary in DINO (unlike BYOL where it is essential).
BYOL with its default configuration on ViT-S/16 (row 7): 71.4% / 66.6%.
MoCo-v2 on ViT-S/16 (row 8): 71.6% / 62.0%.
SwAV on ViT-S/16 (row 9): 71.8% / 64.7%.

Relation to BYOL and MoCo-v2 (Table 14). This ablation cross-references component choices across frameworks on ViT-S/16:

DINO with MSE loss instead of CE (row 2): 62.4% linear — functional but dramatically worse than CE (76.1%).
MoCo-v2 with multi-crop added (row 6): 73.4% linear — improves from 71.4% (row 5), showing multi-crop helps MoCo-v2 on ViT.
BYOL with centering replacing the predictor (row 9): 52.6% linear — functional but much worse than BYOL with predictor (71.4%, row 7), showing centering is not a drop-in replacement for the predictor in BYOL's framework; they operate through different mechanisms.
BYOL with multi-crop added (row 10): 64.8% linear — degrades from 71.4% (row 7), confirming that multi-crop is detrimental for BYOL on ViT in this evaluation.

Relation to SwAV (Table 15). This ablation varies the teacher output operation and the presence of momentum on ViT-S/16:

With momentum: centering achieves 76.1%, softmax(batch) achieves 75.8%, and SK achieves 76.0% — all nearly identical, showing that with momentum, the specific operation matters little as long as it prevents collapse.
Without momentum: centering alone collapses (0.1%), softmax(batch) achieves 72.2%, SK achieves 71.8% (matching SwAV's default). This demonstrates that momentum substitutes for the complexity of SK/softmax(batch) — with momentum, even the simplest operation (centering) suffices; without momentum, SK or softmax(batch) are required.

Projection head design (Appendix C). With the ℓ₂ bottleneck, increasing the number of linear layers from 1 to 4 improves k-NN from 61.6% to 69.3% (100 epochs, ViT-S/16). Without the bottleneck, 1 layer achieves 61.6%, 2 layers achieve 62.9%, and 3–4 layers collapse to 0.1%. The ℓ₂ normalization bottleneck is therefore critical for enabling deep projection heads. Output dimensionality K: 1024 → 67.8%, 4096 → 69.3%, 16384 → 69.2%, 65536 → 69.7%, 262144 → 69.1%. Large K improves performance up to a point, with the default 65536 near-optimal. Using ReLU instead of GELU in the projection head achieves 68.9% vs. 69.7% — a small drop but not catastrophic. Adding batch normalization to the projection head when training ViT achieves 68.6% vs. 69.7% without BN — confirming that BN is not needed and may slightly hurt.

Training length (Appendix D). ViT-S/16 with DINO: 100 epochs → 70.9% k-NN, 300 epochs → 72.8%, 800 epochs → 74.5%. The improvement from extended training is monotonic but exhibits diminishing returns: +1.9 points from 100 to 300 epochs, and +1.7 points from 300 to 800 epochs. The paper notes that BYOL on ViT-S actually degraded with training beyond 300 epochs, which is why BYOL is reported at 300 epochs while DINO, SwAV, and MoCo-v2 are reported at 800 epochs in Table 2 — a methodological choice that favors BYOL. This detail is buried in Appendix D and is not highlighted in the main results tables, which could mislead cursory comparisons.

Multi-crop scale range parameter s (Appendix E). Sweeping s (the boundary between local and global view scales) on ViT-S/16: s = 0.08 → 65.6%, s = 0.16 → 68.0%, s = 0.24 → 69.7%, s = 0.32 → 69.8% (default), s = 0.48 → 69.5%. The optimum around 0.32 is substantially higher than the s = 0.14 used in SwAV with convnets, indicating that ViTs benefit from larger global views (providing more spatial context for self-attention).

Number of attention heads (Appendix D). ViT-S with 6 heads → 72.8% k-NN at 1007 im/sec, 8 heads → 73.1% at 971 im/sec, 12 heads → 73.7% at 927 im/sec, 16 heads → 73.8% at 860 im/sec. More heads improve accuracy at a modest throughput cost. The paper uses the default DeiT-S configuration with 6 heads for all experiments, meaning the reported numbers are a lower bound on what could be achieved with head count tuning.

Ablation Studies and Robustness Checks

Loss function sensitivity (Table 14, rows 1–2): DINO with its default cross-entropy loss on sharpened softmax outputs achieves 76.1% linear top-1. Replacing cross-entropy with mean squared error drops accuracy to 62.4% — a 13.7-point degradation. This is the single largest component-level drop observed in the paper and demonstrates that the matching loss formulation is not incidental. The paper does not provide a mechanistic explanation for why CE outperforms MSE by such a large margin, but the result is consistent with the distillation framing: CE on probability distributions is the standard loss for knowledge distillation because it handles uncertainty in the teacher's predictions more naturally than MSE on ℓ₂-normalized vectors.

Multi-crop incompatibility with BYOL on ViT (Table 14, rows 7 vs. 10, Appendix E): Adding multi-crop (2×224² + 6×96²) to BYOL on ViT-S/16 degrades linear evaluation from 71.4% to 64.8% — a 6.6-point drop. With k-NN, the degradation is even larger: from 66.6% to 59.8% (−6.8 points). Appendix E documents that the training curves show initially better performance with multi-crop, but "the transfer performance growth rate is slowing down and declines after a certain amount of training." Extensive hyperparameter sweeps (learning rates {1e−5, 3e−5, 1e−4, 3e−4, 1e−3, 3e−3}, weight decay {0.02, 0.05, 0.1}, number of small crops {2, 4, 6}) all produce the same pattern. This is not merely a tuning issue — it suggests a fundamental interaction between BYOL's MSE matching loss with predictor and the multi-crop objective that causes training instability or overfitting on ViT architectures. The paper does not diagnose the cause, but the fact that DINO's CE loss works well with multi-crop while BYOL's MSE loss does not points to the loss function as the differentiating factor, possibly because CE's gradient scales with the teacher's confidence (providing an adaptive weighting across views) while MSE treats all views uniformly.

Sinkhorn-Knopp is unnecessary with momentum (Table 15): With a momentum encoder, centering alone (76.1%), softmax(batch) (75.8%), and Sinkhorn-Knopp (76.0%) all achieve nearly identical performance. Without momentum, centering collapses (0.1%), while softmax(batch) (72.2%) and SK (71.8%) remain functional. This is a practically significant finding: the momentum encoder eliminates the need for iterative Sinkhorn-Knopp normalization, which is computationally more expensive (requiring multiple row/column normalization iterations) and which the paper shows can be reduced to a single softmax operation over the batch dimension without performance loss when momentum is present.

Predictor is unnecessary in DINO (Table 7, row 6 vs. row 1): Adding a predictor network to the student (as in BYOL) yields 75.6% linear and 71.8% k-NN, compared to 76.1% and 72.8% without — a small degradation. This contrasts sharply with BYOL, where removing the predictor causes immediate collapse (Table 14, row 8: 0.1% without predictor). The paper attributes this difference to DINO's centering-sharpening mechanism: the centering provides the asymmetry that the predictor provides in BYOL, making the predictor redundant.

Longer training benefits DINO more than BYOL on ViT (Appendix D, Section 5.2): DINO ViT-S/16 k-NN improves from 70.9% (100 epochs) to 72.8% (300 epochs) to 74.5% (800 epochs). BYOL on ViT-S achieves its best at 300 epochs (71.4% linear, 66.6% k-NN) and degrades with longer training. This means the 800-epoch numbers reported for DINO in Table 2 are not directly comparable to the 300-epoch numbers reported for BYOL — BYOL's results represent its peak performance, while DINO's could potentially improve further beyond 800 epochs. The paper acknowledges this asymmetry in Appendix D but does not highlight it in the main results, which is a presentation choice that somewhat favors DINO in the head-to-head comparison.

k-NN evaluation reveals gaps that linear evaluation obscures (Table 13): On ViT-S, DINO outperforms BYOL by +5.6 points in linear evaluation (76.1% vs. 71.4%) but by +7.9 points in k-NN evaluation (72.8% vs. 66.6%). The k-NN gap is 41% larger than the linear gap. Similarly, DINO outperforms SwAV by +4.3 points linear but +8.1 points k-NN. This discrepancy demonstrates that linear evaluation — the standard protocol in self-supervised learning — systematically underestimates the feature quality differences between methods when applied to ViTs. DINO features are not just more linearly separable; they are better clustered in a way that benefits non-parametric classifiers. This finding has methodological implications: k-NN evaluation should be reported alongside linear evaluation as a complementary diagnostic, particularly when comparing across architectures.

Table 10 across-dataset k-NN consistency: DINO ViT-S/16 features consistently outperform DINO ResNet-50 features in k-NN evaluation across diverse datasets, with the gap being largest on ImageNet variants (+7.0 to +14.1 points) and smaller but still positive on PASCAL VOC (+3.1), Places205 (+1.7 to +2.1), and Flowers (+1.2). The average k-NN gap across all datasets is +5.6 points in favor of ViT-S. This demonstrates that the k-NN-friendliness of ViT features is a general property, not an ImageNet-specific artifact.

Self-supervised pre-training initialization benefit over supervised pre-training (Table 11): When fine-tuning on ImageNet, DINO initialization (82.8%) outperforms supervised initialization (81.9%) by +0.9 points, even though both are pre-trained on the exact same ImageNet dataset. This is a well-controlled comparison: same architecture (ViT-B/16), same dataset, same fine-tuning procedure — the only difference is whether the pre-training objective was DINO or supervised classification. The result confirms that self-supervised pre-training provides features that transfer better even to the same dataset they were trained on, consistent with the interpretation that supervised objectives discard information (reducing rich visual scenes to single class labels) while self-supervised objectives preserve it.

ℓ₂ bottleneck necessity for deep projection heads (Appendix C, Section 5.3): With ℓ₂ bottleneck, a 4-layer projection head achieves 69.3% k-NN (100 epochs). Without ℓ₂ bottleneck, a 3-layer head collapses to 0.1%, and a 4-layer head also collapses. The ℓ₂ normalization constrains the representation magnitude before the weight-normalized final layer, preventing logit explosion that would otherwise destabilize the softmax and cause collapse. This is a critical architectural detail for practitioners: if extending DINO to deeper or wider projection heads, the ℓ₂ bottleneck is not optional.

Negative result with ReST^{EM} revision model: Not applicable — this paper does not use revision models or ReST^{EM}. This concern applies to the reference example paper, not DINO.

Critical Assessment

Claim: Self-supervised ViT features contain explicit semantic segmentation information not present in supervised ViTs or convnets.

The evidence for this claim is multi-modal and generally convincing, but with important caveats about what is and is not demonstrated. The qualitative visualizations in Figures 1, 3, and 10 are striking — the self-attention maps clearly delineate object boundaries for diverse scenes. The quantitative evidence in Figure 4 (Jaccard similarity: DINO 45.9 vs. supervised 27.3 vs. random 22.0) provides a controlled comparison that isolates the training objective as the causal factor. The DAVIS-2017 results (Table 5) demonstrate that this spatial information is functional — it can be used for video object segmentation without any fine-tuning — and the large gap between DINO ViT-S/8 (69.9) and supervised ViT-S/8 (66.0) further supports the claim that self-supervision specifically enables this property.

However, the Figure 4 table also shows that this property is not unique to DINO — MoCo-v2 (46.3), BYOL (47.8), and SwAV (46.8) all achieve similar Jaccard scores, with some slightly exceeding DINO (45.9). This means the claim is more accurately stated as "self-supervised training enables segmentation emergence in ViT attention maps" rather than "DINO specifically enables this." The paper acknowledges this implicitly by presenting the cross-method comparison, but the abstract and introduction emphasize DINO's role, which could mislead readers into attributing the property to DINO's specific design choices rather than to self-supervision broadly. The ablation showing DINO without multi-crop still achieves 45.1 suggests the property is robust to the multi-crop component — the core driver appears to be the self-supervised matching objective itself, regardless of the specific loss or collapse prevention mechanism.

The DAVIS evaluation uses a protocol (Jabri et al., 2020) that has known sensitivity to feature resolution and patch size. The paper's finding that /8 models dramatically outperform /16 models (+9.1 points for ViT-B) suggests that part of the segmentation quality may simply be resolution-dependent — smaller patches produce higher-resolution attention maps that happen to align better with object boundaries. This is not a weakness per se (it's a genuine property of the features), but it complicates the interpretation: is the "emergence" genuinely about semantic understanding, or is it partly an artifact of increased spatial resolution in the token grid? The paper does not disentangle these effects.

A missing experiment: train a supervised ViT with the same data augmentation (multi-crop, strong color jittering, etc.) but with the standard cross-entropy classification loss instead of DINO's self-distillation objective. This would test whether the augmentation pipeline alone can induce segmentation emergence, or whether the self-supervised matching objective is necessary. The paper compares DINO to standard supervised ViTs (trained with standard augmentation), but does not control for the augmentation regime.

Claim: Self-supervised ViT features are excellent k-NN classifiers, with DINO ViT-S/8 achieving 78.3% top-1.

The evidence for this claim is strong and well-controlled. The k-NN evaluation protocol is fixed across all models (k = 20, τ = 0.07, no tuning), eliminating the hyperparameter sensitivity that plagues linear evaluation comparisons. The gap between DINO and other methods in k-NN is substantial and consistent across architectures: +7.9 points over BYOL on ViT-S (74.5% vs. 66.6%), +8.2 points over SwAV on ViT-S (74.5% vs. 66.3%), +10.1 points over MoCo-v2 on ViT-S (74.5% vs. 64.4%). The k-NN performance is not far below linear performance (74.5% vs. 77.0% for ViT-S/16; 78.3% vs. 79.7% for ViT-S/8), indicating that the features are inherently clustered by class without requiring a learned linear transformation to separate them.

The critical question: is this k-NN friendliness a property of DINO specifically, or of self-supervised ViTs in general? When the paper compares DINO to BYOL, SwAV, and MoCo-v2 on ViT-S, DINO is the clear winner in k-NN. But the paper also shows in Table 14 that BYOL and MoCo-v2 were tuned for ResNet architectures and transferred with their default configurations to ViT without ViT-specific optimization. The paper performs extensive hyperparameter tuning for DINO on ViT (patch size, crop scale s, temperature schedules, etc.) but does not report similar tuning effort for the baseline methods. The BYOL result with multi-crop (which degrades performance) suggests that BYOL's ViT configuration was suboptimal — it's possible that a ViT-optimized BYOL (with adjusted learning rates, momentum schedules, or loss modifications) could close some of the k-NN gap. The paper's claim that BYOL degrades with longer training on ViT while DINO improves is suggestive that BYOL indeed interacts differently with ViT architectures, but the counterfactual (what would a ViT-optimized BYOL achieve?) is not explored.

The k-NN protocol itself has a potential confound: the temperature τ = 0.07 in the weighted voting is a hyperparameter inherited from Wu et al. (2018) and is not tuned for DINO or any other method. If different methods produce features with different optimal temperature scales (due to differences in feature normalization, dimensionality, or concentration), the fixed τ = 0.07 might favor some methods over others. The paper does not perform a temperature sensitivity analysis for k-NN.

Claim: The combination of momentum encoder, multi-crop, and cross-entropy loss with centering-sharpening achieves state-of-the-art self-supervised performance on ViTs.

This claim is well-supported by the main results in Table 2. DINO ViT-B/8 achieves 80.1% linear top-1, surpassing the previous state of the art (SimCLR-v2 ResNet-152w3+SK at 79.8%) with 9.3× fewer parameters. However, the comparison to SimCLR-v2 is across architectures (ViT vs. ResNet) and the paper does not provide a SimCLR-v2 result on ViT, nor a DINO result on a comparably scaled ResNet (ResNet-152w3+SK is 794M parameters — much larger than ViT-B's 85M). The "state of the art" claim therefore conflates architectural advances (ViT vs. ResNet) with methodological advances (DINO vs. SimCLR-v2). The clean within-architecture comparisons in Table 13 (DINO vs. BYOL/MoCo-v2/SwAV on the same ViT-S) provide stronger evidence for DINO's methodological advantage.

The ablation in Table 7 is comprehensive and well-designed, showing that each component (momentum, multi-crop, CE loss) contributes meaningfully. However, the ablations are performed at 300 epochs on ViT-S/16 — a single architecture and training duration. The paper does not systematically verify that the same component contributions hold at larger scales (ViT-B) or longer training (800 epochs). The diminishing returns from multi-crop at longer training (Table 8: +0.2% from 6 to 10 local crops at 300 epochs) suggest that component importance may shift with scale and duration. A scaling study showing component ablation at ViT-B scale would strengthen the generalizability claim.

Claim: Collapse can be prevented solely through centering and sharpening, without contrastive losses, clustering, predictors, or batch normalization.

This is one of the paper's most novel and well-supported technical claims. Figure 7 directly demonstrates the mechanism — entropy and KL divergence trajectories under different operations show that centering and sharpening have complementary effects that together prevent both mode collapse and uniform collapse. The robustness checks (temperature τ_t, centering rate m, batch size) show that the mechanism works across a wide range of hyperparameters and is not fragile. The finding that DINO works at batch size 8 (35.2% after 50 epochs) is particularly convincing — it rules out the possibility that collapse prevention relies on implicit batch-level statistics.

However, the claim that the system is "entirely BN-free" (Section 3.1) applies only when DINO is used with ViT architectures. When DINO is used with ResNet-50, the backbone still contains batch normalization layers. The paper does not test whether DINO + ResNet-50 would work with the BN layers frozen or removed. Additionally, the ViT architecture uses layer normalization, which also computes batch-independent statistics per token (mean and variance across the feature dimension). The paper does not ablate whether the layer normalization in ViT contributes to collapse prevention. This is probably not the case (layer norm operates on individual tokens, not across the batch), but it means the system is not truly normalization-free — it relies on layer norm, just not batch norm.

General weaknesses in experimental design:

Single dataset for pre-training: All models are pre-trained on ImageNet-1K. The paper does not evaluate DINO pre-training on larger unlabeled datasets (Instagram, YFCC100M, or web-scale image collections). This is a significant gap because one of the paper's central motivations is enabling a "BERT moment" for vision through pre-training on massive unlabeled data. The ImageNet-scale experiments can only suggest this potential; they cannot demonstrate it. The GLDv2 retrieval experiment (Table 3) is the only multi-dataset pre-training result and shows promising domain transfer, but this is a retrieval-specific dataset, not a general-purpose unlabeled corpus.

No error bars or statistical testing: None of the reported numbers include confidence intervals, standard deviations across runs, or statistical significance tests. Given the small gaps between some methods (e.g., DINO ResNet-50 74.5% vs. SwAV 74.1% in Table 13 — a 0.4% difference), it is impossible to determine whether these differences are statistically reliable. Self-supervised training is known to be sensitive to random seeds, data ordering, and initialization, especially for ViTs (which have less built-in inductive bias than convnets). The absence of any statistical characterization weakens all quantitative claims in the paper.

Missing ViT-optimized baselines: As discussed above, BYOL, MoCo-v2, and SwAV are evaluated on ViT-S with their ResNet-optimized configurations. The paper's central finding — that DINO substantially outperforms these methods on ViT — would be more convincing if the baselines had been tuned for ViT to a similar degree as DINO was. The paper's own multi-crop ablation (Table 14, row 7 vs. 10) shows that BYOL's performance on ViT is highly sensitive to whether multi-crop is used, suggesting that ViT-specific tuning could change the ranking. The paper acknowledges this indirectly (Appendix B: "adding multi-crop to BYOL does not work out-of-the-box ... and further adaptation may be required") but does not perform that adaptation.

Linear evaluation sensitivity: The linear evaluation protocol (100 epochs of SGD on frozen features, learning rate swept per model) is standard but has known sensitivity to the learning rate schedule, weight decay, and number of epochs. The paper sweeps only the learning rate. If the optimal linear evaluation hyperparameters differ across methods (as they likely do — DINO features may benefit from different optimization than BYOL features), the comparison could be biased. The k-NN evaluation partially mitigates this concern by eliminating hyperparameter tuning, which is why the paper's emphasis on k-NN results is methodologically sound.

Limited evaluation on dense prediction tasks beyond DAVIS: The paper demonstrates emergent segmentation properties via self-attention visualization and DAVIS video object segmentation, but does not evaluate on standard semantic segmentation benchmarks (PASCAL VOC 2012, COCO, ADE20K) with the standard fine-tuning or linear probe protocols. This limits the practical impact of the segmentation claim — showing that attention maps correlate with object boundaries is interesting, but practitioners need to know whether these features improve actual segmentation models. The DAVIS evaluation uses a nearest-neighbor protocol that tests feature correspondence quality, not standard segmentation with a trained decoder head.

No systematic study of the effect of pre-training data scale: The paper's motivation invokes NLP's BERT moment — pre-training on massive unlabeled corpora. The experiments on ImageNet-1K (1.28M images) are at least an order of magnitude smaller than the datasets that drove NLP's self-supervised revolution (BERT was trained on BooksCorpus + English Wikipedia: ~3.3 billion words). The paper does not include any experiments varying the pre-training data scale (e.g., 10%, 100%, 1000% of ImageNet), which would test whether the emergent properties and k-NN friendliness scale with data quantity.

The self-attention segmentation probe is post-hoc and threshold-dependent: The Jaccard similarity metric in Figure 4 depends on the threshold chosen (60% of the attention mass). The paper does not sweep thresholds to determine sensitivity, nor does it evaluate segmentation quality across different heads and layers (only the "best head" is used). Different heads in different layers might attend to different semantic levels (edges, textures, object parts, whole objects), and the "best head" selection could overestimate the typical quality. The DAVIS evaluation (which uses all patch tokens, not a single attention head) partially addresses this, but the paper does not provide a systematic layer-wise or head-wise analysis of where in the network the segmentation information emerges.

Potential benchmark overfitting: The paper develops DINO through extensive ablation studies on ImageNet validation performance (both k-NN and linear). While this is standard practice, the risk of overfitting to ImageNet-specific properties exists when all design choices (patch size, crop scale s, temperature schedules, projection head depth, output dimension K, number of local crops) are tuned on ImageNet metrics. The transfer learning results (Table 6) partially mitigate this concern by showing improvements on diverse downstream datasets, but the transfer tasks are relatively small and may share ImageNet's distribution. A stronger test would be pre-training on a different dataset (e.g., Places365, or a non-curated dataset like YFCC100M) and evaluating on ImageNet transfer, which would demonstrate that the method's benefits generalize beyond the pre-training distribution.

Missing FLOPs-matched comparison across architectures: The paper uses throughput (im/sec) as an efficiency metric, but this is an imperfect measure — throughput depends on batch size, GPU architecture, and implementation details. A FLOPs-matched comparison between ViT-S/8 (180 im/sec, 21M params) and a comparably expensive ResNet (e.g., ResNet-50 at 1237 im/sec but much lower accuracy) would provide a cleaner efficiency comparison. The paper's Table 2 includes throughput for reference, but does not normalize accuracy by FLOPs or show Pareto frontiers of accuracy vs. compute.

6. Limitations and Trade-offs

Training Data Scale Mismatches the BERT-Moment Motivation

The paper's central motivation — articulated in the abstract and Section 6 — is that self-supervised pre-training could unlock a "BERT moment" for vision, where models pre-trained on massive unlabeled image collections learn rich, structured representations akin to what BERT achieved in NLP. However, the entire experimental validation is conducted on a single curated dataset (ImageNet-1K, ~1.28M images) that is orders of magnitude smaller than the corpora that drove NLP's self-supervised revolution.

The consequence: The paper provides no evidence about how the emergent properties scale with pre-training data quantity. The segmentation emergence, k-NN friendliness, and transfer performance demonstrated on ImageNet-1K may not hold, scale, or even survive when moving to larger, uncurated image collections. The GLDv2 experiment (Table 3) offers one data point for domain-specific pre-training (1.2M landmarks images) and shows strong retrieval performance, but this is a much narrower domain than the general visual understanding that a "vision BERT" would require. Conversely, if the emergent properties do scale favorably with data, the 80.1% linear accuracy on ImageNet with ViT-B/8 may represent only a fraction of what could be achieved with web-scale pre-training. The paper simply cannot say which is true.

Evidence: The entire experimental section (Tables 2–15, Figures 3–11) uses models pre-trained only on ImageNet-1K. The paper acknowledges this aspiration without testing it: Section 6 states "in the future, we plan to explore if pretraining a large ViT model with DINO on random uncurated images could push the limits of visual features." The appendix mentions training on GLDv2 (§4.2.1) and references concurrent work on "self-supervised pretraining of visual features in the wild" (Goyal et al., 2021, Ref. [28]) as future direction, but provides no scaling experiments with varying data quantities.

Mitigation status: Not addressed. The paper explicitly defers data-scale experiments to future work. This is a deliberate scope limitation, but it means the headline motivation — enabling a BERT-like model for vision — remains aspirational rather than demonstrated.

The Emergence of Segmentation Is Demonstrated Only Through Post-Hoc Visualization and Proxy Tasks, Not Standard Segmentation Benchmarks

The paper's most striking qualitative result — that self-attention maps from self-supervised ViTs contain explicit object segmentation — is evaluated through self-attention visualization (Figures 1, 3, 4, 8, 10), a single threshold-based metric on PASCAL VOC 2012 (Figure 4, Jaccard similarity at 60% mass threshold), and a nearest-neighbor video object segmentation protocol on DAVIS-2017 (Table 5). These evaluations demonstrate that spatial information exists in the features, but they do not establish whether these features improve actual segmentation model performance on standard benchmarks with standard protocols.

The consequence: A practitioner deciding whether to adopt DINO pre-training for a segmentation pipeline cannot estimate the practical benefit. The self-attention maps (Figure 1) are correlated with object boundaries, but correlation does not imply that these features will improve a trained segmentation decoder on COCO, ADE20K, or Cityscapes. The Jaccard metric in Figure 4 shows that DINO ViT-S/16 achieves 45.9 Jaccard vs. 27.3 for supervised — a large relative improvement — but this metric depends on an arbitrary threshold (60% of attention mass) and selects the "best head" per image, which may overestimate quality compared to using all heads or a fixed head. The DAVIS-2017 evaluation (Table 5) uses a nearest-neighbor matching protocol between frames (following Jabri et al., 2020) that is a specific, narrow test of feature correspondence quality, not a general segmentation benchmark. DINO ViT-S/8 achieves 69.9 (J&F)_m on DAVIS without any fine-tuning, which is competitive with methods specifically designed for the task, but this protocol does not train a segmentation head and therefore does not measure how the features behave when adapted through standard fine-tuning or transfer learning on pixel-level tasks.

Evidence: The paper does not evaluate on any standard semantic segmentation benchmark (PASCAL VOC 2012 segmentation, COCO-Stuff, ADE20K, Cityscapes) with fine-tuning or linear probe protocols. The DAVIS evaluation is the only dense prediction task. The self-attention metric in Figure 4 is evaluated on 4 methods (supervised, DINO, DINO w/o multicrop, random) but only on ViT-S/16, not on larger models where the effect might be stronger or qualitatively different. The paper's DAVIS results show a large gap between /16 and /8 models (+9.1 (J&F)_m for ViT-B), suggesting the segmentation quality is partly resolution-dependent, but this interaction is not explored on standard segmentation benchmarks.

Mitigation status: Not addressed. The paper presents the emergent segmentation as a property discovery rather than a practical capability claim, and the DAVIS results demonstrate that the property has functional consequences for at least one task. However, the gap between "self-attention maps look like segmentations" and "these features improve trained segmentation models" is substantial and unaddressed.

The Difficulty Estimation Cost of 2048 Samples Per Question Is Not Accounted for in Any Budget Calculation

This limitation applies to the reference example paper (the LLM test-time compute scaling work), not to DINO. For DINO, the analogous unaccounted cost is different.

Correction — for DINO specifically: The paper does not account for the cost of tuning DINO's hyperparameters for ViT architectures versus the baseline methods. The ablation studies in Tables 7, 14, and 15 and Appendices C, D, and E represent a substantial hyperparameter search (crop scale s, temperature τ_t, centering rate m, patch size, projection head depth, output dimension K, number of local crops, training duration) that was performed specifically for DINO on ViT. The baseline methods (BYOL, MoCo-v2, SwAV) were evaluated on ViT with their ResNet-optimized configurations, and the paper acknowledges that these configurations are suboptimal for ViT — Appendix B states "adding multi-crop to BYOL does not work out-of-the-box ... and further adaptation may be required." The tuning effort that produced DINO's strong ViT results was not applied to the baselines, making the comparison asymmetric.

The consequence: The reported performance gaps between DINO and baselines on ViT (e.g., +5.6% linear and +7.9% k-NN over BYOL on ViT-S in Table 13) conflate two effects: (1) genuine methodological superiority of DINO's self-distillation framework for ViT architectures, and (2) the benefit of ViT-specific hyperparameter tuning. It is unknown what fraction of the gap would close if BYOL, MoCo-v2, and SwAV received comparable ViT-specific optimization. The paper's finding that multi-crop degrades BYOL on ViT (Table 14, row 7 vs. 10: 71.4% → 64.8% linear) is the clearest evidence that BYOL's ViT configuration is suboptimal, but the paper does not explore alternative configurations (e.g., changing the loss weighting between global and local views, adjusting the predictor architecture, or modifying the momentum schedule) that might make multi-crop beneficial for BYOL on ViT.

Evidence: Table 14 rows 7–10 document the BYOL multi-crop issue. Appendix E states: "We have performed learning rate, weight decay, multi-crop parameters sweeps for this setting and systematically observe the same pattern." However, the sweep described is limited to learning rate, weight decay, and number of small crops — it does not include architectural modifications to BYOL (predictor depth, projection head design, loss modifications) that might resolve the incompatibility. The paper also documents that BYOL degrades with training beyond 300 epochs on ViT-S (Appendix D), while DINO continues improving to 800 epochs, further suggesting that BYOL's ViT configuration is not optimal.

Mitigation status: Partially acknowledged. The paper states that further adaptation of baselines to ViT "may be required" (Appendix B) and notes that investigating why multi-crop does not combine well with BYOL is "worth investigating" (Appendix E). However, the main results tables (Table 2, Table 13) present the asymmetric comparison without caveats, which could mislead readers about the magnitude of DINO's architectural advantage.

The Implicit Baseline Weakness: Supervised ViTs Are Not Matched for Augmentation or Compute

When comparing self-supervised DINO ViTs to supervised ViTs for properties such as segmentation emergence (Figure 4) or transfer learning (Table 6), the paper does not control for the data augmentation regime or total compute budget. The supervised ViT baselines are trained with standard ImageNet augmentation (random resized crop, horizontal flip) using the DeiT procedure (Touvron et al., 2020), while DINO uses a substantially more aggressive augmentation pipeline: multi-crop (2 global + up to 10 local views per image, each with independent random crops), color jittering, Gaussian blur, and solarization. The supervised model sees each image once per epoch with standard augmentations; DINO sees each image effectively 12 times per epoch (2 global + 10 local views) with much stronger transformations.

The consequence: The claim that self-supervised training produces emergent properties that supervised training does not (the paper's central thesis) cannot be cleanly attributed to the training objective. An alternative hypothesis: the aggressive multi-crop augmentation — which forces the model to match representations across dramatically different views — is the primary driver of spatial understanding and k-NN friendliness, and a supervised model trained with the same augmentation pipeline might exhibit similar properties. The paper's own ablation in Figure 4 (table inset) shows that DINO without multi-crop achieves 45.1 Jaccard similarity, close to the full DINO's 45.9 and far above supervised's 27.3. This suggests that multi-crop is not the primary driver of segmentation emergence, which is reassuring. However, the supervised baseline is not trained with multi-crop or the BYOL-style color augmentations, so it remains unclear whether the gap would narrow if the supervised model received comparable augmentation diversity.

Evidence: The paper describes its augmentation pipeline in Section 3.2 (color jittering, Gaussian blur, solarization, multi-crop) and notes it follows BYOL's augmentations. The supervised ViT baselines (Section 4.2.3, Table 6) use the DeiT training procedure, which employs standard ImageNet augmentations. There is no experiment training a supervised ViT with DINO's exact augmentation pipeline to test whether the observed property differences (Figure 4: 45.9 vs. 27.3 Jaccard; Table 5: 69.9 vs. 66.0 (J&F)_m; Table 6: +1–2% transfer gains) persist under matched augmentation regimes.

Mitigation status: Not addressed. The paper does not discuss this confound. The standard practice in self-supervised learning is to compare against supervised baselines trained with standard augmentations, and the paper follows this convention. However, the specific claim about emergent properties unique to self-supervision requires a stronger baseline: a supervised model trained with equivalent augmentations to isolate the effect of the training objective from the effect of the data pipeline. The DINO-without-multicrop ablation partially addresses this for the segmentation property, but a full controlled experiment is missing.

The System Lacks Statistical Reproducibility Characterization

None of the quantitative results in the paper are accompanied by error bars, confidence intervals, standard deviations across training runs, or any form of statistical significance testing. This applies to all main results tables (Tables 2, 3, 4, 5, 6, 7, 8, 13, 14, 15) and all figures (Figures 4, 5, 6, 7).

The consequence: The paper reports numerous small-margin comparisons where the ranking of methods or the significance of an ablation could be affected by run-to-run variance. Examples include:

DINO ResNet-50 linear evaluation at 75.3% vs. SwAV at 75.3% and DCv2 at 75.2% (Table 2) — a 0.1% difference between three methods. Without variance estimates, these methods cannot be meaningfully ranked.
ViT-S/16 k-NN at 72.8% (DINO) vs. 66.6% (BYOL) — a 6.2 point gap that is likely significant, but the magnitude of the advantage is uncertain.
The ablation in Table 15 showing centering (76.1%), softmax(batch) (75.8%), and Sinkhorn-Knopp (76.0%) as "nearly identical" — without variance estimates, the 0.3% range could be noise or signal.

Self-supervised training is known to be sensitive to random seeds, data ordering, and initialization, particularly for ViT architectures which have weaker inductive biases than convnets and may exhibit higher training variance. The DINO training procedure involves multiple stochastic elements (random crops, color jittering, blur, solarization, batch sampling for centering) and a complex optimization dynamic (student-teacher co-evolution with momentum schedule), all of which could contribute to run-to-run variance.

Evidence: The paper contains no information about the number of training runs per result, whether results are averaged, or what the variance is. The standard practice in the field at the time of publication (2021) was to report single-run results, and the paper follows this convention. However, given the paper's emphasis on precise quantitative comparisons (e.g., +0.2% from 6 to 10 local crops in Table 8), the absence of any variance characterization limits the confidence in the reported rankings and the optimality of the design choices.

Mitigation status: Not addressed. The paper provides no discussion of reproducibility across runs, no mention of fixed random seeds, and no variance estimates. The code release (https://github.com/facebookresearch/dino) partially mitigates this by enabling external reproduction, but the paper itself provides no guidance on expected variance.

k-NN Performance Depends on a Temperature Parameter That Is Not Tuned Per Method

The paper's k-NN evaluation protocol (Section 3.2, Appendix F.1) uses a fixed temperature τ = 0.07 in the weighted voting scheme, inherited from Wu et al. (2018). This temperature controls how sharply the similarity scores are exponentiated when aggregating neighbor votes: α_i = exp(T_i x / τ). The optimal temperature depends on the scale and concentration of the feature space, which can vary across methods and architectures depending on feature dimensionality, normalization, and training objective. Using a single fixed temperature for all methods implicitly favors methods whose feature statistics happen to align well with τ = 0.07.

The consequence: The paper's emphasis on k-NN evaluation as a "no hyperparameter tuning" protocol is somewhat misleading — the temperature is a hyperparameter that is fixed rather than tuned, but it is still a choice that affects relative rankings. If DINO features are better calibrated for τ = 0.07 than BYOL features (e.g., if DINO features have a different typical cosine similarity range or different concentration properties), the reported k-NN gaps may partially reflect temperature mismatch rather than feature quality. The paper reports k-NN gaps that are systematically larger than linear evaluation gaps (Table 13: DINO vs. BYOL on ViT-S is +5.6% linear but +7.9% k-NN; DINO vs. SwAV is +4.3% linear but +8.1% k-NN). This could indicate that DINO features genuinely cluster better for nearest-neighbor retrieval, or it could indicate that DINO features happen to work better with τ = 0.07 while baseline features would benefit from different temperatures.

Evidence: The paper states in Appendix F.1: "We use α_i = exp(T_i x/τ) with τ equals to 0.07 as in [73] which we do not tune." There is no temperature sensitivity analysis — no sweep over τ values for any method on any dataset. The paper does sweep k (the number of neighbors) and finds k = 20 works best, but does not perform the equivalent analysis for τ, which is equally a hyperparameter of the k-NN classifier. The consistent finding that k-NN gaps are larger than linear gaps across multiple method comparisons (Table 13) is suggestive but cannot be attributed to feature quality alone without ruling out temperature mismatch.

Mitigation status: Not addressed. The paper treats k-NN as a hyperparameter-free evaluation, which it is for the practitioner (no training, no data augmentation), but the fixed τ = 0.07 is an untested assumption that could systematically favor DINO if its feature distribution differs from the methods used by Wu et al. (2018), which were all ResNet-based. A simple sensitivity analysis sweeping τ for each method would clarify whether the k-NN advantage is robust to temperature choice.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around self-supervised learning from a architecture-agnostic accumulation of components toward an understanding that the interaction between training objective and architecture is non-trivial, consequential, and can produce qualitatively different kinds of representations. Before DINO, the dominant assumption in self-supervised learning was that methods developed and validated on ResNet-50 would transfer straightforwardly to new architectures — that momentum encoders, multi-crop training, contrastive losses, and predictor networks were general-purpose tools for representation learning, and that architecture choice was largely an implementation detail to be swapped in at evaluation time. The paper's systematic ablation study demolishes this assumption: BYOL loses 6.6 percentage points in linear evaluation when multi-crop is added to ViT-S (Table 14, row 7 vs. 10: 71.4% → 64.8%), MoCo-v2 gains only 1.9 points from multi-crop while DINO gains 3.6 points, and the overall ranking of methods inverts when switching from ResNet-50 (all methods within a ~3-point band) to ViT-S (DINO leads by 5.6 points). This is not a minor hyperparameter sensitivity — it is evidence that the component stack and the architecture are coupled in ways the field had not recognized.

The paper's second landscape-changing contribution is what might be called a representation-level diagnostic: it shows that what a model learns is not just a function of its loss value or downstream accuracy, but can be probed directly in the internal activations to reveal properties — like explicit scene segmentation — that are invisible to standard evaluation protocols. Prior work evaluated self-supervised methods almost exclusively through transfer learning benchmarks (linear classification, fine-tuning, semi-supervised learning). DINO demonstrates that two methods with comparable ImageNet linear accuracy can have fundamentally different internal representations: supervised ViT-S/16 achieves 79.8% linear top-1 versus DINO's 77.0%, yet the supervised model's self-attention maps contain almost no explicit segmentation information (27.3 Jaccard) while DINO's contain clear object boundaries (45.9 Jaccard). This decoupling of classification accuracy from spatial understanding means that the standard evaluation protocol is blind to a major axis of representation quality. The field can no longer assume that better ImageNet linear accuracy implies better features for all purposes — a method might excel at classification while discarding spatial structure, or vice versa.

The conceptual reframing of self-supervised learning as self-distillation with no labels is a third shift, though more of a synthesis than a discovery. The paper connects previously disparate mechanisms — momentum encoders (MoCo, BYOL), clustering constraints (SwAV, DeepCluster), and knowledge distillation (Hinton et al., 2015) — under a single interpretation: the student learns by matching a better, ensemble-quality teacher whose outputs are kept from collapsing through entropy control (centering and sharpening). The key diagnostic supporting this framing is Figure 6 (left), which shows the momentum teacher consistently outperforming the student throughout training. The paper notes explicitly: "This behavior has not been observed by other frameworks also using momentum." This reframing unifies the field and suggests a path toward simpler methods: if the core mechanism is distillation from a better teacher, then the elaborate machinery built to prevent collapse (contrastive losses, clustering, predictor asymmetries) may have been solving the wrong problem — not "how do we prevent collapse" but "how do we ensure the teacher is sufficiently better than the student that matching it provides a useful learning signal." The centering-sharpening mechanism is one minimal answer to the former question; the momentum encoder answers the latter.

The paper also resolves a latent contradiction in the literature. Prior work had shown both that self-supervised convnets could learn segmentation-like information (Gur et al., 2020, requiring post-hoc factorization to extract) and that Transformers could attend to meaningful image regions under supervision (Dosovitskiy et al., 2020). DINO reconciles these by showing that self-supervised training on ViTs makes spatial structure directly accessible — no post-hoc extraction needed — while supervised ViTs suppress it. This explains why the "BERT moment" for vision had not happened: prior work was looking at the wrong architecture (convnets, where spatial information is distributed across weights) or the wrong training signal (supervision, which discards spatial relationships as irrelevant to the classification objective). DINO's contribution is identifying the specific combination — self-supervised ViT — where spatial understanding emerges as a first-class property rather than a latent one.

Finally, the paper makes k-NN evaluation respectable again as a primary diagnostic tool. The finding that DINO ViT features achieve 74.5% k-NN accuracy versus 77.0% linear (ViT-S/16) — and 78.3% k-NN versus 79.7% linear (ViT-S/8) — demonstrates that these features are inherently clustered by class without needing a learned linear transformation. The consistent pattern across Tables 2, 10, and 13 that k-NN gaps between methods are larger than linear gaps (DINO vs. BYOL on ViT-S: +5.6% linear but +7.9% k-NN; DINO vs. MoCo-v2: +4.3% linear but +10.1% k-NN) suggests that linear probing can mask real differences in feature quality by learning to disentangle representations that are not naturally separable. This methodological finding has implications for how the field should evaluate self-supervised methods going forward.

Follow-Up Research This Work Enables

Training a supervised ViT with DINO's exact augmentation pipeline to isolate the effect of the training objective from the data pipeline. The paper's central claim — that self-supervised training produces emergent properties (segmentation, k-NN friendliness) that supervised training does not — is confounded by the fact that DINO uses a dramatically more aggressive augmentation regimen (multi-crop, color jittering, Gaussian blur, solarization) than the supervised ViT baselines (standard ImageNet augmentations). A controlled experiment would train a supervised ViT-B/16 on ImageNet using exactly DINO's augmentation pipeline — 2×224² global views + 10×96² local views per image, all with BYOL-style color augmentations — but with a standard cross-entropy classification loss instead of DINO's self-distillation objective. The key measurements would be: (1) self-attention Jaccard similarity on PASCAL VOC 2012 (Figure 4 protocol, thresholding at 60% mass), testing whether the augmentation alone can induce segmentation emergence or whether the self-supervised matching objective is necessary; (2) k-NN classification accuracy on ImageNet, testing whether strong augmentations alone produce k-NN-friendly features or whether the self-distillation loss is required; (3) DAVIS-2017 video object segmentation with frozen features (Table 5 protocol). The existing Figure 4 table shows DINO without multi-crop achieves 45.1 Jaccard versus 45.9 with multi-crop, suggesting augmentations are not the primary driver of segmentation — but this ablation is within DINO's framework and does not test a supervised model. If the augmentation-matched supervised ViT achieves comparable Jaccard, the paper's narrative about "self-supervised properties" would need revision; if it does not, the claim is strengthened with proper controls.

Scaling DINO pre-training data by 10–100× on uncurated image collections to test whether emergent properties strengthen or qualitatively change. The paper's explicit motivation is enabling a "BERT moment" for vision through pre-training on massive unlabeled data, but all experiments use ImageNet-1K (~1.28M curated images). A natural extension would pre-train DINO ViT-B/8 on YFCC100M (~100M uncurated images; Thomee et al., 2015) or Instagram-1B (if accessible), then evaluate: (1) linear and k-NN classification on ImageNet (does transfer improve with scale, and do k-NN gaps relative to linear probing change?); (2) self-attention segmentation quality on PASCAL VOC and COCO (do attention maps become sharper, cover more object categories, or reveal part-whole hierarchies?); (3) few-shot learning with frozen features on diverse downstream datasets (does the feature generality improve such that 1% ImageNet linear evaluation substantially exceeds the 64.5% reported for ImageNet-1K pre-training?). The GLDv2 experiment in the paper (Table 3) provides one domain-specific scaling data point and shows strong retrieval improvement, but a general uncurated corpus would test whether the properties generalize beyond curated datasets. The key open question is whether the segmentation emergence and k-NN friendliness are properties of the self-supervised objective itself (which should persist or strengthen with scale) or artifacts of ImageNet's specific distribution (which might dilute with uncurated data). A negative result — emergent properties weakening with uncurated data — would suggest that dataset curation plays a larger role than the paper implies, and would motivate research into data filtering or curriculum strategies for large-scale self-supervised ViT pre-training.

Combining DINO pre-training with supervised or semi-supervised segmentation decoders on standard benchmarks. The paper demonstrates that self-attention maps correlate with object boundaries and that frozen patch tokens work for video object segmentation via nearest-neighbor matching, but does not evaluate on standard semantic segmentation benchmarks (PASCAL VOC 2012 segmentation, COCO-Stuff, ADE20K, Cityscapes) with trained decoder heads. A strong follow-up would: (1) attach a standard segmentation decoder (e.g., FPN or DeepLabv3+ head) to a frozen DINO ViT backbone and train only the decoder on PASCAL VOC 2012 and COCO-Stuff, comparing against a supervised ViT backbone with identical architecture and decoder; (2) fine-tune the full DINO ViT + decoder end-to-end, measuring whether DINO initialization provides better segmentation accuracy or faster convergence than supervised ImageNet initialization; (3) evaluate at multiple labeled-data regimes (100%, 10%, 1% of segmentation labels) to test whether the emergent spatial structure in DINO features reduces the annotation burden for dense prediction tasks. The prediction, based on the DAVIS results (Table 5: DINO ViT-S/8 achieves 69.9 (J&F)_m vs. supervised ViT-S/8 at 66.0), is that DINO pre-training should provide a meaningful improvement, particularly in low-label regimes where the pre-existing spatial structure in the features compensates for scarce pixel-level supervision. If DINO does not improve trained segmentation models despite showing clear attention-map segmentation, it would suggest that the attention-map property is a post-hoc interpretability artifact rather than a functionally useful representation for dense prediction — an important negative result that would refine our understanding of what "emergent properties" actually mean for downstream tasks.

Systematic optimization of BYOL, MoCo-v2, and SwAV for ViT architectures to determine whether DINO's advantage is methodological or tuning-based. The paper's ViT baselines use ResNet-optimized configurations, and the BYOL multi-crop failure (Table 14, row 10: 64.8% linear vs. 71.4% without) suggests these configurations are suboptimal. A rigorous follow-up would: (1) optimize each baseline method's hyperparameters specifically for ViT-S/16 — this would include the loss-specific components (BYOL's predictor architecture and momentum schedule, MoCo-v2's queue size and temperature, SwAV's number of Sinkhorn-Knopp iterations and prototype count), the projection head design (depth, width, normalization), and the augmentation pipeline (multi-crop scale range s, number of local views, color augmentation strength); (2) train all methods for 800 epochs on ViT-S/16 and ViT-B/16, reporting both linear and k-NN evaluation at multiple training durations to capture method-specific training dynamics; (3) evaluate the self-attention Jaccard similarity for each optimized method to determine whether the segmentation emergence is universal to self-supervised ViTs (as the Figure 4 cross-method comparison suggests, with BYOL at 47.8, MoCo-v2 at 46.3, SwAV at 46.8, all similar to DINO's 45.9) or whether DINO's specific design produces qualitatively better spatial structure. The result would establish a fair benchmarking baseline for self-supervised ViT methods and either confirm DINO's methodological advantage or reveal that the gap is largely attributable to tuning effort — either outcome would be valuable for the field's self-assessment.

Probing where and how segmentation information emerges across ViT layers, heads, and training checkpoints. The paper's self-attention analysis is limited to the "best head" of the last layer (Figure 4: "We show the best head for both models") and does not characterize how the spatial structure develops during training or across the network depth. A systematic study would: (1) extract self-attention maps from every head in every layer at regular training intervals (e.g., every 50 epochs during a 300-epoch DINO ViT-S/8 training run), computing Jaccard similarity against PASCAL VOC 2012 ground truth for each head; (2) determine whether segmentation information emerges first in early, middle, or late layers, and whether different heads specialize to different semantic levels (edges, textures, object parts, whole objects, background regions); (3) track the "emergence curve" — at what epoch does the Jaccard similarity begin to exceed the random baseline, and does it correlate with specific training dynamics (e.g., when the teacher's k-NN accuracy begins to exceed the student's, as shown in Figure 6 left)? This would transform the paper's qualitative observation ("self-attention maps contain segmentation information") into a quantitative developmental account of how and when spatial structure crystallizes during self-supervised ViT training. If segmentation emerges abruptly at a specific training phase (rather than gradually improving), it would suggest a phase-transition-like phenomenon that could guide training duration decisions. If different heads consistently encode different spatial granularities, it would motivate head-specific downstream usage (e.g., using early-layer heads for edge detection and late-layer heads for instance segmentation).

Practical Applications and Downstream Use Cases

Image retrieval and copy detection at scale without annotated training data. The paper demonstrates that DINO ViT features trained on unlabeled domain-specific data (Google Landmarks v2) achieve 51.5 mAP on revisited Oxford Medium and 75.3 mAP on revisited Paris Medium (Table 3) — substantially outperforming off-the-shelf supervised features (33.5 and 63.0 respectively). For a company with a large collection of unlabeled images (e.g., a stock photo platform, an e-commerce catalog, or a social media platform needing duplicate detection), this means they can train a custom retrieval model without any manual annotation. The practical workflow: collect a dataset of images representative of the target domain (landmarks, products, user-generated content), pre-train DINO ViT on this unlabeled collection using the default configuration (2×224² + 10×96² crops, 300 epochs on 2×8-GPU machines, ~3 days for ViT-S/16), then use the frozen features with cosine similarity for retrieval. For copy detection, DINO ViT-B/8 at 320² resolution achieves 85.5 mAP on Copydays "strong" subset (Table 4), exceeding the specialized MultiGrain model (82.5 at resolution 800) — meaning a general-purpose self-supervised model can match or exceed a purpose-built retrieval system with no task-specific design. The k-NN evaluation speed (63–1007 im/sec depending on model, Table 1) makes this practical for online retrieval with millions of images using approximate nearest neighbor indexes.

Low-shot classification for specialized domains with minimal labeled data. Table 12 shows that a logistic regression trained on frozen DINO ViT-S/16 features with only 1% of ImageNet labels (~12.8 examples per class) achieves 64.5% top-1 accuracy — exceeding fine-tuned BYOL ResNet-50 (53.2%) by 11.3 points. For a biologist classifying rare species with a few dozen labeled examples per category, or a quality-control inspector identifying manufacturing defects from a handful of annotated images, the workflow is: pre-train DINO on a large unlabeled collection of in-domain images (the "unlabeled ImageNet" for that domain), extract frozen features for the few labeled examples, train a lightweight logistic regression or k-NN classifier (no GPU needed for this step, no data augmentation, no hyperparameter tuning beyond the number of neighbors or regularization strength), and deploy. The k-NN protocol in particular requires no training at all — just feature extraction and storage. The paper's across-dataset k-NN results (Table 10) show that this approach works across diverse domains (Places205, PASCAL VOC, Flowers), with ViT-S features consistently outperforming ResNet-50 features by an average of 5.6 k-NN points. The practical limitation is that DINO pre-training still requires substantial compute (60.9 GPU-hours for ViT-S/16 at 300 epochs with 6 local crops; Table 8), but this is a one-time cost amortized over all downstream low-shot tasks — and the pre-training code is open-source and designed to work on limited hardware (the paper reports 35.2% k-NN after 50 epochs with batch size 8 on a single GPU, Section 5.5).

Unsupervised object discovery and weakly-supervised segmentation for image annotation pipelines. The emergent segmentation in DINO ViT self-attention maps (Figure 1, Figure 3, Figure 4) provides a zero-shot object localization signal that requires no segmentation labels. For an annotation team that needs to create bounding box or segmentation masks for a new object category, the workflow would be: (1) run a pre-trained DINO ViT on the unlabeled image collection; (2) threshold the self-attention maps (keeping 60% of the probability mass, as in Figure 4) to generate candidate object masks; (3) use these masks as initial pseudo-labels for a segmentation model, or present them to human annotators for correction rather than drawing masks from scratch. The paper's quantitative evidence: DINO ViT-S/16 achieves 45.9 Jaccard similarity against PASCAL VOC ground truth masks — far from perfect, but substantially better than supervised ViT-S/16 (27.3) and random (22.0). The practical value depends on whether the 45.9 Jaccard represents a useful starting point that reduces annotation time. The DAVIS-2017 results (Table 5) show that the spatial information in the patch tokens is functional for video object segmentation without any training, achieving 69.9 (J&F)_m with ViT-S/8 — competitive with specialized methods. For video annotation pipelines, DINO features enable propagating segmentation masks across frames via nearest-neighbor matching (the Jabri et al., 2020 protocol used in the paper), potentially reducing the annotation burden from per-frame labeling to sparse keyframe labeling with automatic propagation.

Self-supervised pre-training as initialization for supervised ViT training, improving accuracy without external data. Table 11 shows that initializing a supervised ViT-B/16 with DINO pre-training on ImageNet and then fine-tuning with supervision achieves 82.8% top-1, compared to 81.8% from random initialization and 81.9% from supervised pre-training — a +1.0 point improvement over the standard DeiT training procedure using the same data and architecture. For a practitioner training a ViT on a target dataset where they have labels (e.g., a medical imaging dataset, a satellite imagery classification task, or a custom product categorization problem), replacing random initialization with DINO pre-training on the same dataset (without labels) provides a consistent accuracy improvement at the cost of additional pre-training compute. The practical calculation: DINO pre-training on ImageNet takes ~73 GPU-hours for ViT-S/16 (Table 8, 300 epochs with 10 local crops), but this is a one-time cost if using the publicly released pre-trained weights. The paper provides pre-trained models for ViT-S/16, ViT-S/8, ViT-B/16, and ViT-B/8, so a practitioner can simply download the weights and fine-tune on their target dataset — the improvement demonstrated in Table 11 (82.8% vs. 81.8%) suggests this is a free accuracy gain with no annotation cost and minimal implementation effort beyond changing the weight initialization. The transfer learning results (Table 6: DINO outperforms supervised initialization on 5 of 6 downstream tasks for ViT-S/16, with gains of +0.3 to +2.0 points) suggest this benefit generalizes beyond ImageNet to diverse domains including fine-grained classification (Cars, Flowers, iNaturalist).

When to Prefer This Method

The paper positions DINO primarily as a discovery vehicle for studying self-supervised ViT properties rather than as a method competing against a specific named alternative with clear decision boundaries. It demonstrates DINO's competitiveness with BYOL, MoCo-v2, and SwAV (Tables 2, 13) and shows state-of-the-art results, but does not articulate explicit tradeoffs where a practitioner would choose DINO instead of a named baseline for a specific reason. The relevant decision is more about when to prefer self-supervised ViT features over supervised ViT features, which the paper does address:

Prefer self-supervised DINO ViT features when:

The downstream task involves spatial understanding, dense prediction, or retrieval, where the emergent segmentation information (45.9 Jaccard vs. 27.3 for supervised, Figure 4) and k-NN friendliness (74.5% vs. 79.8% for supervised ViT-S on ImageNet classification, but with substantially better retrieval in Tables 3–4) are directly useful. For pure ImageNet classification accuracy, supervised training still holds a small edge (79.8% vs. 77.0% for ViT-S using the DeiT procedure; Table 2).
Unlabeled in-domain data is abundant and labeled data is scarce — the GLDv2 retrieval result (Table 3: 51.5 mAP vs. 33.5 for supervised ImageNet features) and the low-shot frozen-feature results (Table 12: 64.5% at 1% ImageNet labels) demonstrate that DINO pre-training on domain-specific unlabeled data transfers better than supervised pre-training on a generic labeled dataset.
A "BN-free" training pipeline is desirable — DINO + ViT uses no batch normalization anywhere (Section 3.1), which avoids the distributed synchronization overhead that slows down methods like BYOL and SwAV when training on large GPU clusters. The paper notes this as "a great advantage" but does not provide quantitative comparisons of distributed training throughput with vs. without synchronized BN.
The practitioner needs features that work well with simple non-parametric classifiers (k-NN) without any task-specific training — the k-NN accuracy approaches linear probing accuracy for DINO ViTs (74.5% vs. 77.0% for ViT-S/16; Table 2), while the gap is much larger for other methods (BYOL ViT-S: 66.6% vs. 71.4%). This makes DINO attractive for deployment scenarios where training a linear classifier is impractical (e.g., dynamic class sets, streaming data, edge devices with limited training capability).