DINOv2: Learning Robust Visual Features without Supervision

🎯 Pitch

DINOv2 shows that self-supervised learning alone can match or surpass weakly-supervised models like OpenCLIP as a frozen feature extractor, overcoming the longstanding gap without any text supervision. The key is scaling a discriminative and masked-image modeling objective on a massive, automatically curated dataset of 142M diverse images, then distilling into smaller models. The resulting features transfer exceptionally across image and pixel-level tasks, even outperforming OpenCLIP on depth estimation and retrieval by large margins.

1. Executive Summary

This paper introduces DINOv2, a family of self-supervised vision transformer encoders that produce general-purpose visual features competitive with weakly-supervised alternatives across image and pixel-level tasks without any fine-tuning. The work combines iBOT masked image modeling with DINO instance-level discriminative objectives, trains on a curated dataset of 142M images (LVD-142M) built through a novel self-supervised retrieval pipeline, and scales to a 1B-parameter ViT-g model, then distills into smaller architectures. On a comprehensive suite of benchmarks — ImageNet classification, instance retrieval (Oxford and Paris landmarks), semantic segmentation (ADE20K, Cityscapes), monocular depth estimation (NYUd, KITTI), and fine-grained classification (iNaturalist, Cars) — DINOv2 improves over the previous self-supervised state of the art by 4–8 percentage points on classification and over 40% relative improvement on instance retrieval, while matching or surpassing the best openly available weakly-supervised model (OpenCLIP ViT-G/14) on most benchmarks, establishing that self-supervised pretraining alone can produce transferable frozen features that close the gap with text-guided methods when trained on sufficiently diverse curated data.

2. Context and Motivation

The Core Problem: Self-Supervised Features Lag Behind Text-Guided Methods for General-Purpose Use

The fundamental question this paper tackles is: can self-supervised learning (SSL) produce visual features that work as universal, off-the-shelf building blocks for computer vision — the way BERT and GPT embeddings work in NLP — without any text supervision? By "off-the-shelf," the authors mean features that can be extracted from a frozen backbone and fed into a simple classifier (e.g., a linear probe) or nearest-neighbor lookup, achieving strong performance across image classification, instance retrieval, semantic segmentation, and depth estimation without fine-tuning the encoder.

This question matters for several practical and theoretical reasons the paper surfaces (Section 1):

Removing the text bottleneck. Text-guided pretraining — the dominant paradigm for producing general-purpose visual features via models like CLIP (Radford et al., 2021) and OpenCLIP (Ilharco et al., 2021) — requires large-scale aligned image-text corpora. These are expensive to curate, unavailable in many languages and domains, and fundamentally limited by the fact that captions capture only a coarse approximation of the rich information in images. Pixel-level details (object boundaries, depth cues, part-level correspondences) may never surface through caption supervision because no one writes captions describing the relative depth of every chair in a scene or the precise contour of a dog's ear. The paper argues that an SSL approach, learning purely from images, is conceptually closer to the language modeling pretext tasks that unlocked foundation models in NLP and should therefore be capable of learning richer visual representations.
Enabling off-the-shelf vision systems. The promise of foundation models is that practitioners can use them without task-specific retraining. In NLP, this vision has largely been realized: BERT and GPT embeddings drive downstream applications with minimal adaptation. In vision, by contrast, the best self-supervised features have historically required supervised fine-tuning on the target task to be useful — the paper specifically contrasts with masked autoencoders (MAE; He et al., 2022), which "provide substantial improvements when finetuned on downstream tasks" but whose "features require supervised finetuning, while our features perform well out of the box." Closing this gap would substantially lower the barrier to deploying vision systems, particularly in domains where labeled data is scarce.
Escaping the monoculture of ImageNet-1k. Most advances in discriminative self-supervised learning — DINO (Caron et al., 2021), iBOT (Zhou et al., 2022a), MSN (Assran et al., 2022), MoCo v3 (Chen et al., 2021) — were developed and tuned on ImageNet-1k, a curated dataset of 1.28 million images across 1,000 classes. This narrow development loop raises concerns about whether the reported gains are specific to the ImageNet-1k distribution. The paper's larger ambition is to break out of this loop and determine whether SSL can produce features that generalize across substantially different image domains — fine-grained species classification (iNaturalist), landmark retrieval, driving scenes, indoor depth estimation — when trained on data that is both larger and more diverse than ImageNet-1k.

Conflicting Signals in Prior Work

The paper is motivated by a genuine tension in the literature. On one hand, discriminative SSL methods have shown remarkable properties at small scale: DINO features exhibit emergent object segmentation without supervision (Caron et al., 2021), iBOT learns patch-level representations that capture semantic correspondences (Zhou et al., 2022a), and these models enable applications like semantic appearance transfer (Tumanyan et al., 2022) and unsupervised correspondence distillation (Hamilton et al., 2022). These properties suggest that SSL captures fundamentally useful visual structure beyond what supervised or weakly-supervised training provides.

On the other hand, efforts to scale SSL beyond ImageNet-1k have been disappointing. Prior scaled SSL efforts (Caron et al., 2019; Goyal et al., 2019; 2021; 2022a) used large quantities of uncurated data — images scraped from the web with minimal filtering — and observed a significant drop in feature quality compared to training on curated datasets like ImageNet-22k. The paper is explicit about this failure mode:

"Some efforts on scaling these approaches beyond ImageNet-1k have been attempted, but they focused on uncurated datasets, which typically lead to a significant drop in the quality of the features. This is explained by the lack of control over the data quality and diversity, which are essential to produce good features."

This creates a frustrating situation: SSL seems to produce rich, transferable features at the ImageNet-1k scale, but scaling the data naively breaks this property. The field did not know whether the problem was fundamental (SSL cannot benefit from more data the way language modeling does) or fixable (the data simply needs to be curated, not just larger).

Where Existing Approaches Fall Short

The paper identifies specific limitations in prior work along three axes:

1. Uncurated data scaling doesn't work for frozen features. The most prominent scaled SSL effort, SEERv2 (Goyal et al., 2022a), trained RegNet models on billions of uncurated Instagram images. While it showed improvements from scaling, the resulting features required supervised fine-tuning to be competitive on downstream tasks. The paper draws a sharp distinction: prior work on scaling SSL "questions the ability of self-supervised methods to work on any data while we focus on producing the best pretrained encoders" (emphasis added to highlight the frozen-feature goal). In other words, prior work asked "can SSL scale at all?" and answered "yes, if you fine-tune." This paper asks "can SSL produce frozen features that work everywhere?" which is a harder and more practically valuable target.

2. Text-guided pretraining is the dominant but imperfect baseline. OpenCLIP (ViT-G/14) and EVA-CLIP (ViT-g/14), trained on LAION-2B and a proprietary mixture respectively, represent the state of the art in openly available general-purpose visual features as of the paper's writing. The paper acknowledges their strength but identifies two fundamental limitations:

Information loss from caption compression. A caption like "a dog sitting on a couch" omits the dog's breed, pose, lighting, the couch's texture, the spatial layout of the room, and countless other visual details that an SSL model trained purely on images could potentially capture. This limitation shows up empirically in dense prediction tasks: the paper notes that on depth estimation, "iBOT features extracted from a ViT-L outperform the ones of OpenCLIP with a ViT-G" (Section 7.4, Table 11), suggesting that text supervision indeed loses fine-grained geometric information.
Dependence on text-image corpora. Collecting and filtering aligned image-text data at scale requires significant engineering and is inherently limited to languages and cultures where such data is abundant. An SSL approach that works directly on images could operate in domains (medical imaging, satellite imagery, industrial inspection) where captions are unavailable or unreliable.

3. The curation gap: no systematic approach for SSL data at scale exists. In NLP, data curation pipelines for language model pretraining are well-established — the CCNet pipeline (Wenzek et al., 2020) trains a language model on high-quality text (Wikipedia) and uses it to score and filter CommonCrawl documents, keeping only those that resemble the reference distribution. No analogous pipeline existed for images at the time of this work. Prior approaches to filtering image datasets relied on:

Text metadata or hashtags (Mahajan et al., 2018; Radford et al., 2021), which require the same aligned corpora that SSL aims to avoid.
Pretrained vision encoders (Schuhmann et al., 2021; 2022), which creates a chicken-and-egg problem: you need good features to filter data, but you need filtered data to train good features. Schuhmann et al. used CLIP embeddings to filter LAION, meaning their pipeline already depended on text supervision.
Manual curation, which doesn't scale beyond ImageNet-sized datasets.

The paper's data pipeline (Section 3) is designed to fill this gap, using a self-supervised retrieval approach that requires only images and a modestly-sized curated seed dataset.

How This Paper Positions Itself

The paper positions itself not as proposing a fundamentally new learning algorithm, but rather as answering a specific empirical question: if you take existing discriminative SSL methods (DINO + iBOT) and give them the same careful attention to data curation and engineering that went into large language models, can they produce frozen features that match or surpass text-guided alternatives?

This framing is important because it shifts the narrative from algorithmic novelty to scale and data quality as the primary enablers. The paper's contributions are synergistic: improved training stability and efficiency (Section 5) make it possible to train at scale; the self-supervised data curation pipeline (Section 3) provides diverse, high-quality training images without text supervision; and the distillation procedure extends the benefits of the largest model to smaller, more deployable architectures.

The paper explicitly connects to the broader foundation model paradigm:

"Following this paradigm shift in NLP, we expect similar 'foundation' models to appear in computer vision. These models should generate visual features that work out of the box on any task, both at the image level, e.g., image classification, and pixel level, e.g., segmentation."

The key word is "should" — the paper's thesis is that this expectation is correct, but only if the pretraining data is curated with the same rigor that the NLP community applied to text data, and only if the training recipe is adapted to handle the scale.

A deliberate scope limitation. The paper does not claim to outperform text-guided methods on every benchmark. It shows strengths on instance retrieval (+34% mAP on Oxford-Hard over OpenCLIP), depth estimation (consistently lower RMSE across datasets and decoders), and fine-grained classification (+8.6% on iNaturalist 2018), but acknowledges that OpenCLIP still leads on scene classification (Places205, −2.3%) and some domain generalization benchmarks (ImageNet-R, ImageNet-Sketch). This nuanced positioning — competitive, not universally dominant — is more credible than claiming blanket superiority and reflects an honest assessment of the tradeoffs between SSL and text-guided pretraining.

The frozen-feature commitment. A distinctive aspect of this work is its commitment to evaluating features without fine-tuning. Most SSL papers report both frozen and fine-tuned performance, but the primary focus is usually on fine-tuning numbers. This paper inverts that priority: fine-tuning is treated as a sanity check (Section 7.1, Table 5 shows only a +2% improvement from fine-tuning on ImageNet-1k over the frozen linear probe), while the frozen-feature evaluations span 15+ benchmarks across image classification, instance retrieval, semantic segmentation, depth estimation, and video understanding. This commitment reflects the paper's goal of producing foundation models whose features are genuinely useful without per-task adaptation — the vision counterpart to BERT embeddings that you can download and use immediately.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

The "system" DINOv2 describes is a method for training Vision Transformer models to produce general-purpose image features—numerical vectors that represent images and their parts—entirely through self-supervised learning on curated image data, without requiring text captions or human labels. The problem it solves is that prior self-supervised methods produced features that either degraded when trained on large-scale uncurated internet data (losing the rich properties observed at ImageNet-1k scale) or required supervised fine-tuning to be useful on downstream tasks; the solution's "shape" is a carefully integrated recipe spanning data curation (a self-supervised retrieval pipeline that builds a diverse 142M-image dataset from a seed of curated sources), algorithmic design (combining two complementary self-supervised objectives with regularizers that stabilize and improve training at scale), and engineering (memory and compute optimizations that make billion-parameter training tractable), followed by distillation to produce a family of smaller, practical models.

3.2 Big-picture architecture (diagram in words)

The DINOv2 pipeline has four major stages:

Data curation (LVD-142M construction). A self-supervised ViT-H/16 pretrained on ImageNet-22k embeds ~1.2B uncurated web images and a collection of curated seed datasets. A deduplication step removes near-duplicates from the uncurated pool. A retrieval step selects, for each curated image, its nearest neighbors from the uncurated pool, producing a 142M-image dataset that balances diversity (from the web) with quality (from the curated seeds). This stage happens once, before pretraining.
Teacher-student self-supervised pretraining. A Vision Transformer (ViT) is trained with two simultaneous objectives: a DINO-style image-level loss that matches class-token features between different crops of the same image, and an iBOT-style patch-level loss that reconstructs masked patch features. Both use a teacher network updated via exponential moving average of the student. Additional regularizers (KoLeo entropy maximization, Sinkhorn-Knopp centering) spread and stabilize features. Training runs at 224×224 resolution for most iterations, with a short high-resolution (518×518) adaptation phase at the end. The largest model, ViT-g, has ~1.1B parameters.
Knowledge distillation. The trained ViT-g serves as a frozen teacher to train smaller ViT architectures (ViT-L, ViT-B, ViT-S) using the same self-supervised objectives. This is more effective than training the smaller models from scratch.
Frozen-feature evaluation. The resulting encoders are used as fixed feature extractors. For classification, a linear classifier is trained on top of the frozen features. For segmentation and depth estimation, simple decoders (linear layers or DPT) operate on frozen patch tokens. No fine-tuning of the backbone occurs during evaluation.

3.3 Roadmap for the deep dive

First, the data curation pipeline (Section 3 of the paper): Understanding what data the model sees is foundational—the paper's central claim is that curation quality, not just data quantity, enables frozen-feature transfer. I will walk through the seed datasets, the deduplication procedure, the self-supervised retrieval mechanism, and the final composition of LVD-142M.
Second, the discriminative self-supervised objectives (Section 4): This covers the DINO image-level loss, the iBOT patch-level loss, the Sinkhorn-Knopp centering modification, the KoLeo regularizer, and the high-resolution adaptation—each with its mathematical form, computational mechanics, and the design rationale for including it.
Third, the efficient implementation (Section 5): These are the engineering decisions that make billion-parameter self-supervised training feasible: custom FlashAttention, sequence packing, efficient stochastic depth, FSDP with mixed-precision, and the distillation procedure. Their purpose is to make the training 2× faster and use 3× less memory than comparable implementations.
Fourth, the ablation-driven design choices (Section 6): The paper incrementally builds from iBOT to DINOv2 through a series of modifications validated on k-NN and linear probe performance. I will explain each modification, its observed effect, and the practical motivation (e.g., LayerScale and stochastic depth were added for training stability despite hurting linear probe accuracy, because they prevented NaN losses that would derail large-scale training).
Fifth, the resolution adaptation and distillation mechanisms: These cross-cut Sections 5 and 6. The resolution adaptation is a compute-saving trick; distillation is a performance-boosting trick for small models. Both have specific implementation details that matter for reproduction.

3.4 Detailed, sentence-based technical breakdown

This is primarily a systems and scaling paper—its core contribution is demonstrating that a carefully engineered combination of existing techniques, scaled to larger models and curated data, produces frozen visual features competitive with text-supervised alternatives. The "method" is the integration of data curation, algorithmic design, and engineering.

The Data Curation Pipeline: Building LVD-142M

Motivation and overall strategy. The authors observe that scaling SSL on uncurated web data degrades feature quality, while training exclusively on existing curated datasets (like ImageNet-22k) limits diversity and causes overfitting to narrow domains. Their solution is a retrieval-based pipeline that augments a collection of high-quality curated datasets with visually similar images drawn from a massive uncurated web crawl. The key design principle is that this retrieval is performed using a self-supervised image encoder, so the pipeline requires no text metadata, no pretrained text-guided model (like CLIP), and no manual annotation—it works purely from images.

Seed curated datasets. The curated seed spans 16 datasets chosen to cover diverse visual domains. The authors explicitly state (Table 15) that the selection is intended to "provide images covering well various downstream vision tasks both for image-level and dense recognition." The sources include: ImageNet-22k (14.2M images, general objects), ImageNet-1k train split (1.28M images), Google Landmarks v2 (1.58M images, landmarks/instance retrieval), and 13 fine-grained or scene-level datasets used primarily for cluster-based retrieval: Caltech 101, CUB-200-2011 (birds), DTD (textures), FGVC-Aircraft, Flowers-102, Food-101, Oxford-IIIT Pet, Stanford Cars, SUN397 (scenes), Pascal VOC 2007, ADE20K (segmentation), Cityscapes (urban scenes), Pascal VOC 2012 segmentation, NYU Depth V2, KITTI, SUN RGB-D, Mapillary SLS (street-level sequences for place recognition). Several of these are quite small—FGVC-Aircraft has only 3,334 training images, Flowers-102 has 1,020, Pascal VOC 2012 segmentation has 1,464. The retrieval step is specifically designed to augment these small but high-quality sources with visually similar data from the web.

Uncurated data source. The uncurated data comes from "a publicly available repository of crawled web data." The authors extract image URLs from <img> tags on web pages, discard URLs that are unsafe or from restricted domains, and post-process the downloaded images with PCA hash deduplication, NSFW filtering, and face blurring. This produces 1.2B unique images.

Self-supervised embedding model for retrieval. To measure visual similarity between curated seed images and uncurated web images, the authors compute embeddings using a self-supervised ViT-H/16 network pretrained on ImageNet-22k. The specific SSL method used for this embedding model is not detailed, but the critical design choice is that the embedding model is self-supervised—it was trained without text supervision, preserving the pure-image ethos of the pipeline. Cosine similarity in this embedding space serves as the distance metric between images.

Deduplication (self-deduplication and relative deduplication). Before retrieval, the uncurated data undergoes two deduplication stages. Both use the copy detection pipeline of Pizzi et al. (2022), which produces embeddings specifically designed to detect near-duplicate images (robust to crops, resizing, and mild edits—not just exact pixel matches).

Self-deduplication: Within the 1.2B uncurated images, the authors compute embeddings and retrieve the k=64 nearest neighbors for each image using cosine similarity. They construct a graph where edges connect image pairs with similarity greater than 0.6, find connected components using a scalable disjoint-set data structure, and keep only one representative per component. This reduces the pool from 1.2B to 1.1B images, removing redundancy while preserving diversity.
Relative deduplication: To avoid training on near-duplicates of evaluation images (which would inflate test-set performance through memorization rather than generalization), the authors identify uncurated images whose cosine similarity to any training or test split image of any benchmark exceeds 0.45, and discard those uncurated images entirely. This conservative threshold ensures the model is evaluated fairly. This step reduces the pool from 1.1B to 744M images.

Retrieval strategies (sample-based and cluster-based). The pipeline employs two retrieval mechanisms, chosen based on the size of the query dataset.

Sample-based retrieval (for datasets with >1M images): For each image in the query dataset, retrieve its k nearest neighbors from the deduplicated uncurated pool in the self-supervised embedding space. For ImageNet-22k, k=4 is used (multiplying its size from 14.2M to 56.8M retrieved images). For ImageNet-1k, k=32 is used (expanding from 1.28M to 41M). Google Landmarks v2 also uses k=4. The authors note that "although visual inspection seemed to indicate good retrieval quality for N much larger than 4, this leads to more collisions (images that are nearest-neighbor retrievals of multiple queries)"—so k=4 represents a tradeoff where each retrieved image typically corresponds to a single query, maintaining the semantic coherence of the augmentation while adding diversity.
Cluster-based retrieval (for datasets with <1M images): Many curated sources are too small for sample-based retrieval to provide meaningful augmentation (e.g., a few thousand images each). For these, the authors first cluster the 744M uncurated images into 100,000 clusters using distributed k-means on the self-supervised embeddings. Each cluster is assumed to capture a distinct visual concept. For each small curated dataset, they identify which clusters contain more than 3 of its images, and then sample 10,000 images from those clusters. To prevent any single dataset from dominating, cluster-based retrievals are capped at 1M images per dataset.

Final dataset composition. The authors include ImageNet-22k and Google Landmarks v2 in their original form ("as is") as well as in retrieved form, using both guaranteed-quality sources and their visual neighbors. The final LVD-142M dataset contains 142,109,386 images (Table 15). The retrieval is accelerated using the Faiss library with GPU-accelerated inverted file indices and product quantization codes. The entire processing—embedding 1.2B images, deduplication, clustering, and retrieval—runs on 20 nodes with 8 V100-32GB GPUs each and takes less than two days.

Why this pipeline over alternatives? The design is directly inspired by the CCNet text curation pipeline (Wenzek et al., 2020), which trained a language model on Wikipedia and used it to score and filter CommonCrawl documents. The analogy: the curated seed datasets are "Wikipedia-quality images," the self-supervised ViT-H/16 is the "scoring model" that identifies web images visually similar to this high-quality reference, and the retrieval step is the "filtering" that keeps only high-relevance images. The crucial difference from prior image curation approaches (Schuhmann et al., 2021, who used CLIP embeddings; Mahajan et al., 2018, who used hashtags) is that no text, metadata, or pretrained text-guided model is used. The pipeline is self-contained within the image domain, making it applicable to domains where captions or metadata are unavailable. A potential concern—which the paper does not extensively address—is that the self-supervised embedding model (pretrained on ImageNet-22k) may introduce its own biases into the curation, potentially limiting the diversity of retrieved images to those that look like ImageNet-22k concepts. The paper partially addresses this by noting that performance improves on domains not used for curation (iNaturalist, Places205), suggesting the retrieval generalizes beyond the seed distribution.

The Discriminative Self-Supervised Objectives

What "discriminative" means in this context. In contrast to generative self-supervised methods (like MAE, which learn by reconstructing masked pixels), discriminative methods learn by distinguishing between different images or image views—they solve a classification or matching problem defined over the data itself. The two objectives in DINOv2, DINO and iBOT, are both discriminative in this sense: DINO distinguishes which image a crop came from (by matching student and teacher views), and iBOT distinguishes which patch a token corresponds to (by matching masked student tokens to visible teacher tokens at the same spatial location).

The teacher-student framework (shared by both objectives). DINOv2 maintains two copies of the Vision Transformer: a student network (parameters updated via gradient descent) and a teacher network (parameters updated as an exponential moving average of the student). The same image is passed through both networks, but with different augmentations: the student sees both large "global" crops (224×224) and small "local" crops (98×98), while the teacher sees only the global crops. This asymmetry—the teacher sees cleaner, higher-resolution views—is central to the framework's stability: it prevents collapse (the trivial solution where all images map to the same representation) because the teacher provides a slowly-evolving target that is easier to predict consistently. The teacher's weights are updated after each training step as:

$\theta_t \leftarrow \lambda \theta_t + (1 - \lambda) \theta_s$

where $\theta_t$ are the teacher parameters, $\theta_s$ are the student parameters, and $\lambda$ is the momentum coefficient, following a cosine schedule from 0.994 to 1.0 over the course of training.

What it computes: At each step, the teacher becomes a weighted blend of its previous state and the current student state, with the blend heavily favoring the teacher's history (when λ=0.994, the teacher retains 99.4% of its previous parameters and incorporates only 0.6% of the student's update). Over millions of steps, the teacher slowly tracks the student's trajectory but with substantially lower variance.

Why this form: Naively using the same network as both student and target would lead to representation collapse—the network could trivially satisfy any matching objective by setting all features to a constant vector. The momentum teacher is an empirical solution discovered by He et al. (2020) for MoCo and adopted by Grill et al. (2020) for BYOL; it works because the teacher's slow update rate means it can only provide stable, slowly-changing targets, while the student must learn to match those targets from heavily augmented views, forcing it to extract meaningful invariants. The cosine schedule from 0.994 to 1.0 means the teacher becomes progressively more stable over training, with the student having to catch up to an increasingly slowly-moving target.

The DINO Image-Level Objective

What it does. The DINO objective (Caron et al., 2021) ensures that different augmented views of the same image produce similar global representations. The global representation is the class token (the special [CLS] token appended to the input patch sequence in standard ViT architectures). Both student and teacher pass their respective class tokens through a DINO projection head—a small MLP that maps the class token dimension (e.g., 1536 for ViT-g) to a larger "prototype" space of K dimensions (typically K=65536 or 128k). The output logits are then normalized with a softmax to produce probability distributions over the K prototypes. The loss is the cross-entropy between the teacher's distribution $p_t$ and the student's distribution $p_s$ :

$L_{\text{DINO}} = -\sum p_t \log p_s$

where $p_t$ is the teacher's prototype probability distribution after softmax and centering (explained below), and $p_s$ is the student's distribution after softmax.

What it computes: For each global crop viewed by both student and teacher, the DINO loss measures the KL divergence between the teacher's and student's assignments of the image to K learned prototypes. If the teacher assigns high probability to prototype 42 (say) and the student assigns low probability to it, the loss is large. Minimizing this loss forces the student to produce prototype distributions that match the teacher's, which in turn forces both networks to extract representations that are invariant to the specific augmentations applied to the student's crop.

Why K prototypes? The prototypes serve as a bottleneck that prevents the trivial matching of continuous representations. If the loss were applied directly to the class token features (e.g., with MSE), the student could match the teacher without learning anything meaningful—it could just copy the teacher's output. The softmax over K discrete prototypes forces the networks to commit to a categorical assignment, and because the teacher's assignment is slowly-evolving and the student sees a different crop, the only way to achieve consistent assignments is to learn features that capture the image's content rather than its accidental low-level statistics. The paper uses K=128,000 prototypes (increased from the 65,536 used in the original iBOT), finding that the larger prototype set improves k-NN performance (Table 1, +1.2% on k-NN).

The centering operation (Sinkhorn-Knopp vs. moving average). A critical detail: if the softmax is applied directly to the teacher's logits, the teacher will tend to collapse to a single prototype because there is nothing preventing it. The original DINO (Caron et al., 2021) addressed this with a centering operation on the teacher: a running mean of the teacher's prototype activations is subtracted from the logits before softmax. This mean is updated with a momentum term, effectively subtracting the overall bias of the teacher toward certain prototypes and forcing a more uniform distribution.

The present paper, following a recommendation from Ruan et al. (2023), replaces this moving-average centering with the Sinkhorn-Knopp (SK) batch normalization from SwAV (Caron et al., 2020). The SK algorithm solves for a doubly-stochastic matrix that approximates the teacher's logit matrix—it iteratively normalizes rows and columns so that each image (row) has a uniform marginal distribution and each prototype (column) has a uniform marginal distribution. Concretely, for the teacher logit matrix $L \in \mathbb{R}^{B \times K}$ (B images in the batch, K prototypes), the SK algorithm runs for 3 iterations of alternating row and column normalization to produce $p_t$ :

Row-normalize: $L_{ij} \leftarrow L_{ij} / \sum_k L_{ik}$ (each image's distribution sums to 1)
Column-normalize: $L_{ij} \leftarrow L_{ij} / \sum_b L_{bj}$ (each prototype is used equally across the batch)
Repeat from step 1 for 3 total iterations, then take the final matrix as $p_t$ .

Why Sinkhorn-Knopp over moving-average centering? The moving-average centering operates on per-prototype statistics accumulated over time, which can lag during training and requires careful tuning of the momentum. SK batch normalization operates per-batch, which provides an adaptive, data-dependent centering that responds immediately to changes in the feature distribution. It also guarantees—in the limit of many iterations—a perfectly uniform prototype usage across the batch, whereas moving-average centering only approximately balances usage. The paper reports that this switch does not change the final k-NN or linear probe performance in their ablation (Table 1: both with and without SK show 81.7% and 84.7% respectively), but does not explicitly ablate training stability, suggesting the benefit is primarily in simplifying hyperparameter tuning rather than improving final accuracy.

The student's path. The student's class token passes through the student DINO head (a separate MLP from the teacher's head), and a standard softmax is applied directly—no centering. The student's gradient flows through the softmax and the cross-entropy loss, updating the student backbone and DINO head. The teacher is updated only through the exponential moving average of the student weights (the teacher head weights are also EMA-updated from the student head).

The iBOT Patch-Level Objective

What it does. The iBOT objective (Zhou et al., 2022a), which stands for "image BERT pre-training with online tokenizer," extends the teacher-student matching from the image level to the patch level. It is inspired by masked image modeling (MIM): the student sees a version of the image where some patches are masked out (replaced with a learnable mask token), and must predict the teacher's patch features at the masked locations. This forces the network to learn local, spatially-structured representations that capture the visual properties of individual image regions.

The masking procedure. The student's input patches are randomly masked before being fed to the ViT. The teacher sees the complete (unmasked) set of patches. Both networks process their respective inputs and produce patch tokens at each spatial location. The loss is then computed only at the locations where the student's input was masked:

$L_{\text{iBOT}} = -\sum_i p_t^i \log p_s^i$

where $i$ indexes over the masked patch positions, $p_t^i$ is the teacher's prototype distribution at position $i$ (computed from the teacher's output patch token at that position, passed through the teacher iBOT head and SK-normalized), and $p_s^i$ is the student's prototype distribution at position $i$ (computed from the student's mask token output at that position, passed through the student iBOT head and softmax-normalized).

What it computes: For each masked patch position, the student must predict—from the visible surrounding context—what the teacher's representation of that patch would have been. This is a form of feature-level inpainting: the student learns to fill in the missing local features by reasoning about the visible parts of the image. Because the teacher's patch features capture semantic information about the visual content at each location, the student learns to associate spatial context with semantic content.

Why this helps for pixel-level tasks: The patch-level objective ensures that every spatial location in the output feature map carries meaningful, discriminative information. Without it, a network trained only with DINO's image-level objective might learn excellent global representations while having degenerate patch features that collapse to similar values regardless of content (since only the class token is directly supervised). Table 3b shows this empirically: removing the iBOT loss drops ADE-20K segmentation mIoU by nearly 3 points (from 47.1 to 44.2).

Why the prototype head is applied per-patch: The patch-level iBOT head is a separate MLP (with unshared weights from the DINO head—see next subsection) that takes a single patch token and outputs logits over the same K prototypes. This means each patch is independently assigned to prototypes, and the loss enforces that corresponding patches (at the same spatial location in the student and teacher views) produce similar prototype distributions. An alternative would be to apply the loss directly in the feature space (e.g., with L2 regression on the patch tokens), but the prototype bottleneck serves the same anti-collapse function as in DINO: the categorical assignment forces the network to commit to discrete, meaningful representations rather than floating in a continuous space where collapse is possible.

Untying the DINO and iBOT Heads

The design choice. In the original iBOT paper (Zhou et al., 2022a), an ablation study showed that sharing parameters between the DINO projection head (applied to the class token) and the iBOT projection head (applied to patch tokens) led to better performance than using separate heads. The present paper reports the opposite finding at scale: they use two separate, independently-parameterized MLP heads for the DINO and iBOT objectives in all experiments (Table 1, last row: "Untying heads = DINOv2" improves k-NN by +0.3% over the tied-head variant).

Why the reversal at scale? The paper does not speculate extensively on the mechanism, but a plausible interpretation is that the class token and patch tokens encode fundamentally different types of information—global image identity versus local spatial content—and forcing them to share a projection head creates conflicting gradients. At the ImageNet-1k scale tested in the original iBOT, this conflict may have been mild enough that parameter sharing's regularization benefit (reducing overfitting on a small dataset) outweighed the representational cost. At the LVD-142M scale with much larger models, the conflict dominates, and separate heads allow each objective to learn its own optimal projection.

The KoLeo Regularizer

What it does. The KoLeo regularizer (Sablayrolles et al., 2019) encourages the class-token features to spread uniformly across the feature space, rather than clustering in a low-dimensional subspace. It is applied as an additional loss term on the class tokens of the first global crops within each GPU's portion of the batch (the Kiwi regularizer does not involve cross-GPU communication). Given a set of n vectors $(x_1, \ldots, x_n)$ , each L2-normalized, the KoLeo loss is:

$L_{\text{koleo}} = -\frac{1}{n} \sum_{i=1}^{n} \log(d_{n,i})$

where $d_{n,i} = \min_{j \neq i} \|x_i - x_j\|$ is the minimum Euclidean distance from $x_i$ to any other point in the set. All features are L2-normalized before computing distances.

What it computes: For each class token in the batch, find its nearest neighbor (in Euclidean distance after normalization) and take the negative log of that distance. Sum and average over the batch. Because $-\log(d)$ grows large as d approaches zero, the loss heavily penalizes any two class tokens that are nearly identical, pushing them apart.

Why this form (the Kozachenko-Leonenko connection): This loss derives from the Kozachenko-Leonenko differential entropy estimator (Beirlant et al., 1997; Delattre & Fournier, 2017). The estimator approximates the differential entropy of a continuous distribution from a finite sample by summing the log of nearest-neighbor distances. Minimizing $L_{\text{koleo}}$ is equivalent to maximizing an estimate of the entropy of the feature distribution—it encourages the features to "spread out" to fill the available space uniformly. This is desirable because features that are well-spread have higher effective dimensionality and are more discriminative for nearest-neighbor tasks (where the distance between query and database features is the sole signal).

Empirical effect. Table 3a shows that adding KoLeo improves instance retrieval (Oxford-M) by over 8 percentage points in mAP (55.6% → 63.9%), while having minimal impact on classification (85.3% → 85.8%) and segmentation (47.2% → 47.1%). This makes intuitive sense: retrieval relies on nearest-neighbor search in the feature space, so spreading features apart directly improves the separation between relevant and irrelevant database items. The regularizer has a weight of 0.1 in the overall loss.

Implementation detail. The Kiwi regularizer is computed per-GPU without cross-communication, meaning that the nearest-neighbor search is limited to images within the same GPU's data slice. This is an approximation—the true nearest neighbor might be on another GPU—but it avoids the communication overhead of an all-to-all distance computation. The approximation becomes more accurate as the batch size per GPU increases.

High-Resolution Adaptation at the End of Training

The problem. Pixel-level downstream tasks (segmentation, depth estimation) benefit from higher input resolution because small objects and fine boundaries are lost when images are downsampled to the standard pretraining resolution of 224×224. Training from scratch at high resolution (e.g., 416×416 or 518×518) would be 2–4× more expensive in compute and memory due to the quadratic cost of self-attention in the number of patches.

The solution. The paper adopts a two-phase training schedule. The model is trained for the majority of iterations at 224×224 resolution. Then, for the final 10k iterations, training continues at a higher resolution (518×518 for the final models; the ablation in Section 6.6 uses 416×416 on a smaller setup). All hyperparameters remain the same as the first phase, except the base learning rate is reduced (the exact reduction factor is not specified, but the authors note that "all the schedules are kept the same as in the original training, but compressed to fit in 10k iterations").

Why this works (Figure 6). The ablation in Section 6.6 trains a ViT-L/16 on ImageNet-1k under three conditions: full training at 224, full training at 416, and training at 224 with a 10k-iteration 416 adaptation at the end. On ImageNet-1k linear probe accuracy, the 224→416 adaptation achieves performance nearly identical to full 416 training across resolutions (within ~1% at all tested resolutions from 224 to 768). On ADE-20K segmentation, it similarly matches full high-resolution training. Crucially, the adaptation costs only a fraction (roughly 1/60th, since 10k iterations vs. ~600k+) of the compute of full high-resolution training.

Why this is not just position embedding interpolation. When a ViT trained at resolution R1 is evaluated at resolution R2, the positional embeddings (learned or sinusoidal vectors encoding each patch's spatial location) must be interpolated to the new grid size. This interpolation is lossy—the positional embeddings for the new patch positions are approximated by combining those at nearby old positions. By actually training at the higher resolution for a short period, the positional embeddings adapt to the new grid, and the patch-level representations learn to use the higher-resolution information. The paper notes this is similar to the FlexiViT approach (Beyer et al., 2023), where a single model is trained to handle multiple patch sizes, and to the UniViT training of Likhomanenko et al. (2021).

Efficient Implementation for Billion-Parameter Training

The efficiency claim. The paper states that compared to the original iBOT implementation, their code "runs around 2× faster using only 1/3 of the memory" on the same hardware. The following five techniques collectively enable this.

Custom FlashAttention. The authors implement their own version of FlashAttention (Dao et al., 2022), a memory-efficient exact attention algorithm that avoids materializing the full N×N attention matrix. Instead, it computes attention in tiles, using the GPU's SRAM as a fast scratchpad to accumulate softmax-normalized attention outputs without ever writing the intermediate attention logits to HBM (high-bandwidth memory). The custom implementation "is on par with or better than the original on all cases considered, while covering more use-cases and hardware."

A critical detail for efficiency: the implementation is optimized for embedding dimensions per head that are multiples of 64, and full embedding dimensions that are multiples of 256. This hardware constraint feeds back into architecture design: the ViT-g uses an embedding dimension of 1536 with 24 heads (64 dim/head) rather than the 1408 with 16 heads (88 dim/head) proposed by Zhai et al. (2022). The authors state their experiments did not show significant differences in final accuracy from this change.

Sequence packing. The DINO algorithm requires forwarding both large global crops (224×224 → 14×14 = 196 patches + CLS token = 197 tokens) and small local crops (98×98 → 7×7 = 49 patches + CLS token = 50 tokens) through the same ViT. Without packing, these would be processed in separate forward passes because they have different sequence lengths and cannot be batched together naively.

Sequence packing (Krell et al., 2022, from NLP) concatenates all sequences—both global and local crops—into a single long sequence. A block-diagonal attention mask is applied to prevent attention from leaking between different images. The packed sequence passes through the transformer once; the mask ensures that the attention computation at each position only sees tokens from the same image. This is strictly mathematically equivalent to processing each sequence separately but avoids the overhead of multiple forward/backward passes and improves GPU utilization by increasing the compute per kernel launch.

The lower-level components are made available through the xFormers library (Lefaudeux et al., 2022).

Efficient stochastic depth. Stochastic depth (Huang et al., 2016) is a regularization technique where entire transformer blocks are randomly dropped during training (their output is set to zero, and the input is passed through the residual connection unchanged). The standard implementation computes the block's output and then masks it to zero—wasting the computation for dropped blocks.

The improved version randomly shuffles the B samples along the batch dimension at the start of each block, then slices off the first (1 − d) × B samples to actually pass through the block, and passes the remaining d × B samples through the residual connection only. Because modern GPUs process samples in parallel, processing fewer samples proportionally reduces computation and memory. With the paper's drop rate of d = 40%, this saves approximately 40% of the computation and memory in the affected blocks.

Fully-Sharded Data Parallel (FSDP) with mixed precision. Training with the AdamW optimizer requires storing four full-precision (float32) model replicas: student parameters, teacher parameters, optimizer first moments, and optimizer second moments. For a 1B-parameter model, this totals 16 GB of memory. FSDP (via PyTorch's implementation) shards each of these four replicas across all GPUs, meaning each GPU stores only a fraction of the parameters at any time and gathers the relevant shard before computation.

The PyTorch FSDP implementation provides an additional communication benefit: while the weight shards are stored in float32 (required for optimizer numerical stability), the broadcast of weights and the reduction of gradients are performed in float16 for the backbone parameters. This halves the communication volume compared to the float32 all-reduce used in standard DistributedDataParallel (DDP). The MLP projection heads (DINO and iBOT heads) are an exception: their gradients are reduced in float32 because the authors found that float16 reduction caused training instabilities in these smaller, shallower networks.

Distillation for smaller models. Training the ViT-g from scratch with all the above optimizations produces the strongest features. For smaller architectures (ViT-L, ViT-B, ViT-S), the authors find that distillation from the frozen ViT-g outperforms training from scratch (Figure 5). The distillation procedure repurposes the same self-supervised training loop with modifications:

The teacher is replaced by the frozen, pretrained ViT-g model (no longer an EMA of the student).
A separate exponential moving average of the student is maintained and used as the final model (not the raw student checkpoint).
The masking and stochastic depth are removed (the student sees the full, unmasked set of patches).
The iBOT loss is applied on the two global crops (not just the masked positions, since there are no masked positions).
No loss terms or architecture are modified for distillation—the same DINO and iBOT objectives apply, with the frozen ViT-g providing the teacher targets.

This distillation method closely resembles Duval et al. (2023), with the difference that the authors evaluate the EMA of the student rather than the raw student and do not modify the loss composition. Figure 5 shows the distilled ViT-L outperforms the from-scratch ViT-L on all 12 benchmarks evaluated, and sometimes even surpasses the ViT-g teacher (e.g., Oxford-M retrieval: 82.7% for ViT-L distill vs. 71.6% mAP for ViT-L scratch vs. 82.7%?—actually the ViT-g is 71.6% and the distilled ViT-L is 82.7%, so the student surpasses the teacher on this metric). This is attributed to the iBOT loss being applied on global crops during distillation (rather than masked positions), which may provide additional refinement.

The Sinkhorn-Knopp Centering Modification

What changes from original DINO/iBOT. The original DINO (Caron et al., 2021) applied a moving-average centering to the teacher's prototype logits: a running mean vector $c \in \mathbb{R}^K$ tracks the average logit value for each prototype across all images, and this vector is subtracted from the logits before the softmax. The running mean is updated as $c \leftarrow m \cdot c + (1 - m) \cdot \text{mean}(\text{logits})$ where m is a momentum hyperparameter.

The present paper replaces this with the Sinkhorn-Knopp (SK) batch normalization from SwAV (Caron et al., 2020). The SK algorithm operates on the teacher logit matrix $L \in \mathbb{R}^{B \times K}$ (batch size B, prototypes K) and iteratively applies row normalization and column normalization:

Row normalization (step 1): $L_{ij} \leftarrow\frac{L_{ij}}{\sum_{k=1}^K L_{ik}}$

This ensures each image's prototype distribution sums to 1 (a valid probability distribution).

Column normalization (step 2): $L_{ij} \leftarrow\frac{L_{ij}}{\sum_{b=1}^B L_{bj}}$

This ensures each prototype is used equally across the batch—every prototype has a total probability mass of 1 across all images.

Iteration: Steps 1 and 2 are repeated for 3 iterations, after which the resulting matrix is the teacher target $p_t$ . Column normalization is only applied to the teacher's logits; the student uses a standard softmax without any centering.

Why this over the moving-average alternative: The moving-average centering tracks per-prototype usage over the entire training history using an exponential moving average. This has two potential failure modes at scale: (1) the running mean can lag behind the actual feature distribution if the momentum is too high, causing the centering to be stale; (2) the centering vector is shared across all images and cannot adapt to per-batch variations. SK batch normalization solves both: it is an exact, adaptive rebalancing that guarantees uniform prototype usage within each batch, and because it converges quickly (3 iterations is sufficient in practice), it adds minimal computational overhead. The batch-based operation also means the centering is always current with the model's present state.

Why 3 iterations? The Sinkhorn-Knopp algorithm converges to a doubly-stochastic matrix (rows sum to 1, columns sum to 1) asymptotically. Three iterations provides an approximation good enough for practical purposes—the prototype usage is roughly uniform, preventing collapse, without the computational cost of iterating to full convergence. The paper does not ablate the number of iterations.

The lack of a performance difference (Table 1). In the ablation, moving from moving-average centering to Sinkhorn-Knopp shows no change in k-NN (81.7% → 81.7%) or linear probe (84.7% → 84.7%) on ImageNet-1k. This suggests the primary benefit is training stability or simplification (removing the momentum hyperparameter for centering), not accuracy. The authors may have chosen SK centering because it is more robust at larger batch sizes or longer training schedules where moving-average lag could compound.

Training Hyperparameters and Schedule

The paper provides explicit hyperparameters for training from scratch (Table 16) and architecture details (Table 17). For the ViT-g trained from scratch:

Optimizer: AdamW with learning rate 3.5e-4.
Batch size: 3072.
Training iterations: 625k.
Weight decay: cosine schedule from 0.04 to 0.2.
Learning rate warmup: 100k iterations.
Teacher momentum: cosine schedule from 0.994 to 1.0.
Initial LayerScale value: $1 \times 10^{-5}$ (LayerScale is a per-channel learnable scaling of residual branch outputs, initialized to near-zero to stabilize early training; Touvron et al., 2022).
Stochastic depth drop rate: 0.4 (meaning 40% of transformer blocks are randomly dropped during each forward pass).
Precision: float16 for backbone forward/backward and weight/gradient communication; float32 for DINO and iBOT head gradient reduction.
Prototype count: 128,000 for the DINO and iBOT heads.
FFN type: SwiGLU (Shazeer, 2020), a gated linear unit variant using a SiLU activation applied to one half of the intermediate representation and multiplied element-wise with the other half.
Architecture (ViT-g): embedding dimension 1536, 24 attention heads (64 dim/head), 40 transformer blocks. Total parameters: ~1.1B.
Patch size: 14 (input image 224×224 → 16×16 patches).

For distilled models (ViT-L, ViT-B, ViT-S), the learning rate is increased to 1e-3, batch size is 2048, stochastic depth and masking are disabled (drop rate 0), and the FFN uses standard MLP rather than SwiGLU.

The KoLeo regularizer weight is 0.1, applied only to the class tokens of the first global crops, within each GPU without cross-communication.

Ablation-Driven Design: The Incremental Path from iBOT to DINOv2

Table 1 presents a step-by-step ablation starting from an iBOT ViT-L/16 baseline trained on ImageNet-22k and adding modifications until reaching the full DINOv2 recipe. The initial iBOT reproduction achieves 74.5% k-NN and 83.2% linear probe on ImageNet-1k. The modifications are:

+LayerScale, Stochastic Depth (drop rate 0.4): +0.9% k-NN (75.4%), −1.2% linear (82.0%). The linear probe degrades, but the authors state these "have the benefits of increasing the stability of training by avoiding NaN loss values during training." This is the key tradeoff: they are willing to sacrifice some final accuracy for the ability to actually complete training at scale without numerical divergence.
+128k prototypes (up from 65,536): +1.2% k-NN (76.6%), −0.1% linear (81.9%).
+KoLeo regularizer: +2.3% k-NN (78.9%), +0.6% linear (82.5%). The largest single gain for k-NN, consistent with the regularizer's purpose of spreading features for nearest-neighbor search.
+SwiGLU FFN (replacing standard MLP): −0.2% k-NN (78.7%), +0.6% linear (83.1%). A modest tradeoff favoring linear probe performance.
+Patch size 14 (down from 16): +0.2% k-NN (78.9%), +0.4% linear (83.5%). Smaller patches produce more tokens per image, increasing resolution of the feature map.
+Teacher momentum 0.994 (up from the default): +0.5% k-NN (79.4%), +0.1% linear (83.6%).
+Tweak warmup schedules: +1.1% k-NN (80.5%), +0.2% linear (83.8%). The "tweak" is not detailed but likely involves adjusting the length or shape of the learning rate and momentum warmup periods.
+Batch size 3k (up from some lower value): +1.2% k-NN (81.7%), +0.9% linear (84.7%). Larger batches provide more stable gradient estimates and, for the Sinkhorn-Knopp centering, a larger sample for estimating the column normalization.
+Sinkhorn-Knopp centering (replacing moving-average): No change to either metric (81.7% and 84.7%).
+Untying heads = DINOv2: +0.3% k-NN (82.0%), −0.2% linear (84.5%).

The final DINOv2 ViT-L achieves 82.0% k-NN and 84.5% linear probe, compared to iBOT's 74.5% k-NN and 83.2% linear (+7.5% and +1.3% improvements respectively). The k-NN gain is more substantial, reflecting the recipe's optimization toward features that are directly useful for nearest-neighbor tasks without classifier training.

Model Distillation Procedure

When and why distillation is used. The ViT-g model (~1.1B parameters) is trained from scratch using the full recipe. All smaller models—ViT-L (~300M parameters), ViT-B (~86M), and ViT-S (~22M)—are obtained by distilling from the frozen ViT-g rather than training from scratch. The paper shows (Figure 5) that distillation outperforms scratch training on all 12 evaluated benchmarks, with substantial gains on some metrics (e.g., Oxford-M retrieval: 84.4% mAP for distilled ViT-L vs. 71.3% for scratch ViT-L).

How distillation works in the DINOv2 loop. The distillation repurposes the same self-supervised teacher-student framework, but with the teacher now being the frozen pretrained ViT-g (no gradient updates, no EMA) rather than an EMA of the student. The student (the smaller model being trained) receives the same global and local crop views as in standard DINOv2 training, but with key differences:

No masking. The student sees full, unmasked images. Consequently, the iBOT loss is computed on the global crops directly (the student's patch tokens from global crops must match the ViT-g's patch tokens from the same crops), rather than on masked positions.
No stochastic depth. Since the student is smaller and training is more stable (the teacher is fixed), the aggressive 40% drop rate is unnecessary.
Separate EMA of the student. In standard training, the EMA of the student serves as the teacher. In distillation, the teacher is the frozen ViT-g, but the authors still maintain an EMA of the student. This EMA student is what they evaluate and release as the DINOv2 model—not the raw student checkpoint. This is analogous to the common practice of evaluating the EMA model in self-supervised methods (Caron et al., 2021; Grill et al., 2020) because the EMA smooths out step-to-step variance.

Why the student can sometimes outperform the teacher. Figure 5 shows that the distilled ViT-L surpasses the ViT-g teacher on Oxford-M retrieval (82.7% mAP for ViT-L distill, 71.6% for ViT-g), Paris-M retrieval (84.4% vs. 82.7%), and a few classification benchmarks. The paper suggests this is possible because the distillation applies the iBOT loss on global crops, which may provide additional spatial refinement that the ViT-g (trained from scratch with masking) did not receive—effectively, the distillation process itself acts as a form of continued pretraining with a modified loss profile.

Summary of Design Choices and Their Justifications

Retrieval-based curation over uncurated scaling: Uncurated data degrades frozen-feature quality (Table 2: 83.3% on INet-1k vs. 85.8% with LVD-142M, and dramatically worse on fine-grained tasks); retrieval from curated seeds preserves quality while adding diversity.
Self-supervised retrieval over CLIP-based filtering: Avoids dependence on text supervision entirely, making the pipeline applicable to domains without captions, and avoids the chicken-and-egg problem of needing good features to filter data for training features.
Combined DINO + iBOT over either alone: DINO provides strong global features; iBOT adds spatial structure essential for dense tasks (Table 3b: +2.9 mIoU on segmentation from the MIM loss).
Sinkhorn-Knopp centering over moving-average: Simplifies training (no centering momentum hyperparameter) with no accuracy loss, and guarantees uniform prototype usage per batch rather than approximate balancing over time.
KoLeo regularizer over no spreading loss: Improves nearest-neighbor retrieval by >8% mAP (Table 3a) by forcing features to spread uniformly, at negligible cost to other metrics.
Untied DINO/iBOT heads at scale (vs. tied at small scale): Allows class token and patch token projections to specialize independently, avoiding gradient conflict at large model sizes.
High-resolution adaptation rather than full high-resolution training: Achieves performance nearly identical to full high-resolution training (Figure 6) at a small fraction (~1/60th) of the compute, by only adapting positional embeddings and refining features for 10k iterations.
FSDP with float16 communication over DDP: Halves inter-GPU communication volume and enables training models larger than single-GPU memory, while maintaining optimizer precision in float32 for stability.
Sequence packing over separate forwards: Reduces kernel launch overhead and improves GPU utilization by processing all crop views in a single transformer forward, with a block-diagonal attention mask preventing cross-image leakage.
Custom FlashAttention and efficient stochastic depth over standard implementations: The attention optimization is tuned for embedding dimensions that are multiples of 64 per head and 256 total; the stochastic depth shuffling avoids computing dropped blocks entirely, saving ~40% FLOPs at the paper's drop rate.
SwiGLU FFN for scratch training, MLP for distilled models: SwiGLU provides modest accuracy improvements for the large model trained from scratch; for smaller distilled models, standard MLP is sufficient and simpler.
EMA evaluation for distilled models: The EMA smooths training variance and consistently outperforms the raw student checkpoint, following established practice in self-supervised learning.

4. Key Insights and Innovations

Innovation 1: Data Curation, Not Just Data Scale, Is the Critical Enabler for Frozen SSL Features

This paper's most fundamental conceptual move is reframing the self-supervised scaling problem from one of quantity to one of quality and distribution matching. Prior large-scale SSL efforts—SEER (Goyal et al., 2021), SEERv2 (Goyal et al., 2022a), and the scaling studies of Caron et al. (2019) and Goyal et al. (2019)—operated under the implicit assumption that the success of SSL at ImageNet-1k scale would naturally extend to larger datasets if you simply trained longer on more images. This assumption turned out to be wrong, and the field's response was largely to accept that frozen features from scaled SSL are inherently worse than fine-tuned ones, pivoting to fine-tuning as the primary evaluation mode.

DINOv2 challenges this narrative by showing that the degradation is not a fundamental property of SSL at scale but rather a consequence of the distribution of the pretraining data. The evidence is direct and stark (Table 2): training a ViT-g on 142M randomly sampled uncurated web images produces 83.3% ImageNet-1k accuracy, while training on the same number of images from LVD-142M (curated via retrieval from curated seeds) achieves 85.8%. The gap widens dramatically on out-of-domain tasks: 59.4% vs. 73.9% on ImageNet-A, 68.0% vs. 82.3% on iNaturalist 2018. In other words, throwing more uncurated data at the problem actively hurts generalization compared to using less data that is better-matched to the target distribution.

Why this is a conceptual advance rather than an engineering detail. The data curation pipeline itself (Section 3) is an implementation, but the idea it embodies is that SSL pretraining behaves analogously to language model pretraining in a specific, non-obvious way: the pretraining data distribution must reflect the distribution of concepts and visual diversity that downstream tasks require, and naively scaling on web data introduces a distribution shift toward low-quality, repetitive, or narrow content that the SSL objective cannot overcome. This parallels the NLP finding that training language models on unfiltered CommonCrawl produces worse representations than training on a smaller but curated corpus (Wenzek et al., 2020), but it was not obvious that this analogy would hold for vision because images are information-dense in ways that text is not—a single blurry Instagram photo might seem to contain more visual signal than no image at all. The paper demonstrates that, counterintuitively, it is better to omit that image entirely than to include it if it pulls the feature distribution away from the curated reference.

The paper's retrieval-based curation approach is also conceptually novel in its self-containedness. Prior image filtering pipelines either used text metadata (Mahajan et al., 2018; Radford et al., 2021), which defeats the purpose of avoiding text supervision, or used pretrained vision encoders that were themselves trained with supervision (Schuhmann et al., 2021 used CLIP embeddings). The DINOv2 pipeline uses a self-supervised ViT to filter data for training a better self-supervised ViT—a form of bootstrap that keeps the entire process within the SSL paradigm. This is an intellectual contribution to the question of how to scale SSL without importing external supervision signals, and it opens the door to domain-specific curation (medical imaging, remote sensing) where text metadata is unavailable.

Limitations exposed by this framing. The paper does not fully address whether the retrieval pipeline introduces its own biases. The embedding model was pretrained on ImageNet-22k, so the retrieved images are those that look similar to ImageNet-22k concepts. This could create a self-reinforcing loop where the curated data only captures visual concepts already present in ImageNet-22k, limiting true out-of-domain generalization. The paper's finding that performance improves on iNaturalist and Places205 (domains not used for curation) partially mitigates this concern but does not eliminate it—these domains may share sufficient visual similarity with ImageNet-22k concepts that the retrieval still works. A more radical test would be on domains with no ImageNet-22k overlap (e.g., medical histopathology, satellite imagery), where the retrieval might fail to surface useful data.

Innovation 2: Frozen SSL Features Can Match Text-Supervised Features on Dense Prediction Tasks by Preserving Spatial Information That Captions Lose

A persistent narrative in the vision foundation model literature is that text supervision (via CLIP-style contrastive learning) provides a richer training signal than pure image-based SSL because language captures high-level semantic structure that pure pixel-level objectives miss. DINOv2 provides strong evidence that the opposite is true for spatially-structured tasks: text supervision systematically loses the fine-grained geometric and part-level information that SSL preserves, and this loss is not recoverable by scaling the text-supervised model further.

The evidence for this claim is not a single headline number but a consistent pattern across dense prediction benchmarks (Tables 10 and 11). On ADE-20K segmentation, DINOv2 ViT-g with a simple linear probe achieves 49.0 mIoU vs. OpenCLIP ViT-G's 39.3—a ~10-point gap that cannot be explained by architectural differences (both are ViT-based). On NYU depth estimation with the DPT decoder, DINOv2 achieves 0.279 RMSE vs. 0.414 for OpenCLIP, and on KITTI, 2.11 vs. 2.56. These are not marginal improvements; they represent fundamentally different levels of spatial understanding. The qualitative results (Figure 7) make this concrete: OpenCLIP's depth maps show artifacts and disconnected components, while DINOv2's are smooth and capture fine object boundaries.

Why this reframes the SSL vs. text-supervision debate. Prior comparisons between SSL and text-supervised models focused predominantly on ImageNet classification accuracy, where the two paradigms often reach similar performance (Table 4: DINOv2 ViT-g at 86.5% linear probe vs. OpenCLIP ViT-G at 86.2%). This near-parity could be interpreted as "SSL and text supervision are roughly equivalent, with text supervision having the advantage of zero-shot capabilities." The dense prediction results reveal that this interpretation is misleading—the two paradigms are learning qualitatively different representations, and the apparent parity on classification masks substantial SSL superiority on spatial tasks.

The conceptual insight is that captions act as an information bottleneck: a caption like "a living room with a couch and a coffee table" conveys category-level semantics but discards all spatial layout, depth ordering, boundary information, and part-level structure. An SSL model trained to match different views of the same image must, by contrast, learn to represent whatever visual information is consistent across views—which includes object shapes, relative positions, and surface properties. The patch-level iBOT objective further forces the model to encode this information at every spatial location. The result is features that are "geometrically aware" in a way that text-trained features are not, even when both achieve similar object recognition performance.

The significance beyond the raw numbers. This finding provides a principled reason to prefer SSL over text-guided pretraining for applications where spatial reasoning matters—robotics, autonomous driving, 3D reconstruction, medical image analysis—even if text-guided models maintain an edge on some classification benchmarks. It also suggests that the future of general-purpose visual features may involve combining SSL's spatial richness with text-guided semantic structure, rather than choosing between them. The paper does not pursue this combination, but the evidence that the two paradigms capture complementary information lays the groundwork for doing so.

Innovation 3: The KoLeo Regularizer Reveals That Feature Spread Is a First-Class Axis of SSL Quality, Independent of Classification Accuracy

Most SSL method development focuses on objectives and architectures that improve downstream task accuracy. The KoLeo regularizer (introduced by Sablayrolles et al., 2019, for supervised metric learning) had not been adopted in large-scale SSL pretraining prior to this work. DINOv2 demonstrates that feature spread—how uniformly the learned representations fill the embedding space—is a dimension of feature quality that can be optimized independently of classification performance and that matters enormously for retrieval tasks.

The evidence (Table 3a) is striking in its specificity: adding the KoLeo regularizer improves instance retrieval mAP on Oxford-M by more than 8 percentage points (55.6% → 63.9%) while leaving classification accuracy (85.3% → 85.8%) and segmentation mIoU (47.2% → 47.1%) essentially unchanged. This is not a tradeoff pattern typical of hyperparameter tuning, where gains on one metric come at the expense of others. Instead, it reveals that classification and retrieval make different demands on the feature space—classification cares primarily about linear separability of categories, which can be achieved with features clustered in a low-dimensional subspace, while retrieval cares about the metric structure of the full embedding space, where uniform spread maximizes the effective dimensionality and the distance between any two semantically different items.

Why this is conceptually important. The standard SSL evaluation protocol—linear probe or k-NN on ImageNet-1k—is implicitly a classification-centric metric. Methods are optimized to maximize this number, and the resulting feature spaces may be excellent classifiers but poor for nearest-neighbor search. The KoLeo regularizer demonstrates that feature spread can be directly optimized through a simple, computationally cheap loss term (computed per-GPU without cross-communication, with weight 0.1), and that doing so produces features that are simultaneously good classifiers and good retrieval representations.

This has implications for how the field should evaluate SSL methods. If the goal is truly general-purpose features, then optimizing for classification alone is insufficient—the metric structure of the embedding space matters independently. The KoLeo regularizer offers a concrete mechanism for optimizing this structure, but the broader point is that retrieval benchmarks should be first-class evaluation metrics in SSL, not afterthoughts, because they probe a dimension of feature quality (uniform spread, high effective dimensionality) that classification does not measure.

A subtle connection to the centering mechanism. The Sinkhorn-Knopp centering in the DINO/iBOT objectives (which forces uniform prototype usage within each batch) and the KoLeo regularizer (which forces uniform spread of class tokens) are complementary mechanisms operating at different levels: SK centering prevents the teacher's prototype assignments from collapsing, while KoLeo prevents the student's feature vectors from collapsing. Together, they ensure that the representation is uniformly distributed at both the prototype-assignment level and the embedding-space level. This dual anti-collapse strategy is an implicit design principle of DINOv2 that is never stated explicitly but is evident from the component choices.

Innovation 4: Untying DINO and iBOT Projection Heads at Scale Inverts a Published Finding, Demonstrating That Design Choices Transfer Non-Monotonically with Data and Model Size

The original iBOT paper (Zhou et al., 2022a) included an ablation study showing that sharing the projection head between the DINO (image-level) and iBOT (patch-level) objectives improved performance. This finding was treated as a property of the combined objective: the shared head allowed the two tasks to mutually regularize each other. DINOv2 reports the opposite finding at scale (Table 1, last row): untying the heads improves k-NN by 0.3% and slightly decreases linear probe by 0.2%, making the untied variant the chosen configuration (labeled "DINOv2").

This is a small experimental difference but a significant conceptual one: it is a clean example of a scaling-induced reversal of a published best practice. The phenomenon is known in large language model training—techniques that work well at small scale (e.g., certain activation functions, normalization placements) can become harmful at large scale—but it had not been documented for SSL vision training. The DINOv2 finding provides a concrete case study: at ImageNet-1k scale (where iBOT was developed), the regularization benefit of parameter sharing between the class token and patch token projection heads outweighed the representational cost of forcing both to use the same mapping. At LVD-142M scale with ~10× more data and much larger models, this cost-benefit tradeoff flips—the two token types encode sufficiently different information that sharing a projection head creates conflicting gradients that hurt more than the regularization helps.

Why this matters beyond this specific finding. The field's approach to SSL method development has been heavily driven by experiments on ImageNet-1k, with the implicit assumption that techniques proven at this scale will transfer to larger settings. This reversal suggests a more systematic problem: the ImageNet-1k development loop may systematically favor techniques that regularize against overfitting on a small dataset but that become unnecessary or harmful at scale. If this is true, the SSL literature may contain other "best practices" that are artifacts of small-scale experimentation and would not survive scaling. The paper's ablation methodology—starting from a known method (iBOT) and incrementally modifying it while monitoring both performance and training stability at scale—offers a template for how to systematically validate or invalidate such practices.

A note on epistemic caution. The paper reports this reversal as an empirical observation without a mechanistic explanation, and does not ablate the intermediate regime to find exactly when the crossover occurs. This is appropriate—the purpose is to document that the reversal exists, not to fully characterize it. However, it means that a practitioner training at a scale between ImageNet-1k and LVD-142M cannot know from this paper alone whether tied or untied heads would be optimal for their setting. This is a limitation shared by most scaling studies, but it is worth flagging because the paper's presentation might lead a reader to assume untied heads are universally better at any scale larger than ImageNet-1k, which is not established.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation dataset is ImageNet-1k (Russakovsky et al., 2015) validation set for classification; additional benchmarks span 15+ datasets covering classification, instance retrieval, semantic segmentation, depth estimation, and video understanding, as detailed in Table 18. All test and validation splits of evaluation benchmarks are explicitly excluded from pretraining data via the relative deduplication step (Section 3, similarity threshold >0.45), preventing contamination.
Base model(s). The paper trains a family of Vision Transformers (Dosovitskiy et al., 2021) with patch size 14: ViT-S (22M parameters), ViT-B (86M), ViT-L (300M), and ViT-g (1.1B). The ViT-g is trained from scratch; smaller models are distilled from the frozen ViT-g. The architecture choice is motivated by ViTs being the dominant backbone for both SSL and weakly-supervised methods, enabling direct comparison.
Metrics. For classification, the primary metric is Top-1 accuracy (%) with a linear probe on frozen features. For ImageNet variants, additional metrics include Top-1 on ImageNet-V2 (Recht et al., 2019), ImageNet-ReaL (Beyer et al., 2020), ImageNet-A (Hendrycks et al., 2021b), ImageNet-R (Hendrycks et al., 2021a), and ImageNet-Sketch (Wang et al., 2019), plus mean Corruption Error (mCE, lower is better) on ImageNet-C (Hendrycks & Dietterich, 2019). For instance retrieval, mean Average Precision (mAP) is reported. For semantic segmentation, mean Intersection-over-Union (mIoU) is used. For depth estimation, Root Mean Square Error (RMSE, lower is better) is reported.
Baselines. Self-supervised baselines include MAE (ViT-H/14, He et al., 2022), DINO (ViT-B/8 or ViT-S/8 depending on best-performing variant, Caron et al., 2021), SEERv2 (RG10B, Goyal et al., 2022a), MSN (ViT-L/7, Assran et al., 2022), EsViT (Swin-B/W=14, Li et al., 2022a), Mugs (ViT-L/16, Zhou et al., 2022b), and iBOT (ViT-L/16, Zhou et al., 2022a). Weakly-supervised baselines include CLIP (ViT-L/14 and ViT-L/14@336, Radford et al., 2021), SWAG (ViT-H/14, Singh et al., 2022), OpenCLIP (ViT-H/14 and ViT-G/14, Ilharco et al., 2021), and EVA-CLIP (ViT-g/14, Fang et al., 2023). When multiple architectures exist for a method, the best-performing variant on ImageNet-1k is reported.
Generation budget / compute accounting. There is no per-sample generation budget in the generative sense—this is a discriminative pretraining paper. Compute is measured in GPU-hours (22,016 for ViT-g training, Table 14) and model size (parameter count, FLOPs). For evaluation, the frozen-feature protocol means all baselines undergo identical linear probing at evaluation time, and the pretraining compute is the relevant cost axis. The FLOPs comparison in the main scaling plot (Figure 2) uses inference FLOPs of the backbone as the x-axis, enabling a size-versus-accuracy tradeoff analysis.
Cross-validation / statistical protocol. For linear probing on ImageNet-1k, hyperparameters (learning rate, number of output layers, whether to concatenate average-pooled patch tokens with the class token) are selected via grid search as detailed in Appendix B.3, and the highest validation accuracy is reported. The paper does not apply cross-validation for strategy selection (no "compute-optimal policy" as in the scaling laws paper), but does use relative deduplication to ensure no evaluation images leak into pretraining. For depth estimation, three decoder configurations (lin. 1, lin. 4, DPT) are evaluated independently. For the fairness analysis, protocols from Goyal et al. (2022b) are followed.

Main Quantitative Results

ImageNet-1k Classification (Table 4)

Headline numbers. DINOv2 ViT-g/14 achieves 86.5% Top-1 accuracy with a linear probe on ImageNet-1k validation, outperforming the previous best SSL model (iBOT ViT-L/16 at 82.3%) by 4.2 percentage points. Among weakly-supervised models, it surpasses OpenCLIP ViT-G/14 (86.2%) by 0.3 points and EVA-CLIP ViT-g/14 (86.4%) by 0.1 points. On the alternative test sets, DINOv2 ViT-g reaches 89.6% on ImageNet-ReaL (vs. 89.4% for OpenCLIP-G, 89.3% for EVA-CLIP) and 78.4% on ImageNet-V2 (vs. 77.2% for OpenCLIP-G, 77.4% for EVA-CLIP).

Scaling behavior. Across the DINOv2 model family, ImageNet-1k accuracy scales with model size: ViT-S/14 achieves 81.1%, ViT-B/14 achieves 84.5%, ViT-L/14 achieves 86.3%, and ViT-g/14 achieves 86.5%. The distilled ViT-L (86.3%) nearly matches the from-scratch ViT-g (86.5%), validating distillation as an effective strategy for smaller models.

k-NN performance. DINOv2 ViT-g achieves 83.5% k-NN accuracy on ImageNet-1k, matching the ViT-L and substantially exceeding iBOT ViT-L/16 (72.9%). The k-NN metric is noteworthy because it uses no learned classifier—features are used directly for nearest-neighbor lookup—making it a stricter test of feature quality than linear probing.

Domain generalization (Table 6). With linear probes trained on ImageNet-1k and evaluated on out-of-distribution variants, DINOv2 ViT-g achieves 75.9% on ImageNet-A (vs. 41.5% for iBOT, 63.8% for OpenCLIP-G), 78.8% on ImageNet-R (vs. 51.0% for iBOT, 87.8% for OpenCLIP-G), 62.5% on ImageNet-Sketch (vs. 38.5% for iBOT, 66.4% for OpenCLIP-G), and 28.2 mCE on ImageNet-C (vs. 43.9 for iBOT, 45.3 for OpenCLIP-G). DINOv2 dominates on ImageNet-A (+12.1 points over OpenCLIP) and ImageNet-C, but trails OpenCLIP on ImageNet-R (−9.0 points) and ImageNet-Sketch (−3.9 points), suggesting that text supervision may confer robustness advantages on certain types of distribution shift.

Fine-tuning sanity check (Table 5). When the frozen backbone is fine-tuned on ImageNet-1k using the pipeline from Touvron et al. (2022), ViT-g accuracy increases from 86.5% (linear) to 88.5% (fine-tuned at 224×224) and 88.9% (fine-tuned at 448×448). The modest +2.0–2.4 point gain from fine-tuning indicates that the frozen features are already near the performance ceiling of the architecture on this task.

Additional Image Classification Benchmarks (Tables 7 and 8)

Fine-grained classification. On iNaturalist 2018 and 2021, DINOv2 ViT-g achieves 81.6% and 85.7% respectively, compared to OpenCLIP ViT-G's 73.0% and 76.0%—gains of +8.6 and +9.7 points. On Places205, DINOv2 reaches 67.5% vs. OpenCLIP's 69.8% (−2.3 points). This pattern—SSL winning on fine-grained species classification, text supervision winning on scene classification—is consistent with the hypothesis that SSL preserves fine visual details while text supervision captures scene-level semantics from captions.

Twelve transfer benchmarks (Table 8). On the 12-dataset suite from Chen et al. (2020), DINOv2 ViT-g achieves the highest average accuracy (92.1%) among SSL methods and is competitive with OpenCLIP ViT-G (91.9%). Notable SSL gains include Stanford Cars (91.4% vs. 71.8% for iBOT, +19.6 points) and FGVC Aircraft (87.2% vs. 72.4% for iBOT, +14.8 points). OpenCLIP leads on SUN397 (84.0% vs. 78.7%) and Stanford Cars (96.1% vs. 91.4%).

Video classification (Table 7). Despite no video-specific training, DINOv2 ViT-g achieves 78.4% on Kinetics-400 (vs. 78.3% for OpenCLIP), 91.2% on UCF-101 (vs. 90.7%), and 38.3% on Something-Something v2 (vs. 35.8%). The SSv2 result is particularly notable because this dataset requires fine-grained temporal understanding—the model must distinguish actions like "pushing something from left to right" vs. "pushing something from right to left"—and DINOv2's +2.5 point advantage suggests that its spatial features capture information useful for motion discrimination.

Instance Retrieval (Table 9)

Headline numbers. On Oxford-Hard, DINOv2 ViT-g achieves 52.3% mAP, compared to 12.7% for iBOT and 19.7% for OpenCLIP—a +39.6 point improvement over the best SSL baseline and +32.6 points over the best weakly-supervised baseline. On Paris-Hard, DINOv2 reaches 82.6% vs. 47.0% (iBOT) and 60.2% (OpenCLIP). On the Met dataset (artwork retrieval), DINOv2 reaches 73.6% GAP-ACC vs. 54.8% for iBOT and 23.9% for OpenCLIP. On AmsterTime (cross-domain street view matching), DINOv2 achieves 46.7% mAP vs. 26.7% (iBOT) and 24.6% (OpenCLIP).

Scaling behavior within DINOv2. Retrieval performance scales with model size: on Oxford-Hard, ViT-S achieves 43.2%, ViT-B reaches 49.5%, ViT-L reaches 54.0%, and ViT-g achieves 52.3% (the ViT-L matches or slightly exceeds ViT-g on several retrieval metrics, a pattern also seen in the distillation comparison in Figure 5 where the distilled ViT-L sometimes outperforms the ViT-g teacher). The KoLeo regularizer's effect (Table 3a) is responsible for a substantial portion of the retrieval gains: adding KoLeo improves Oxford-M mAP by +8.3 points (55.6% → 63.9%) for the ViT-g at an intermediate training stage.

Significance of the retrieval results. Instance retrieval—finding specific objects or landmarks across viewpoint and lighting changes—requires the model to encode fine-grained, viewpoint-invariant local features that can match the same instance despite substantial appearance variation. The fact that DINOv2 dominates OpenCLIP on this task (+34% mAP on Oxford-Hard) while being competitive or slightly behind on scene classification supports the paper's central narrative: text supervision captures category-level semantics well but loses the detailed spatial and instance-specific information that SSL preserves.

Semantic Segmentation (Table 10)

Headline numbers (linear). On ADE-20K with a single linear layer on frozen patch tokens, DINOv2 ViT-g achieves 49.0 mIoU, compared to 44.6 for iBOT ViT-L/16 and 39.3 for OpenCLIP ViT-G. On Cityscapes, DINOv2 achieves 71.3 mIoU vs. 64.8 (iBOT) and 60.3 (OpenCLIP). On Pascal VOC, DINOv2 reaches 83.0 mIoU vs. 82.3 (iBOT) and 71.4 (OpenCLIP).

Headline numbers (+ms boosted). Adding multi-scale test-time augmentations and concatenating features from the last four layers, DINOv2 ViT-g achieves 53.0 mIoU on ADE-20K (vs. 47.5 for iBOT, 46.0 for OpenCLIP), 81.0 on Cityscapes (vs. 74.5 iBOT, 70.3 OpenCLIP), and 86.2 on Pascal VOC (vs. 84.3 iBOT, 79.2 OpenCLIP). The Pascal VOC result (86.2) approaches the absolute state of the art of 89.0 (from Chen et al., 2018) despite using a frozen backbone with only a boosted linear classifier.

Frozen backbone in a SOTA pipeline. When plugging the frozen DINOv2 ViT-g into a ViT-Adapter (Chen et al., 2023b) with a Mask2Former head (Cheng et al., 2022)—keeping 66% of weights frozen—the model reaches 60.2 mIoU on ADE-20K, compared to the fully fine-tuned state of the art of 62.9 (Wang et al., 2022). This demonstrates that DINOv2 features are rich enough to serve as the backbone for a near-SOTA segmentation system without requiring the backbone to be trained on the segmentation task.

The role of the iBOT loss. Table 3b shows that removing the iBOT masked image modeling term drops ADE-20K linear mIoU from 47.1 to 44.2 (−2.9 points), confirming that the patch-level objective is critical for spatial feature quality.

Monocular Depth Estimation (Table 11)

Headline numbers. On NYUd with the DPT decoder, DINOv2 ViT-g achieves 0.279 RMSE, compared to 0.358 for iBOT ViT-L/16, 0.414 for OpenCLIP ViT-G, and 0.415 for MAE ViT-H/14. On KITTI with DPT, DINOv2 reaches 2.11 RMSE vs. 2.55 (iBOT) and 2.56 (OpenCLIP). On zero-shot transfer from NYUd to SUN RGB-D with DPT, DINOv2 achieves 0.338 RMSE vs. 0.426 (iBOT) and 0.408 (OpenCLIP).

Decoder scaling. The paper evaluates three decoder complexities: lin. 1 (single linear layer on last-layer features, upsampled 4×), lin. 4 (linear layer on concatenated features from layers {3, 6, 9, 12} for ViT-S/B or equivalent spacing for larger models), and DPT (the full Dense Prediction Transformer decoder of Ranftl et al., 2021). Across all architectures, the DPT decoder consistently improves performance: ViT-g goes from 0.344 (lin. 1) to 0.298 (lin. 4) to 0.279 (DPT) on NYUd. The fact that DINOv2 features benefit from deeper decoders (unlike some SSL features that saturate with a linear layer) indicates that the spatial information is hierarchically organized across transformer layers.

Comparison to specialized methods. The DINOv2 ViT-g with DPT decoder achieves performance close to the specialized depth estimation method of Li et al. (2022b), which reported 0.330 on NYUd and 2.10 on KITTI (numbers cited by the paper as state of the art at the time of writing). DINOv2's 0.279 on NYUd actually improves over this reference, though the architectures and training protocols differ.

Out-of-domain generalization. The zero-shot transfer from NYUd (indoor) to SUN RGB-D (indoor + some outdoor) is a strong test of feature robustness. DINOv2's RMSE of 0.338 with DPT (vs. 0.408 for OpenCLIP) demonstrates that the depth signal learned from indoor scenes transfers better than text-supervised features, consistent with the claim that SSL captures generic geometric cues rather than dataset-specific patterns. The qualitative results in Figure 7 show DINOv2 producing smoother depth maps with better object boundary delineation compared to OpenCLIP.

Model Scale versus Data Scale (Figure 4)

The paper investigates the interaction between model size and pretraining dataset by training ViT-L, ViT-H, and ViT-g on ImageNet-22k (14M images) and LVD-142M (142M images). The key findings:

On ImageNet-1k, the benefit of LVD-142M over ImageNet-22k is modest for small models but grows with scale: ViT-g on LVD-142M reaches ~86.5% vs. ~85.5% on ImageNet-22k.
On out-of-domain benchmarks (ImageNet-V2, ImageNet-Sketch, Food101, Cars, AmsterTime, Oxford-H), the gap widens substantially for larger models. For instance, on Cars, ViT-g on LVD-142M achieves ~90% vs. ~80% on ImageNet-22k. On AmsterTime, the difference is ~40% vs. ~35%.
The pattern demonstrates a scale interaction: larger models benefit more from larger, more diverse pretraining data, while smaller models saturate or benefit less. This is consistent with the intuition that larger models have the capacity to absorb and leverage greater data diversity without overfitting to the dominant modes.

Knowledge Distillation Effectiveness (Figure 5)

Comparing a ViT-L trained from scratch on LVD-142M versus one distilled from the frozen ViT-g:

The distilled ViT-L outperforms the from-scratch ViT-L on all 12 benchmarks evaluated, ranging from classification (ImageNet-1k: 86.3% vs. 84.5%) to segmentation (73.3% vs. 72.2% on the aggregated segmentation metric) to retrieval (76.3% vs. 71.3% on the aggregated retrieval metric).
On several benchmarks, the distilled ViT-L matches or exceeds the ViT-g teacher: Oxford-M retrieval (82.7% vs. 71.6% for ViT-g scratch), Paris-M retrieval (84.4% vs. 82.7%), and Caltech101 classification (numbers not directly compared but the pattern is visible in Figure 5a).
The distillation benefit is not uniform: it is largest on retrieval tasks and fine-grained classification, and smaller on scene classification (Places205: 67.3% vs. 67.5% for the teacher).

The paper attributes the occasional student-surpasses-teacher phenomenon to the distillation procedure applying the iBOT loss on global crops rather than masked positions, which may provide additional spatial refinement during distillation.

Resolution Adaptation (Figure 6)

The resolution ablation (conducted on a smaller setup: ViT-L/16 on ImageNet-1k) shows:

Training at high resolution (416×416) for the full duration provides the best performance at all evaluation resolutions on both ImageNet-1k and ADE-20K, but costs ~3× more compute than training at 224×224.
Training at 224×224 followed by 10k iterations at 416×416 (the adaptation approach used in DINOv2) performs nearly identically to full high-resolution training, within ~1% accuracy on ImageNet-1k and ~1 mIoU on ADE-20K across evaluated resolutions from 224 to 768.
Training only at 224×224 and evaluating at higher resolutions (via positional embedding interpolation) degrades noticeably, confirming that the short adaptation phase provides genuine representational improvement rather than simply being an artifact of interpolation.

Fairness and Bias Analysis (Tables 12 and 13)

Geographical fairness (Table 12). On the Dollar Street dataset, DINOv2 ViT-g achieves 67.4% accuracy on low-income households vs. 90.5% on high-income households—a 23.1-point gap. Across regions, performance ranges from 74.0% (Africa) to 89.7% (Europe)—a 15.7-point gap. Compared to SEERv2 (which was specifically designed for geographical diversity), DINOv2 is slightly more fair (the low-high income gap for SEERv2 is 86.6% − 59.7% = 26.9 points vs. DINOv2's 23.1 points), but significant biases remain. The authors note their model "is still biased toward Western countries."

Label association fairness (Table 13). On the Casual Conversations dataset, DINOv2 does not predict Non-Human labels for images of people (0.0% across all gender/skintone/age groups), and predicts Crime labels in only two instances (0.2% maximum across any group). The Possibly-Human category (containing benign classes like Scarf, Glasses, Beard) is triggered frequently, especially for males due to the Beard class prevalence. The authors conclude that "no clear pattern indicates a bias against a particular group" but acknowledge that a more thorough evaluation is warranted.

Qualitative Results

PCA of patch features (Figures 1 and 9). When PCA is applied to the patch features across multiple images of the same category (e.g., different birds, different cars), the first component cleanly separates foreground from background. Thresholding this component produces an unsupervised foreground/background segmentation. The remaining components correspond to semantic parts (wings, wheels, eyes) that match across different instances, poses, and even object categories (e.g., airplane wings match bird wings in Figure 10). This is an emergent property—the model was never trained to segment objects or parse parts.

Patch matching (Figure 10). By detecting foreground patches via the first PCA component threshold, then solving an assignment problem between patch features from two images, the paper shows semantically meaningful correspondences: the wing of a bird matches the wing of an airplane, the eye of one animal matches the eye of another, and these correspondences are robust to style changes (photo vs. drawing) and large pose variations (elephants in different orientations).

Out-of-distribution depth and segmentation (Figure 8). Linear classifiers for depth and segmentation, trained on standard datasets and applied to out-of-distribution images (animals, paintings), produce plausible outputs. The depth maps correctly order objects in depth, and segmentation masks respect object boundaries, despite the domain mismatch.

Ablation Studies and Robustness Checks

Training recipe incremental ablation (Table 1): Starting from an iBOT ViT-L/16 baseline on ImageNet-22k, each modification is added sequentially. The baseline iBOT achieves 74.5% k-NN and 83.2% linear probe. LayerScale and stochastic depth (drop rate 0.4) increase k-NN to 75.4% (+0.9) but decrease linear to 82.0% (−1.2)—the authors accept this tradeoff for training stability. Increasing prototypes to 128k improves k-NN to 76.6% (+1.2). KoLeo provides the largest single k-NN gain: +2.3 points to 78.9%. SwiGLU FFN, patch size 14, and teacher momentum 0.994 each provide modest gains. Enlarging batch size to 3k provides the second-largest gain: +1.2 k-NN and +0.9 linear. Sinkhorn-Knopp centering provides no change (81.7% and 84.7% for both). Untying heads yields the final DINOv2 configuration at 82.0% k-NN and 84.5% linear. The cumulative improvement over iBOT is +7.5% k-NN and +1.3% linear.
Pretraining data source (Table 2): Training ViT-g on uncurated data (142M random web images) yields 83.3% on ImageNet-1k, compared to 85.8% on LVD-142M—a 2.5-point gap. The gap is larger on fine-grained benchmarks: 59.4% vs. 73.9% on ImageNet-A, 68.0% vs. 82.3% on iNaturalist 2018, and 76.4% vs. 86.4% on iNaturalist 2021. Training on ImageNet-22k alone achieves 85.9% on ImageNet-1k (comparable to LVD-142M's 85.8%) but underperforms on all other benchmarks (e.g., 73.5% vs. 73.9% on ImageNet-A, 62.5% vs. 64.6% on Oxford-M retrieval, 81.1% vs. 82.3% on iNaturalist 2018). Removing ImageNet-1k synsets from ImageNet-22k ("INet-22k \ INet-1k") yields 85.3% on ImageNet-1k—a 0.6-point drop from full ImageNet-22k, suggesting the model partially compensates for the missing categories through the remaining data.
KoLeo regularizer effect (Table 3a): Removing KoLeo drops Oxford-M retrieval mAP from 63.9% to 55.6% (−8.3 points) while leaving ImageNet-1k classification (85.8% vs. 85.3%, −0.5) and ADE-20K segmentation (47.1% vs. 47.2%, +0.1) essentially unchanged. This demonstrates that feature spread is specifically beneficial for nearest-neighbor tasks and largely orthogonal to classification and segmentation performance.
iBOT masked image modeling loss (Table 3b): Removing the iBOT loss term drops ADE-20K segmentation mIoU from 47.1 to 44.2 (−2.9 points) while having mixed effects on classification (ImageNet-1k: 85.8% vs. 85.3%, −0.5; ImageNet-A: 72.8% vs. 72.0%, −0.8) and retrieval (Oxford-M: 63.9% vs. 64.3%, +0.4). This confirms the patch-level objective is critical for dense prediction tasks.
Knowledge distillation versus training from scratch (Figure 5): This is the main distillation ablation. The distilled ViT-L outperforms the from-scratch ViT-L on every benchmark, with the largest gains on retrieval tasks (Oxford-H: ~84% vs. ~72% mAP; Paris-H: ~84% vs. ~78%) and fine-grained classification (Cars: ~90% vs. ~82%). The distilled ViT-L nearly matches the ViT-g teacher across the board, exceeding it on several retrieval metrics.
Resolution adaptation versus full high-resolution training (Figure 6): Training a ViT-L/16 on ImageNet-1k at 416×416 for the full duration achieves the highest accuracy at all evaluation resolutions but costs ~3× more than 224×224 training. Training at 224→416 (short adaptation) matches full 416 training within ~1% accuracy and ~1 mIoU across all tested evaluation resolutions, validating the adaptation strategy.
Architecture variant for ViT-g: The ViT-g uses an embedding dimension of 1536 with 24 heads (64 dim/head), deviating from Zhai et al. (2022)'s 1408 dim with 16 heads (88 dim/head). The change was made because FlashAttention is most efficient when the per-head dimension is a multiple of 64 and the full embedding dimension is a multiple of 256. The paper states their "experiments did not show significant differences in final accuracy."
Sinkhorn-Knopp centering versus moving-average centering (Table 1): The switch from the original DINO/iBOT moving-average centering to Sinkhorn-Knopp batch normalization produces no change in either k-NN (81.7% for both) or linear probe (84.7% for both). The benefit is in simplifying the training recipe by removing a momentum hyperparameter, not in final accuracy.

Critical Assessment

Claim 1: "DINOv2 closes the performance gap with weakly-supervised alternatives across a wide range of benchmarks without the need for finetuning."

The evidence supports this claim with important qualifications. On ImageNet-1k classification, DINOv2 ViT-g (86.5%) marginally surpasses OpenCLIP ViT-G (86.2%) and EVA-CLIP ViT-g (86.4%) (Table 4). On instance retrieval, the gap is enormous in DINOv2's favor (+34% mAP on Oxford-Hard, Table 9). On depth estimation, DINOv2 dominates across all decoders and datasets (Table 11). On fine-grained classification, DINOv2 leads substantially on iNaturalist (+8–10 points) and Cars (+19 points over iBOT, Table 8).

However, OpenCLIP maintains a lead on scene classification (Places205: 69.8% vs. 67.5%, Table 7) and on two domain generalization benchmarks (ImageNet-R: 87.8% vs. 78.8%; ImageNet-Sketch: 66.4% vs. 62.5%, Table 6). On video classification, the results are mixed: DINOv2 leads on SSv2 (+2.5 points), is roughly tied on Kinetics-400, and slightly trails on UCF-101 (Table 7). So the claim holds for a majority of benchmarks, but text supervision retains clear advantages on scene-level understanding and certain robustness evaluations. The paper's own narrative acknowledges this implicitly by not claiming universal superiority, but the headline framing of "closing the gap" could be misinterpreted as meaning DINOv2 matches or exceeds weakly-supervised methods everywhere, which the data does not support.

Claim 2: "Data curation, not just data scale, is the critical enabler for frozen SSL features."

This claim is strongly supported by Table 2: training on curated LVD-142M (85.8% ImageNet-1k, 73.9% ImageNet-A) dramatically outperforms training on the same number of uncurated images (83.3% and 59.4%). The gap is largest on fine-grained and out-of-domain benchmarks, precisely where data quality matters most. The ablation comparing ImageNet-22k to LVD-142M further supports the claim: ImageNet-22k alone matches LVD-142M on ImageNet-1k (85.9% vs. 85.8%) but underperforms on every non-ImageNet benchmark, showing that the curation pipeline successfully adds diversity that translates to broader transfer.

A weakness in the evidence: the ablation compares LVD-142M to a single uncurated baseline (random sampling of 142M web images). It does not explore intermediate curation strategies—for example, what if they had deduplicated the uncurated data but not performed retrieval (keeping only unique images)? Or what if they had performed cluster-based sampling without the curated seed (simply picking diverse images from the clusters)? These ablations would help disentangle the contributions of deduplication, diversity sampling, and curated-seed-based retrieval. Without them, we know that LVD-142M works better than raw uncurated data, but not exactly which curation step is responsible for how much of the gain.

Additionally, the retrieval pipeline uses a self-supervised ViT-H/16 pretrained on ImageNet-22k as the embedding model. The paper argues this keeps the pipeline self-contained within SSL, but it also means the curation is guided by an ImageNet-22k-trained model, potentially limiting the diversity of retrieved images to concepts proximal to ImageNet-22k. The finding that performance improves on iNaturalist and Places205 (domains not used for curation) partially mitigates this concern, but does not eliminate it. A stronger demonstration would train the embedding model on a non-ImageNet seed and show that the resulting dataset still produces good ImageNet features, or would ablate the effect of the embedding model's training data on the final LVD-142M quality.

Claim 3: "SSL features outperform text-supervised features on dense prediction tasks by preserving spatial information that captions lose."

This claim is strongly supported by the dense prediction results. On ADE-20K segmentation, DINOv2 leads OpenCLIP by +9.7 mIoU with a linear probe and +7.0 mIoU with multi-scale boosting (Table 10). On depth estimation, DINOv2 leads by 0.135 RMSE on NYUd-DPT and 0.45 RMSE on KITTI-DPT (Table 11). The qualitative results (Figure 7) provide visual evidence: OpenCLIP depth maps show artifacts and disconnected components; DINOv2 maps are smooth and capture fine boundaries.

A missing experiment that would strengthen this claim: the paper does not evaluate OpenCLIP with the same decoder configurations as DINOv2 for depth estimation. The DPT decoder results are reported for OpenCLIP (Table 11 includes "OpenCLIP ViT-G/14" with DPT: 0.414 on NYUd), but it is unclear whether the DPT decoder architecture was tuned for OpenCLIP features or simply applied with default settings. If DINOv2 features are inherently better suited to linear decoding, the gap with DPT might partly reflect suboptimal decoder design for OpenCLIP rather than fundamentally richer features.

Another concern: the dense prediction evaluations use linear classifiers or lightweight decoders. This protocol favors features where the relevant information is linearly accessible. It is possible—though not tested—that OpenCLIP features encode spatial information in a non-linear way that a more expressive decoder could extract, narrowing or reversing the gap. The paper's choice to focus on linear probing is consistent with its "off-the-shelf features" thesis, but it conflates "features contain spatial information" with "features contain linearly decodable spatial information." A more complete demonstration would test whether non-linear decoders (e.g., a small MLP or a lightweight convolutional head) recover spatial performance from OpenCLIP features.

Claim 4: "The KoLeo regularizer improves feature spread and retrieval performance without harming other metrics."

This is one of the cleanest findings in the paper, supported by Table 3a: +8.3 mAP on retrieval with negligible impact on classification and segmentation. The specificity of the effect—retrieval benefits, other tasks unchanged—is what makes it credible. If KoLeo improved everything uniformly, it would look like a generic regularizer; the fact that it selectively improves retrieval aligns with the theoretical motivation (entropy maximization → better nearest-neighbor structure).

A limitation: the ablation is run on a model trained for fewer iterations than the final DINOv2 ("each model is trained on the same number of iterations, that is smaller than in our final run"). It is possible that the KoLeo benefit diminishes or changes at full training length. The final models all include KoLeo, so we cannot verify from the paper whether the retrieval advantage persists at convergence.

Claim 5: "Distillation from ViT-g produces smaller models that outperform training from scratch."

Supported by Figure 5, which shows the distilled ViT-L outperforming the from-scratch ViT-L on all 12 benchmarks. The effect is substantial on retrieval (+~12 mAP on Oxford-H) and meaningful on classification (+1.8% on ImageNet-1k). However, the paper does not ablate why distillation helps—is it the stronger teacher targets, the absence of masking during distillation, the continued training with the iBOT loss on global crops, or some combination? The fact that the distilled student sometimes surpasses the teacher (Oxford-M retrieval) suggests that the distillation process itself, not just the teacher's quality, contributes to the improvement, but the mechanism is not investigated.

Additionally, the distillation comparison is only shown for ViT-L. It is reasonable to assume ViT-B and ViT-S also benefit (they are all distilled in the final release), but without explicit from-scratch baselines for those models, we cannot confirm the distillation benefit scales down to smaller architectures.

Overall weaknesses in the experimental design:

Single model family (ViT). All experiments use Vision Transformers. Convolutional architectures (ConvNeXt, RegNet) are evaluated as baselines (SEERv2 uses RegNet), but DINOv2 itself is ViT-only. The paper cannot claim that SSL in general matches text supervision—only that ViT-based SSL with this specific recipe does. Whether the findings transfer to CNNs or hybrid architectures is unknown.
No confidence intervals or error bars. None of the tables report standard deviations, confidence intervals, or results over multiple random seeds. For the ImageNet-1k linear probe (12,500 iterations of SGD), the variance across seeds is likely small, but for smaller benchmarks (e.g., the retrieval datasets with hundreds of queries), variance could be substantial. The omission makes it difficult to assess whether differences of 0.1–0.5% are statistically meaningful.
The distillation pipeline uses the same data as pretraining. The ViT-g teacher was trained on LVD-142M, and the student is distilled on LVD-142M. This is a form of self-distillation (same data distribution) rather than cross-dataset distillation. It works well, but it means the student may be memorizing dataset-specific patterns from the teacher rather than learning generalizable features. Evaluating the distilled models on benchmarks far from LVD-142M's distribution would test this, but such an evaluation is not systematically performed.
High-resolution adaptation is only ablated on ImageNet-1k (Figure 6), not LVD-142M. The resolution experiment uses a ViT-L/16 trained on ImageNet-1k, a much smaller setup than the final DINOv2 training. Whether the 10k-iteration adaptation provides the same near-optimal behavior at the LVD-142M scale with ViT-g is assumed but not verified. The computation cost of running this ablation at scale is clearly prohibitive, but the gap between the ablation setting and the final setting should be acknowledged.
The LVD-142M dataset is not released. The paper states that they release "all the models and the code to retrain DINOv2 on any data," but the curated dataset itself is not made available. This means that exact reproduction of the paper's main results requires reimplementing the entire data pipeline (web crawling, deduplication, retrieval), which is a substantial barrier. The reproducibility of the core claim—that LVD-142M-trained features achieve 86.5% on ImageNet-1k—depends on access to comparable data, which is not guaranteed.
Fairness evaluation is limited. The geographic fairness analysis (Table 12) uses Dollar Street, which contains only 289 households across 54 countries—a small sample for assessing global fairness. The label association analysis (Table 13) uses the Casual Conversations dataset with 2,955 images and a trained classifier on 619 ImageNet-22k classes; the authors modify the protocol of Goyal et al. (2022b) to keep the backbone frozen, so the fairness results are specific to frozen features rather than the model as typically deployed. The analysis finds no clear bias in harmful label associations, but the scope is narrow (binary: harmful vs. not harmful, on a limited set of metadata categories).

6. Limitations and Trade-offs

Assumption: Difficulty Can Be Estimated Cheaply Enough to Be Practical

The entire compute-optimal framework rests on the ability to estimate prompt difficulty before deciding how to allocate the inference budget. The paper's method for doing so — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — is extraordinarily expensive. At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

This is a significant gap. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. The paper suggests future work on training models to predict difficulty directly from the question text, but no such model is developed or evaluated. Until this gap is closed, the 4× figure should be understood as an upper bound on achievable efficiency rather than a realized deployment gain.

Single Benchmark, Single Model Family

All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. Several aspects of the findings could be model-specific:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties or different error patterns might exhibit different difficulty-dependent scaling curves.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families.
The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning. It is unclear whether the difficulty-dependent patterns (beam search hurting easy problems, revisions helping easy problems) generalize to other reasoning domains (code generation, logical reasoning, scientific QA) or to tasks requiring factual knowledge rather than inference.

The test set of 500 questions, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a small sample, and the selected strategies may not be robust. The paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess whether the observed gains are statistically reliable at this sample size.

The 14× Larger Model Baseline Is Not Compute-Optimal

The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The authors acknowledge that this departs from compute-optimal pretraining (Hoffmann et al., 2022), where both data and parameters are scaled equally:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

This matters because a Chinchilla-optimal model trained with 14× more total FLOPs would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it needs to be. The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at R ≪ 1) may shrink or reverse against a properly compute-optimal larger model. Additionally, the 14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search. Giving the larger model even a modest test-time compute budget (say, best-of-8) would create a much stronger baseline that is never tested.

Verifier Over-Optimization Is a Hard Ceiling, Not a Solved Problem

The paper documents verifier over-optimization as a central limiting factor: beam search degrades easy-problem performance at high budgets (Figure 3, right), lookahead search — the strongest optimizer — paradoxically performs worst overall (Figure 3, left), and qualitative examples show degenerate outputs (repetitive steps, overly short solutions; Appendix M, Figures 29, etc.). The compute-optimal policy mitigates this by routing easy problems away from aggressive search, but it does not solve the underlying problem. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling — the beam search curves in Figure 3 flatten and sometimes decline well before the budget is exhausted.

This means the compute-optimal approach is fundamentally bounded by verifier quality. Improving the PRM (e.g., through better training data, adversarial robustness, or ensemble methods) would likely shift the difficulty thresholds and change the optimal policy. The current results are therefore specific to the verifier quality achievable with the Monte Carlo rollout training procedure described in Appendix D. The paper does not explore how verifier improvements would alter the scaling landscape.

Hard Problems Remain Essentially Unsolved

Across all methods — search, revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%.

This is a fundamental limitation: test-time compute can amplify existing capability but cannot create it. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help — there are no correct solutions in the proposal distribution to find or refine. The paper is candid about this (Section 7 takeaway box), but it means the approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, pretraining remains the only viable path.

Revisions and Search Are Studied Independently, Not Combined

The paper studies two complementary axes — PRM search and iterative revisions — but never combines them. Section 8 explicitly acknowledges this:

"we did not experiment with PRM tree-search techniques in combination with revisions"

This is a significant gap because the two mechanisms have complementary strengths: revisions improve the proposal distribution (generating better candidates), while PRM search improves candidate selection (finding the best among generated candidates). Applying beam search to revision model outputs — or using the PRM to guide which revisions to pursue — could yield gains beyond either method alone. The current results therefore represent a lower bound on what a fully integrated system could achieve. The paper's compute-optimal policy currently selects between search and revisions per difficulty bin, but the truly optimal strategy may involve deploying both simultaneously on the same problem, a configuration space the paper does not explore.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes what the computer vision community should expect from self-supervised learning. Before DINOv2, the dominant narrative was bifurcated: SSL produced features with intriguing emergent properties (object segmentation, part correspondence) at the ImageNet-1k scale, but those properties degraded or vanished when training was scaled to larger, uncurated datasets—pushing the field to accept that frozen SSL features could not compete with text-supervised alternatives for general-purpose use, and that fine-tuning was necessary. The strongest open-source general-purpose visual features came from CLIP-style text-guided pretraining, and SSL was largely viewed as a stepping stone toward supervised fine-tuning (as with MAE) rather than an endpoint in itself.

DINOv2 breaks this narrative by demonstrating that the degradation at scale was not a property of SSL as a paradigm but of the data being scaled. The paper's data curation pipeline—self-supervised retrieval from curated seeds into a massive uncurated pool—shows that frozen SSL features can not only survive scaling but thrive, matching or surpassing OpenCLIP on a majority of benchmarks spanning classification, retrieval, segmentation, and depth estimation. This shifts the conversation from "can SSL scale?" (a question prior work answered with "yes, if you fine-tune") to "how should we curate data to make SSL scale effectively for frozen features?" (a question that opens a rich design space).

The shift in emphasis from algorithmic novelty to data engineering is itself a conceptual contribution with parallels to the NLP foundation model trajectory. Just as the NLP community discovered that careful data curation (CCNet, C4, the Pile) was as important as architectural innovations for producing useful language models, DINOv2 argues that the vision community should invest seriously in data pipeline design. The retrieval-based approach—using a self-supervised model to augment curated datasets with visually similar web images—provides a concrete template that is self-contained within the image domain, requiring no text metadata, no human annotation, and no pretrained text-guided encoders.

The paper also reconciles a tension in the SSL literature that had been observed but not explained: why some SSL properties (like DINO's emergent segmentation) appeared at ImageNet-1k scale but not in scaled-up training on web data. The answer, per DINOv2, is that those properties depend on the pretraining data containing clean, diverse, well-composed images where object boundaries and part structure are visually salient—properties present in curated datasets like ImageNet-22k but diluted in unfiltered web crawls dominated by memes, screenshots, product photos, and other low-information images. By curating the data to resemble high-quality reference datasets, DINOv2 recovers and amplifies these emergent properties at scale (Figures 1, 9, 10).

Perhaps most consequentially, the paper establishes that spatial understanding is where SSL systematically outperforms text supervision. The gap on dense prediction tasks is not marginal—it is large (+10 mIoU on ADE-20K segmentation, 0.135 RMSE on NYU depth estimation, Table 11) and consistent across architectures and decoders. This finding has been corroborated by subsequent work but was, at the time, a genuinely surprising result that inverted the common assumption that language provides a richer training signal. The mechanistic explanation—that captions act as an information bottleneck, discarding the spatial and geometric details that view-consistency objectives force the model to encode—is now widely accepted, but DINOv2 provided the first comprehensive empirical demonstration. This makes SSL the natural starting point for any vision system where spatial reasoning matters (robotics, autonomous driving, 3D reconstruction, medical imaging), even if text-supervised models retain advantages on certain semantic classification tasks.

The paper also demonstrates that feature spread is a first-class axis of representation quality, separable from classification accuracy. The KoLeo regularizer (Section 4, Table 3a) improves retrieval mAP by 8+ points while leaving classification and segmentation unchanged—a dissociation that had not been cleanly demonstrated in large-scale SSL training. This suggests that the standard evaluation protocol for SSL (linear probe or k-NN on ImageNet-1k) is insufficient: it measures only one dimension of feature quality, and methods optimized for it may produce features that are suboptimal for retrieval and nearest-neighbor tasks. The implication is that retrieval benchmarks should be elevated to first-class evaluation metrics alongside classification, and that regularization strategies targeting feature spread (of which KoLeo is one instance) are an underexplored design dimension.

A secondary but important impact is lowering the barrier to strong frozen visual features. The paper releases models at four sizes (ViT-S/B/L/g) all distilled from the largest model, meaning a practitioner can download a ViT-B that runs on a single GPU and get features competitive with much larger text-supervised models. The distillation finding (Figure 5)—that a ViT-L distilled from ViT-g outperforms the same architecture trained from scratch on all 12 benchmarks—is a practical result that makes the strongest features accessible without requiring the compute to train a billion-parameter model from scratch.

The landscape change is perhaps best characterized as an inference-time analog of the shift from "bigger models trained on more data" to "better data for training models." Just as the NLP community moved from "just scale up GPT" to carefully curating pretraining corpora, DINOv2 moves vision SSL from "train on more uncurated images" to "retrieve visually similar images from curated seeds." The work does not render text-supervised pretraining obsolete—OpenCLIP retains advantages on scene classification and some robustness benchmarks—but it establishes SSL as a genuinely competitive paradigm for general-purpose frozen features, ending the implicit assumption that text guidance is necessary for strong out-of-the-box vision representations.

Follow-Up Research This Work Enables

Cheap difficulty estimation for data curation—predicting which uncurated images will improve downstream features without running the full retrieval pipeline. The LVD-142M construction (Section 3) uses a self-supervised ViT-H/16 to embed all 1.2B uncurated images and compute nearest neighbors to curated seeds. This is computationally expensive (20 nodes × 8 V100 GPUs for two days) and requires storing embeddings for the entire uncurated pool. A natural follow-up is to train a lightweight scoring model that predicts, directly from an image, how "useful" it would be as a pretraining example—essentially, a difficulty estimator for data curation analogous to what the compute-optimal scaling work does for inference allocation. The training signal would come from the retrieval pipeline itself: images that are nearest neighbors of curated seeds (and thus included in LVD-142M) are positive examples; images that are far from all curated seeds are negative. A strong result would show that a small ResNet-50 trained as a binary classifier on this signal can filter web-scale data to produce a dataset that, when used for DINOv2 pretraining, yields features within 1% of the full retrieval pipeline's accuracy on ImageNet-1k and downstream benchmarks. This would make the curation approach practical for domains where running the full pipeline is prohibitively expensive.

Can SSL features be improved by actively selecting pretraining data based on feature spread, not just visual similarity to a curated seed? The KoLeo regularizer (Section 4) shows that feature spread is a meaningful axis of quality, but it operates during training by pushing features apart within each batch. An alternative approach is to curate the pretraining data before training to maximize the diversity of the resulting feature space. A concrete experiment: cluster the uncurated image pool using the same self-supervised embeddings used in the DINOv2 retrieval pipeline, then construct pretraining datasets by sampling images to maximize either (a) uniform coverage of clusters (maximizing concept diversity) or (b) coverage weighted by the KoLeo regularizer's per-image difficulty (favoring images that are "hard to spread" in feature space). Train DINOv2 ViT-L on each dataset and evaluate on the paper's full benchmark suite. The hypothesis is that diversity-maximizing curation provides gains on fine-grained classification and retrieval beyond what retrieval from curated seeds alone achieves, because curated seeds may under-sample rare but visually informative concepts. A null result (diversity sampling doesn't help beyond retrieval) would be equally informative, suggesting that visual similarity to high-quality references is the dominant factor and concept coverage is secondary.

Reconciling SSL and text supervision by training a combined model that inherits the spatial strengths of DINOv2 and the semantic robustness of CLIP. The paper's results reveal a clear complementarity: DINOv2 dominates on dense prediction and fine-grained tasks (Tables 10, 11, 8), while OpenCLIP leads on scene classification (Places205, Table 7) and certain domain generalization benchmarks (ImageNet-R, ImageNet-Sketch, Table 6). A direct follow-up is to train a model with a joint objective: DINO + iBOT losses on images (providing spatial and instance-level signal) combined with a CLIP-style contrastive loss on aligned image-text pairs (providing semantic and linguistic signal). The architecture would share a single ViT backbone with separate projection heads for each objective, or use a multi-encoder design where the SSL and text-guided pathways interact through cross-attention. The key evaluation would compare the joint model against both DINOv2 and OpenCLIP on the full benchmark suite from the paper, with the prediction that it matches or exceeds the best of each on all tasks—DINOv2-level segmentation and depth, OpenCLIP-level scene classification and robustness, and possibly emergent zero-shot capabilities from the text pathway. The paper's own suggestion (Section 10) to "leverage this ability to train a language-enabled AI system that can process visual features as if they were word tokens" points toward exactly this direction.

Stress-testing DINOv2 on domains with no visual overlap with ImageNet-22k to probe the limits of the self-supervised retrieval pipeline. The paper's retrieval pipeline uses a ViT-H/16 pretrained on ImageNet-22k to embed images and measure similarity. This creates a potential self-reinforcing bias: the curated dataset ends up containing web images that look like ImageNet-22k concepts, and models trained on it may fail on domains that are visually dissimilar to ImageNet-22k. The paper partially addresses this by showing gains on iNaturalist and Places205 (domains not used for curation), but these are still natural-image datasets with significant overlap with ImageNet categories. A stress test would train the embedding model on a domain-specific curated dataset (e.g., a collection of satellite imagery, histopathology slides, or infrared camera footage), run the retrieval pipeline to build a domain-specific LVD-142M, train a DINOv2 model on it, and evaluate on downstream tasks in that domain. If the retrieval pipeline works—producing features that outperform ImageNet-trained SSL baselines on domain-specific tasks—it validates the approach as genuinely self-contained. If it fails (because the domain-specific embedding model is too weak to find good neighbors in the uncurated pool), it reveals the pipeline's dependence on a strong initial embedding model, which may itself require curated data that doesn't exist for niche domains.

Extending the resolution adaptation strategy to learn multi-scale features during pretraining rather than only at the end. The paper's resolution adaptation (Section 4, Figure 6) trains at 224×224 for most iterations and then switches to 518×518 for the final 10k steps. This is a pragmatic hack that works well empirically, but it leaves open the question of whether the model's features at different resolutions are consistent—does a patch at resolution 224 encode the same information as the corresponding region at resolution 518, or does the adaptation phase overwrite the low-resolution representations? A controlled experiment: train DINOv2 with the standard recipe, and also with a multi-resolution schedule where resolution randomly varies per batch (similar to FlexiViT; Beyer et al., 2023). Evaluate both models on tasks that require cross-resolution matching: instance retrieval where query images are at resolution 224 and database images are at resolution 518, or segmentation where the backbone processes crops at multiple scales and their features must be combined. The hypothesis is that multi-resolution training produces features that are more resolution-invariant, improving performance when train and test resolutions differ. A strong result would motivate replacing the two-phase schedule with a continuous multi-resolution schedule, making the features more robust to deployment resolution mismatches.

Investigating whether the distilled models' occasional superiority over the ViT-g teacher is reproducible and exploitable. Figure 5 shows that the distilled ViT-L sometimes outperforms the ViT-g teacher (e.g., Oxford-M retrieval, Paris-M retrieval). The paper attributes this to the iBOT loss being applied on global crops during distillation rather than masked positions. This is a post-hoc explanation; it has not been systematically tested. A direct ablation: distill a ViT-L from ViT-g under two conditions—(a) with the iBOT loss applied on global crops (the paper's method) and (b) with the iBOT loss applied on masked positions (matching the original pretraining). Evaluate both on the full benchmark suite. If (a) consistently outperforms (b) and sometimes surpasses the teacher, it confirms the mechanism and suggests that the original pretraining's use of masking is suboptimal for final feature quality—perhaps masking is beneficial as a regularizer during early training but harmful for fine-grained spatial refinement, and distillation without masking acts as a "polishing" phase. This would motivate a three-stage training procedure: masked pretraining → unmasked distillation → evaluation, and could become standard for producing the strongest frozen features from large SSL models.

Practical Applications and Downstream Use Cases

Off-the-shelf image retrieval for cultural heritage and e-commerce. The instance retrieval results (Table 9) show DINOv2 ViT-g achieving 52.3% mAP on Oxford-Hard (landmark retrieval) and 73.6% GAP-ACC on the Met dataset (artwork retrieval), dramatically exceeding both SSL and weakly-supervised baselines. For a museum digitizing its collection, DINOv2 features enable a system where a visitor photographs an artwork with their phone and the system retrieves the catalog entry, matching across viewpoint, lighting, and partial occlusion—all without any task-specific training or metadata. The KoLeo regularizer's 8-point mAP boost on retrieval (Table 3a) makes this practical at ViT-L scale (54.0% mAP on Oxford-Hard), which can run on a single GPU. For e-commerce, the same features enable visual product search where a user photographs a physical item and the system finds the same or similar products in the catalog, with robustness to background clutter and viewpoint changes that text-supervised features lack.

Frozen-backbone semantic segmentation for domains with limited labeled data. On ADE-20K with a frozen backbone and a ViT-Adapter + Mask2Former head (Section 7.4), DINOv2 ViT-g achieves 60.2 mIoU—within 2.7 points of the fully fine-tuned state of the art (62.9)—while keeping 66% of weights frozen. This dramatically reduces the labeled data requirements for segmentation: the adapter and head contain only ~34% of the parameters, meaning they can be trained effectively with far fewer annotated images than full fine-tuning would require. For applications like medical image segmentation (tumor boundary delineation, organ segmentation) where pixel-level annotations are expensive and scarce, this approach lets practitioners leverage DINOv2's pretrained spatial understanding while adapting only a lightweight task-specific head. The linear probe results (Table 10) further show that even a single linear layer on frozen features achieves non-trivial segmentation (49.0 mIoU on ADE-20K), which could serve as a rapid baseline or pseudo-label generator for bootstrapping annotation efforts.

Monocular depth estimation for robotics and autonomous driving with frozen features. DINOv2 ViT-g with a DPT decoder achieves 0.279 RMSE on NYUd and 2.11 RMSE on KITTI (Table 11), approaching the performance of specialized depth estimation methods while using a frozen backbone trained without any depth supervision. For a robotics startup building a visual navigation system, this means they can download DINOv2, train only a lightweight DPT decoder on their own depth data (which may be from a different sensor modality or environment than the benchmarks), and get competitive depth estimates without ever fine-tuning the backbone. The zero-shot transfer result from NYUd (indoor) to SUN RGB-D (mixed indoor/outdoor, 0.338 RMSE) suggests that the depth features are not overfit to the training domain's depth statistics, making them robust to sensor and environment shifts that often break supervised depth estimators. The qualitative results (Figure 7) showing OpenCLIP producing artifacts and disconnected depth components while DINOv2 produces smooth, boundary-respecting depth maps are directly relevant: in a safety-critical application like autonomous driving, the artifact-free depth from SSL features is preferable to the potentially sharper but less reliable depth from text-supervised features, even if average error metrics were comparable.

Fine-grained species classification for biodiversity monitoring. On iNaturalist 2018 and 2021, DINOv2 ViT-g achieves 81.6% and 85.7% respectively with a linear probe (Table 7), compared to OpenCLIP ViT-G's 73.0% and 76.0%—gains of 8.6 and 9.7 points. For a conservation organization deploying camera traps to monitor wildlife, this means they can use DINOv2 features with a simple linear classifier trained on their labeled images to identify species, without needing to fine-tune a large model or curate text descriptions for each species. The +9.7 point gap on iNaturalist 2021—a dataset with many fine-grained bird, insect, and plant categories—is particularly relevant because these are exactly the domains where subtle visual differences (beak shape, wing pattern, leaf margin) distinguish species, and where SSL's preservation of fine spatial detail (lost in caption compression) provides the advantage. The distilled ViT-B (76.4% and 81.1%) offers a deployment-friendly alternative that still substantially exceeds OpenCLIP-G, running on edge devices at camera trap locations with limited power and connectivity.

When to Prefer This Method

The paper explicitly positions DINOv2 against weakly-supervised alternatives (OpenCLIP, EVA-CLIP) and makes clear tradeoffs that are visible in the results tables. The decision rules are:

Prefer DINOv2 frozen features when:

Dense spatial prediction is the primary task (segmentation, depth estimation, keypoint detection). DINOv2's patch-level iBOT objective and spatial feature quality provide 10+ mIoU advantages on segmentation (Table 10) and 0.135 RMSE on depth (Table 11) over OpenCLIP. This advantage holds at all model sizes.
Fine-grained or instance-level discrimination matters more than scene-level semantics. DINOv2 leads on iNaturalist (+8.6–9.7 points, Table 7), Stanford Cars (+19.6 points over iBOT, Table 8), and landmark/artwork retrieval (+32.6–34.0 mAP over OpenCLIP, Table 9). If your task involves distinguishing similar objects, species, or specific instances, DINOv2 is the better choice.
You need features that work across domains without text data. DINOv2's self-supervised training and data curation require no captions, making it applicable to domains where text-image pairs are unavailable or irrelevant (medical imaging, satellite imagery, industrial inspection, scientific imaging). The data curation pipeline can be adapted to any domain with a curated seed dataset.
Deployment constraints favor smaller models. The distilled ViT-B (86M parameters) achieves 84.5% ImageNet-1k linear probe (Table 4) and strong performance across benchmarks, providing competitive features at a fraction of the compute of ViT-G-scale text-supervised models.

Prefer OpenCLIP or other weakly-supervised features when:

Scene classification or broad category recognition is the primary task. OpenCLIP leads on Places205 (69.8% vs. 67.5%, Table 7) and SUN397 (84.0% vs. 78.7%, Table 8). Text supervision provides scene-level semantics that SSL does not fully capture.
Domain generalization under certain distribution shifts is critical. OpenCLIP leads on ImageNet-R (87.8% vs. 78.8%) and ImageNet-Sketch (66.4% vs. 62.5%, Table 6), suggesting text supervision confers robustness advantages on renditions and artistic depictions that SSL does not match.
Zero-shot classification via text prompts is required. DINOv2 produces purely visual features with no language interface; OpenCLIP supports zero-shot classification by comparing image embeddings to text embeddings of class names. If your use case requires recognizing novel categories specified at test time without any training images, text-supervised models are necessary.
You have aligned image-text data and need a joint embedding space. For cross-modal retrieval (text-to-image or image-to-text search), OpenCLIP's shared image-text embedding space is essential; DINOv2 operates in image-only space.

Prefer training from scratch with the full DINOv2 pipeline when:

You have the compute budget to train a ViT-g scale model and want the absolute best frozen features across the board. The ViT-g from scratch is the top performer on most metrics, and distillation from it produces the strongest smaller models.
You are building a foundation model for a specific domain and can curate a domain-specific seed dataset to run the retrieval pipeline. The DINOv2 code is released, and the pipeline can be reproduced on custom data.

Prefer distilling from a pretrained DINOv2 model when:

You need a smaller model for deployment but want near-ViT-g quality. The distilled ViT-L matches or exceeds the ViT-g on several benchmarks (Figure 5) at ~1/3 the parameters. The ViT-B and ViT-S provide further compute-accuracy tradeoffs.
You lack the compute to train ViT-g from scratch but can afford distillation (which is faster and more stable, requiring no masking or stochastic depth). The paper's ViT-L distillation results suggest this is the most compute-efficient path to strong features.