VerifiedarXiv:2304.0719330 min
Vision · Self-supervised learning

DINOv2: Learning Robust Visual Features without Supervision

An image encoder trained without labels whose frozen features work for classification, segmentation, retrieval and depth.

DINOv2 trains a Vision Transformer on 142 million web images, chosen automatically for their similarity to curated datasets, with no labels or captions. A linear layer on top of the frozen network then matches or beats caption-trained models such as OpenCLIP on most of the paper's benchmarks.

Explaining the paperDINOv2: Learning Robust Visual Features without SupervisionOquab, Darcet, Moutakanni, Vo, Szafraniec, Khalidov, et al. · TMLR 2024 · arXiv:2304.07193 ↗

Meta AI's 2023 follow-up to DINO. The loss reuses DINO and iBOT; most of the work went into a data pipeline and into fixes that let the method train a 1.1-billion-parameter model, which is then distilled into the small models most people download.

What DINOv2 is for

A feature extractor is a network you run once per image to get a vector (or a grid of vectors) that other, smaller models consume. In language, pretrained models are routinely used this way: their outputs feed a classifier with no further training of the big model. The paper asks whether vision can have the same thing, features that work "out of the box" for classification, retrieval, segmentation and depth, with the backbone frozen.

The test throughout is the linear probe. Freeze the pretrained network, compute features for a labelled dataset, and train only a linear layer (one matrix multiply) on top. On ImageNet-1k that layer maps the feature (1,536 numbers for the largest model's [CLS] token) to 1,000 class scores. If the probe reaches high accuracy, the information was already in the features in a form a linear map can read; the backbone learned it without seeing a label. The paper also uses k-nearest-neighbour classification and nearest-neighbour retrieval, which train nothing at all.

In 2023 the best frozen features came from weakly supervised models trained on image-caption pairs: CLIP and its open reproduction OpenCLIP. Self-supervised methods, which learn from images alone, were behind. The best one available, iBOT with a ViT-L/16 trained on ImageNet-22k, reached 82.3% ImageNet linear-probe accuracy against OpenCLIP ViT-G/14's 86.2%. DINOv2's largest model reaches 86.5%.

The paper gets there with four pieces, which this page takes in order:

  1. a dataset, LVD-142M, built by retrieving web images that look like images from curated datasets;
  2. a loss that combines DINO's image-level self-distillation with iBOT's masked-patch prediction, plus Sinkhorn-Knopp target balancing and a KoLeo spreading penalty;
  3. engineering that makes a 1.1-billion-parameter ViT-g trainable at batch size 3,072;
  4. distillation of that ViT-g into ViT-S, ViT-B and ViT-L.

How the LVD-142M dataset is built

Earlier attempts to scale self-supervised learning beyond ImageNet trained on uncurated web images and got worse features. A web crawl is unbalanced: near-copies of the same photo and a few popular kinds of picture make up a large share of it, while most visual concepts appear rarely. A model trained on a random sample spends most of its updates on the common kinds. DINOv2's answer is to keep the scale of the crawl but choose which images to keep by their similarity to images from curated datasets, using no labels, captions or metadata.

The pipeline, with the counts from the paper's Section 3 and Appendix A:

  1. Raw pool. Image URLs from <img> tags in a public web crawl, filtered for safety, with faces blurred: about 1.2 billion images (Appendix A says 1.3 billion).
  2. Self-deduplication. Embed every image with a copy-detection network (Pizzi et al., 2022), link each image to its 64 nearest neighbours with cosine similarity above 0.6, and keep one image per connected component: 1.1 billion left.
  3. Relative deduplication. Drop every component that contains a near-copy (similarity above 0.45) of an image from the train or test split of any benchmark the paper evaluates on: 744 million left.
  4. Retrieval. Embed everything with a self-supervised ViT-H/16 trained on ImageNet-22k and, for each curated dataset, pull pool images that are close to its images under cosine similarity.

Cosine similarity between two images ss and rr is the normalized dot product of their embeddings f(⋅)f(\cdot):

m(s,r)=f(s)⋅f(r)∥f(s)∥2 ∥f(r)∥2m(s,r) = \frac{f(s)\cdot f(r)}{\lVert f(s)\rVert_2\,\lVert f(r)\rVert_2}

Retrieval comes in two forms. For a curated dataset with more than a million images, sample-based retrieval keeps the NN nearest pool images of every query image: N=4N = 4 for ImageNet-22k (14,197,086 queries, 56,788,344 retrieved) and Google Landmarks v2, and N=32N = 32 for the ImageNet-1k training set (1,281,167 queries, 40,997,344 retrieved). For a small dataset, cluster-based retrieval first runs k-means with 100,000 clusters over the whole pool, then takes up to 10,000 images from each cluster that contains more than 3 retrieved neighbours of the dataset, capped at 1 million per dataset. The 3,334 training images of FGVC-Aircraft, for example, pull in 1,170,000 candidates, cut to 1,000,000.

The 27 rows of Table 15 cover classification, fine-grained classification, segmentation, depth and retrieval datasets. ImageNet-22k and Google Landmarks v2 appear twice, once added as they are and once as retrieval queries, and Mapillary street-level images are added as they are. The rows sum to 142,109,386 images. The pipeline ran with the Faiss similarity-search library on 20 nodes of 8 V100 GPUs in under two days.

Figure 1 is a two-dimensional toy of the retrieval step. The grey pool has three dense clusters, standing in for the overrepresented kinds of web image, and 26 sparse ones. The amber diamonds are the curated queries, one per sparse concept plus four on the dense clusters. Compare the share of selected images that come from the dense clusters under retrieval and under a random sample of the same size, then raise NN.

Figure 1 · retrieval-based curation
4
Curated queries each keep their NN nearest pool images. The readout counts collisions (a pool image retrieved by more than one query) and the share taken from the three dense clusters. Switch to the random sample to see the uncurated baseline with the same number of images.

Retrieval keeps the sparse concepts at the weight the curated datasets give them, while the random sample mirrors the pool's imbalance. As NN grows, the retrieved sets of nearby queries start to overlap; the paper reports that retrieval quality looked good on visual inspection for NN much larger than 4, but collisions grew, and it chose 4 as the trade-off.

Table 2 of the paper trains the same ViT-g/14 for the same number of iterations on four datasets. Against a random 142-million sample of the same crawl, LVD-142M gains 2.5 points of ImageNet-1k linear accuracy (85.8 vs 83.3), 14.5 on ImageNet-A (73.9 vs 59.4), 10.3 on Oxford retrieval and 14.3 on iNaturalist 2018 (82.3 vs 68.0). iNaturalist 2018, iNaturalist 2021 and Places205 were not used as retrieval queries, and all three improve. The uncurated sample is better on one column, ADE20k segmentation (48.5 vs 47.7). Against ImageNet-22k, LVD-142M is level on ImageNet-1k (85.8 vs 85.9) and ahead on every other benchmark in the table.

The image-level loss, inherited from DINO

The training objective has three terms. The first is DINO's image-level loss, derived in full in the DINO explainer and run here with the settings below.

Two copies of a Vision Transformer (ViT) are trained together. The student is updated by gradient descent. The teacher has the same architecture and gets no gradient; after each step its weights move toward the student's,

θt←λ θt+(1−λ) θs\theta_t \leftarrow \lambda\,\theta_t + (1-\lambda)\,\theta_s

an exponential moving average (EMA) with λ\lambda rising from 0.994 to 1 on a cosine schedule over the 625,000 iterations of the ViT-g run. The teacher is a smoothed copy of recent students, which makes it a slightly better and more stable target than the student itself.

Every image is cut into random crops. Two global crops cover 32% to 100% of the image and are resized to 224×224; eight local crops cover 5% to 32% and are resized to 98×98. DINOv2 uses 14×14-pixel patches, so a global crop becomes 16×16=25616 \times 16 = 256 patch tokens plus one [CLS] token (257 tokens) and a local crop 7×7=497 \times 7 = 49 patches plus [CLS] (50 tokens). The student sees all ten crops; the teacher sees only the two global ones.

The [CLS] token's output vector (1,536 numbers for ViT-g) goes through a projection head: a 3-layer MLP from 1,536 to 2,048 to 2,048 to a 384-dimensional bottleneck, an ℓ2\ell_2 normalization, and a weight-normalized linear layer to K=131,072K = 131{,}072 outputs. The paper calls these outputs prototype scores. Each is the cosine similarity between the normalized bottleneck vector and one learned prototype direction, times a learned per-prototype scale that starts at 1 (the weight normalization's gain), and a softmax with a temperature turns the 131,072 scores into a distribution. That last layer alone holds 384×131,072≈50.3384 \times 131{,}072 \approx 50.3 million weights. The head is thrown away after training.

With ptp_t the teacher's target distribution for one global crop and psp_s the student's softmax for a different crop of the same image, the image-level term is the cross-entropy

LDINO=−∑k=1Kpt(k)log⁡ps(k)\mathcal{L}_{\mathrm{DINO}} = -\sum_{k=1}^{K} p_t^{(k)} \log p_s^{(k)}
(1)

summed over the KK prototypes and averaged over crop pairs. The pairs per image are global crop A against B, B against A, and each of the 8 local crops against both global crops: 18 terms. The student's softmax uses temperature 0.1; the teacher's, 0.04 rising linearly to 0.07 over the first 37,500 iterations. The lower teacher temperature makes the target more peaked than the student's prediction. The gradient of (1) with respect to the student's scores is (ps−pt)/0.1(p_s - p_t)/0.1, which pulls each crop's prediction toward what the teacher says about the other crops of the same image. A 98-pixel local crop of a dog's ear is trained to give the distribution the teacher gives a crop showing most of the dog.

DINOv2 turns the teacher's scores into ptp_t with Sinkhorn-Knopp normalization (two sections below), where DINO subtracted a moving-average center before the softmax.

The patch-level loss, from iBOT

Loss (1) only touches the [CLS] token. The 256 patch tokens of a global crop, which segmentation and depth models read, get gradient only indirectly. iBOT (Zhou et al., 2022) added a term that trains them directly, in the style of masked language modelling: hide some patches from the student and make it predict, at each hidden position, what the teacher outputs for the visible patch there.

In the code, half of the global crops in a batch get a mask and half do not. The masked ones get between 10% and 50% of their 256 patches masked, in rectangular blocks of at least 4 patches. At a masked position the student receives a learned mask-token embedding instead of the patch. The teacher receives the unmasked crop. A separate iBOT head (same shape as the DINO head) maps patch outputs to 131,072 prototype scores, and the target ptip_{ti} for patch ii is the teacher's balanced distribution for that patch. The patch term is

LiBOT=−∑i∑k=1Kpti(k)log⁡psi(k)\mathcal{L}_{\mathrm{iBOT}} = -\sum_{i} \sum_{k=1}^{K} p_{ti}^{(k)} \log p_{si}^{(k)}
(2)

with ii running over masked positions only. The code averages the masked positions within each crop, then averages over all global crops in the batch, counting unmasked crops as zero. A crop masked at 30% contributes 77 cross-entropies to its average; an unmasked crop contributes only to (1).

Figure 2 shows one global crop as the two networks receive it. The right grid colours each patch by its target (a toy: six scene regions stand in for prototypes), and the dark outlines mark the positions that enter (2). Raise the mask ratio and press new mask; click a masked patch to see what its visible neighbours suggest.

Figure 2 · the masked-patch term
0.30
The masked patches are the only positions with a loss term. The context guess is a majority vote of visible patches within two cells, a stand-in for how much of a hidden patch is predictable from its surroundings, not the network's computation. Large blocks hide whole regions and make the guess fail.

Predicting a masked patch's target requires knowing what object part sits there, which is information the dense tasks need. Table 3b of the paper removes the term at otherwise equal settings: ADE20k linear segmentation drops from 47.1 to 44.2 mIoU, and ImageNet linear accuracy from 85.8 to 85.3. Oxford retrieval moves the other way, 63.9 with the term and 64.3 without.

iBOT used one head for both terms and reported that sharing helped. DINOv2 found the opposite at scale and uses two heads in all experiments. In Table 1 untying is the last change, worth +0.3 k-NN and −0.2 linear on ImageNet.

Balancing the teacher with Sinkhorn-Knopp

Self-distillation has a trivial solution: if the student and the teacher output the same distribution for every image, the student matches the teacher on every pair of crops, and the features carry no information about the image. This is collapse. It comes in two forms: every image puts all its mass on one prototype, or every image outputs the uniform distribution. DINO prevents it with two operations on the teacher's scores. Centering subtracts a running mean of the scores, so no prototype can win for every image; sharpening (the low temperature) stops the output from drifting to uniform.

DINOv2 replaces centering with the Sinkhorn-Knopp normalization from SwAV, following Ruan et al. (2023). It treats the teacher's targets for a batch as a matrix with one column per teacher crop and one row per prototype, and asks for a matrix where every column is a probability distribution and every prototype's row receives the same total mass. Start from the exponentiated scores,

Qkb=exp⁡ ⁣(zkb/τt)Q_{kb} = \exp\!\big(z_{kb}/\tau_t\big)

where zkbz_{kb} is the teacher's score for prototype kk on crop bb. Scale all entries so they sum to 1, then repeat three times: divide each row by its sum and by KK, so each prototype row sums to 1/K1/K; divide each column by its sum and by BB, so each column sums to 1/B1/B. Finally multiply by BB, so each crop's column is a distribution that sums to 1. Each normalization is cheap (one division per entry), and alternating them converges to a matrix of the form diag(u) Q diag(v)\mathrm{diag}(u)\,Q\,\mathrm{diag}(v) that satisfies both constraints. In DINOv2 the sums are all-reduced across GPUs, so BB counts the whole batch: 6,144 global crops (two per image) for the image-level targets, and every masked patch in the batch for the patch-level ones. The student keeps an ordinary softmax at temperature 0.1.

Equal row sums mean every prototype receives, in total over the batch, the same share of target mass. If the teacher starts to prefer one prototype for most images, its row is scaled down and the others up before the targets are used. Centering pushes in the same direction through a running mean; Sinkhorn-Knopp enforces the balance on the current batch.

Figure 3 runs the plain softmax, centering and Sinkhorn-Knopp on a toy batch of 10 images and 5 prototypes, transposed so that rows are images (each row sums to 1) and the bars underneath are the prototype totals. Equal shares would give each prototype B/K=2B/K = 2. Start with the plain softmax at the code's τ=0.07\tau = 0.07, then switch to Sinkhorn-Knopp and drag the iteration count.

Figure 3 · Sinkhorn-Knopp vs centering
0.07
3
Each row is one image's target distribution over five prototypes. The bars sum each column: the amber ones are more than 0.1 away from the equal share, the teal ones within it. Lower the temperature and count how many iterations it takes to balance.

On this batch the plain softmax gives prototype 1 a total of 5.47 and prototype 3 only 0.34. Three Sinkhorn-Knopp iterations bring the totals to 3.05, 2.98, 1.89, 1.03 and 1.06; at τ=0.2\tau = 0.2 three iterations already land within 0.12 of 2 everywhere. The sharper the temperature, the more iterations full balance needs, and DINOv2 stops at three, so the balancing is partial. In the real setting the matrix is 131,072 prototypes by 6,144 crops, many more prototypes than crops, and each prototype's equal share is 0.047 of a crop.

Table 1 measures the switch to Sinkhorn-Knopp at no change in ImageNet accuracy (81.7 k-NN and 84.7 linear, before and after). The paper keeps it on the recommendation of Ruan et al. and gives no separate stability measurement.

Spreading features with KoLeo

The third term acts on the backbone's [CLS] features directly, before the head. Nearest-neighbour retrieval and k-NN classification rank images by distance, and they work poorly if many images sit almost on top of each other in feature space. The KoLeo regularizer (Sablayrolles et al., 2019) penalizes that. Given nn feature vectors x1,…,xnx_1, \ldots, x_n, each ℓ2\ell_2-normalized to unit length, let dn,id_{n,i} be the distance from xix_i to its nearest neighbour in the set:

dn,i=min⁡j≠i∥xi−xj∥d_{n,i} = \min_{j \neq i} \lVert x_i - x_j \rVert
Lkoleo=−1n∑i=1nlog⁡dn,i\mathcal{L}_{\mathrm{koleo}} = -\frac{1}{n}\sum_{i=1}^{n} \log d_{n,i}
(3)

The name comes from the Kozachenko-Leonenko estimator of differential entropy. For nn samples from a density, that estimator is a positive constant times the average of log⁡dn,i\log d_{n,i} plus another constant, both depending only on nn and the dimension. Minimizing (3) therefore maximizes an estimate of the entropy of the features: it pushes the batch toward covering the unit sphere evenly.

The logarithm sets how hard each pair is pushed. The gradient of −1nlog⁡d-\tfrac{1}{n}\log d with respect to either point of a pair has length 1/(n d)1/(n\,d). In the code n=12n = 12, the images on one GPU in the ViT-g run, and the loss is computed on each GPU separately. Evenly spaced on a circle, 12 points sit 2sin⁡(π/12)=0.5182\sin(\pi/12) = 0.518 apart and (3) equals 0.658. Two near-duplicates at d=0.01d = 0.01 each contribute −log⁡0.01=4.6-\log 0.01 = 4.6, adding 2×4.6/12=0.772 \times 4.6 / 12 = 0.77 to the loss, and the push on each is 1/(12×0.01)=8.31/(12 \times 0.01) = 8.3, about 50 times the push at d=0.5d = 0.5.

Figure 4 runs gradient descent on (3) alone for points on a circle, the two-dimensional version of normalized features, starting from two clumps. Play it, scrub back, and change nn.

Figure 4 · the KoLeo regularizer
0
12
Features on the unit circle, each joined to its nearest neighbour by an amber link drawn thicker when shorter. The readout compares the loss and the closest pair with evenly spaced points. At n=2n = 2 the two points end opposite each other, the largest possible distance.

DINOv2 adds 0.1 times (3) to the loss, computed separately for the [CLS] features of each global crop, so the two crops of the same image are never each other's neighbours. In Table 1 adding KoLeo is the largest single k-NN gain, from 76.6 to 78.9. Table 3a isolates it: Oxford-M retrieval rises from 55.6 to 63.9 mAP, ImageNet-A from 70.6 to 72.8, ImageNet linear from 85.3 to 85.8, and ADE20k is flat (47.2 vs 47.1).

One training step, with shapes

Put together, one step of the ViT-g run on one GPU looks like this. The shapes are for the released configuration: 12 images per GPU and a global batch of 3,072, which implies 256 GPUs.

# One DINOv2 step on one GPU: ViT-g/14, 12 images (256 GPUs -> batch 3,072)
g = [aug_global(x), aug_global(x)]    # 2 x (12,3,224,224): 256 patches + CLS each
l = [aug_local(x) for _ in range(8)]  # 8 x (12,3,98,98):    49 patches + CLS each
m = block_masks(g)                    # 12 of the 24 global crops get a mask,
                                      # ratios spread over 0.1..0.5 of 256 patches
with no_grad():                       # teacher = EMA weights, full global crops
    tc, tp = vit_t(g)                 # CLS (24,1536), patches (24,256,1536)
    qc = sinkhorn(head_dino_t(tc), tau_t, iters=3)     # (24, 131072)
    qp = sinkhorn(head_ibot_t(tp[m]), tau_t, iters=3)  # (n_masked, 131072)
sc, sp = vit_s(g, mask=m)             # masked patches replaced by a mask token
lc = vit_s(l)                         # in practice packed with g, one sequence
# 18 image-level pairs: crop A->B, B->A, and each of 8 locals -> A and -> B
L_dino = mean_pairs(CE(qc, log_softmax(head_dino_s([sc, lc]) / 0.1)))
L_ibot = mean_masked(CE(qp, log_softmax(head_ibot_s(sp[m]) / 0.1)))
L_kol = koleo(norm(sc[:12])) + koleo(norm(sc[12:]))  # 12 vectors per call
loss = L_dino + L_ibot + 0.1 * L_kol
loss.backward(); adamw_step(student)  # gradient reaches the student only
theta_t = lam * theta_t + (1 - lam) * theta_s  # lam: 0.994 -> 1, cosine

Only the student receives gradients, from three terms with weights 1, 1 and 0.1. The teacher's output enters only as targets. The optimizer is AdamW. Over the run the student sees 625,000×3,072≈1.92625{,}000 \times 3{,}072 \approx 1.92 billion images, about 13.5 passes over LVD-142M.

One more phase follows. Segmentation and detection need high resolution, where small objects survive, but training at high resolution throughout costs too much: on a smaller ViT-L/16 ablation, training at 416 pixels was about 3 times the compute of 224. DINOv2 instead trains at 518×518 for the last 10,000 iterations, with all schedules compressed into that window and a lower learning rate. A 518-pixel crop is 37×37=1,36937 \times 37 = 1{,}369 patches. In the paper's Figure 6 ablation, a short high-resolution phase at the end came close to training at high resolution throughout, at a fraction of the cost.

What each change bought

Table 1 of the paper starts from iBOT and adds the changes one at a time, reporting ImageNet-1k accuracy of a k-NN classifier and a linear probe on frozen features after each. The authors tuned for k-NN, which they found to bound the linear probe from below. Figure 5 plots the twelve rows. Click a row or drag the slider to read its change and the deltas.

Figure 5 · the training recipe, Table 1
5 / 12
Each row adds one change to the row above. k-NN and linear-probe accuracy, with a faint segment from the previous row's value.

From top to bottom k-NN rises 9.1 points (72.9 to 82.0) and linear accuracy 2.2 (82.3 to 84.5). The authors' reimplementation of iBOT alone gains 1.6 k-NN. LayerScale (a learned per-channel scale on each residual branch, initialized at 10−510^{-5} in the code) and stochastic depth (randomly skipping residual blocks during training) cost 1.2 points of linear accuracy; the caption of Table 1 says they were kept because they prevent NaN losses in large runs. KoLeo (+2.3 k-NN), the batch size of 3,072 (+1.2 k-NN, +0.9 linear) and the retuned warm-up schedules (+1.1) give the largest steps. Two rows change nothing measurable on ImageNet: Sinkhorn-Knopp, and 128,000 prototypes on the linear side.

The table is a single incremental path, so each delta is measured in the context of the changes above it. A component placed later might have helped more or less on its own.

Training a 1.1B-parameter ViT

The largest model, ViT-g/14, has 40 blocks of width 1,536 with 24 attention heads, 1.1 billion parameters, and SwiGLU feed-forward layers (a gated MLP: one linear map, passed through SiLU, multiplies a second linear map elementwise). The usual ViT-g from Zhai et al. has width 1,408 and 16 heads, 88 dimensions per head. DINOv2 changed it to 24 heads of 64 because its attention kernels are fastest when the per-head dimension is a multiple of 64 and the width a multiple of 256; the paper saw no accuracy difference. Against the iBOT code on the same hardware, the DINOv2 code runs about 2 times faster with a third of the memory. Four changes account for it:

The smaller models are distilled from the ViT-g; see the next section.

The paper estimates one ViT-g pretraining run at 22,016 A100 GPU-hours, 9.7 MWh of electricity with a data-centre overhead factor of 1.1, and 3.7 tonnes of CO2-equivalent at the US-average grid intensity of 0.385 kg per kWh. On 256 GPUs that is about 86 hours.

Distilling the smaller models

The released ViT-S/14 (21M parameters), ViT-B/14 (86M) and ViT-L/14 (300M) are not trained from scratch with the recipe above. They are distilled from the ViT-g. Knowledge distillation trains a small network to reproduce a large network's outputs, and DINOv2's loss is already a distillation loss with a teacher made from the student, so the same training loop works with four changes: the teacher is the trained ViT-g, frozen instead of an EMA; masking and stochastic depth are removed; the iBOT term is applied to the two global crops (with no mask, the student predicts the teacher's output for every patch); and a separate EMA of the student is kept and released as the final model. The distilled models use a standard MLP instead of SwiGLU, a learning rate of 1e-3 and a batch of 2,048.

Figure 5 of the paper compares a ViT-L trained from scratch with one distilled from the ViT-g on 12 benchmarks. The distilled ViT-L wins on all 12: ImageNet-1k 86.3 vs 84.5 (the ViT-g itself has 86.5). The figure's table also averages scores within task groups, without listing each group's members: the segmentation average is 73.3 vs 72.2, and the average labelled "ARSketch" (by its name, the ImageNet-A, ImageNet-R and ImageNet-Sketch robustness sets) is 74.5 vs 69.5. On instance retrieval the distilled ViT-L, at 76.3, also beats its teacher's 75.2, with about a quarter of the teacher's parameters.

How frozen DINOv2 features compare with CLIP

Every number below comes from a frozen backbone with a small trained head (a linear layer or logistic regression; retrieval trains nothing). For ImageNet the paper grid-searches the probe: learning rate over 13 values, the last 1 or 4 layers, and the [CLS] token alone or concatenated with the average patch token. Since the backbone runs once per image and every candidate probe is one matrix multiply on the same features, the search costs little.

Figure 6 collects eleven benchmarks from the paper's tables. Pick one to compare the four DINOv2 sizes with OpenCLIP ViT-G/14 (the strongest open caption-trained model) and with the best earlier self-supervised models.

Figure 6 · frozen-feature benchmarks
DINOv2 sizes against OpenCLIP ViT-G/14 and three self-supervised baselines. Depth is error, so shorter bars are better there. Places205, ImageNet-R, Sketch and SSv2 are the ones to check for where DINOv2 does not win.

Some of the numbers, with what they mean:

The paper also shows qualitative properties nobody trained for. A principal component analysis of the patch features of an image, thresholded on the first component, separates the main object from the background; the next three components, computed jointly over three images of one category, colour matching object parts consistently across the images. Matching patches between two images by feature distance pairs the wing of a plane with the wing of a bird.

Limits and open points

Provenance Verified against primary literatureHow we verify
Caron et al. (2021), DINOThe image-level term: cross-entropy between a teacher softmax on a global crop and the student softmax on another crop, with the teacher an exponential moving average of the student. DINOv2 keeps the loss and multi-crop and changes the head size, temperatures schedule and normalization of the targets.
Zhou et al. (2022), iBOTThe patch-level term: the student sees a global crop with blockwise-masked patches, the teacher sees it whole, and each masked position is trained to match the teacher's softmax for that patch. iBOT shared one head for both terms; DINOv2 reports that separate heads work better at scale.
Caron et al. (2020), SwAV; Ruan et al. (2023)Sinkhorn-Knopp normalization of the targets so that every prototype receives an equal share of the batch. SwAV introduced it for clustering-based self-supervision; Ruan et al. recommended it in place of DINO's centering. DINOv2 runs 3 iterations on the teacher only.
Sablayrolles et al. (2019)The KoLeo regularizer, Eq. (2) of "Spreading vectors for similarity search": L = -(1/n) sum_i log(rho_n,i), with rho_n,i the distance from point i to its nearest neighbour, obtained by dropping the constants alpha_n and beta_n of the Kozachenko-Leonenko entropy estimate H_n = (alpha_n / n) sum_i log(rho_n,i) + beta_n.
dinov2/loss/*.pydino_clstoken_loss.py: sinkhorn_knopp_teacher computes Q = exp(t / tau) with no center subtracted, then alternates prototype-row and image-column normalization for n_iterations = 3; student temperature 0.1. ibot_patch_loss.py: masked-patch cross-entropies weighted by 1 / (masked patches in that image), summed, divided by the number of global crops. koleo_loss.py: l2-normalize, nearest neighbour by max dot product, -log(distance + 1e-8), mean.
dinov2/train/ssl_meta_arch.pyLines 131-342: the DINO term averages 18 crop pairs per image (2 global-global, 16 local-global); KoLeo is applied with weight 0.1 to the backbone [CLS] tokens of each global crop separately ("we don't apply koleo loss between cls tokens of a same image"). The paper's Appendix B.1 says the first global crop only.
configs/train/vitg14.yaml, ssl_default_config.yaml131,072 prototypes per head with a 384-d bottleneck, separate iBOT head, sinkhorn_knopp centering, patch 14, drop-path 0.4, SwiGLU, teacher momentum 0.994 -> 1, teacher temperature 0.04 -> 0.07 over 30 epochs of 1,250 iterations, 500 such epochs (625,000 iterations), 12 images per GPU, local crops 98 px, base LR 2e-4 at batch 1,024 with square-root scaling (2e-4 x sqrt(3) = 3.46e-4, the 3.5e-4 of Table 16), mask ratio 0.1 to 0.5 on half the crops.
Sec. 4 vs codeThe paper describes the teacher as "a softmax followed by a centering". In the code the moving-average center is subtracted from the logits before the softmax (softmax((t - c) / tau)), as in DINO, and the Sinkhorn-Knopp path used for the released models has no center at all.
Sec. 7 text vs tablesSeveral deltas in the prose do not match the tables they cite. Robustness vs iBOT: text +29.6 / +22.1 / +23.0, Table 6 gives ViT-g 34.4 / 27.8 / 24.0. Video vs OpenCLIP: text "+0.1% and +0.5%" for UCF and Kinetics, Table 7 gives UCF +0.5 and K400 +0.1. ImageNet-V2 vs EVA-CLIP: text +1.1, Table 4 gives 78.4 - 77.4 = 1.0. Fairness: text 25.7% (Africa vs Europe) and 31.7% (low vs high income), Table 12 gives 15.7 and 23.1 points. Present in v1 and v2. This page quotes the tables.
Table 7"Amongst self-supervised approaches, our model clearly sets a new state of the art" on video holds for UCF-101 and Kinetics-400 but not for Something-Something v2, where iBOT ViT-L/16 scores 38.7 and DINOv2 ViT-g/14 38.3.
correctionArchitecture table typo. Table 17 (both arXiv versions) lists the distilled ViT-B/14 with 18 blocks. The released model has 12: vit_base in dinov2/models/vision_transformer.py sets depth=12, the hub model dinov2_vitb14 is built from it, the README gives it 86M parameters (a standard 12-block ViT-B), and the paper's own depth evaluation reads ViT-B features from layers {3, 6, 9, 12}. This page uses 12 blocks for ViT-B.

Questions you might still have

?

Is DINOv2 just DINO trained on more data?
No. The loss is DINO's image-level term plus iBOT's masked-patch term, with separate heads, Sinkhorn-Knopp targets and a KoLeo penalty; the model is a 1.1-billion-parameter ViT-g; the data is a new 142-million-image set built by retrieval; and the released small models are distilled from the large one. Table 1 measures the recipe changes alone (k-NN 72.9 to 82.0 on ImageNet), and Table 2 measures the data alone at fixed recipe.

?

Why not just train on the 1.2 billion uncurated images?
The paper tried a random 142-million subset of the same crawl. Against the curated LVD-142M it lost 2.5 points on ImageNet-1k (83.3 vs 85.8), 14.5 on ImageNet-A and 14.3 on iNaturalist 2018, and was slightly better on ADE20k segmentation (48.5 vs 47.7). A crawl is dominated by a few kinds of image, so a random sample spends most of its budget on them.

?

Does the curation leak the test sets into training?
The pipeline removes pool images that are near-duplicates (cosine similarity above 0.45 under a copy-detection embedding) of the train and test splits of every evaluation dataset before retrieval, which cut the pool from 1.1 billion to 744 million. Retrieval then uses the training splits of many evaluation datasets as queries, so the pretraining data is deliberately similar in distribution to those benchmarks, though not to their test images. iNaturalist and Places205 were not used as queries and still improved.

?

Which output should I use: the [CLS] token or the patch tokens?
Both are trained. The [CLS] token gets the image-level loss and KoLeo and is the feature for classification and retrieval; the patch tokens get the masked-patch loss and are the features for segmentation and depth. The paper's linear probes on ImageNet search over using the [CLS] token alone or concatenated with the average of the patch tokens, and over the last 1 or 4 layers.

?

How do I get DINOv2 features for my own images?
The official repository publishes the backbones through torch.hub, for example torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14'), with _reg variants that include register tokens. Calling the model returns the [CLS] token after the final LayerNorm (768 numbers for ViT-B); model.forward_features returns a dictionary that also holds the normalized patch tokens, one per 14 by 14 patch, so the image sides should be multiples of 14. No projection head ships with the backbone.

?

How expensive is it to reproduce?
The paper estimates 22,016 A100 GPU-hours for one ViT-g pretraining run, 9.7 MWh and 3.7 tonnes of CO2-equivalent under a US-average grid, and 0.5 to 1 thousand tonnes for the whole project (about 200,000 GPU-days). The curation pipeline ran on 20 nodes of 8 V100s in under two days.

?

What is the difference between DINOv2 and CLIP features?
CLIP-style models are trained to match images with their captions, so their features carry what captions describe; DINOv2 is trained on images alone. In the paper's tables the frozen DINOv2 ViT-g beats OpenCLIP ViT-G on fine-grained species classification, instance retrieval, segmentation and depth, and loses on scene classification (Places205, SUN397) and on renditions and sketches (ImageNet-R, Sketch). DINOv2 has no text encoder, so it cannot do zero-shot classification from class names the way the CLIP explainer describes.

?

Why distill the small models instead of training them directly?
Because it worked better at every size the paper tried. A ViT-L distilled from the frozen ViT-g beat a ViT-L trained from scratch on all 12 benchmarks of Figure 5 (ImageNet 86.3 vs 84.5), and on retrieval it beat the ViT-g teacher itself (76.3 vs 75.2). The distillation reuses the same training loop with the ViT-g as a frozen teacher.

Footnotes & further reading

  1. The paper: Oquab, Darcet, Moutakanni, Vo, Szafraniec, Khalidov, et al., DINOv2: Learning Robust Visual Features without Supervision (arXiv April 2023; v2 February 2024, TMLR). Official code, the source for the loss implementations, schedules and configurations quoted on this page.
  2. DINO: Caron et al., Emerging Properties in Self-Supervised Vision Transformers (ICCV 2021); see the DINO explainer for the centering, sharpening and multi-crop details.
  3. iBOT: Zhou et al., iBOT: Image BERT Pre-Training with Online Tokenizer (ICLR 2022).
  4. Sinkhorn-Knopp in self-supervision: Caron et al., Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (SwAV, NeurIPS 2020), Section 3.1; Ruan et al., Weighted Ensemble Self-Supervised Learning (ICLR 2023).
  5. KoLeo: Sablayrolles, Douze, Schmid & Jégou, Spreading vectors for similarity search (ICLR 2019), Section 3.1, Eq. (1) and (2).
  6. Registers: Darcet, Oquab, Mairal & Bojanowski, Vision Transformers Need Registers (ICLR 2024).
  7. Related explainers: the Vision Transformer, CLIP, and MAE, whose features need fine-tuning (ImageNet linear probe 76.6% for ViT-H/14 in this paper's Table 4).