DINOv2: Learning Robust Visual Features without Supervision
An image encoder trained without labels whose frozen features work for classification, segmentation, retrieval and depth.
DINOv2 trains a Vision Transformer on 142 million web images, chosen automatically for their similarity to curated datasets, with no labels or captions. A linear layer on top of the frozen network then matches or beats caption-trained models such as OpenCLIP on most of the paper's benchmarks.
Explaining the paperDINOv2: Learning Robust Visual Features without SupervisionMeta AI's 2023 follow-up to DINO. The loss reuses DINO and iBOT; most of the work went into a data pipeline and into fixes that let the method train a 1.1-billion-parameter model, which is then distilled into the small models most people download.
What DINOv2 is for
A feature extractor is a network you run once per image to get a vector (or a grid of vectors) that other, smaller models consume. In language, pretrained models are routinely used this way: their outputs feed a classifier with no further training of the big model. The paper asks whether vision can have the same thing, features that work "out of the box" for classification, retrieval, segmentation and depth, with the backbone frozen.
The test throughout is the linear probe. Freeze the pretrained network, compute features for a labelled dataset, and train only a linear layer (one matrix multiply) on top. On ImageNet-1k that layer maps the feature (1,536 numbers for the largest model's [CLS] token) to 1,000 class scores. If the probe reaches high accuracy, the information was already in the features in a form a linear map can read; the backbone learned it without seeing a label. The paper also uses k-nearest-neighbour classification and nearest-neighbour retrieval, which train nothing at all.
In 2023 the best frozen features came from weakly supervised models trained on image-caption pairs: CLIP and its open reproduction OpenCLIP. Self-supervised methods, which learn from images alone, were behind. The best one available, iBOT with a ViT-L/16 trained on ImageNet-22k, reached 82.3% ImageNet linear-probe accuracy against OpenCLIP ViT-G/14's 86.2%. DINOv2's largest model reaches 86.5%.
The paper gets there with four pieces, which this page takes in order:
- a dataset, LVD-142M, built by retrieving web images that look like images from curated datasets;
- a loss that combines DINO's image-level self-distillation with iBOT's masked-patch prediction, plus Sinkhorn-Knopp target balancing and a KoLeo spreading penalty;
- engineering that makes a 1.1-billion-parameter ViT-g trainable at batch size 3,072;
- distillation of that ViT-g into ViT-S, ViT-B and ViT-L.
How the LVD-142M dataset is built
Earlier attempts to scale self-supervised learning beyond ImageNet trained on uncurated web images and got worse features. A web crawl is unbalanced: near-copies of the same photo and a few popular kinds of picture make up a large share of it, while most visual concepts appear rarely. A model trained on a random sample spends most of its updates on the common kinds. DINOv2's answer is to keep the scale of the crawl but choose which images to keep by their similarity to images from curated datasets, using no labels, captions or metadata.
The pipeline, with the counts from the paper's Section 3 and Appendix A:
- Raw pool. Image URLs from
<img>tags in a public web crawl, filtered for safety, with faces blurred: about 1.2 billion images (Appendix A says 1.3 billion). - Self-deduplication. Embed every image with a copy-detection network (Pizzi et al., 2022), link each image to its 64 nearest neighbours with cosine similarity above 0.6, and keep one image per connected component: 1.1 billion left.
- Relative deduplication. Drop every component that contains a near-copy (similarity above 0.45) of an image from the train or test split of any benchmark the paper evaluates on: 744 million left.
- Retrieval. Embed everything with a self-supervised ViT-H/16 trained on ImageNet-22k and, for each curated dataset, pull pool images that are close to its images under cosine similarity.
Cosine similarity between two images and is the normalized dot product of their embeddings :
Retrieval comes in two forms. For a curated dataset with more than a million images, sample-based retrieval keeps the nearest pool images of every query image: for ImageNet-22k (14,197,086 queries, 56,788,344 retrieved) and Google Landmarks v2, and for the ImageNet-1k training set (1,281,167 queries, 40,997,344 retrieved). For a small dataset, cluster-based retrieval first runs k-means with 100,000 clusters over the whole pool, then takes up to 10,000 images from each cluster that contains more than 3 retrieved neighbours of the dataset, capped at 1 million per dataset. The 3,334 training images of FGVC-Aircraft, for example, pull in 1,170,000 candidates, cut to 1,000,000.
The 27 rows of Table 15 cover classification, fine-grained classification, segmentation, depth and retrieval datasets. ImageNet-22k and Google Landmarks v2 appear twice, once added as they are and once as retrieval queries, and Mapillary street-level images are added as they are. The rows sum to 142,109,386 images. The pipeline ran with the Faiss similarity-search library on 20 nodes of 8 V100 GPUs in under two days.
Figure 1 is a two-dimensional toy of the retrieval step. The grey pool has three dense clusters, standing in for the overrepresented kinds of web image, and 26 sparse ones. The amber diamonds are the curated queries, one per sparse concept plus four on the dense clusters. Compare the share of selected images that come from the dense clusters under retrieval and under a random sample of the same size, then raise .
Retrieval keeps the sparse concepts at the weight the curated datasets give them, while the random sample mirrors the pool's imbalance. As grows, the retrieved sets of nearby queries start to overlap; the paper reports that retrieval quality looked good on visual inspection for much larger than 4, but collisions grew, and it chose 4 as the trade-off.
Table 2 of the paper trains the same ViT-g/14 for the same number of iterations on four datasets. Against a random 142-million sample of the same crawl, LVD-142M gains 2.5 points of ImageNet-1k linear accuracy (85.8 vs 83.3), 14.5 on ImageNet-A (73.9 vs 59.4), 10.3 on Oxford retrieval and 14.3 on iNaturalist 2018 (82.3 vs 68.0). iNaturalist 2018, iNaturalist 2021 and Places205 were not used as retrieval queries, and all three improve. The uncurated sample is better on one column, ADE20k segmentation (48.5 vs 47.7). Against ImageNet-22k, LVD-142M is level on ImageNet-1k (85.8 vs 85.9) and ahead on every other benchmark in the table.
The image-level loss, inherited from DINO
The training objective has three terms. The first is DINO's image-level loss, derived in full in the DINO explainer and run here with the settings below.
Two copies of a Vision Transformer (ViT) are trained together. The student is updated by gradient descent. The teacher has the same architecture and gets no gradient; after each step its weights move toward the student's,
an exponential moving average (EMA) with rising from 0.994 to 1 on a cosine schedule over the 625,000 iterations of the ViT-g run. The teacher is a smoothed copy of recent students, which makes it a slightly better and more stable target than the student itself.
Every image is cut into random crops. Two global crops cover 32% to 100% of the image and are resized to 224×224; eight local crops cover 5% to 32% and are resized to 98×98. DINOv2 uses 14×14-pixel patches, so a global crop becomes patch tokens plus one [CLS] token (257 tokens) and a local crop patches plus [CLS] (50 tokens). The student sees all ten crops; the teacher sees only the two global ones.
The [CLS] token's output vector (1,536 numbers for ViT-g) goes through a projection head: a 3-layer MLP from 1,536 to 2,048 to 2,048 to a 384-dimensional bottleneck, an normalization, and a weight-normalized linear layer to outputs. The paper calls these outputs prototype scores. Each is the cosine similarity between the normalized bottleneck vector and one learned prototype direction, times a learned per-prototype scale that starts at 1 (the weight normalization's gain), and a softmax with a temperature turns the 131,072 scores into a distribution. That last layer alone holds million weights. The head is thrown away after training.
With the teacher's target distribution for one global crop and the student's softmax for a different crop of the same image, the image-level term is the cross-entropy
summed over the prototypes and averaged over crop pairs. The pairs per image are global crop A against B, B against A, and each of the 8 local crops against both global crops: 18 terms. The student's softmax uses temperature 0.1; the teacher's, 0.04 rising linearly to 0.07 over the first 37,500 iterations. The lower teacher temperature makes the target more peaked than the student's prediction. The gradient of (1) with respect to the student's scores is , which pulls each crop's prediction toward what the teacher says about the other crops of the same image. A 98-pixel local crop of a dog's ear is trained to give the distribution the teacher gives a crop showing most of the dog.
DINOv2 turns the teacher's scores into with Sinkhorn-Knopp normalization (two sections below), where DINO subtracted a moving-average center before the softmax.
The patch-level loss, from iBOT
Loss (1) only touches the [CLS] token. The 256 patch tokens of a global crop, which segmentation and depth models read, get gradient only indirectly. iBOT (Zhou et al., 2022) added a term that trains them directly, in the style of masked language modelling: hide some patches from the student and make it predict, at each hidden position, what the teacher outputs for the visible patch there.
In the code, half of the global crops in a batch get a mask and half do not. The masked ones get between 10% and 50% of their 256 patches masked, in rectangular blocks of at least 4 patches. At a masked position the student receives a learned mask-token embedding instead of the patch. The teacher receives the unmasked crop. A separate iBOT head (same shape as the DINO head) maps patch outputs to 131,072 prototype scores, and the target for patch is the teacher's balanced distribution for that patch. The patch term is
with running over masked positions only. The code averages the masked positions within each crop, then averages over all global crops in the batch, counting unmasked crops as zero. A crop masked at 30% contributes 77 cross-entropies to its average; an unmasked crop contributes only to (1).
Figure 2 shows one global crop as the two networks receive it. The right grid colours each patch by its target (a toy: six scene regions stand in for prototypes), and the dark outlines mark the positions that enter (2). Raise the mask ratio and press new mask; click a masked patch to see what its visible neighbours suggest.
Predicting a masked patch's target requires knowing what object part sits there, which is information the dense tasks need. Table 3b of the paper removes the term at otherwise equal settings: ADE20k linear segmentation drops from 47.1 to 44.2 mIoU, and ImageNet linear accuracy from 85.8 to 85.3. Oxford retrieval moves the other way, 63.9 with the term and 64.3 without.
iBOT used one head for both terms and reported that sharing helped. DINOv2 found the opposite at scale and uses two heads in all experiments. In Table 1 untying is the last change, worth +0.3 k-NN and −0.2 linear on ImageNet.
Balancing the teacher with Sinkhorn-Knopp
Self-distillation has a trivial solution: if the student and the teacher output the same distribution for every image, the student matches the teacher on every pair of crops, and the features carry no information about the image. This is collapse. It comes in two forms: every image puts all its mass on one prototype, or every image outputs the uniform distribution. DINO prevents it with two operations on the teacher's scores. Centering subtracts a running mean of the scores, so no prototype can win for every image; sharpening (the low temperature) stops the output from drifting to uniform.
DINOv2 replaces centering with the Sinkhorn-Knopp normalization from SwAV, following Ruan et al. (2023). It treats the teacher's targets for a batch as a matrix with one column per teacher crop and one row per prototype, and asks for a matrix where every column is a probability distribution and every prototype's row receives the same total mass. Start from the exponentiated scores,
where is the teacher's score for prototype on crop . Scale all entries so they sum to 1, then repeat three times: divide each row by its sum and by , so each prototype row sums to ; divide each column by its sum and by , so each column sums to . Finally multiply by , so each crop's column is a distribution that sums to 1. Each normalization is cheap (one division per entry), and alternating them converges to a matrix of the form that satisfies both constraints. In DINOv2 the sums are all-reduced across GPUs, so counts the whole batch: 6,144 global crops (two per image) for the image-level targets, and every masked patch in the batch for the patch-level ones. The student keeps an ordinary softmax at temperature 0.1.
Equal row sums mean every prototype receives, in total over the batch, the same share of target mass. If the teacher starts to prefer one prototype for most images, its row is scaled down and the others up before the targets are used. Centering pushes in the same direction through a running mean; Sinkhorn-Knopp enforces the balance on the current batch.
Figure 3 runs the plain softmax, centering and Sinkhorn-Knopp on a toy batch of 10 images and 5 prototypes, transposed so that rows are images (each row sums to 1) and the bars underneath are the prototype totals. Equal shares would give each prototype . Start with the plain softmax at the code's , then switch to Sinkhorn-Knopp and drag the iteration count.
On this batch the plain softmax gives prototype 1 a total of 5.47 and prototype 3 only 0.34. Three Sinkhorn-Knopp iterations bring the totals to 3.05, 2.98, 1.89, 1.03 and 1.06; at three iterations already land within 0.12 of 2 everywhere. The sharper the temperature, the more iterations full balance needs, and DINOv2 stops at three, so the balancing is partial. In the real setting the matrix is 131,072 prototypes by 6,144 crops, many more prototypes than crops, and each prototype's equal share is 0.047 of a crop.
Table 1 measures the switch to Sinkhorn-Knopp at no change in ImageNet accuracy (81.7 k-NN and 84.7 linear, before and after). The paper keeps it on the recommendation of Ruan et al. and gives no separate stability measurement.
Spreading features with KoLeo
The third term acts on the backbone's [CLS] features directly, before the head. Nearest-neighbour retrieval and k-NN classification rank images by distance, and they work poorly if many images sit almost on top of each other in feature space. The KoLeo regularizer (Sablayrolles et al., 2019) penalizes that. Given feature vectors , each -normalized to unit length, let be the distance from to its nearest neighbour in the set:
The name comes from the Kozachenko-Leonenko estimator of differential entropy. For samples from a density, that estimator is a positive constant times the average of plus another constant, both depending only on and the dimension. Minimizing (3) therefore maximizes an estimate of the entropy of the features: it pushes the batch toward covering the unit sphere evenly.
The logarithm sets how hard each pair is pushed. The gradient of with respect to either point of a pair has length . In the code , the images on one GPU in the ViT-g run, and the loss is computed on each GPU separately. Evenly spaced on a circle, 12 points sit apart and (3) equals 0.658. Two near-duplicates at each contribute , adding to the loss, and the push on each is , about 50 times the push at .
Figure 4 runs gradient descent on (3) alone for points on a circle, the two-dimensional version of normalized features, starting from two clumps. Play it, scrub back, and change .
DINOv2 adds 0.1 times (3) to the loss, computed separately for the [CLS] features of each global crop, so the two crops of the same image are never each other's neighbours. In Table 1 adding KoLeo is the largest single k-NN gain, from 76.6 to 78.9. Table 3a isolates it: Oxford-M retrieval rises from 55.6 to 63.9 mAP, ImageNet-A from 70.6 to 72.8, ImageNet linear from 85.3 to 85.8, and ADE20k is flat (47.2 vs 47.1).
One training step, with shapes
Put together, one step of the ViT-g run on one GPU looks like this. The shapes are for the released configuration: 12 images per GPU and a global batch of 3,072, which implies 256 GPUs.
# One DINOv2 step on one GPU: ViT-g/14, 12 images (256 GPUs -> batch 3,072)
g = [aug_global(x), aug_global(x)] # 2 x (12,3,224,224): 256 patches + CLS each
l = [aug_local(x) for _ in range(8)] # 8 x (12,3,98,98): 49 patches + CLS each
m = block_masks(g) # 12 of the 24 global crops get a mask,
# ratios spread over 0.1..0.5 of 256 patches
with no_grad(): # teacher = EMA weights, full global crops
tc, tp = vit_t(g) # CLS (24,1536), patches (24,256,1536)
qc = sinkhorn(head_dino_t(tc), tau_t, iters=3) # (24, 131072)
qp = sinkhorn(head_ibot_t(tp[m]), tau_t, iters=3) # (n_masked, 131072)
sc, sp = vit_s(g, mask=m) # masked patches replaced by a mask token
lc = vit_s(l) # in practice packed with g, one sequence
# 18 image-level pairs: crop A->B, B->A, and each of 8 locals -> A and -> B
L_dino = mean_pairs(CE(qc, log_softmax(head_dino_s([sc, lc]) / 0.1)))
L_ibot = mean_masked(CE(qp, log_softmax(head_ibot_s(sp[m]) / 0.1)))
L_kol = koleo(norm(sc[:12])) + koleo(norm(sc[12:])) # 12 vectors per call
loss = L_dino + L_ibot + 0.1 * L_kol
loss.backward(); adamw_step(student) # gradient reaches the student only
theta_t = lam * theta_t + (1 - lam) * theta_s # lam: 0.994 -> 1, cosineOnly the student receives gradients, from three terms with weights 1, 1 and 0.1. The teacher's output enters only as targets. The optimizer is AdamW. Over the run the student sees billion images, about 13.5 passes over LVD-142M.
One more phase follows. Segmentation and detection need high resolution, where small objects survive, but training at high resolution throughout costs too much: on a smaller ViT-L/16 ablation, training at 416 pixels was about 3 times the compute of 224. DINOv2 instead trains at 518×518 for the last 10,000 iterations, with all schedules compressed into that window and a lower learning rate. A 518-pixel crop is patches. In the paper's Figure 6 ablation, a short high-resolution phase at the end came close to training at high resolution throughout, at a fraction of the cost.
What each change bought
Table 1 of the paper starts from iBOT and adds the changes one at a time, reporting ImageNet-1k accuracy of a k-NN classifier and a linear probe on frozen features after each. The authors tuned for k-NN, which they found to bound the linear probe from below. Figure 5 plots the twelve rows. Click a row or drag the slider to read its change and the deltas.
From top to bottom k-NN rises 9.1 points (72.9 to 82.0) and linear accuracy 2.2 (82.3 to 84.5). The authors' reimplementation of iBOT alone gains 1.6 k-NN. LayerScale (a learned per-channel scale on each residual branch, initialized at in the code) and stochastic depth (randomly skipping residual blocks during training) cost 1.2 points of linear accuracy; the caption of Table 1 says they were kept because they prevent NaN losses in large runs. KoLeo (+2.3 k-NN), the batch size of 3,072 (+1.2 k-NN, +0.9 linear) and the retuned warm-up schedules (+1.1) give the largest steps. Two rows change nothing measurable on ImageNet: Sinkhorn-Knopp, and 128,000 prototypes on the linear side.
The table is a single incremental path, so each delta is measured in the context of the changes above it. A component placed later might have helped more or less on its own.
Training a 1.1B-parameter ViT
The largest model, ViT-g/14, has 40 blocks of width 1,536 with 24 attention heads, 1.1 billion parameters, and SwiGLU feed-forward layers (a gated MLP: one linear map, passed through SiLU, multiplies a second linear map elementwise). The usual ViT-g from Zhai et al. has width 1,408 and 16 heads, 88 dimensions per head. DINOv2 changed it to 24 heads of 64 because its attention kernels are fastest when the per-head dimension is a multiple of 64 and the width a multiple of 256; the paper saw no accuracy difference. Against the iBOT code on the same hardware, the DINOv2 code runs about 2 times faster with a third of the memory. Four changes account for it:
- Memory-efficient attention. An in-house implementation in the style of FlashAttention, computing attention in tiles without storing the full attention matrix.
- Sequence packing. Global crops are 257 tokens and local crops 50, so they cannot share a batch tensor, and earlier code ran a separate forward and backward pass for each size. DINOv2 concatenates all the sequences into one long sequence (for one image, tokens) and applies a block-diagonal attention mask, so a token attends only within its own crop. The feed-forward layers and projections, which act on each token independently, run once over everything; the attention computes score entries per image instead of the of unmasked attention over the packed sequence, and the outputs match separate per-crop passes exactly.
- Stochastic depth that skips the work. With a drop rate of 40%, each residual block runs on a random 60% of the batch and the rest pass through unchanged. Standard implementations compute the block for every sample and zero out the dropped ones; DINOv2 shuffles the batch and computes the block on the first 60% only, saving about 40% of that block's compute and memory.
- Fully sharded data parallel (FSDP). AdamW keeps four float32 copies of the weights in play: student, teacher, first moments and second moments, 16 bytes per parameter, or about 16 GB for a billion parameters. FSDP splits these across GPUs instead of replicating them, the same idea as ZeRO. Weights are stored in float32 but broadcast and gradients reduced in float16 for the backbone (the heads' gradients are reduced in float32 to avoid instabilities), which halves the communication compared with float32 all-reduce.
The smaller models are distilled from the ViT-g; see the next section.
The paper estimates one ViT-g pretraining run at 22,016 A100 GPU-hours, 9.7 MWh of electricity with a data-centre overhead factor of 1.1, and 3.7 tonnes of CO2-equivalent at the US-average grid intensity of 0.385 kg per kWh. On 256 GPUs that is about 86 hours.
Distilling the smaller models
The released ViT-S/14 (21M parameters), ViT-B/14 (86M) and ViT-L/14 (300M) are not trained from scratch with the recipe above. They are distilled from the ViT-g. Knowledge distillation trains a small network to reproduce a large network's outputs, and DINOv2's loss is already a distillation loss with a teacher made from the student, so the same training loop works with four changes: the teacher is the trained ViT-g, frozen instead of an EMA; masking and stochastic depth are removed; the iBOT term is applied to the two global crops (with no mask, the student predicts the teacher's output for every patch); and a separate EMA of the student is kept and released as the final model. The distilled models use a standard MLP instead of SwiGLU, a learning rate of 1e-3 and a batch of 2,048.
Figure 5 of the paper compares a ViT-L trained from scratch with one distilled from the ViT-g on 12 benchmarks. The distilled ViT-L wins on all 12: ImageNet-1k 86.3 vs 84.5 (the ViT-g itself has 86.5). The figure's table also averages scores within task groups, without listing each group's members: the segmentation average is 73.3 vs 72.2, and the average labelled "ARSketch" (by its name, the ImageNet-A, ImageNet-R and ImageNet-Sketch robustness sets) is 74.5 vs 69.5. On instance retrieval the distilled ViT-L, at 76.3, also beats its teacher's 75.2, with about a quarter of the teacher's parameters.
How frozen DINOv2 features compare with CLIP
Every number below comes from a frozen backbone with a small trained head (a linear layer or logistic regression; retrieval trains nothing). For ImageNet the paper grid-searches the probe: learning rate over 13 values, the last 1 or 4 layers, and the [CLS] token alone or concatenated with the average patch token. Since the backbone runs once per image and every candidate probe is one matrix multiply on the same features, the search costs little.
Figure 6 collects eleven benchmarks from the paper's tables. Pick one to compare the four DINOv2 sizes with OpenCLIP ViT-G/14 (the strongest open caption-trained model) and with the best earlier self-supervised models.
Some of the numbers, with what they mean:
- ImageNet-1k linear probe: 86.5% for ViT-g/14, against 86.2% for OpenCLIP ViT-G/14 and 82.3% for iBOT. On ImageNet-V2, a re-collected test set that punishes overfitting to the original one, the gap is wider: 78.4 vs 77.2. Fine-tuning the whole backbone adds 2.0 points at 224 pixels (88.5%) and 2.2 at 448 (88.9%), so the frozen features are close to what the network can reach.
- Robustness: the same ImageNet probe applied to ImageNet-A (natural images that fool ResNets) scores 75.9, above OpenCLIP's 63.8. On ImageNet-R (renditions such as paintings and cartoons) and Sketch, OpenCLIP stays ahead, 87.8 vs 78.8 and 66.4 vs 62.5.
- Fine-grained and scenes: iNaturalist 2018 (species) 81.6 vs 73.0 for OpenCLIP; Places205 67.5 vs 69.8. Over the 12 small classification sets of SimCLR, the averages are 92.1 vs 91.9, with OpenCLIP ahead on SUN397 by 5.3 and Stanford Cars by 4.7.
- Instance retrieval (rank database images by cosine similarity to a query): Oxford-Hard mAP 54.0 for ViT-L/14 against 19.7 for OpenCLIP and 13.7 for DINO. The ViT-L beats the ViT-g here (52.3).
- Segmentation: a linear layer on frozen patch tokens gets 49.0 mIoU on ADE20k (OpenCLIP 39.3). With a trained ViT-Adapter and Mask2Former head on the frozen backbone, 60.2, against a state of the art of 62.9 at the time.
- Depth: a linear layer on frozen patch tokens predicting depth over 256 bins on NYU Depth v2 reaches RMSE 0.344 m against OpenCLIP's 0.541; with a DPT decoder, 0.279.
- Video: averaging the features of 8 frames and training a linear layer gives 78.4% on Kinetics-400 and 91.2% on UCF-101, level with OpenCLIP; on Something-Something v2, which depends on motion, 38.3 vs OpenCLIP's 35.8, with iBOT slightly ahead at 38.7.
The paper also shows qualitative properties nobody trained for. A principal component analysis of the patch features of an image, thresholded on the first component, separates the main object from the background; the next three components, computed jointly over three images of one category, colour matching object parts consistently across the images. Matching patches between two images by feature distance pairs the wing of a plane with the wing of a bird.
Limits and open points
- Fairness. On the Dollar Street dataset, which photographs the same household objects in homes across 54 countries and income levels, ViT-g scores 89.7% on images from Europe and 74.0% on images from Africa, and 90.5% for high-income households against 67.4% for low-income ones. That is fairer than SEERv2 (85.6 and 65.9; 86.6 and 59.7), but the gaps are 15.7 and 23.1 points. Curation by similarity to existing curated datasets inherits their geographic skew.
- Curation depends on the benchmarks. The retrieval queries are the training splits of many of the evaluation datasets, so LVD-142M is built to resemble the test distributions (test images themselves are removed). The improvements on iNaturalist and Places205, which were not queries, are the cleaner evidence of general benefit.
- No text. DINOv2 features have no link to language, so zero-shot classification from class names, which CLIP does, requires training a separate alignment. The paper names a language-enabled system as future work.
- Attention artifacts. A follow-up by four of the authors (Darcet, Oquab, Mairal and Bojanowski, 2023) found that large DINOv2 models, like other large ViTs, produce a few high-norm tokens in background patches that the model uses for global information, which shows up as artifacts in attention maps. Adding extra learnable "register" tokens removes them; the official code has a
num_register_tokensoption and released checkpoints with registers. - Attribution. Table 1 adds changes in one fixed order, and the data ablation (Table 2) and the loss ablations (Table 3) use shorter training runs than the final model, so the per-component numbers are estimates in context rather than independent effects.
Questions you might still have
Is DINOv2 just DINO trained on more data?
No. The loss is DINO's image-level term plus iBOT's masked-patch term, with separate heads, Sinkhorn-Knopp targets and a KoLeo penalty; the model is a 1.1-billion-parameter ViT-g; the data is a new 142-million-image set built by retrieval; and the released small models are distilled from the large one. Table 1 measures the recipe changes alone (k-NN 72.9 to 82.0 on ImageNet), and Table 2 measures the data alone at fixed recipe.
Why not just train on the 1.2 billion uncurated images?
The paper tried a random 142-million subset of the same crawl. Against the curated LVD-142M it lost 2.5 points on ImageNet-1k (83.3 vs 85.8), 14.5 on ImageNet-A and 14.3 on iNaturalist 2018, and was slightly better on ADE20k segmentation (48.5 vs 47.7). A crawl is dominated by a few kinds of image, so a random sample spends most of its budget on them.
Does the curation leak the test sets into training?
The pipeline removes pool images that are near-duplicates (cosine similarity above 0.45 under a copy-detection embedding) of the train and test splits of every evaluation dataset before retrieval, which cut the pool from 1.1 billion to 744 million. Retrieval then uses the training splits of many evaluation datasets as queries, so the pretraining data is deliberately similar in distribution to those benchmarks, though not to their test images. iNaturalist and Places205 were not used as queries and still improved.
Which output should I use: the [CLS] token or the patch tokens?
Both are trained. The [CLS] token gets the image-level loss and KoLeo and is the feature for classification and retrieval; the patch tokens get the masked-patch loss and are the features for segmentation and depth. The paper's linear probes on ImageNet search over using the [CLS] token alone or concatenated with the average of the patch tokens, and over the last 1 or 4 layers.
How do I get DINOv2 features for my own images?
The official repository publishes the backbones through torch.hub, for example torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14'), with _reg variants that include register tokens. Calling the model returns the [CLS] token after the final LayerNorm (768 numbers for ViT-B); model.forward_features returns a dictionary that also holds the normalized patch tokens, one per 14 by 14 patch, so the image sides should be multiples of 14. No projection head ships with the backbone.
How expensive is it to reproduce?
The paper estimates 22,016 A100 GPU-hours for one ViT-g pretraining run, 9.7 MWh and 3.7 tonnes of CO2-equivalent under a US-average grid, and 0.5 to 1 thousand tonnes for the whole project (about 200,000 GPU-days). The curation pipeline ran on 20 nodes of 8 V100s in under two days.
What is the difference between DINOv2 and CLIP features?
CLIP-style models are trained to match images with their captions, so their features carry what captions describe; DINOv2 is trained on images alone. In the paper's tables the frozen DINOv2 ViT-g beats OpenCLIP ViT-G on fine-grained species classification, instance retrieval, segmentation and depth, and loses on scene classification (Places205, SUN397) and on renditions and sketches (ImageNet-R, Sketch). DINOv2 has no text encoder, so it cannot do zero-shot classification from class names the way the CLIP explainer describes.
Why distill the small models instead of training them directly?
Because it worked better at every size the paper tried. A ViT-L distilled from the frozen ViT-g beat a ViT-L trained from scratch on all 12 benchmarks of Figure 5 (ImageNet 86.3 vs 84.5), and on retrieval it beat the ViT-g teacher itself (76.3 vs 75.2). The distillation reuses the same training loop with the ViT-g as a frozen teacher.
Footnotes & further reading
- The paper: Oquab, Darcet, Moutakanni, Vo, Szafraniec, Khalidov, et al., DINOv2: Learning Robust Visual Features without Supervision (arXiv April 2023; v2 February 2024, TMLR). Official code, the source for the loss implementations, schedules and configurations quoted on this page.
- DINO: Caron et al., Emerging Properties in Self-Supervised Vision Transformers (ICCV 2021); see the DINO explainer for the centering, sharpening and multi-crop details.
- iBOT: Zhou et al., iBOT: Image BERT Pre-Training with Online Tokenizer (ICLR 2022).
- Sinkhorn-Knopp in self-supervision: Caron et al., Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (SwAV, NeurIPS 2020), Section 3.1; Ruan et al., Weighted Ensemble Self-Supervised Learning (ICLR 2023).
- KoLeo: Sablayrolles, Douze, Schmid & Jégou, Spreading vectors for similarity search (ICLR 2019), Section 3.1, Eq. (1) and (2).
- Registers: Darcet, Oquab, Mairal & Bojanowski, Vision Transformers Need Registers (ICLR 2024).
- Related explainers: the Vision Transformer, CLIP, and MAE, whose features need fine-tuning (ImageNet linear probe 76.6% for ViT-H/14 in this paper's Table 4).
How could this explainer be improved? Found an error, or something unclear? I read every message.