Emerging Properties in Self-Supervised Vision Transformers

A network learns image features by predicting the output of a slowly averaged copy of itself, with no labels.

DINO trains a Vision Transformer, the student, to match the softmax output of a teacher whose weights are a moving average of the student's own. Two crops of the same image go in, the student sees one and the teacher the other, and two small operations on the teacher's output keep the model from giving every image the same answer.

The paper behind the attention maps that outline objects nobody labelled, and a k-NN classifier on frozen features that reaches 78.3% on ImageNet.

The question and the two findings

In 2020 the Vision Transformer (ViT) matched convolutional networks on image classification, but it needed more training data and compute to get there, and its features showed nothing a convnet's did not. Language Transformers had taken a different route: BERT and GPT were pretrained on raw text with a self-supervised objective (predict hidden or next words) and fine-tuned afterwards. Self-supervised means the training signal is made from the data itself, with no human labels. Caron and colleagues at Facebook AI Research and Inria asked whether the same would help ViTs: pretrain on ImageNet's 1.28 million images with the labels thrown away, and see what the features can do.

They report two properties that supervised ViTs and convnets do not show as clearly:

1. The attention of the last layer outlines objects. Take the [CLS] token (an extra token whose output summarizes the image) and look at how much it attends to each image patch: the high-attention patches trace the foreground object, although no segmentation label was ever used.
2. The features work as a nearest-neighbour index. Store the frozen features of the ImageNet training set, classify a test image by a vote of its 20 most similar training images, and a ViT-S/8 gets 78.3% top-1, close to the 79.7% of a linear classifier trained on the same features.

The method they built to get there is DINO, for self-distillation with no labels. It has four parts, and the sections below take them in order: a student network trained to match a teacher's softmax output (distillation); many crops of each image, with the teacher seeing only the large ones (multi-crop); a teacher whose weights are a moving average of the student's (the momentum teacher); and two operations on the teacher's output, centering and sharpening, that stop the trivial solution. After that come the evaluations, the attention maps, and which parts the ablations say matter.

The network: a ViT and a projection head

The network $g$ is two pieces, $g = h \circ f$. The backbone $f$ is a standard ViT (or a ResNet-50; DINO works with both). A ViT cuts a 224×224 image into non-overlapping $N\times N$ patches, flattens each and maps it linearly to a vector, adds a position embedding, prepends the learnable [CLS] vector, and runs the sequence through a stack of Transformer blocks. With 16×16 patches that is $14 \times 14 = 196$ patch tokens plus [CLS], 197 tokens; with 8×8 patches it is 785. The paper names models by size and patch: ViT-S/16 is the small model (12 blocks, width 384, 6 attention heads, 21M parameters) with 16-pixel patches; ViT-B is 12 blocks, width 768, 12 heads, 85M parameters. ViT-S was chosen because it resembles a ResNet-50 in size (21M vs 23M parameters) and supervised accuracy (79.8% vs 79.3%).

The output of $f$ is the [CLS] token's final vector, 384 numbers for ViT-S. This is the feature every evaluation uses. During training it passes through the projection head $h$: a 3-layer MLP (384 → 2048 → 2048 → 256, GELU activations), an $\ell_2$ normalization of the 256-dimensional result, and a weight-normalized linear layer from 256 to $K = 65{,}536$ outputs with no bias. That last layer alone has $256 \times 65{,}536 \approx 16.8$ million weights, and the whole head about 22.3 million, more than the ViT-S backbone it sits on. It is discarded after training.

Nothing in the network uses batch normalization. ViTs normalize with LayerNorm, and DINO keeps the head free of BatchNorm too (the paper measured 69.7% k-NN without it and 68.6% with it at 100 epochs). A BatchNorm-free system does not need to synchronize statistics across GPUs, which is one of the things that makes earlier self-supervised methods slow to train.

Distillation, and the output it matches

Knowledge distillation trains a student network $g_{\theta_s}$ to reproduce the outputs of a teacher network $g_{\theta_t}$. Both produce $K$ unnormalized scores (logits) for an input $x$, and a softmax with a temperature turns them into a probability distribution:

The superscript $(i)$ picks one of the $K$ entries. The temperature $\tau_s > 0$ divides every logit before the exponential: a small temperature stretches the gaps between logits and concentrates the probability on the largest ones; a large temperature shrinks the gaps and flattens the distribution toward uniform. The teacher's $P_t$ is computed the same way with its own temperature $\tau_t$. With the teacher fixed, the student minimizes the cross-entropy between the two distributions:

The paper writes $H(a,b) = -a \log b$, with the sum over the $K$ entries implied. Cross-entropy is smallest when $P_s = P_t$, so the gradient pushes the student's distribution toward the teacher's, entry by entry. Its gradient with respect to the student's logits has the form $(P_s - P_t)/\tau_s$, which is the whole signal the student receives.

In Hinton's distillation both sides use the same raised temperature, so the student learns the teacher's relative preferences among wrong classes. DINO uses different temperatures on purpose. The student runs at $\tau_s = 0.1$, the teacher at $\tau_t$ between 0.04 and 0.07, so the teacher's target is more peaked than anything the student produces at its own temperature. A four-entry example with logits $(0.30,\ 0.25,\ 0.10,\ -0.05)$, the scale this head works at: with the code's default normalized last layer, every logit is a cosine similarity in $[-1, 1]$:

The entropy drops from 0.96 to 0.56 nats (the maximum for four entries is $\ln 4 = 1.39$). At temperature 1 the same logits would give $(0.29,\ 0.27,\ 0.24,\ 0.20)$, close to uniform, so a head whose outputs are bounded by 1 needs temperatures well below 1 to produce a peaked distribution. The paper calls the low teacher temperature sharpening; the collapse section shows it is one of the two things that keep training from failing.

Views: multi-crop and the loss

Distillation needs a teacher output to match, and with no labels the only thing to agree on is the image itself. DINO feeds the two networks different random views of the same image and asks the student's output on one view to match the teacher's on another. A view is a random crop, resized, then flipped, color-jittered, blurred or solarized with some probability (the augmentations of BYOL). Two views of one image share their content and differ in everything the augmentations change, so matching them rewards features that describe what is in the image and ignore where the crop landed or what the color balance was.

The views come from multi-crop, a scheme from SwAV. Each image yields two global crops, each covering a large part of the image (the code draws the area fraction from 0.4 to 1) and resized to 224×224, and several local crops covering a small part (0.05 to 0.4) resized to 96×96. A local crop costs much less than a global one: at patch size 16 it is $6\times 6 = 36$ patches plus [CLS], 37 tokens against 197. The teacher sees only the two global crops; the student sees all of them. With $V$ the set of all crops, the loss is

Every teacher crop is paired with every student crop except itself. With 2 global and $L$ local crops that is $2(L+2) - 2 = 2L + 2$ cross-entropy terms per image; the official code averages them instead of summing, which only rescales the loss. For the 10 local crops of the paper's best ViT-S/16 run, each image contributes 22 terms: the two global crops predict each other, and each of the 10 local crops is asked to predict both global ones. Because the teacher only sees crops covering most of the image, a student looking at a 96-pixel piece of it is trained to output what the teacher says about the whole image. The paper calls this local-to-global correspondence.

In Figure 1 the teal rectangles are the global crops and the amber ones the local crops. Click a cell in the pair grid (or press next pair) to see the two crops that term compares, as the networks receive them.

Table 8 of the paper measures what the extra crops buy on ViT-S/16 trained on two 8-GPU machines. With only the two global crops, 300 epochs take 45.9 hours and reach 72.5% linear accuracy. Adding 10 local crops reaches 74.6% after 100 epochs and 24.2 hours, and 76.1% after 300 epochs (72.6 hours), at the cost of peak memory rising from 9.3 GB to 15.4 GB per GPU. Training longer without multi-crop does not catch up. The gain shrinks at the top: going from 6 to 10 local crops adds 0.2 points at 300 epochs.

A teacher built from the student

Distillation assumes a teacher that is already good. DINO has none, so it builds the teacher from the student. The teacher has the same architecture and its own copy of the weights, initialized equal to the student's. It receives no gradient (the stop-gradient in the paper's diagram); after every optimizer step on the student, its weights move a small fraction of the way toward the student's:

This is an exponential moving average (EMA) of the student's weights over training, the momentum encoder of MoCo. The rate $\lambda$ follows BYOL's cosine schedule from 0.996 to 1 over the course of training, recomputed at every iteration $k$ of the $T$ iterations in the run (0.004 is $1 - 0.996$):

At $\lambda = 0.996$ the teacher is a weighted average over roughly $1/(1-\lambda) = 250$ recent student iterations. At batch size 1024, one ImageNet epoch is 1,251 iterations, so early in training the teacher averages over about a fifth of an epoch. By the end $\lambda$ reaches 1 and the teacher stops moving.

Section 5.2 compares teachers on ViT-S/16 after 300 epochs, by k-NN accuracy. A teacher that is an exact copy of the student collapses to 0.1%, and so does the student from the previous iteration. The student from the previous epoch, held fixed for a whole epoch, gets 66.6%, on par with MoCo-v2 and BYOL. The momentum teacher gets 72.8%.

The paper also plots the teacher and the student during training and finds the teacher ahead the whole time, for ViT and for ResNet-50. Its reading is Polyak-Ruppert averaging: the student's weights wander around a good region because every SGD step is taken on a noisy minibatch gradient, and an average of recent weights sits closer to the middle of that region than any single iterate. The paper describes this weight averaging as a way to simulate an ensemble of the recent models. The student is then trained toward the outputs of this better model, and the teacher, built from the student, improves with it. Figure 2 isolates the averaging half of that loop in two dimensions.

In the toy, $\lambda = 0.98$ brings the teacher's mean loss over the last 100 steps to 0.003 against the student's 0.073. The best $\lambda$ depends on the length of the run: a teacher with a 250-iteration memory is too slow for a 400-step toy and right for a training run of hundreds of thousands of iterations. The toy's student ignores the teacher; in DINO the student is pulled toward the teacher's outputs.

Collapse, centering and sharpening

Everything so far admits a trivial solution. If both networks output the same distribution for every image, the student matches the teacher perfectly on every pair of crops and the loss is as low as it can go. The features then say nothing about the image. This is collapse, and every self-supervised method of this kind carries a mechanism against it: a contrastive loss that pushes different images apart (MoCo, SimCLR), a clustering constraint that forces images to spread over clusters (SwAV), a predictor network and batch normalization (BYOL). DINO uses only two operations on the teacher's output.

The paper separates two ways to collapse. In the first, one output dimension dominates for every image: the softmax puts almost all its mass on the same entry regardless of input. In the second, the output is uniform: every entry gets $1/K$ for every image.

Centering targets the first. The teacher keeps a running mean $c$ of its own logits over batches,

with rate $m = 0.9$ and $B$ the batch size (in the code, all images on all GPUs, both global crops), and subtracts it from the teacher's logits before the softmax. The teacher's distribution is

A dimension that is large for every image is large in the batch mean too, so subtracting $c$ removes the part every image shares and keeps only how each image differs from the average. No single entry can then win for all inputs. The center depends only on a batch mean, a first-order statistic, and it is smoothed over batches, which is why DINO keeps working at small batch sizes (Section 5.5).

Centering pushes toward the second kind of collapse. If the network gives every image the same logits, those logits equal the batch mean, and after centering every entry is zero: the softmax of a zero vector is uniform. Sharpening, the low teacher temperature, counters this. Dividing the centered logits by 0.04 multiplies each image's small deviations from the mean by 25 before the softmax, so an image that is slightly above average on some dimension gets a peaked target on it.

Figure 3 applies the two operations to one batch. Every image carries a +1.0 logit on dimension 4, standing in for a component all images share, and a +0.35 logit on a dimension of its own.

A static batch does not show why the balance matters over training, since collapse is a feedback loop. Suppose dimension 4 wins for every image. The student is trained to put its mass on 4 for every crop, the teacher follows the student through the EMA, and dimension 4 wins by a wider margin the next time. The loss goes to zero with the output ignoring the input. The uniform direction has its own loop. The student has to predict the teacher's output on one crop from a different crop, and when a crop is ambiguous (a local crop that could belong to several images) the best prediction is an average over the possible targets, softer than any of them. The teacher averages those softer students, and the targets soften further.

To see which loop is running, the paper splits the loss. The cross-entropy of Eq. (2) is the teacher's entropy plus the Kullback-Leibler divergence from the teacher to the student:

The entropy $h(P_t)$ measures how spread out the target is: 0 for a one-hot target, $\log K$ for a uniform one. The KL measures how far the student is from the target. Collapse of either kind drives the KL to zero, because a student that outputs a constant can match a teacher that outputs the same constant exactly. The entropy tells the two kinds apart: it goes to 0 when one dimension wins and to $\log K$ when the output is uniform. In the paper's Figure 7, removing either operation sends the KL to zero, with the entropy at 0 without centering and at $\log K$ without sharpening.

Figure 4 runs that experiment on a toy: six kinds of image, 10 output dimensions, a network that is a table of logits (one shared bias plus one row per image), crops that are confused with a neighbouring image 35% of the time, and the DINO update with a student temperature of 0.1. Press Play in each setting and watch the entropy and the KL.

In the toy, the working setting ends with an entropy of about 0.6 nats and a KL of about 0.8: the targets are peaked but not one-hot, and the student, working from a confusable crop, never matches them exactly, so the student's gradient never vanishes. The paper's ablations show how narrow the working range is (ViT-S/16, 100 epochs, k-NN): a constant teacher temperature of 0.04 gives 69.6%, 0.06 gives 68.7%, and 0.08 collapses to 0.1% with the training loss sitting at $\ln K = \ln 65{,}536 \approx 11.09$, the cross-entropy of a uniform target. A temperature of 0 (a one-hot argmax target) trains but reaches only 43.9%. The published schedule warms $\tau_t$ from 0.04 to 0.07 over the first 30 epochs (69.7%), since a temperature above 0.06 does not collapse if training starts below it. For the center, $m$ anywhere from 0 to 0.99 works (69.1% to 69.7%); at 0.999 the center moves too slowly to track the logits and the model collapses.

Momentum is the third safeguard, and the others depend on it. Appendix B shows centering alone with a copied teacher collapses (0.1%), while the same network with Sinkhorn-Knopp balancing, SwAV's heavier normalization, reaches 71.8% without momentum. With the momentum teacher, centering is enough (76.1% linear).

One training step, with shapes

Put the pieces together for ViT-S/16 at the paper's batch size of 1024, spread over 16 GPUs as 64 images each. Every image gives 2 global crops and 10 local crops. The student runs on all 12 crops and the teacher on the 2 global ones; each network output is a row of 65,536 logits. Per GPU the step computes:

# One DINO step on one GPU (ViT-S/16, 64 images; 16 GPUs make batch 1024)
# crops: 2 global (3x224x224) + 10 local (3x96x96) per image
g   = [aug_global(x), aug_global(x)]           # 2 tensors, each 64x3x224x224
l   = [aug_local(x) for _ in range(10)]         # 10 tensors, each 64x3x96x96
s   = [head_s(vit_s(v)) / 0.1 for v in g + l]   # 12 x (64 x 65536) logits
with no_grad():
    zt = [head_t(vit_t(v)) for v in g]          # 2 x (64 x 65536), no gradient
    pt = [softmax((z - c) / tau_t) for z in zt]  # center, then sharpen
loss, n = 0, 0
for i, p in enumerate(pt):                      # teacher crop i
    for j, sj in enumerate(s):                  # student crop j
        if j == i: continue                     # skip a crop paired with itself
        loss += (-p * log_softmax(sj)).sum(-1).mean(); n += 1
loss = loss / n                                  # n = 2*12 - 2 = 22 terms
loss.backward(); adamw_step(student)            # gradient reaches the student only
theta_t = lam * theta_t + (1 - lam) * theta_s   # teacher: EMA of the student
c = 0.9 * c + 0.1 * all_gpus_mean(cat(zt))      # center: EMA of raw teacher logits

The gradient flows from the loss into the student's head and backbone only; the teacher's probabilities are treated as constants. The optimizer is AdamW. The learning rate warms up linearly over 10 epochs to $0.0005 \times \text{batch}/256$, 0.002 at batch 1024, then decays on a cosine schedule; weight decay rises on a cosine schedule from 0.04 to 0.4. The official code also clips gradients at norm 3 and keeps the last layer of the head frozen during the first epoch. The center update uses the teacher's raw logits, before centering, averaged over all 2,048 global crops in the global batch.

One practical note from the code: the default teacher_temp is 0.04 with no warm-up, and the paper's 0.04 → 0.07 schedule is the README's "boosting" recipe, which sets --teacher_temp 0.07 and --warmup_teacher_temp_epochs 30. The README's 100-epoch ViT-S/16 run on one 8-GPU node takes 1.75 days and reaches 69.3% k-NN and 74.0% linear.

Evaluating features without training on them

A self-supervised model is judged by what its frozen features support. The standard test is linear evaluation: freeze the backbone, train one linear layer on ImageNet's labels on top of the features, report top-1 accuracy. The paper finds this protocol sensitive to its own learning rate and adds a second one with nothing to tune, a weighted k-nearest-neighbour (k-NN) classifier. Compute the $\ell_2$-normalized [CLS] feature of every training image once. For a test image, find the $k$ training features with the largest cosine similarity $s_i$ and let them vote, each with weight $\exp(s_i/\tau)$, where $\tau = 0.07$, $k = 20$, and $\mathcal{N}_k$ is the set of neighbours:

The weight grows fast with similarity. A neighbour at cosine 0.9 counts $e^{0.1/0.07} \approx 4.2$ times as much as one at 0.8. The protocol needs no training, no augmentation and one pass over the data.

On ViT-S (800 epochs), DINO gets 77.0% linear and 74.5% k-NN. The same backbone trained with BYOL, MoCo-v2 and SwAV by the authors (BYOL for 300 epochs, since it got worse when trained longer) gets 71.4 / 66.6, 72.7 / 64.4 and 73.5 / 66.3. The k-NN margin (7.9 points over the best of them) is more than twice the linear margin (3.5 points). On ResNet-50, DINO is level with the best prior method (75.3% linear, like SwAV; 67.5% k-NN), and its k-NN accuracy sits 7.8 points below its linear accuracy, the usual gap for convnets. The small gap between k-NN and linear is specific to DINO with a ViT. With 8×8 patches, ViT-S/8 reaches 79.7% linear and 78.3% k-NN, and ViT-B/8 80.1% linear, against a supervised ViT-S at 79.8%.

A good k-NN score means images of the same class already sit near each other in feature space, so the features can index a database with no classifier trained on top. The paper tests this directly on retrieval. On the revisited Oxford and Paris landmark benchmarks, DINO ViT-S/16 features pretrained on ImageNet beat supervised ViT-S/16 features (41.8 vs 33.5 mAP on Oxford, medium split), and pretrained instead on 1.2 million Google Landmarks images, with no labels, reach 51.5, above the best previously published off-the-shelf descriptor (49.8). On copy detection (INRIA Copydays, strong subset, with 10,000 distractors) ViT-B/8 features at 320×320 get 85.5 mAP.

Attention maps that segment objects

In a ViT, each attention head of each block computes, for every token, a softmax distribution over all tokens: how much of each token's value vector to mix in. Take the last block, the [CLS] token as the query, and drop the [CLS]-to-[CLS] entry. What is left is one weight per image patch, a map the size of the patch grid, for each of the heads (6 in ViT-S). For a 480×480 image at patch size 8 the sequence is $60 \times 60 + 1 = 3601$ tokens and each map is 60×60.

In DINO ViTs these maps light up on the foreground objects, and different heads pick out different objects or parts, including small or partly hidden ones. To measure it, the paper turns a map into a mask: sort the patches by attention weight and keep the smallest set whose weights add up to a fixed share of the total (60% in Figure 4 of the paper; the official visualize_attention.py does this with a sorted cumulative sum). Then it compares the mask with the ground-truth object masks of PASCAL VOC 2012 by the Jaccard index, the area of the intersection divided by the area of the union. Figure 6 walks through that procedure.

On ViT-S/16, masks from the best head score a Jaccard of 45.9 for DINO, 27.3 for the same network trained with labels, and 22.0 for random weights (ViT-S/8: 44.7, 23.7, 21.8). The paper's examples show the supervised ViT attending poorly to the object in cluttered scenes. Its motivation points at the training signal: an image-level label reduces a picture to one concept out of a fixed list, and any feature that predicts that concept is rewarded, while matching two views of the same image rewards describing whatever the views share.

The property is shared across self-supervised methods. The paper's appendix trains MoCo-v2, BYOL and SwAV on the same ViT-S/16 and gets 46.3, 47.8 and 46.8, all at or above DINO's 45.9; DINO without multi-crop gets 45.1. DINO stands apart from those methods on k-NN accuracy (Table 2 on ViT-S: 74.5% against at most 66.6%).

The patch features carry spatial information beyond the attention. On the DAVIS-2017 video object segmentation benchmark, the paper propagates the first frame's mask through the video by nearest-neighbour matching of patch features between consecutive frames, with no training for the task. DINO ViT-S/8 scores 69.9 (the mean of region and contour accuracy), above a supervised ViT-S/8 at 66.0, and ViT-B/8 scores 71.4.

Smaller patches

Patch size changes the results without changing the parameter count. A 16×16 patch on a 224 image leaves a 14×14 grid; an 8×8 patch leaves 28×28, four times as many tokens, and self-attention compares every token with every other, so its cost grows with the square of the token count. Figure 7, from the ViT explainer, shows the count and the attention cost across patch sizes.

The parameter count barely moves. The Transformer blocks do not depend on sequence length; the patch embedding shrinks (an 8×8 RGB patch is 192 numbers instead of 768, so about 221,000 fewer weights at width 384); the position embedding gains 588 rows of 384, about 226,000 more. The paper lists both ViT-S models at 21M. Going from /16 to /8, ViT-S goes from 77.0% to 79.7% linear and from 74.5% to 78.3% k-NN, and on DAVIS ViT-B goes from 62.3 to 71.4. Speed drops: inference throughput on a V100 falls from 1,007 images per second for ViT-S/16 to 180 for ViT-S/8, and to 44 with 5×5 patches. ViT-B/8 runs at 63 images per second, which is still 1.4 times faster than the previous best self-supervised model on the linear benchmark, a ResNet-152 three times wider with 794M parameters (79.8%), at a tenth of the parameters.

Which components matter

Table 7 adds and removes one component at a time on ViT-S/16 (300 epochs; k-NN / linear). The default DINO gets 72.8 / 76.1.

• No momentum teacher: 0.1 / 0.1, collapse.
• Sinkhorn-Knopp balancing added on top of the momentum teacher: 72.2 / 76.0, no gain.
• No multi-crop: 67.9 / 72.5.
• Mean squared error in place of cross-entropy (BYOL's loss): 52.6 / 62.4.
• A predictor MLP on the student (BYOL's asymmetry): 71.8 / 75.6, slightly worse.

The authors' own runs of the reference methods under the same schedule give 66.6 / 71.4 for BYOL, 62.0 / 71.6 for MoCo-v2 and 64.7 / 71.8 for SwAV. Multi-crop is not a generic add-on either: with 6 local crops, MoCo-v2 gains and BYOL loses (66.6 → 59.8 k-NN), and DINO gains most (72.5 → 75.9 linear). Other settings: more attention heads help a little at the same width (6 heads 72.8, 16 heads 73.8 k-NN), longer training helps (100, 300, 800 epochs: 70.9, 72.8, 74.5 k-NN), and batch size matters little (k-NN at 100 epochs without multi-crop: 57.9 at batch 128 on one GPU, 59.9 at 1024).

Fine-tuned on downstream classification datasets, DINO-pretrained ViTs beat supervised pretraining on ImageNet itself (ViT-S/16: 81.5% vs 79.9%; ViT-B/16: 82.8% vs 81.8%) and on most of the six transfer datasets, with ViT-B/16 on iNaturalist 2018 the exception (72.6% vs 73.2%).

Limits and open points

The collapse defenses are narrow. A constant teacher temperature of 0.08 or a center rate of 0.999 each end in total collapse, and the paper's explanation of why centering plus sharpening suffices with a momentum teacher is empirical, from the entropy and KL curves, with no proof. The paper expects small batches to need re-tuned momentum rates, and the code suggests an EMA rate of 0.9995 at batch 256 instead of 0.996.

The best features come from the smallest patches, which are the slowest to run: 180 images per second for ViT-S/8 against 1,007 for ViT-S/16. The segmentation evidence is one metric on one dataset (PASCAL VOC masks from the best head, at a mass threshold stated as 60% in one place and 80% in another), plus pictures, and it does not separate DINO from other self-supervised methods. The attention map is also not trained to be a mask; the paper describes the maps as smooth and not optimized to produce one, and names weakly supervised segmentation as a possible use.

Apart from the landmark-retrieval run, every model is pretrained on ImageNet, a curated dataset with one main object per image. The paper's conclusion names training larger ViTs on uncurated images as the next step. The follow-up, DINOv2 (2023), instead built a curated dataset automatically and scaled the ViT to 1 billion parameters. Other directions for self-supervised ViTs followed within a year or two: MAE reconstructs masked pixels, and I-JEPA predicts the features of masked regions.