Masked Autoencoders Are Scalable Vision Learners

Hide most of an image, learn to reconstruct the missing patches, and the encoder becomes a strong vision model.

Most self-supervised vision pre-training either matches two augmented crops of an image or predicts discrete tokens. MAE does the plainest thing: mask random patches, reconstruct the missing pixels. Two design choices make it scale. The encoder processes only the patches that are still visible, and the masking ratio sits near 75 percent.

A vanilla ViT-Huge, pre-trained this way on ImageNet-1K alone, reaches 87.8% top-1. No labels, no extra data, no augmentation pipeline. And the encoder does it while looking at a quarter of each image at a time.

Language models learned to read by playing fill-in-the-blank. Hide some words, predict them from the rest, and a model trained that way on enough text turns out to have learned grammar, facts, and a good deal of reasoning, all without a single label. Its internal features then transfer to almost any language task. The recipe is old and simple, and it made GPT and BERT possible. Vision spent years trying to copy it and mostly could not. Convolutional networks, which dominated the field, have no natural way to represent a “this patch is missing” marker dropped into a regular grid of pixels. So image pre-training stayed supervised, hungry for the hundreds of millions of hand-labeled images that only a few labs have.

MAE, from Kaiming He and colleagues at FAIR, is the paper that made fill-in-the-blank finally work for images, and work at scale. It cuts an image into patches, hides a large random fraction of them, and trains a network to reconstruct the missing pixels. What comes out is a plain Vision Transformer encoder that, once fine-tuned, beats every prior result trained on ImageNet alone, and improves the more you scale the model up.

A handful of choices explain the method, and each one is a design decision you could have argued for from scratch: what gets hidden, what the encoder is allowed to see while it learns, where the mask markers are allowed to enter, and how you read the trained model out at the end. The rest of this piece takes them one at a time.

The idea: hide most of the image

Start with the thing MAE is a member of: an autoencoder. An autoencoder is a network in two halves. An encoder compresses an input into a small internal code, and a decoder expands that code back into the original input. Train the pair to reproduce their own input and the code has to capture whatever is worth keeping, since the decoder has nothing else to work from. You keep the trained encoder: a learned way to turn raw data into a compact, useful vector.

MAE is a masked autoencoder, meaning the input it is handed is a damaged version of the image, with most patches removed, and the target it must reconstruct is the full image. The word self-supervised describes exactly this arrangement: the training signal comes from the data itself (the pixels you hid are the answer key) rather than from a human label. Nobody had to annotate anything, so any pile of unlabeled images is a training set.

MAE hides a surprising amount: about 75 percent of the patches. The exact fraction matters enough that it gets its own section below. For now hold the shape of the method: cut the image into patches, throw away three-quarters of them, reconstruct what is missing, and keep the encoder.

Autoencoders, corrupted on purpose

Why corrupt the input at all? Because a plain autoencoder has a lazy escape route. If the internal code is wide enough, the encoder can pass the input straight through and the decoder can copy it back out, learning the identity function: zero reconstruction error, nothing useful. The usual fix is to force a narrow bottleneck so copying is impossible, but that caps how much the code can represent.

The denoising autoencoder, introduced by Vincent and colleagues in 2008, removes the escape route a different way: damage the input first, then ask for the clean original back. Now the input and the target are not the same object, so copying does not work, and to fill in what was destroyed the network has to learn how the pieces of an input relate. A common kind of damage picks a random fraction of the input entries and sets them to zero. Vincent's 2010 paper names this masking noise. Masking, in other words, is not a new idea invented for BERT; it is one corruption in a toolbox that also includes adding Gaussian noise and flipping random values.

MAE says as much about itself: it is “a form of denoising autoencoding.” The lineage runs straight through. A classical autoencoder needs a bottleneck to avoid copying; a denoising autoencoder gets that for free by corrupting the input; and MAE is masking noise applied to the patches of an image, with a Transformer doing the encoding. What MAE changes, and what earns it a paper rather than a footnote, is the three decisions that follow: how much to mask, what the encoder is shown, and where the mask markers go.

BERT's trick, and two changes for vision

The closest ancestor is BERT, which brought masked prediction to language. BERT hides about 15 percent of the words in a sentence and trains a Transformer to predict them from the words left standing, scoring itself only on the hidden positions. (The details are finicky: of that 15 percent, 80 percent become an actual [MASK] token, 10 percent a random word, and 10 percent are left unchanged, a split that keeps the model from over-relying on a marker it will never see once it is fine-tuned. So only about 12 percent of all tokens ever become the real [MASK].) BERT's features became the default starting point for language tasks almost overnight.

Porting this to images runs into a difference the paper keeps returning to: information density. A sentence is dense. Every word carries meaning, so blanking even one forces the model to reason about the rest. An image is not dense in the same way. It is recorded light, heavy with spatial redundancy, and a missing 16-by-16 patch can usually be guessed by extending the lines and textures of its neighbors, with no understanding of what object it belongs to. Mask 15 percent of an image and you have posed a coloring-in exercise, not a comprehension test.

So MAE makes two changes. The first: mask a very high fraction, around 75 percent, so a patch's neighbors are usually gone too and filling the hole needs a more global read of the scene. The second change is structural, and it is the one that makes MAE more than a port of BERT: keep the mask markers out of the encoder entirely. In BERT the [MASK] tokens flow through the entire Transformer stack. In MAE the encoder never sees a mask token at all. It processes only the real, visible patches, and the markers for the holes are introduced later, in a separate decoder. The next section builds out that asymmetry: an encoder on the visible patches, a small decoder on everything.

The pipeline, patch by patch

Trace the full forward pass with concrete shapes. Take a 224-by-224 image. ViT-Large uses 16-by-16 patches, so the image becomes a 14-by-14 grid: $196$ patches, each holding $16\times16\times3 = 768$ raw pixel numbers. A single shared linear layer maps each patch to a 1024-dimensional token, and a fixed position vector is added to record where each patch sat.

Now the masking, done with a trick that needs no special sparse operations. Shuffle the 196 tokens into a random order and keep the first $49$ of them, a quarter. Those are the visible patches; the other $147$ are simply dropped. The encoder, a full 24-layer ViT-Large, runs on those 49 tokens and nothing else. Two things follow: the encoder does a quarter of the work, and it never encounters a placeholder.

To reconstruct, the holes have to come back. Project the 49 encoded vectors down to the decoder's width of 512, then append $147$ copies of a single shared, learned mask token, one stand-in for each missing patch. Unshuffle all 196 back into the original grid order so every slot lines up with the patch it represents, add the decoder's own position vectors (without them the decoder could not tell the identical mask tokens apart), and run eight lightweight decoder blocks. A final linear layer emits 768 numbers per slot: the reconstructed pixels. The loss compares them to the true pixels, but only on the 147 masked patches.

# one MAE forward pass (adapted from the official models_mae.py;
# the [cls] token is dropped here for clarity)
def forward(imgs, mask_ratio=0.75):            # imgs: [N, 3, 224, 224]
    x = patch_embed(imgs)                      # [N, 196, 1024] patch tokens
    x = x + pos_embed                          # fixed 2D sin-cos positions

    # shuffle, keep the first 25% -> sampling without replacement
    ids  = argsort(rand(N, 196))               # a random order per image
    keep = ids[:, :49]                          # 49 = round(196 * 0.25)
    x = gather(x, keep)                         # [N, 49, 1024] visible only
    x = encoder(x)                             # ViT-L on 49 tokens, no [M]

    # put the holes back: append shared mask tokens, undo the shuffle
    x = decoder_embed(x)                       # [N, 49, 512]
    x = cat([x, mask_token.repeat(N, 147, 1)]) # [N, 196, 512]
    x = unshuffle(x, ids)                      # realign to patch order
    x = x + decoder_pos_embed
    pred = decoder(x)                          # [N, 196, 768] pixels/patch
    return pred

Two details in that code repay a closer look. The shuffle-and-keep-first-49 step is exactly “sample 49 patches at random without replacement”, done with a sort instead of a gather over a sparse mask, so it runs fast on ordinary dense tensors. And the mask token is a single learned 512-dimensional vector, repeated 147 times; the decoder tells the copies apart only by the position vector added to each. The position embeddings here are fixed 2D sine-cosine patterns, computed once from the grid coordinates and never trained, on both the encoder and the decoder. (Standard ViT, by contrast, learns its position embeddings; MAE does not inherit that choice.)

The loss is as simple as it gets: mean squared error between predicted and true pixels, averaged over the masked patches. Write $\mathcal{M}$ for the set of masked patches and $x_i,\hat{x}_i$ for a patch's true and predicted pixels:

The visible patches were handed to the model, so scoring it on reproducing them would only reward copying; computing the loss on every pixel instead costs about half a point of final accuracy. An optional refinement helps: normalize each target patch by its own mean and standard deviation before comparing. That sharpens local contrast and keeps the high-frequency detail the plain target would smear, and it lifts accuracy a little. (The code toggles it with norm_pix_loss.)

# loss: pixel-space MSE on the masked patches only (BERT-style)
def loss(imgs, pred, mask):                    # mask: 1 = masked, 0 = seen
    target = patchify(imgs)                     # [N, 196, 768] true pixels
    if norm_pix_loss:                           # per-patch normalized target
        mean = target.mean(-1, keepdim=True)
        var  = target.var(-1, keepdim=True)
        target = (target - mean) / (var + 1e-6) ** 0.5
    err = ((pred - target) ** 2).mean(-1)       # [N, 196] per-patch MSE
    return (err * mask).sum() / mask.sum()      # average over masked only

Why 75 percent

The masking ratio produces the paper's signature result, and language would not have predicted it. Where BERT masks 15 percent, MAE's sweet spot is a startling 75 percent, and it comes back to the spatial redundancy from two sections ago. Erase a little and the task collapses to copying from neighbors. Erase most of the image and there is no nearby patch to copy from, so the model has to infer the missing content from the object and scene, which forces it to learn something useful.

The two evaluation numbers on the figure below deserve a word first, because the rest of the piece rests on them. Fine-tuning means you take the pre-trained encoder and keep training the entire network, weights and all, on the labeled task; it measures how good a starting point the pre-training gives. Linear probing means you freeze the encoder and train only a single linear layer on top; it measures how linearly separable the frozen features already are, without any further learning in the network. Watch how differently the two respond to the masking ratio.

Fine-tuning is forgiving: anywhere from 40 to 80 percent masking, the final accuracy barely moves, and every point on the curve beats training the same ViT-Large from scratch. Linear probing is fussier, rising steadily to a clear peak at 75 percent, with a span of roughly 20 points between the low and high ends. A ratio of 75 percent happens to be near-best for both, which is why the paper picks it. It also means the encoder sees a quarter of each image, so the training is cheap, a coincidence the paper calls a win-win.

Slide the control to the far ends and the story holds. Near 10 percent the linear probe bottoms out: the task is easy, the features are weak. Push past 80 and the reconstructions get worse and both curves start to drop, since eventually too little is left to infer from. The interesting middle is wide and flat for fine-tuning and sharply peaked for linear probing, and the gap between those two behaviors is a hint the last section picks up.

Keep the placeholder out of the encoder

Return to the structural choice from the BERT section: the encoder never sees a mask token. That choice separates MAE from a naive port of BERT, and it does two things at once.

Picture the alternative, the BERT-style version, where the mask tokens do flow through the encoder. During pre-training that encoder's input is 49 real patches plus 147 placeholders, so three-quarters of what it reads is a token that stands for “something was here.” At deployment, when you actually use the encoder for recognition, you feed it a whole, uncorrupted image: 196 real patches and zero placeholders. The encoder was trained on inputs that look nothing like the ones it is asked to handle in the end. That mismatch degrades the features. MAE removes it by construction: its encoder only ever sees real patches, in training and in deployment alike.

Feeding the placeholders through the encoder drops linear-probe accuracy from 73.5 to 59.6 percent, a 14-point fall, while fine-tuning barely moves. The gap hits the frozen features hardest, exactly because fine-tuning can retrain the last layers to paper over the mismatch and linear probing cannot.

Skipping the placeholders pays off in compute too. An encoder that skips them processes a quarter of the tokens, which cuts the pre-training FLOPs by about 3.3 times and the wall-clock time by 2.8 times for the default configuration, more for bigger encoders or smaller decoders. The token count enters twice: the linear, per-token work (the projections and the feed-forward layers) scales with the number of tokens, so a quarter of them is a $4\times$ floor, and the self-attention map, which pairs every token with every other, scales with the square, so its cost drops by $16\times$. The two combine to a saving strictly above four times, though how far above depends on the resolution; at ViT-Large's size the per-token work is actually the larger share, so the realized speedup grows as you scale to more patches.

The decoder you discard

The decoder exists only during pre-training. Once training is done it is thrown away and only the encoder is kept. That freedom lets the decoder be small, and the paper's default is a lightweight stack: eight blocks at 512 dimensions, under 10 percent of the encoder's per-token compute. Since it runs on all 196 tokens rather than 49, keeping it cheap matters.

Small does not mean careless, though, because the decoder's depth sets how abstract the encoder's features become. The depth works by dividing labor. Reconstructing exact pixels is a low-level job, and if the decoder is shallow the encoder has to do that job itself, dragging its representation down toward pixels. Give the decoder more depth and it can absorb the reconstruction specialization, which leaves the encoder free to hold a more abstract representation, the kind a frozen linear probe can read.

Fine-tuning is nearly indifferent to decoder depth: it reaches 84.8 percent even with a one-block decoder, because it can retune the encoder's last layers on the target task regardless. The frozen linear probe cannot retune anything, so decoder depth moves it a full 8 points, from 65.5% at one block to 73.5% at the default eight. This is the same linear-probing-versus-fine-tuning split the masking-ratio figure hinted at, seen from a second angle, and it says something the next section makes central: how you evaluate a frozen encoder can disagree sharply with how good it actually is once you let it learn.

Linear probing is not the whole story

Linear probing became the default yardstick for self-supervised vision because it is clean: freeze the features, fit one linear layer, report accuracy. But it only measures one thing, whether a single straight cut separates the classes, and the paper shows that this can be nearly uncorrelated with how well the same encoder does once you let a little non-linearity in.

The paper demonstrates this with partial fine-tuning: unfreeze just the last few Transformer blocks and tune those, leaving the rest frozen. Zero blocks is linear probing; all 24 is full fine-tuning. The compelling comparison is against MoCo v3, a contrastive method, which learns by pulling together the embeddings of two augmented crops of one image and pushing apart crops of different images. Contrastive methods top the linear-probing charts, and MoCo v3 does here too.

At zero blocks MoCo v3 wins, 77.6 to 73.5. Unfreeze a single block and MAE leaps to 81.0 percent, a 7.5-point jump from one block of tuning, and passes MoCo. From there MAE leads everywhere: by 2.6 points at four blocks, and up to full fine-tuning. Even tuning half of the last block, just its feed-forward sub-layer, reaches 79.1 percent, comfortably above the linear probe. MAE's features are less linearly separable and, at the same time, stronger once any non-linear head is allowed.

Scale, and what transfers

The paper is titled “scalable” because the gains grow with model size, the way they do for large language models and rarely had for vision. Fine-tuned on ImageNet-1K alone, a ViT-Large reaches 85.9 percent and a ViT-Huge 86.9 percent at the usual 224-pixel resolution; push the Huge model to 448 pixels and it hits 87.8 percent, the best result anyone had reported using only ImageNet-1K, past a previous best of 87.1 that relied on a more elaborate architecture. MAE gets there with a completely vanilla ViT. And the bigger the model, the larger the margin over training from scratch, which is the property that lets you keep spending compute and keep getting returns.

Two more results round it out. The features transfer: swap the pre-trained encoder into an object detector on COCO and a ViT-Large reaches 53.3 box AP against 49.3 for the same model pre-trained with supervision, a 4-point gain; semantic segmentation on ADE20K improves by 3.7 points the same way. And it needs almost no data augmentation. Contrastive methods lean hard on it (BYOL loses 13 points and SimCLR 28 when you strip augmentation back to cropping), because two augmented views of one image are their entire training signal. MAE uses only random cropping, and works nearly as well with none at all, because the random mask is different every iteration and supplies all the variety the model needs.

What MAE settled is that the language recipe was never the problem for vision. Fill-in-the-blank works on pixels once you mask most of the image and keep the placeholder out of the encoder. The encoder you keep is a plain Transformer, cheaper to train than the contrastive methods it beats, that gets better as it gets bigger. The line of work it opened, from pixel targets to the embedding targets of I-JEPA, is still arguing about what exactly the model should be asked to predict. MAE's answer, reconstruct the raw pixels and keep it simple, set the baseline everyone else now has to beat.