V-JEPA: Revisiting Feature Prediction for Learning Visual Representations from Video

A video encoder learns by predicting the features of hidden parts of a clip, with no labels, text, or pixel targets.

An encoder sees a video with large regions removed. A predictor guesses what a slowly updated copy of the same encoder outputs for those regions. The prediction error trains the encoder and the predictor, and the copy only follows the encoder.

Trained on 2 million unlabeled videos, the largest V-JEPA model reaches 81.9% on Kinetics-400 and 72.2% on Something-Something-v2 with its weights frozen. This page walks one training step with real tensor shapes, runs a small model that collapses and one that does not, and checks the paper's tables.

V-JEPA (Video Joint-Embedding Predictive Architecture) is a recipe for pretraining a video encoder from raw video. Take a 16-frame clip, delete most of it, and ask a network to predict the internal representation the encoder would produce for the deleted part. The target is a vector of features at each hidden position, not the hidden pixels. The paper, from Meta FAIR with Inria, ENS, Université Gustave Eiffel and NYU, asks whether this feature-prediction objective works on its own for video, without pretrained image encoders, text, negative examples, reconstruction, or any other supervision.

The paper answers with frozen-backbone numbers: after pretraining, the encoder's weights are never changed again, and only a small classifier on top is trained per task. A Vision Transformer of the Huge size (ViT-H/16, 16-pixel patches), pretrained at 384-pixel resolution, reaches 81.9% top-1 accuracy (the share of test clips whose highest-scoring class is the right one) on Kinetics-400, 72.2% on Something-Something-v2 and 77.9% on ImageNet-1K. Kinetics actions can often be named from the objects in view; Something-Something-v2 labels such as "pushing something from left to right" are decoupled from object appearance and need motion.

Five pieces explain the method: the training step and its loss, the choice of features over pixels as the target, the stop-gradient and moving-average copy that keep the encoder from outputting a constant, the mask that decides what to hide, and the attentive probe used to read out a frozen encoder. The results section then goes through the paper's comparisons with image models, pixel-prediction video models, and label-scarce training.

What V-JEPA learns from

Three families of self-supervised methods dominated vision before this paper, and V-JEPA uses none of them. Contrastive learning trains an encoder to map two augmented views of the same image or clip close together and views of different ones far apart. The "far apart" examples are called negatives, and the method needs many of them per step. V-JEPA's loss contains no negatives: nothing in it compares one clip to another.

Language supervision, as in CLIP, trains an image or video encoder to match the embedding of a caption. It transfers knowledge from text, and it needs paired text. V-JEPA sees only video.

Masked reconstruction, as in MAE and its video version VideoMAE, hides patches and trains a decoder to output their pixels. V-JEPA hides patches the same way and replaces the pixel target with a feature target.

Without a reconstruction to look at, the paper measures quality the standard self-supervised way: freeze the pretrained encoder, train a small classifier on its outputs for each downstream task, and report accuracy. A higher score means the frozen features already contain the information the task needs. The paper also reports full fine-tuning, where every weight is retrained, but frozen evaluation is its main protocol.

Pretraining data is VideoMix2M, the union of HowTo100M, Kinetics-400/600/700 and Something-Something-v2 with any overlap with their validation sets removed: about 2 million videos. Their labels are never used during pretraining.

How V-JEPA works: one training step

A JEPA has three parts. An encoder $E_\theta$ maps an input to a representation. A predictor $P_\phi$ maps the representation of one input $x$ to a guess of the representation of a related input $y$, given a variable $z$ that says how $x$ and $y$ relate. In V-JEPA, $x$ is the visible part of a clip, $y$ is the hidden part, and $z$ is the list of space-time positions of the hidden tokens, written $\Delta_y$. The training objective is equation (1) of the paper:

The norm is an L1 distance, averaged over every hidden token and every feature dimension. $\bar E_\theta$ is a second copy of the encoder, the y-encoder, whose weights are an exponential moving average (EMA) of the trained encoder's weights: after each optimizer step every weight is set to $\tau$ times its old value plus $1-\tau$ times the trained encoder's current value. $\operatorname{sg}$ is a stop-gradient: in the backward pass the target side is treated as a constant, so the loss changes $\theta$ only through the prediction side.

The paper's Figure 3 and Appendix B pin down what each part computes. Follow one clip through a ViT-L/16 (24 transformer blocks, width 1024):

1. Tokenize. Sample 16 frames with a stride of 4 (about 3 seconds of video) at 224×224. A 3D convolution cuts the clip into 16×16-pixel patches two frames deep, giving an 8×14×14 grid: 1568 tokens of width 1024, each with a fixed 3D sin-cos position embedding. There is no [cls] token.
2. Mask. Draw two masks for the clip, a short-range and a long-range one (the masking section below explains both). A long-range draw might hide 1280 of the 1568 tokens and keep 288.
3. Encode the context. The x-encoder, which is $E_\theta$, runs on the 288 visible tokens only and outputs a 288×1024 matrix. The hidden tokens are deleted from its input, not replaced with placeholders, so the encoder never processes them.
4. Predict. The predictor is a narrow transformer: 12 blocks of width 384, with as many attention heads as the backbone. It projects the 288 context vectors to width 384 and appends 1280 mask tokens, each a learnable vector plus the sin-cos position of one hidden patch. After the 12 blocks it keeps the 1280 outputs at the mask tokens and projects them back to width 1024.
5. Build the targets. The y-encoder $\bar E_\theta$ runs on all 1568 tokens of the unmasked clip without gradients. The loss uses its outputs at the 1280 hidden positions. Because the y-encoder saw the whole clip, each target vector already reflects the patch's surroundings; the paper calls these contextualized targets, following data2vec.
6. Update. The L1 loss over 1280×1024 numbers, averaged over the two masks, is backpropagated into the predictor and the x-encoder, which AdamW updates. The y-encoder is then moved toward the x-encoder by the EMA rule.

The x-encoder and predictor run once per mask; the y-encoder runs once per clip and both masks reuse its output, which the paper uses to amortize the cost of computing targets. Click a box or a stage in Figure 1 to see each tensor's shape; the loss stage shows which modules receive gradients.

In code, the step looks like the listing below. It follows the official implementation, which adds one detail Eq. (1) leaves out: the y-encoder's outputs pass through a layer norm with no learned parameters (each token's 1024 features are shifted to mean 0 and scaled to variance 1) before the loss. The hidden-token count changes from draw to draw; 288 and 1280 are one example.

# One V-JEPA pretraining step, ViT-L/16 at 224 (shapes for one clip).
clip = sample_clip()                    # 16 frames, stride 4, 224x224x3
tok = patchify(clip) + pos3d            # [1568, 1024]  (8 x 14 x 14 tubelets)

with no_grad():                         # y-encoder: EMA copy, full clip
    h = layer_norm(y_encoder(tok))      # [1568, 1024]

loss = 0
for m in (short_range_mask(), long_range_mask()):  # 2 masks per clip
    ctx_idx, tgt_idx = m.visible, m.hidden         # e.g. 288 and 1280
    z = x_encoder(tok[ctx_idx])                    # [288, 1024]
    pred = predictor(z, ctx_idx, tgt_idx, mask_token[m.kind])
    loss += (pred - h[tgt_idx]).abs().mean()       # L1 on [1280, 1024]
loss = loss / 2

loss.backward()                         # grads: x_encoder and predictor only
adamw.step()
tau = 0.998 + 0.002 * it / 112_500      # momentum schedule; training stops
for p_y, p_x in zip(y_encoder.params, x_encoder.params):  # at it = 90_000
    p_y.data = tau * p_y.data + (1 - tau) * p_x.data

Optimization uses AdamW, with a learning rate warmed up from 2×10⁻⁴ to 6.25×10⁻⁴ over 12,000 iterations and then cosine-decayed, and weight decay raised from 0.04 to 0.4; training runs 90,000 iterations at batch 3072 (2400 for the 384-resolution model). All schedules are laid out for 112,500 iterations and cut at 90,000, because the authors found the last quarter of a schedule changed hyperparameters too aggressively; the learning rate therefore ends near 7.5×10⁻⁵ instead of the nominal 10⁻⁶. Augmentation is limited to random resized crops and horizontal flips.

Equation (1) has no reconstruction term, no comparison with other clips, and no text. With nothing else in the loss, an encoder that outputs the same vector for every input, and a predictor that outputs that vector too, would score zero. The next two sections cover why the target is features at all, and why training does not slide into that constant solution.

Why predict features instead of pixels

A patch of video contains detail that its surroundings do not determine: the grain of a carpet, sensor noise, the exact pattern of leaves. A model asked to reproduce the pixels of a hidden patch pays for every such detail it cannot guess, and the best it can do is output the per-pixel conditional average (or, under an L1 loss, the conditional median). That residual error never goes away, however good the model gets. A feature target can omit those details: the encoder that produces the targets is trained by the same loss, and the paper argues that predicting in representation space gives the model the flexibility to drop irrelevant or unpredictable pixel-level detail from the target.

Figure 2 makes that concrete on a 10×10 patch built from a soft edge, which the surrounding context determines, plus random texture, which it does not. Watch the readout as the texture slider moves: with the pixel target, the best possible loss rises in step with the texture, and with the feature target it stays at zero.

At texture 0.6 the pixel floor is 0.152 per pixel, 65% of the 0.236 that a flat-gray guess pays. In a real network that floor shows up as loss the model keeps trying to reduce and cannot, and the paper argues that pixel models spend capacity and compute chasing it.

Table 1 tests the claim directly. Two ViT-L/16 models are trained identically (VideoMix2M, 90,000 iterations, batch 3072, the same masks), one with the V-JEPA feature loss and one with mean squared error on normalized pixel values as in MAE, each with its own learning-rate and weight-decay sweep. Frozen, with the attentive probe:

• Kinetics-400: 73.7 with features, 68.6 with pixels (+5.1).
• Something-Something-v2: 66.2 vs 66.0 (+0.2).
• ImageNet-1K: 74.8 vs 73.3 (+1.5).
• Kinetics-400 after full fine-tuning: 85.6 vs 85.4 (+0.2).

The feature target wins every column, by a large margin only on frozen Kinetics. Under full fine-tuning the two are nearly tied, and the paper's own summary is that feature prediction beats pixel prediction in frozen evaluation and is competitive with it under fine-tuning.

Why the encoder does not collapse

Suppose equation (1) had no stop-gradient and used the trained encoder on both sides. Then the gradient could lower the loss by changing the target as easily as the prediction, and the cheapest way to lower it is to make all representations the same: an encoder that outputs a constant vector $\mathbf{c}$ for every clip, plus a predictor that outputs $\mathbf{c}$, has loss zero. This failure is called representation collapse, and the paper states that the naive objective admits it.

V-JEPA uses the recipe introduced by BYOL for images: a predictor, a stop-gradient on the target, and a target encoder that is an EMA of the trained one. The stop-gradient removes the route where the target moves toward the prediction; the target changes only through the EMA. The momentum $\tau$ is not fixed. It starts at 0.998 and rises linearly toward 1.0 over the 112,500-iteration schedule, so at the 90,000th iteration, where training stops, it is about 0.9996. A momentum of 0.998 averages the encoder over roughly the last 1/(1 − 0.998) = 500 steps; 0.9996 averages over roughly 2,500.

The stop-gradient alone does not rule out collapse: the encoder could still drift toward a constant through its own gradient. Section 3.1 adapts BYOL's argument for why it does not. Ignore $z$, treat representations as one-dimensional, and call the target $Y = \bar E_\theta(y)$. For a fixed encoder, the predictor that minimizes the expected L1 loss outputs the conditional median of the target:

Substituting this optimal predictor back into the loss, the expected gradient for the encoder is the gradient of a median absolute deviation (MAD), a measure of spread like the standard deviation but built from medians:

Read the right side as: how uncertain the target remains once the encoder's output for the visible part is known. The target $Y$ is held fixed by the stop-gradient, so the only way for the encoder to shrink this spread is to make $E_\theta(x)$ more informative about the hidden part. In the paper's words, with an optimal predictor the encoder must learn to capture as much information about the video as possible. A constant encoder tells the predictor nothing about $Y$ and leaves the spread at its maximum.

The argument assumes the predictor stays optimal while the encoder changes. The paper offers the EMA as the reason it does, and labels this a hypothesis: a slowly moving target lets the predictor evolve faster than the encoder and remain close to optimal, which prevents collapse. No experiment in the paper varies the momentum, so the claim is untested there.

Figure 3 is a small experiment of this page's own, run in your browser: a linear encoder from 4 inputs to 2 features, a 2×2 linear predictor, two noisy views of the same latent as context and target, and the L1 loss of Eq. (1). Start with the default setting (stop-gradient on, predictor learning 50× slower than the encoder, $\tau$ = 0.998) and press Play, then lower $\tau$ to 0.9 or 0, then turn the stop-gradient off.

In the toy the stop-gradient is necessary: without it every setting collapses. The EMA matters only in the regime the hypothesis describes, a predictor that adapts slowly relative to the encoder; when the predictor adapts quickly, even $\tau$ = 0 (target equal to the current encoder, with the gradient still stopped) trains fine, which matches the image result of SimSiam (Chen & He, 2021). How these regimes map onto a 300-million-parameter ViT is not something the paper measures.

The multi-block mask

Which tokens to hide decides how hard the prediction task is. Video is redundant in space and time: a hidden patch usually looks like the patch next to it, and like itself one frame earlier. If the visible context includes either, the predictor can copy instead of reasoning about the scene. The paper follows VideoMAE in hiding large contiguous blocks that run through the entire clip, which removes both shortcuts.

A V-JEPA mask is built from blocks. Each block is a rectangle on the 14×14 spatial grid, repeated at all 8 token time steps, so it hides a tube through the whole clip. A mask is the union of several such blocks, possibly overlapping, with aspect ratios drawn from 0.75 to 1.5. There are two mask types, and every clip gets one of each:

• Short-range: the union of 8 blocks, each covering 15% of a frame.
• Long-range: the union of 2 blocks, each covering 70% of a frame.

The two masks are separate prediction problems. For each one the x-encoder and predictor run on that mask's visible tokens, and both share the single y-encoder pass. Appendix E.4 finds two masks per clip better than one, and several smaller blocks better than one large block at the same masking ratio.

Section 3.2 puts the average masking ratio at about 90%. The official mask generator gives numbers per mask type. With the paper's settings, a 15% block on the 14×14 grid is about 5×5 patches and a 70% block about 12×12. Simulating the generator, the union of 8 short-range blocks covers about 63% of a frame on average and the union of 2 long-range blocks about 81%. The code then trims every clip's list of visible tokens to the length of the shortest list in its per-GPU batch of 24, so the share of tokens withheld from the x-encoder averages about 77% for short-range masks and 94% for long-range ones. At a round 90%, the x-encoder sees about 157 of the 1568 tokens.

Figure 4 draws masks with the official generator's rules on the real 14×14 grid. Besides the two V-JEPA mask types it offers random-tube, the weakest strategy in the paper's ablation. Compare the "nearest visible" readout across the three: it is the average distance, in patches, from a hidden patch to the closest visible one.

Table 4 compares four strategies on a ViT-L/16 pretrained on Kinetics-710 plus Something-Something-v2. Random-tube[0.9] hides 90% of the spatial positions chosen at random, each through the whole clip, so it has full temporal extent like multi-block but scatters the hidden region into isolated tubes. Causal multi-block[p] restricts the visible context to the first p of the 16 frames (6 or 12) and masks those with random blocks too, so the model predicts the rest of the clip from its beginning.

On Something-Something-v2 the order is random-tube 46.4, causal multi-block[6] 49.8, causal multi-block[12] 63.6, multi-block 67.4. Kinetics-400 follows the same order (51.5, 61.3, 71.9, 72.9), as does ImageNet (55.6, 66.9, 72.2, 72.8). The random-tube gap (21 points on SSv2) separates scattered tubes from large contiguous blocks; both hide every frame, so it does not measure per-frame masking against tubes. The paper reports the causal result only as a decrease. Whether it says anything general about predicting the future from the past, the language-model recipe, is this page's question, not a claim the paper makes.

The masking ratio itself has a failure mode at the low end. Appendix E.4 trains a ViT-B/16 with different spatial and temporal coverage and finds that low coverage makes the prediction task trivial and degrades downstream accuracy, which is why the default masks large blocks through the full duration of the clip, at an average masking ratio of about 90%.

Reading out a frozen encoder: the attentive probe

A frozen V-JEPA encoder outputs 1568 vectors per clip (more for longer evaluation inputs). A classifier needs one. The common choice, averaging the vectors and training a linear layer, assumes the useful information is linearly readable after averaging. The paper argues there is no reason to expect that here: Eq. (1) has no normalization that would arrange features into linearly separable groups, as contrastive objectives tend to.

So every frozen evaluation uses an attentive probe, following the learnable pooling of CAE (Chen et al., 2022). A single learnable query vector attends over all the frozen output tokens through one cross-attention layer with 12 heads. Its output is added back to the query (a residual connection), passed through a two-layer MLP with one GeLU, a LayerNorm and a linear classifier. The probe can weight the few tokens that matter for a class instead of diluting them in an average of 1568.

Table 3 measures the difference on V-JEPA ViT-L/16: average pooling gives 56.7 on Kinetics-400 and 50.1 on Something-Something-v2; the attentive probe gives 73.7 and 66.2, gains of 17.0 and 16.1 points. These numbers are for V-JEPA L/16 on those two benchmarks. The probe also helps the baselines on average, which keeps the comparisons fair, but not in every case: VideoMAE L/16 rises from 52.5 to 77.8 on K400, while OpenCLIP falls slightly on SSv2 (35.8 to 34.8) and ImageNet (86.2 to 85.3), and DINOv2 on ImageNet (86.5 to 86.2).

Results: motion, appearance, few labels

Against pixel-prediction video models, same architecture. Table 5 compares ViT-L/16 class models, frozen: V-JEPA beats OmniMAE, VideoMAE and Hiera on every task except ImageNet, where it scores 74.8 against 75.1 for an OmniMAE model trained directly on ImageNet. On Something-Something-v2 the scores are V-JEPA 69.5, VideoMAE 65.5, Hiera 64.2, OmniMAE 60.6. V-JEPA processes 270M samples in 90,000 iterations; VideoMAE uses 410M, Hiera 770M over 1.5M iterations, and OmniMAE 2400M.

Against everything, any size. Table 6 drops the architecture constraint. The three V-JEPA models score as follows (K400 / SSv2 / AVA / ImageNet / Places205 / iNat21; AVA is action localization scored in mean average precision, Places205 is scene classification, iNat21 is fine-grained species recognition):

• ViT-L/16 at 224: 80.8 / 69.5 / 25.6 / 74.8 / 60.3 / 67.8
• ViT-H/16 at 224: 82.0 / 71.4 / 25.8 / 75.9 / 61.7 / 67.9
• ViT-H/16 at 384: 81.9 / 72.2 / 25.0 / 77.4 / 62.8 / 72.6

The 384 model is a separate pretraining run at the higher resolution, not a fine-tune of the 224 model. Its ImageNet score rises from 77.4 to 77.9 with a two-layer attentive probe, and the abstract reports that two-layer score.

Something-Something-v2 needs temporal understanding, and on it V-JEPA H/16 at 384 scores 72.2 against 50.6 for DINOv2 ViT-g/14 (the successor of DINO), 50.0 for I-JEPA ViT-H/16 at 512 and 34.8 for OpenCLIP ViT-G/14: 21.6 points over the best of them. The 224 H/16 leads by 20.8. Against the largest public pixel-prediction video models (VideoMAE, VideoMAEv2, OmniMAE, MVD, Hiera), the paper reports a lead of at least 5 points on SSv2; those models differ in size, and the matched-architecture lead in Table 5 is 4.0 points over VideoMAE.

On appearance, image models still lead. DINOv2 g/14 scores 83.4 on Kinetics-400 with the same attentive probe, above every V-JEPA model (82.0 and 81.9 for the two H/16s), and 86.2 on ImageNet, where V-JEPA's best one-layer-probe score is 77.4. Among video models V-JEPA leads Kinetics-400: the best baselines are VideoMAE H/16 at 79.8 and MVD L/16 at 79.4. The authors hypothesize that public video datasets are too narrow and lack the visual diversity of internet-scale image data, and call for building more diverse ones.

With few labels. Table 7 trains the attentive probe on 5%, 10% or 50% of each training set (three random splits each) while the backbone stays frozen. On Kinetics-400 that is about 29, 58 and 287 labeled clips per class; on Something-Something-v2, 48, 96 and 440. There is no 100% column. Going from 50% to 5% of the labels, a 10× reduction:

• V-JEPA ViT-H/16 at 384: K400 80.6 → 68.2 (−12.4), SSv2 67.9 → 54.0 (−13.9)
• MVD ViT-L/16: K400 77.2 → 62.6 (−14.6), SSv2 61.0 → 42.9 (−18.1)
• VideoMAE ViT-H/16: K400 78.2 → 62.3 (−15.9), SSv2 60.5 → 41.4 (−19.1)
• VideoMAEv2 ViT-g/14: K400 67.8 → 37.0 (−30.8), SSv2 54.0 → 28.0 (−26.0)

The paper rounds these drops to whole percentages ("drops by 12%", "drops by 30%"); they are percentage points. On SSv2 the gap between V-JEPA and the next-best model grows from 6.9 points at 50% of the labels to 11.1 at 5%. Read Figure 6 at its left edge for the scarce-label case.

Cost. V-JEPA reaches these numbers on far fewer training samples: about 210M clips seen for the H/16 at 384, against 1600M for VideoMAEv2, 1900M images for DINOv2 and 39,000M for OpenCLIP (Table 16; those models also train on different datasets). In wall-clock time the paper estimates roughly a 2× speedup over the large pixel-prediction models, from timings on one GPU at a batch of 10 clips, extrapolated linearly to a global batch of 2400, not from a full-scale run.

Data. Table 2 trains the ViT-L/16 on subsets of VideoMix2M for the same 90,000 iterations. Average accuracy over K400, SSv2 and ImageNet rises with dataset size, from 70.9 on Kinetics-710 (700K videos) to 71.5 on all of VideoMix2M (2M), but each single benchmark peaks on a different mix: K400 is best at 75.8 when pretraining on Kinetics-710 alone, SSv2 at 67.4 on Kinetics-710 plus SSv2.

What the predictions contain

V-JEPA outputs features, so its predictions cannot be looked at directly. Section 6 trains a conditional diffusion model (a generative model of the kind in DDPM) to turn predicted features into pixels, with the pretrained ViT-H/16 encoder and its predictor frozen. The decoder receives only the predictor's outputs for the hidden region and never sees the visible part of the clip, so whatever its samples share across random seeds was carried by the predicted features.

The samples in the paper's Figure 6 are spatially and temporally consistent with the unmasked video: the decoded objects sit at plausible positions, keep their trajectory across frames, and in some samples reappear after partial occlusion. They also vary between seeds in position and appearance, which the authors read as the predictor representing positional uncertainty. The analysis is qualitative, and no number in the paper depends on it.

I-JEPA, V-JEPA, V-JEPA 2

The JEPA framing comes from Yann LeCun's 2022 position paper. I-JEPA (Assran et al., 2023) built it for images: an encoder on visible patches, a narrow predictor that fills in masked positions given their locations, an EMA target encoder, an L2 loss, and 2D block masks. V-JEPA keeps that structure and changes the input to 16-frame clips of tokens two frames deep, the masks to blocks that run through the whole clip, and the loss to L1. Its collapse-prevention recipe goes back to BYOL, and its contextualized targets to data2vec.

V-JEPA 2 (Assran et al., 2025) pretrains on over 1 million hours of internet video and adds V-JEPA 2-AC, an action-conditioned model post-trained on under 62 hours of unlabeled robot video from the Droid dataset and used for planning robot actions.