V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

A video model trained on over a million hours of internet video and 62 hours of robot video plans a robot arm's motions in labs it has never seen.

An encoder learns by predicting the features of hidden parts of video clips. A second network, trained on robot video with the encoder frozen, predicts how those features change when the arm moves, and a search over candidate moves picks the one whose predicted features come closest to a goal image.

Meta's 2025 paper in two stages: a 1-billion-parameter encoder pretrained without labels, then a 300-million-parameter predictor that turns it into a planner for a Franka arm, with eight interactive figures.

V-JEPA 2 comes from FAIR at Meta and is the third paper in a line that starts with LeCun's 2022 proposal for joint-embedding predictive architectures (JEPA): train a network by hiding part of the input and asking it to predict the hidden part's representation, a vector produced by an encoder, instead of the hidden pixels. I-JEPA applied this to images and V-JEPA to video. V-JEPA 2 keeps V-JEPA's objective, scales it to a 1-billion-parameter encoder trained on 22 million videos and images, and then adds an action-conditioned model, which the earlier papers did not have.

The paper has two stages. Stage one, V-JEPA 2, pretrains a video encoder on internet video with no labels. Stage two, V-JEPA 2-AC (action-conditioned), freezes that encoder and trains a new predictor on 62 hours of robot video to predict the next frame's features from the current ones and the arm's next move. A planner then searches for the move whose predicted features come closest to the features of a goal photo. Deployed on Franka arms in two labs that contributed no training data, it picks up and places a cup in 80% of trials, against 15% for a behavior-cloning baseline.

The page follows the paper's order: why the loss is computed on features, the pretraining step with real tensor shapes, the four changes that make it scale, the action-conditioned predictor and its two losses, planning by minimizing an energy with the cross-entropy method, the robot and benchmark results, and the limitations, including the sensitivity to camera placement.

Why predict features instead of pixels

The next frame of a video is partly predictable and partly not. A cup that has been sliding keeps sliding; the positions of a thousand leaves in a breeze do not follow from anything in the previous frames. A model trained with a per-pixel squared error is scored on both. For the unpredictable pixels the lowest achievable error comes from predicting their average, which renders as a blur, and the error left over is a floor no amount of understanding can lower. Every gradient step still carries that leftover error, so the leaves add noise to the signal from the cup.

A JEPA moves the loss into a learned space. An encoder $E$ maps a clip to a grid of vectors, and a predictor is scored on how well it predicts the encoder's vectors for the hidden part. If the encoder represents the leaves as "foliage, moving" and drops their exact positions, the target becomes predictable and the floor goes away. The encoder is trained by the same loss, so it is pushed toward keeping what can be predicted from context. Figure 1 is a one-dimensional toy of this argument: a bright object that moves predictably next to a patch of random "leaf" pixels.

With leaf noise at zero, both losses respond only to the ball error. Raise the noise and the pixel loss gains a floor that the ball term becomes a sliver on top of, while the feature loss is unchanged. The toy's encoder is hand-written; in V-JEPA it is learned. The comparison has been measured: V-JEPA v1 trained the same ViT-L with a pixel target and with a feature target and scored 68.6 versus 73.7 on Kinetics-400 with a frozen encoder.

Moving the loss into feature space has two costs. The model never learns to output pixels, so nothing in V-JEPA 2 can show you an imagined frame without a separately trained decoder (the paper trains one for visualization only, covered in the limitations). And training the encoder against its own outputs has a trivial solution: if the encoder maps every clip to the same vector, the predictor outputs that vector and the loss is zero. This failure is called representation collapse, and the next section shows the two terms in the loss that prevent it.

How V-JEPA 2 pretraining works

Take one training clip from the main phase of pretraining: 16 frames at 256 by 256 pixels, sampled at 4 frames per second, so 4 seconds of video. The clip is cut into tubelets, blocks of 2 frames by 16 by 16 pixels, giving 8 positions in time and a 16 by 16 grid in space: 2,048 tokens. A strided convolution turns each tubelet into a vector of 1,408 numbers, the width of the ViT-g encoder.

Next a mask hides most of those tokens. V-JEPA 2 uses V-JEPA's multiblock masking: it picks several rectangles of patches, takes their union, and hides that region in every one of the 8 time positions, so a hidden patch stays hidden for the whole clip. Short-range masks combine 8 rectangles that each cover about 15% of the frame; long-range masks combine 2 that each cover about 70%. Because the hidden region is a large block that persists in time, the predictor cannot fill a hidden patch by copying the same position from another frame, and most hidden patches are several patches away from anything visible.

Three networks then run. The encoder $E_\theta$ reads only the visible tokens (the masked view $x$). The predictor $P_\phi$, a 12-layer transformer of width 384 (about the size of a ViT-small), receives the encoder's outputs plus a learned vector $\Delta_y$, the mask token, placed at each hidden position along with that position's coordinates, and outputs a 1,408-number prediction for every hidden token. A third network $E_{\bar\theta}$, a copy of the encoder, reads the full clip $y$, and its outputs at the hidden positions are the targets. The loss is the L1 distance (the sum of absolute differences) between predictions and targets:

The loss is applied only at the hidden positions. Two pieces of (1) prevent collapse. The first is $\mathrm{sg}(\cdot)$, stop-gradient: backpropagation treats the targets as constants, so the encoder cannot reduce the loss by changing what the targets are. The second is the bar on $\bar\theta$. The target encoder's weights change only by averaging: after every step they move a small fraction of the way toward the trained weights, an exponential moving average (EMA):

With that coefficient the target weights average the trained weights over roughly the last $1/\tau$ = 1,333 steps, and a change in $\theta$ takes about 924 steps to reach the targets halfway. The targets therefore move slowly and the predictor has time to catch up with them. V-JEPA presents this as a working recipe, not a proof that collapse cannot happen; the same stop-gradient and EMA pair is used by BYOL, DINO and I-JEPA. ^[1]

Figure 2 draws one step. Switch between the two mask types and a random-patch mask with the same number of hidden patches, and watch the count of hidden patches that sit next to a visible one.

With random patches, nearly every hidden patch has a visible neighbor, and smooth video lets the predictor interpolate. With multiblock masks most hidden patches are several patches from anything visible, so the predictor must infer what is there from the scene. In V-JEPA's ablation, random tubes at 90% masking reached 51.5 on Kinetics-400 and multiblock reached 72.9, with the same model and data.

Two choices differ from V-JEPA. The encoder encodes position with 3D-RoPE, rotary embeddings applied separately to three slices of each vector for time, height and width, instead of fixed sinusoidal embeddings; the paper reports this stabilized training of the largest models. ^[3] And the EMA coefficient and weight decay stay fixed for the whole run, where V-JEPA ramped both on a schedule; the authors found the ramps made little difference. The full step, in pseudocode:

# One V-JEPA 2 pretraining step (primary phase: 16 frames, 256 x 256)
clip    = sample_video()                 # (16, 3, 256, 256), 4 fps = 4 seconds
tokens  = tubelet_embed(clip)            # 2x16x16 tubelets -> (2048, 1408)
ctx, tgt = multiblock_mask()             # index lists; tgt = hidden positions
h       = E_theta(tokens[ctx])           # encoder sees visible tokens only
z_hat   = P_phi(h, ctx, tgt, delta_y)    # mask token delta_y fills each tgt slot
with no_grad():                          # sg(.): no gradient into the targets
    z   = E_ema(tokens)[tgt]             # EMA encoder reads the full clip
loss    = abs(z_hat - z).mean()          # L1, masked positions only, eq (1)
loss.backward()                          # gradients reach theta, phi, delta_y
opt.step()
ema     = 0.99925 * ema + 0.00075 * theta   # teacher trails the student

Four changes that scale V-JEPA to a 1B encoder

The paper measures each change against a baseline: a ViT-L/16 (300 million parameters) trained with the V-JEPA objective on V-JEPA's 2-million-video dataset. Every model is scored the same way. The encoder is frozen and a small classifier called an attentive probe is trained on its output for each task: four transformer blocks, the last of which uses cross-attention from one learned query token to pool the encoder's tokens into a single vector. The score is the average accuracy over six tasks: Something-Something v2 (which hand motion is happening), Diving-48, Jester (hand gestures), Kinetics (action classes), COIN (instructional steps) and ImageNet. The baseline averages 84.2.

• Data, 2M to 22M samples: +1.0. VideoMix22M combines Something-Something v2 (168K clips), Kinetics (733K), HowTo100M (1.1M), 19M clips from YT-Temporal-1B and 1M ImageNet images, each image repeated as a 16-frame still video. YT-Temporal-1B is filtered by retrieving scenes that resemble Kinetics, Something-Something v2, COIN and Epic-Kitchens, the cluster-based method of DINOv2; at ViT-L size the curated subset scores 1.4 points above the uncurated one.
• Model, 300M to 1B parameters: +1.5. ViT-g: 40 layers, width 1,408, 22 attention heads. The predictor stays the same small network.
• Training length, 90K to 252K iterations: +0.8. Measured with ViT-g models, at a batch of 3,072 clips.
• Resolution, 256 to 384 pixels and 16 to 64 frames: +0.7. Applied in pretraining and evaluation.

The resolution change has the model see four times as many frames at 2.25 times as many pixels per frame, and the learning-rate schedule keeps its cost down. A cosine schedule, the common default, decays the learning rate along a curve whose shape depends on the total number of steps, which has to be fixed in advance. V-JEPA 2 uses warmup-constant-decay instead: a linear warmup over 12K iterations from $1\times10^{-4}$ to $5.25\times10^{-4}$, a constant learning rate for 228K iterations, and a 12K-iteration cooldown in which the rate decays linearly to $10^{-6}$. Following Hägele et al., the paper finds this matches a half-cosine schedule, with one practical difference: a cooldown can be started from any checkpoint of the constant phase, so trying a longer run does not mean training again from scratch. ^[2]

The cooldown is also where resolution goes up. Warmup and constant phase use 16 frames at 256 pixels, 2,048 tokens per clip. The cooldown uses 64 frames at 384 pixels: 32 time positions on a 24 by 24 grid, 18,432 tokens, nine times as many. Attention cost grows with the square of the token count, so one high-resolution iteration costs between 9 and 81 times a low-resolution one. The paper estimates that training ViT-g on 64 × 384 × 384 inputs throughout would take roughly 60 GPU-years and that the progressive schedule cuts GPU time by 8.4 times, to about 7 GPU-years.

If both runs last 252K iterations and a high-resolution iteration costs $r$ low-resolution ones, the saving is

which is 6.5 times at $r = 9$, 16.8 times at $r = 81$, and can never exceed 252/12 = 21 times, because the 12K cooldown iterations are paid at full price in both runs. The paper's measured 8.4 times corresponds to $r \approx 13.3$.

How V-JEPA 2-AC learns what actions do

Nothing so far involves actions. The pretrained encoder has watched people cook and dive, but it has no notion that a robot can choose what happens next. Stage two adds that from the Droid dataset: 23k trajectories of a teleoperated 7-joint Franka Panda arm, successes and failures together, just under 62 hours after discarding clips shorter than 4 seconds. The paper uses only the video from the left camera and the arm's logged end-effector state. No task descriptions, success flags or rewards are used, which is what "unlabeled" means here.

Each training sample is a 4-second clip at 4 frames per second, so 16 frames. The state $s_k$ at frame $k$ has 7 numbers: the end effector's 3D position, its orientation as 3 Euler angles, and the gripper's opening. The action $a_k$ is the change in that state from frame $k$ to frame $k+1$, also 7 numbers, so the 16 frames give 15 actions. Every frame goes through the frozen V-JEPA 2 encoder on its own, as a still image, producing a feature map $z_k$ of 16 × 16 × 1,408: 256 tokens of 1,408 numbers.

The new predictor is a 300M-parameter transformer: 24 layers, 16 heads, width 1,024, GELU activations. Three separate linear layers map an action, a state and each patch token to width 1,024, so one time step contributes 258 tokens (1 action, 1 state, 256 patches) and 15 steps make a sequence of 3,870. Attention is block-causal: a token at time $t$ attends to every token at time $t$ and earlier, never later, as in a language model with whole time steps in place of words. Patch tokens carry 3D-RoPE positions; action and state tokens carry only the time component, since they have no place in the image. A final linear layer maps the outputs back to 1,408 numbers per patch, a predicted feature map $\hat z_{k+1}$.

Training uses two losses. Teacher forcing gives the predictor the real history at every step and asks for the next feature map, over $T = 15$ steps:

When a plan covers more than one action, the predictor has no real frames for the later steps; from the second step on it reads its own output. A model trained only with (2) never sees its own errors as input. The rollout loss runs the predictor autoregressively from the real $(s_1, z_1)$ through $T$ actions, feeding each prediction back in, and compares the last prediction with the truth:

The two $T$s differ. The paper uses $T = 15$ in (2) and $T = 2$ in (3), so the rollout backpropagates through one recurrent step: the prediction of $z_3$ is made from a predicted $\hat z_2$, and the gradient trains the predictor to cope with input of that quality. Figure 5 shows why a longer chain is harder.

In the toy, the teacher-forcing error stays at 0.04 per step while the rollout error grows with every step, because each prediction inherits the previous one's offset and adds its own; at $T = 15$ it reaches 0.93, 23 times the one-step error. The paper's rollout loss uses $T = 2$, and its robot experiments plan one step ahead, so at deployment no prediction is fed back in.

The predictor trains with AdamW at batch size 256 for 94,500 iterations: 4,500 of linear warmup from $7.5\times10^{-5}$ to $4.25\times10^{-4}$, 85,500 constant, and 4,500 decaying to zero, with weight decay 0.04. Training on both Droid camera views without telling the model which view it was seeing made results worse, which is why only the left view is used. A training step in pseudocode:

# One V-JEPA 2-AC training step (encoder E frozen, predictor P trained)
frames  = droid_clip()                   # 16 frames: 4 s at 4 fps, left camera
s       = ee_state()                     # (16, 7): xyz, 3 Euler angles, gripper
a       = s[1:] - s[:-1]                 # (15, 7): action = change in state
with no_grad():
    z   = E(frames)                      # each frame alone -> (16, 16, 16, 1408)
# teacher forcing, eq (2): real history, predict every next frame
z_hat   = P(a, s[:15], z[:15])           # block-causal -> (15, 16, 16, 1408)
L_tf    = abs(z_hat - z[1:16]).mean()    # T = 15
# rollout, eq (3): feed the prediction back in for one more step
z2      = P(a[:1], s[:1], z[:1])[-1]     # predicted z_2 from real z_1
z3      = P(a[:2], s[:2], cat(z[:1], z2))[-1]   # z_3 from the PREDICTED z_2
L_ro    = abs(z3 - z[2]).mean()          # T = 2
(L_tf + L_ro).backward()                 # eq (4); only phi changes

How the robot plans: energy minimization with CEM

A task is given as a photo of the goal, $x_g$, encoded once into $z_g = E(x_g)$. At control step $k$ the robot encodes its current camera frame into $z_k$ and reads its state $s_k$. A candidate plan is a sequence of $T$ actions $\hat a_{1:T}$; the predictor rolls it out and the plan is scored by how far the predicted final feature map lands from the goal's:

The paper calls (5) an energy: a number that is low for good plans. It takes the place of a reward function, and no one writes it by hand: an action scores well if the predictor expects it to make the camera view look like the goal photo, as measured by the frozen encoder. With $T = 1$, scoring one candidate means one predictor pass and an absolute difference summed over 16 × 16 × 1,408 = 360,448 numbers.

The minimization uses the cross-entropy method (CEM), a sampling search from Rubinstein (1997). It keeps a Gaussian over actions, starting at zero mean and unit variance. Each iteration draws 800 candidate actions, clips each to an L1 ball of radius 0.075 (at most about 13 cm of end-effector motion, because larger moves are rare in the training data), scores all 800 with (5), keeps the best $k$, and refits the Gaussian's mean and variance to them. After 10 iterations the mean is the chosen action. CEM uses only the energy values. The predictor is differentiable, and the paper lists gradient-based planning as a way to cut planning time in future work.

# Choosing one action with the cross-entropy method (horizon T = 1)
z_g  = E(goal_image)                     # encoded once per goal
z_k  = E(camera_frame());  s_k = ee_state()
mu, sd = zeros(7), ones(7)               # Gaussian over the 7-D action
for it in range(10):                     # 10 refinement iterations
    A = clip_l1(mu + sd * randn(800, 7), 0.075)  # 800 candidates, ~13 cm max
    E_ = [abs(P(a, s_k, z_k) - z_g).sum() for a in A]   # energy, eq (5)
    elite = A[argsort(E_)[:k]]           # keep the k lowest-energy actions
    mu, sd = elite.mean(0), elite.std(0) # refit the Gaussian to them
robot.execute(mu)                        # run it, then plan again from scratch

To see what CEM is searching, the paper plots (5) for a reaching task over one action, sweeping the horizontal components $\Delta x$ and $\Delta y$ with $\Delta z = 0$. The surface is smooth and locally convex, a single bowl. The action that relates the start and goal images, the ground truth, is $(\Delta x, \Delta y) = (0, -0.1)$; the energy minimum is at about $(0, -0.05)$, the right direction and about half the distance.

CEM converges to the energy minimum rather than the ground truth, so a single step covers about half the needed motion, and replanning makes up the rest. After the chosen action is executed, the robot takes a new photo, encodes it, and plans again from the new state; this is receding-horizon control, the scheme behind model-predictive control (MPC). A step that falls short, or goes slightly off direction, is corrected by the next plan. In the paper's three reaching examples the arm ends within 4 cm of the goal, with the distance shrinking at every step.

All the robot experiments plan with horizon $T = 1$: one action, executed, then a fresh plan. Figure 7 also offers a 4-step horizon to show what the receding part means when the plan is longer than one step: only the first action is executed and the rest is discarded. Its second slider anticipates the limitations section. If the model's sense of which way is "x" in the image is rotated relative to the robot's actual axes, each executed step goes off at an angle.

With no axis error the arm reaches the goal in five steps. At 30 degrees every step leaves at an angle and the path curves in, entering the 4 cm ring at step 7; at 45 degrees it takes 11 steps. The distance still shrinks at every step, because each new observation re-aims the next action, but by less than it could. At 60 degrees the arm is still 15 cm away after 12 steps, and at 90 degrees each step is perpendicular to the direction it should take, so the distance grows.

One action costs 16 seconds on a single RTX 4090: 10 iterations of 800 candidates is 8,000 energy evaluations, 2 ms each on average. Pick-and-place, the longest task, is 18 actions (4 toward the first sub-goal, 10 toward the second, 4 toward the final goal), about 5 minutes of planning. The video-generation baseline, Cosmos, needs 4 minutes per action with 10 iterations of only 80 candidates, about 0.3 s per candidate, so a pick-and-place with Cosmos would take over an hour.

Zero-shot robot results against Octo and Cosmos

The test robots are Franka Panda arms with RobotiQ grippers in two labs, neither of which appears in Droid. Input is one uncalibrated, low-resolution RGB camera, and no data is collected in either lab before the trials. Each skill gets 10 trials per lab, varying the object's location and the arm's start pose:

• Reach: move the gripper to the place shown in the goal image.
• Grasp a cup or a box, from one goal image showing it grasped.
• Reach with object: move while holding the object without dropping it, from one goal image.
• Pick-and-place, from three images: the object grasped, the object near its destination, and the final placement. The planner switches images after 4 and then 10 actions. ^[4]

The main baseline is Octo, a transformer robot policy trained by behavior cloning (imitating the recorded actions) on Open-X Embodiment, over 1M trajectories, then fine-tuned with goal images on all of Droid. V-JEPA 2-AC trained its predictor on 23k Droid trajectories, roughly 40 times fewer than Octo's pretraining set, and never imitates an action: it only learns what actions do. Cosmos, a 7B video-generation model trained on 20M hours of video and fine-tuned on Droid with its official action-conditioned code, was run in Lab 2 with the same CEM planner.

Both methods solve reach. Averaging cup and box, V-JEPA 2-AC succeeds on 45% of grasps against 7.5% for Octo, 75% of reach-with-object trials against 42.5%, and 72.5% of pick-and-place trials against 12.5%: margins of 37.5, 32.5 and 60 points. Per object the margins range from 5 points (reach with the box, 75 against 70) to 65 (pick-and-place with the cup, 80 against 15). The box is the hardest grasp for both, 25% at best, because the fingers have to open wide enough and close precisely. Cosmos reaches in 80% of Lab 2 trials but succeeds on 0% of cup grasps and 0% of pick-and-place trials.

What the frozen encoder does on benchmarks

The same encoder also does well without the robot. On Something-Something v2, a benchmark of 174 hand-motion classes where a single frame rarely gives the answer, ViT-g at 384 pixels scores 77.3 top-1 with the frozen encoder and an attentive probe (75.3 at 256 pixels).

On Epic-Kitchens-100 action anticipation, the model watches a kitchen video that stops 1 second before an action starts and must name the action. Here the pretraining predictor is used directly: it receives the encoded context plus mask tokens at the positions of a frame 1 second in the future and predicts that frame's features, and an attentive probe reads the encoder and predictor outputs together. ViT-g at 384 pixels reaches 39.7 recall-at-5 (the right action among the top 5 guesses), against 27.6 for PlausiVL, an 8B model built for this task: a 44% relative improvement.

Finally, the paper connects V-JEPA 2 to a language model, the recipe of LLaVA-style video assistants: an MLP projects the encoder's output tokens (288 per frame for ViT-g at 384 pixels) into the input space of Llama 3.1 8B Instruct, and the system is trained on 88.5M alignment samples with the PerceptionLM recipe. It scores 84.0 on PerceptionTest, 76.9 on TempCompass, 44.5 on MVP, 36.7 on TemporalBench and 40.3 on TOMATO, ahead of PerceptionLM 8B on all five by 1.3 to 8.4 points, but not ahead on TVBench or MVBench. ^[5] In the paper's controlled comparisons, fine-tuning the encoder during alignment beats keeping it frozen, and the paper does not say which of the two produced these final numbers.

The encoder saw no text during pretraining. Many video assistants use encoders trained against text captions, as CLIP is. The paper reads these results as evidence that an encoder pretrained without language supervision can be aligned to a language model and lead its size class. The evidence is at one model scale (8B) and one recipe.

Limitations: camera placement, long horizons, image goals

Camera position. Actions are Cartesian moves in the robot's frame, and the camera is not calibrated, so the model has to infer from the image which direction on screen corresponds to +x. When the robot's base is out of view that inference is ill-defined. In practice the authors tried several camera positions by hand, picked one that worked, and kept it for every experiment. Appendix B.4 measures the effect. For camera angles from about 35 to 85 degrees around the table, the robot made 200 random horizontal moves; for each, the action that minimized (5) was compared with the action the robot took. The inferred actions differ from the true ones by about 1.6 cm on moves of about 5 cm, the error is systematic, and it is mostly a rotation, whose angle changes almost linearly with camera position. Figure 7's axis-error slider shows what a rotation does to a reach. According to the authors, measuring this rotation with random moves and undoing it would calibrate the model without supervision; they did not do this in their experiments.

Long horizons. Prediction error accumulates over autoregressive rollouts (Figure 5), and the number of action sequences to search grows exponentially with the horizon. Both push the planner toward short horizons, so tasks that require non-greedy moves need sub-goal images: pick-and-place without them is left to future work.

Goals are images. The energy (5) measures distance to an encoded goal photo, so a task must come with a photo of its end state. Goals given in language would need the action-conditioned model to be aligned with a language model, which the paper lists as future work.

No pixel output. Neither stage trains a decoder, so each prediction of the world model is a feature map of 360,448 numbers. To inspect them, Appendix B.3 trains a separate feedforward decoder (a ViT-L, L2 loss, frozen V-JEPA 2 features to 256 × 256 frames from Droid). Decoded rollouts show the arm moving and a held cup moving with it, a cup left in place when the gripper is open, and errors accumulating: after several steps the predicted cup sits slightly lower than the real one.