V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

A robot that learned to plan by watching a million hours of internet video and 62 hours of itself.

Predict masked patches of a video in a learned representation space. Add actions on top. The same model that scored state of the art on action anticipation drives a Franka arm it has never seen, picking up a cup from a goal image alone.

A 1-billion-parameter video encoder, trained without a single label, that you can bolt 300 million parameters of action prediction onto and plan with.

Most of the recent wins in machine learning come from one idea: predict the missing piece. Hide a word and you get a language model. Hide a patch of an image and you get an image encoder. The obvious next step was video, which is mostly missing pieces by definition: a billion pixels of next-frame, all of which you could try to predict. Pixels, though, are mostly noise.

A leaf flutters in the wind. The exact position of every leaf in the next frame is unpredictable, and that unpredictability is irrelevant; what you wanted to know was that there was wind. A pixel-level loss spends most of its capacity on the wrong question. V-JEPA 2, out of Meta's FAIR lab, is a clean-up of an old answer to this: predict the missing piece in a learned representation space, where unpredictable noise has been dropped, instead of in pixels. Then scale it up. The result is a video encoder that, once frozen, can be aligned with a language model for video question answering, or have an action-prediction head bolted on top and be used to plan zero-shot manipulation on a robot arm it has never seen.

The model is called V-JEPA 2, the second video instance of LeCun's joint-embedding predictive architecture. The robotics version is V-JEPA 2-AC, where AC stands for action-conditioned. The rest of this page covers five things: why predict-in-features beats predict-in-pixels, what the mask-denoising loss actually is, how four scaling choices add up to a four-point gain, how you stitch actions onto a frozen feature space, and what planning means when your dynamics live in a latent rather than a pixel buffer.

Why not just predict pixels

Imagine the next frame of a video as a draw from some conditional distribution over images, given everything you have seen so far. Most of that distribution's entropy lives in details you do not care about: the exact configuration of leaves, the precise placement of every blade of grass, the speckle pattern in a shadow. A model trained to minimize pixel error has to spend its capacity learning to render all of that, because predicting the wrong fine grain costs it just as much as missing the actual content of the scene. Latent diffusion has the same problem and gets around it by running in a perceptual VAE's compressed code, which throws away high-frequency noise on purpose.

JEPA pushes the same idea further. Train an encoder $E_\theta$ that maps a frame to a representation. Now ask a predictor to predict the encoded version of the missing piece, not the pixels. A blade of grass in slightly the wrong place is the same vector as a blade in the right place if the encoder has learned to ignore the difference, so the predictor only ever has to commit to predictable structure. The encoder and the predictor are trained together, and the loss is in representation space.

One thing this does not buy is a way to generate pixels, which is what pixel-based world models do. JEPA has no decoder, so it cannot draw the next frame. It can only tell you what the next frame would look like to its own encoder. That is fine for the downstream tasks the paper cares about: classification, anticipation, and planning by latent distance all consume the encoded version directly, so the fluttering leaves never have to be re-rendered.

Training encoder and predictor together has a well-known failure mode called representation collapse: the encoder can cheat by mapping every input to the same vector, which makes the predictor's job free. JEPA dodges this with two details that come back in the loss equation. The prediction target comes from a frozen copy of the encoder rather than the live one, and a stop-gradient on that target keeps the loss from pulling the encoder toward the easy constant.

The mask-denoising objective in representation space

One training step looks like this: a video clip is patchified into a sequence of tubelets, three-dimensional patches of size $2\times 16\times 16$ covering two adjacent frames at a time. A subset of those tubelets is dropped, leaving a masked view $x$. The unmasked view $y$ is the full clip. The encoder reads $x$; the predictor takes the encoder's output and fills in embeddings at the masked positions, guided by a learnable mask token $\Delta_y$ that marks where to predict. The supervision comes from running the full clip $y$ through a separate copy of the encoder, an exponential moving average of the live weights, written $E_{\bar\theta}$. The objective minimizes the L1 distance between the predictor's outputs and the EMA encoder's targets, at the masked positions only:

Read it left to right. $E_\theta(x)$ is the encoder run on the masked view, so it returns a sequence of token embeddings with the masked positions empty. $\Delta_y$ is a learned vector that gets inserted at each masked position, telling the predictor "a missing patch was here, predict its embedding." $P_\phi$ is the predictor, a smaller transformer that reads the encoded visible tokens plus the mask tokens and outputs an embedding for each masked position. On the other side, $E_{\bar\theta}(y)$ is the EMA encoder applied to the full, unmasked clip; its outputs at the masked positions are the targets.

The $\mathrm{sg}(\cdot)$ is the stop-gradient. Gradients of the L1 loss flow through the predictor and the student encoder, but not through the EMA teacher: the EMA copy is updated separately, by $\bar\theta \leftarrow (1-\tau)\bar\theta + \tau\theta$ at every step. Combined with the stop-gradient, this is what stops the trivial collapse. The student cannot make its task easier by pushing the targets around, because the targets' gradient path is cut, and the teacher only ever drifts toward the student slowly through the EMA average. ^[1]

In the figure below the mask ratio is the slider. The amber patches are what the encoder sees, the teal patches are what the predictor has to fill in (in feature space), and the EMA teacher on the bottom is fed the full clip with its outputs at the masked positions used as the targets:

Two design choices in (1) deserve a sentence each. The L1 norm rather than L2 is a robustness call: representation targets that are slightly off should not blow the loss up the way an L2 would. And the masking is multiblock, meaning the dropped tokens come as contiguous spatiotemporal blobs rather than as a salt-and-pepper sprinkle, which forces the predictor to actually use temporal context to fill in long stretches of motion rather than copy from an immediate neighbor. Both choices are inherited from V-JEPA v1; the contribution of this paper is what you do once they are in place.

What makes it work at scale

V-JEPA v1 already trained well at a smaller scale. V-JEPA 2 takes the same objective and scales it across four axes simultaneously: more data, more parameters, longer training, and higher resolution at the very end. Each axis is worth something between half a point and two points of average accuracy across six probe tasks (Something-Something v2, Diving-48, Jester, Kinetics, COIN, ImageNet); together they cumulate to about four points over a ViT-L/16 baseline at 84.2.

The numbers below are what the paper reports for each ingredient added on top of the baseline, all from §2.2. Toggle them on and off to see the stack climb:

The least glamorous of the four ingredients is the schedule. V-JEPA 2 uses warmup-constant-decay rather than cosine, with a long constant phase and a brief cosine-shaped cooldown at the end. That shape matters operationally because from a single constant-phase checkpoint you can launch a dozen different cooldowns at different resolutions and clip lengths without retraining the constant phase. Spending the high-resolution budget only in the final cooldown is what the paper calls progressive resolution training, and it cuts the ViT-g run from a projected 60 GPU-years naively to roughly 7 at 384 pixels and 64 frames (an 8.4× reduction). The anticipation, classification, and planning results all use that same encoder. ^[2]

One caveat. Scaling at this level is a Meta-only result in practice: ViT-g at 22 million clips on their cluster is not reproducible in a university lab. The paper's contribution to the rest of us is the schedule and the progressive-resolution recipe; both transfer to smaller budgets and both are simple.

Adding actions: V-JEPA 2-AC

So far there are no actions anywhere. V-JEPA 2 is trained on internet video, plain sequences of frames; there is no robot in it and no notion of what someone did between frames. To control a robot you have to predict not just what comes next in a video, but what comes next given that I take this action. The post-training stage stitches that on top.

The data is 62 hours of unlabeled robot video from the Droid dataset, a public collection of teleoperated 7-DoF Franka Panda arm trajectories. "Unlabeled" means the paper does not use what task each clip was trying to do, whether the demonstration succeeded, or any reward. What it uses is the raw video plus the seven-dimensional end-effector state at each frame: three numbers of Cartesian position, three of orientation, one for the gripper. The action between consecutive frames is the change in end-effector state, $a_k = s_{k+1} - s_k$, which means the model learns velocity-style control even though nobody ever wrote that down as a label.

The V-JEPA 2 encoder is frozen; only a new predictor is trained on top. Call this predictor $P_\phi$ again, by analogy with the pretraining one, even though it is a different network: 300M parameters, 24 layers, 16 heads, 1024 hidden dimension, block-causal attention so each timestep can attend to past actions, past states, past patches. The state at time $k$ is the encoded frame $z_k = E(x_k)$, a feature map of shape $16\times 16\times 1408$. The predictor consumes the interleaved sequence of actions, proprioceptive states, and feature maps, and outputs a prediction for the next feature map.

Now training has two losses. The first is teacher forcing: at every step $k$, feed the predictor the true history and ask it to predict $z_{k+1}$, with the truth supplied by the frozen encoder. Average over $T$ steps:

The second is a rollout loss. Run the predictor autoregressively for $T$ steps starting from $(s_1, z_1)$, feeding each output back in as the next input, and compare the final imagined feature map to the true one:

The training objective is the sum:

Why two losses, when (2) already supervises every step? Because of how each one will be used at deployment. The teacher forcing in (2) trains the predictor to do one step accurately given a real previous frame; that one-step skill is what an MPC controller needs at the start of every plan. The rollout in (3) trains the predictor to recover from its own mistakes, because at planning time it will spend most of its rollouts looking at its own imagined feature maps, not at real frames. Without (3), one-step accuracy compounds badly. The paper uses $T=2$ for the rollout, so backpropagation only has to chain through one recurrent step; even with that mild rollout, the planning rollouts at inference time can run much longer because the predictor has been exposed to at least one of its own errors during training. In the figure below the rollout length T is interactive, and the drift visibly accumulates with T:

The predictor uses 3D-RoPE for the video patch positions but only the temporal component of RoPE for the action and proprio tokens. Actions and end-effector states have no spatial location of their own, so giving them a fake one would just confuse the relative-position read. ^[3] The block-causal attention pattern is the same as in language models: at time $k$, the patch features attend to all earlier patches and to the action and proprio tokens at the same and earlier times.

Planning by energy minimization

With a trained dynamics model the planning loop is short. The agent is given a goal image $x_g$, encoded once into $z_g = E(x_g)$. At control step $k$, the current frame is encoded into $z_k$ and the current proprio state $s_k$ is read off the robot. A candidate plan is a sequence of $T$ actions $\hat a_{1:T}$. We score it by how close the imagined final feature map ends up to the goal, in L1:

The best plan is the one that minimizes that energy:

This picture has no reward function, no task label, and no inverse model that asks "what action would get me to the goal" in closed form. Because the world model predicts the consequence of action sequences and the L1 energy scores the result, the controller reduces to an optimizer over those scores. The L1-in-feature-space "energy" replaces a hand-engineered task reward, and the energy minimum implicitly defines what success means: an action sequence that makes the world look the way the goal image does, to the encoder.

We never see the analytic minimum, because $P_\phi$ is a transformer and the energy in (5) is not differentiable in any way you would want to follow. So the paper optimizes (5) with the cross-entropy method (CEM), a gradient-free population search. CEM keeps a Gaussian distribution over candidate action sequences, samples a batch from it, scores them with (5), refits the Gaussian to the top-K best, and repeats. Eight or ten iterations of that, with a population of 800 sequences each iteration, is enough to land near the minimum. CEM is old (Rubinstein 1997), simple, parallelizable, and the only thing that needed adapting for V-JEPA 2-AC was that each sampled action is clipped to an L1 ball of radius 0.075, about 13 cm of end-effector displacement, because larger jumps go out of the training distribution.

The paper visualizes (5) on a single-action slice for a reaching task: sweep $\Delta x$ and $\Delta y$, hold $\Delta z = 0$, and the energy bowl is smooth and locally convex with a minimum near the ground-truth direction $(0, -0.05)$. CEM exploits that smoothness. Below, advancing the CEM iteration slider shows the sample population (white dots) collapsing onto the elite top-K (teal) as the current Gaussian (teal ellipse) shrinks onto the bowl:

The other knob that planning gives you is the horizon $T$. With $T=1$, you optimize one action, execute it, observe, repeat: this is visual servoing in latent space. With $T=4$ the planner imagines four steps into the future and commits only to the first; if its imagined fourth step is wrong, the next replan corrects. That is the receding-horizon control loop, classical MPC, with the dynamics swapped for a video model. The figure below shows it from above: a tabletop, a goal mark, the executed path so far in gray, the imagined four-step plan in teal, and a ring on the one action the controller will actually execute before re-encoding and re-planning:

CEM with 800 samples and 10 refinement iterations takes about 16 seconds per action on a single RTX 4090, which is what makes a full pick-and-place trajectory feasible (40-or-so actions, about ten minutes). The Cosmos baseline, a latent video diffusion model that has to denoise an entire imagined video for every scored plan, takes four minutes per action. That 15× planning-cost gap is, before accuracy even enters the picture, why JEPA-style latent dynamics beat pixel-generation dynamics for control.

Two arms, neither seen in training

Zero-shot means something specific here. V-JEPA 2-AC is trained on Droid, which is a teleoperation dataset collected at other labs, with other arms, in other rooms, under other lighting. The paper deploys the same checkpoint on two Franka arms in Meta's own labs, neither of which appears in Droid, with no additional fine-tuning. The goal is given as a single image; the robot sees only a monocular RGB feed; the controller plans with CEM and (5). Three skills are evaluated: grasp a cup or box, reach with an object held in the gripper, and pick-and-place using three sub-goal images stitched in sequence (gripper near object, object near goal, object at goal). ^[4]

The success rates below average across two labs and ten trials each. The Octo baseline is a vision-language-action transformer trained on the much larger Open-X Embodiment dataset (1M trajectories) and fine-tuned on Droid with behavior cloning. V-JEPA 2-AC is trained from 23k Droid clips, no behavior cloning, no language. On the harder skills the gap widens sharply:

Numbers from the figure (averaging cup and box trials together): V-JEPA 2-AC reaches 45% on grasp (vs 7.5% for Octo), 75% on reach-with-object (vs 42.5%), and 72.5% on pick-and-place (vs 12.5%). All zero-shot, in environments the model has never seen. Pick-and-place is the task that matters for practice, because it requires composition: a sequence of sub-goals must each be hit in turn, and a failure on any of them sinks the trajectory.

Aligning the frozen encoder with a language model

The same V-JEPA 2 encoder, frozen, also drives understanding benchmarks. The recipe is the standard one for video LLMs: project encoder outputs into the embedding space of a language model with a small MLP, then train the projector plus the language model on video-instruction pairs while leaving the encoder fixed. With an 8B-class language model behind it, the paper hits a state-of-the-art 84.0 on PerceptionTest, 76.9 on TempCompass, 44.5 on MVP, 36.7 on TemporalBench, and 40.3 on TOMATO. These all live in the band of benchmarks that need temporal reasoning, not just frame-by-frame appearance. ^[5]

V-JEPA 2 was pretrained without any language supervision at all. Conventional wisdom from CLIP-trained encoders is that an encoder that has never seen text cannot be aligned to a language model competitively. V-JEPA 2 says otherwise. The encoder gets language alignment in a separate, cheap, late stage; the language model meets the video encoder for the first time during instruction tuning, and the resulting video-LLM still leads the 8B class on tasks that depend on motion understanding. Language is not required for learning to see, even when the downstream task is to talk about what was seen. That implication runs alongside the planning result and is the second claim of the paper.

On the probe benchmarks the encoder itself is evaluated on, a single frozen V-JEPA 2 ViT-g hits 77.3 on Something-Something v2 (motion understanding) and 39.7 recall-at-5 on Epic-Kitchens-100 (action anticipation), the latter a 44% relative improvement over the previous best task-specific anticipation model. Anticipation is the telling result here, because from a clip ending at time $t$ the model has to predict what action will begin at $t + \tau$ for $\tau$ a second into the future. A pretrained representation that has learned to fill in missing pieces of video, by construction, knows something about what tends to come next; the anticipation number is the cleanest evidence that the predictor module learned dynamics rather than just texture.

Where it still breaks

The paper's own limitations section is clear-eyed, so this one is too.

Camera position is load-bearing. V-JEPA 2-AC takes Cartesian end-effector deltas as actions, but it has no camera calibration. To predict the visual consequence of an action, it has to implicitly infer which way the world's x-axis points in the image. When the robot base is out of frame, that inference is under-determined and the model mis-predicts which way a positive $\Delta x$ will move things on screen. The paper's appendix shows the success rate degrading sharply with camera angle perturbations. In practice the authors hand-tuned the camera position until everything worked, which is an asterisk on "zero-shot."

Long horizons compound. The autoregressive predictor was trained with a two-step rollout. At planning time it runs many more steps, and its representation-space drift grows with the rollout length. Combined with the exponential growth of the search space (a horizon of $T$ actions in a 7D continuous space is a vast hypercube for CEM to sample), this caps the practical planning horizon well below what you would want for, say, a multi-stage assembly task without sub-goals. The pick-and-place result relies on three sub-goal images precisely because the single final goal does not yield enough signal to plan all the way through. Free pick-and-place from a single end goal is the obvious next step and the paper does not solve it.

Goals must be images. The energy in (5) is L1 distance to an encoded goal frame. That requires you to have a goal frame, which is fine in a lab and awkward in the world. Asking a robot to "pour me a coffee" in natural language is not what this model can do; that is left to future work that would marry a language model's goal-encoding to V-JEPA 2-AC's dynamics.

No way to render predictions. V-JEPA 2 has no decoder, so it cannot draw an imagined future frame back into pixels for a human to inspect. For planning by latent distance this is the right architectural call (everything you save by not predicting pixels you can spend on dynamics), but it does mean any sanity-check on the world model has to happen in feature space, which is hard to read. The energy landscape figure above is partly a workaround, and partly evidence that there is no neater way to do it.

The pipeline itself is what the paper does prove. Internet-scale video, plus a feature-space prediction objective, plus a small amount of robot interaction data, plus model-predictive control, plus a single goal image, gives a robot that picks up cups it has never seen in a lab it has never been in. The individual pieces are not new; the assembly is, and the pick-and-place gap to the next baseline is large enough that the assembly itself is the contribution.