OpenVLA: An Open-Source Vision-Language-Action Model

A language model that writes the robot's next move.

Tokenize the seven numbers that control a robot arm, hand them to a 7B model that already speaks images and English, and the same machinery that predicts the next word predicts the next action. The open weights and the recipe matter more than the headline gap.

Robotics has lacked the open, fine-tunable backbone that language modeling got from Llama. This paper builds it.

For a year before OpenVLA, the best generalist robot policy was RT-2: a vision-language model fine-tuned on robot demonstrations, with the continuous control signal turned into discrete tokens so the language model could write actions the same way it writes words. RT-2 worked. It generalized to unseen objects. It could be given a new instruction and follow it. And it was closed. The weights were never released, the recipe was sketched but not reproducible end-to-end, and adapting it to a new robot meant going through an API that nobody outside Google could call. Robotics, which is bottlenecked on the cost of new data, was looking at its best model from the outside.

OpenVLA is the open version of that idea, with a few choices that change the numbers. The recipe it inherits from RT-2 is the same one in spirit: take a 7B vision-language model, train it on robot trajectories, treat each action as a short string of tokens. What OpenVLA adds is the open Llama 2 backbone, a curated 970,000-trajectory slice of the Open X-Embodiment dataset (versus the 350k subset RT-2-X used), and a visual encoder that fuses two pretrained backbones instead of one. The headline result is that with one-seventh the parameters of RT-2-X (7B versus 55B), it wins 16.5 percentage points of average success rate across 29 manipulation tasks. The contribution that probably matters more in the long run is the fine-tuning story: with LoRA at rank 32, a single A100 fine-tunes OpenVLA on a new robot setup in 10 to 15 hours, at 1.4% of the parameters, within the error bar of full fine-tuning's success rate.

To see how this works, a few pieces carry the story: what a vision-language-action model is and how the three parts of OpenVLA fit together, how continuous robot actions become discrete tokens a language model can predict, what the training loop actually does, the gap to RT-2-X and where it comes from, and the two efficiency results, LoRA and 4-bit quantization, that put a 7B robot policy on a single GPU.

A language model that writes actions

A robot controller, at the lowest useful level, is a function that takes one camera image and one instruction string and returns seven numbers. The seven numbers are the next move, expressed as small changes the end-effector should apply: three for translation along x, y, z, three for rotation in roll, pitch, yaw, and one for the gripper (open versus closed). Call that 7-vector $\mathbf{a}_t \in \mathbb{R}^7$. A policy $\pi(\mathbf{a}_t \mid \mathbf{o}_t, \ell)$ maps the current observation $\mathbf{o}_t$ (the image) and the instruction $\ell$ (English text) to a distribution over those seven numbers. Run the policy in a loop, integrate the small moves, and the arm goes somewhere.

The trouble with classical imitation learning is the supervision signal. You collect demonstrations: a person teleoperates the arm through a task and you record the camera and the actions, hundreds of times. Then you train a network to map the camera to the actions. The networks were small, the data was small, and the resulting policies fell apart the moment the scene changed: a new background, a new object, a new way of asking. They had never seen the Internet.

A vision-language model has. CLIP, SigLIP, and DINOv2 read images by training on billions of image-text or image-only pairs scraped from the web; Llama 2 reads English the same way through text. Glue them together and you have a model that already knows what a coffee cup is, what "to the left of the bowl" means, and that a green apple is different from a red one. The vision-language-action recipe, due to RT-2, asks: what if the policy were a vision-language model fine-tuned to emit actions? The Internet-scale knowledge is already in the weights, and the fine-tuning only has to teach the model what an action is.

For that to work, the actions have to be something the language model can already write. A language model writes tokens, integers picked from a fixed vocabulary, one at a time. So OpenVLA turns each of the seven action numbers into one such integer, discretizing each dimension into 256 bins. Then the action becomes a 7-token string the LM can autoregress alongside the image and the instruction. Predict, integrate, repeat.

The three parts

OpenVLA is three modules stacked left to right (the paper's Figure 1, redrawn below). On the left, a visual encoder turns an image into a sequence of patch features. In the middle, a small projector maps those features into the language model's embedding space so the LM can read them as if they were text tokens. On the right, the language model takes the patches, the instruction, and the partial action sequence, and decodes the next action token.

The visual encoder is the one design choice that materially changes the headline gap to RT-2-X. OpenVLA does not use a CLIP or a SigLIP backbone in isolation. It uses both SigLIP and DINOv2, in parallel, on the same image. Each model produces a per-patch feature vector; the two vectors are concatenated along the channel dimension (so each spatial location gets one wider feature). SigLIP is contrastively trained on image-text pairs and is good at "what is this"; DINOv2 is self-supervised, meaning it learns from images alone with no labels, and is good at "where is what." Image-text contrastive training teaches the encoder to compress out spatial detail that does not change the caption (a cup is a cup whether it is two centimeters left or right), which is exactly the detail a controller needs. Stapling DINOv2 alongside puts that detail back. The recipe is not OpenVLA's own; the fused encoder, the 2-layer MLP projector, and the channel-wise concat are the Prismatic VLM's design (Karamcheti et al., 2024), and OpenVLA chose Prismatic over IDEFICS-1 and LLaVA in early experiments. Prismatic beat LLaVA by about 10 percentage points on Bridge tasks, in absolute success rate, and that 10-point gap is most of the lead OpenVLA carries over RT-2-X.

The projector is the unglamorous part that matters: a 2-layer MLP that maps the fused per-patch feature into the same dimension Llama 2 uses for its token embeddings. Without it, the visual features live in their own space and the LM cannot read them. With it, the patches become "visual tokens" that the LM sees in exactly the same way it sees text tokens, and the same attention mixes both. This is the standard VLM construction LLaVA popularized.

The language model is Llama 2 7B. The choice is consequential mostly because it is open: the weights are public, the tokenizer is documented, the inference stack is the same one a thousand other projects use. The choice is not, on its own, what makes the policy good; the Bridge ablation showed that swapping in IDEFICS-1 or LLaVA gives a worse model on the same recipe. Llama 2 7B is the size where the pretrained world knowledge is wide enough to recognize Internet objects without bloating inference past the point where a single GPU can serve it at 6 Hz.

Continuous actions, 256 bins, one Llama token each

A language model writes integers from a fixed vocabulary. Robot actions are floats. To make the two compatible, OpenVLA discretizes each of the seven action dimensions independently into 256 uniform bins, then assigns each bin to a specific Llama token id. The recipe is:

1. For each dimension, compute the 1st and 99th percentile of that dimension's values across the entire training dataset. Call them $q_1$ and $q_{99}$.
2. Divide the interval $[q_1, q_{99}]$ into 256 equal-width bins. Any value outside the interval saturates to the nearest end bin.
3. Pick the 256 least-used token ids in the Llama tokenizer (the last 256 of its 32,000) and overwrite their embeddings to mean "bin $k$ of dimension $d$." That way an action vector becomes 7 token ids the LM can produce by ordinary next-token prediction.

Two details earn their place. The 1st-to-99th-percentile clip, rather than the min-to-max range RT-1 used, calibrates the bin width to typical motion instead of to the worst data point; one operator who flailed the arm to the limits of the workspace can no longer stretch every bin into uselessness. The least-used-token overwrite exists because the Llama 2 SentencePiece tokenizer only leaves 100 free slots for tokens added during fine-tuning, and 256 will not fit in 100. The tokens it overwrites essentially never appear in Internet text; the pretraining knowledge lost by repurposing them is negligible.

Drag a dimension across its range below and watch where its discretized value lands. At the edges of the slider, past the 99th percentile, the value saturates: the same token comes out no matter how much further you push it. Saturating outliers is the deliberate cost of the quantile-based scheme, and a strong argument against ever clipping at the min/max:

With the encoding fixed, the rest of the architecture stays as a vision-language model. The image flows through the encoders and the projector and arrives at Llama as a sequence of "visual tokens," one per spatial patch. The instruction flows through the tokenizer and arrives as a few dozen text tokens. The previous action tokens, if any, arrive too. Llama predicts the next action token from all three. At training time, the loss is a vanilla per-token cross-entropy evaluated only on the seven action tokens; the model is not asked to reconstruct the image or the instruction, only to write the right action. In code:

# one training step (per-token cross-entropy on action tokens only)
img, instr, action = sample_batch()        # img 224x224, action in R^7

# fused visual encoder (Prismatic): both backbones, per-patch concat
feat_s = siglip(img)                       # [N_s, d_s]
feat_d = dinov2(img)                       # [N_d, d_d]
feat   = concat_channels(feat_s, feat_d)   # per spatial location
visemb = mlp_projector(feat)               # mapped into Llama embed space

# discretize each of the 7 action dims into one of 256 bins
clipped = clip_to_percentiles(action, p1, p99)
ids     = discretize(clipped, 256)         # 7 ints in [0, 255]
toks    = action_token_ids(ids)            # last-256 ids of Llama vocab

text_ids = tokenize(instr)
seq      = [visemb, embed(text_ids), embed(toks)]  # one long sequence

logits = llama2_7b(seq)
loss   = cross_entropy(logits[-7:], toks)  # only the 7 action tokens
loss.backward(); opt.step()                # vision encoder is NOT frozen

How it is trained

OpenVLA starts from a Prismatic-7B VLM that has already been jointly trained on the LLaVA 1.5 image-text mixture, so the visual encoder, projector, and Llama 2 backbone are already speaking the same language at initialization. From there, the whole stack is fine-tuned on 970k robot demonstrations curated from Open X-Embodiment, with cross-entropy on the action tokens. Three choices in the recipe are not obvious until they are written down.

Should the vision encoder be frozen or fine-tuned along with the rest? The conventional answer for VLM training is to freeze it: the Prismatic paper, which gave OpenVLA its initialization, found freezing gave better question-answering numbers, because the pretrained encoder already captures what the text task cares about. The OpenVLA authors started there too, and it did not work: the frozen-vision ablation (Table 1) gets 47.0% rollout success versus 69.7% for full fine-tuning. The hypothesis is that image-text contrastive pretraining throws away exactly the spatial precision a controller needs, a few centimeters of offset between a gripper and a handle, and the only way to put it back is to let the encoder learn from the robot demonstrations. The Sandwich and full-fine-tune rows in Table 1 outperform Last-layer-only for the same reason: only the strategies that touch the encoder give it room to learn the controller's spatial geometry.

The model trains for 27 epochs through the 970k-trajectory mixture, not the one or two passes typical of LLM and VLM pretraining. That is a lot of repetition. The authors found that real-world success kept climbing as long as training action-token accuracy did, and that did not plateau until accuracy was past 95%. A standard one-pass schedule would have stopped well short of the policy you can actually deploy. The signal is that on physical tasks, the bottleneck is not how many trajectories you train on; it is how precisely you fit the trajectories you already have.

The learning rate is held fixed at $2 \times 10^{-5}$, the same one Prismatic used for VLM fine-tuning, with no warm-up. The compute budget is 21,500 A100-hours: 64 A100s, 14 days, batch size 2048. That is roughly the cost of a small open-source LLM pretraining run, and exactly the cost the open-source community absorbs once and then forks. The inference footprint is 15 GB of GPU memory at bfloat16, which fits on a single consumer card.

What it actually does

The evaluation runs across three suites. The first is the BridgeData V2 WidowX setup: 17 tasks, 170 rollouts per method, designed to test four axes of generalization that a person would distinguish (visual, motion, physical, semantic) plus a language-grounding axis. The second is a Google robot with 12 tasks and 60 rollouts, split into in-distribution and out-of-distribution. The third is Franka fine-tuning, where every method is fine-tuned on the same 10-to-150 demonstrations and the comparison is against the data-efficient state of the art, Diffusion Policy.

The headline number is the BridgeData V2 average: OpenVLA at about 71% success, RT-2-X at about 55%, a 16.5 percentage-point gap with a seventh of the parameters. That gap is not uniform across axes. OpenVLA leads on visual generalization (new backgrounds, distractors), motion (unseen positions), physical (unseen shapes), and language (multiple objects, name the right one). It trails on semantic generalization: the case where the instruction refers to a category the robot never saw in its demonstrations and has to lean on Internet world knowledge. RT-2-X edges OpenVLA there because PaLI-X is co-fine-tuned with Internet image-text data alongside robot actions, which keeps more of the pretraining knowledge alive. OpenVLA fine-tunes only on robot data and pays a small semantic tax for it.

Toggle the suites below. Bridge is where the 16.5-point headline lives, Google is the tie that confirms the result is not a Bridge-specific artifact, and the Franka fine-tune is where the more interesting question gets asked: is OpenVLA also the right starting point when you have new data and want to adapt? Drag the axis slider to read off the gap per category. Watch the semantic column on Bridge: that is the only one where the bigger model still wins.

The Franka fine-tune story deserves its own gloss. When the task is narrow and dexterous, a single instruction on a single object that has to be done precisely, Diffusion Policy still wins. The likely reason is that Diffusion Policy uses action chunking (it predicts a chunk of future actions and executes the first few open-loop) and temporal smoothing, both of which give it more stable trajectories than OpenVLA's one-step autoregressive output. When the task is diverse, multiple objects, multiple instructions in the same setup, Diffusion Policy collapses because it has no way to ground language. OpenVLA wins those by 20 absolute points on average. The authors are explicit that this is not a clean "OpenVLA always wins" result. Diffusion Policy is the right tool for narrow dexterous tasks; OpenVLA is the right tool for diverse language-grounded ones; OpenVLA alone clears 50% on every task tested.

Where does the 16.5-point Bridge gap actually come from? The authors break it into three sources. The training set is larger and more carefully cleaned (970k trajectories versus RT-2-X's 350k, with all-zero Bridge actions filtered out instead of left in). The encoder is fused DINO+SigLIP, which adds spatial precision RT-2-X does not have. And OpenVLA is fine-tuned only on robot data, so it does not waste capacity preserving Internet-text behavior the robot will never need. The first two help everywhere; the third costs them semantic generalization but pays for itself elsewhere.

Fine-tuning, cheaply

Out-of-the-box performance is half the story; the other half is whether a lab with a different robot can pick this up. Full fine-tuning of OpenVLA on a new Franka task takes 8 A100s for 5 to 15 hours, depending on dataset size. That is not nothing. The paper's second contribution is showing this can be cut by roughly 8× without measurable quality loss using LoRA at rank 32.

LoRA inserts a small rank-$r$ update $\Delta W = B A$ into each weight matrix, training only those rank-$r$ factors and leaving the original weights frozen. For OpenVLA at $r=32$, that comes out to 97.6M trainable parameters, about 1.4% of the 7.2B-parameter total. The trade-off the paper measures (Table 1) is success rate on the Franka-Tabletop tasks versus GPU memory at batch size 16. Plot it and the sweet spot is obvious: LoRA $r=32$ sits in the top-left corner at 68.2%, statistically indistinguishable from full fine-tuning's 69.7% (the 1.5-point gap is well inside each method's ±7-point standard error across 33 rollouts), at 59.7 GB instead of 163.3 GB, on a single GPU instead of two with FSDP.

The other strategies in the table are instructive failures. "Last layer only" (fine-tune just the final transformer block and the embedding table, freeze the rest) is cheap at 51.4 GB but barely halves the performance gap to a from-scratch model. "Frozen vision" (fine-tune the LM but freeze the encoder) does worse than full fine-tuning because, as the authors flagged earlier, the visual encoder needs to learn the controller's spatial geometry. "Sandwich" (fine-tune the encoder, the embedding table, and the last layer) is the runner-up at 62.1% and 64.0 GB; it does better than the frozen-vision row precisely because the encoder is unfrozen. LoRA wins because it lets every weight matrix adapt a little, including the encoder's, without paying for the full backward pass through 7B parameters.

4-bit inference, no rollout hit

Once the model is fine-tuned, the next question is what you need to run it. At bfloat16 (the default), OpenVLA takes 16.8 GB of memory at batch 1 and runs at about 6 Hz on an RTX 4090. That fits on a consumer card. It does not fit comfortably on something smaller. The paper benchmarks two LLM-style quantization recipes, 8-bit and 4-bit, on the same 8 BridgeData V2 tasks.

4-bit quantization halves the memory to 7.0 GB, and rollout success comes out at 71.9% versus the bfloat16 71.3%. Within the error bars, those are identical. 8-bit, which you might expect to be safer, comes out at 58.1%, and the explanation is not about precision. On the A5000 used for the rollouts, 8-bit inference runs at 1.2 Hz (versus 3 Hz for 4-bit and 5 Hz for bfloat16) because the 8-bit quantization kernels add more overhead than the memory bandwidth they save. The BridgeData V2 controller expects actions at 5 Hz. At 1.2 Hz, the closed loop is so out of phase that the rollouts go off the rails. The paper confirms this by measuring offline action-token accuracy, which is comparable for all three precisions: the tokens 8-bit emits are correct, the robot just sees them too late.

That lesson does not show up in ordinary LLM serving. For text generation, an extra hundred milliseconds per token is a UX issue. For closed-loop control at 5 Hz, an extra two hundred milliseconds is the difference between grasping the cup and missing it; the system dynamics the policy was trained against require a particular cadence. Quantization for VLAs is a latency question on top of the usual memory and accuracy questions. 4-bit wins because it is the only setting fast enough to keep the closed loop in phase.

What it cannot do yet

The paper names its own ceiling. OpenVLA takes a single image observation: no history, no proprioception, no second camera. Real robot setups are heterogeneous; the next iteration would have to support all of them, and doing it through a VLA fine-tune likely needs a VLM pretrained on interleaved image-text data so it can read multiple images natively.

Inference is the second ceiling. 6 Hz on a 4090 is fine for tabletop manipulation and useless for the bimanual, dexterous tasks that an ALOHA-class setup runs at 50 Hz. The proposed escape routes are action chunking (predict $k$ actions at once, execute open-loop, which spreads one forward pass over $k$ control cycles and effectively multiplies the policy's control frequency by $k$) and speculative decoding (let a smaller model draft tokens, have the big model verify), both of which would have to be ported from LLM serving and re-evaluated for whether the open-loop assumption holds when a robot is touching things.

And the policy does not yet hit very-high reliability on the tested tasks. Most successes land in the 70-to-90% band, with few above 90%, which is the band where a method is interesting to robotics researchers and not yet ready to ship behind a customer-facing demo. That is exactly where the open weights, the open codebase, and the LoRA recipe give the next research group a place to start.

None of the individual components are new. The fused vision encoder is Prismatic's. The action discretization is RT-1's, packaged into a VLM by RT-2. The multi-embodiment data pool is Open X-Embodiment's. LoRA is from the LLM-tuning world. What was missing was the assembly into one open system: an open backbone running open weights, with a 970k-trajectory training mix that beats the closed 55B-parameter incumbent, and a fine-tuning recipe that puts adaptation onto one GPU for a day. OpenVLA is that assembly, and it shortens the next paper's starting line.