OpenVLA: An Open-Source Vision-Language-Action Model

A 7B vision-language model that outputs a robot arm's next move as seven tokens.

OpenVLA maps each of the seven numbers that command a robot arm to one of 256 tokens, then fine-tunes an open vision-language model (Llama 2 7B with two image encoders) to predict those tokens from a camera frame and an instruction. Trained on 970k robot episodes, it beats the closed 55B RT-2-X.

The weights and training code are public, and the paper shows how to fine-tune the model for a new robot on one GPU and how to run it with 4-bit weights.

In 2023 Google DeepMind's RT-2 (project page here) showed that a vision-language model fine-tuned on robot demonstrations can control a robot arm. The arm's continuous commands are converted to integer tokens, and the model predicts them the way it predicts words. Its successor RT-2-X, a 55B-parameter model trained on the pooled Open X-Embodiment robot dataset, was the state of the art for generalist manipulation in early 2024. Neither model's weights were released, and RT-2-X could be queried only through an inference API that does not support fine-tuning.

OpenVLA, by Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti and 15 co-authors, rebuilds the recipe from open parts. It starts from Prismatic-7B, an open vision-language model built on Llama 2 7B, and trains it on 970k robot episodes curated from Open X-Embodiment. With 7B parameters to RT-2-X's 55B, it scores 16.5 points higher in absolute task success, averaged over 29 tasks on two different robots. The paper then measures two ways to make the model cheaper. LoRA fine-tuning on one A100 in 10 to 15 hours matches full fine-tuning on a new robot, and 4-bit weights cut inference memory from 16.8 GB to 7.0 GB with no loss in rollout success. Both were measured on a smaller variant of the model, described in the LoRA section.

The page follows the paper's order: what a manipulation policy outputs; the three parts of the model and the token sequence it trains on; how seven action numbers become seven tokens and back; the training data and recipe; inference; the comparisons with RT-2-X, Octo, RT-1-X and Diffusion Policy; and the fine-tuning and quantization experiments, including why 8-bit inference lost rollouts that 4-bit won.

What a robot manipulation policy outputs

A manipulation policy is a function called in a loop. Each call takes the current camera image and an instruction such as "put the eggplant in the pot" and returns one command for the arm. In OpenVLA the command is seven numbers: the change in the gripper's position along x, y and z, the change in its orientation as roll, pitch and yaw, and the gripper opening, where 0 is closed and 1 is open in the data pipeline OpenVLA uses. The arm's controller turns each relative command into joint motion, and the loop repeats from the new camera frame.

On BridgeData V2, the WidowX dataset behind the paper's main evaluations, 98% of recorded x-axis steps lie between −0.0287 and +0.0283, under 3 cm per command. The demonstrations were recorded with a 5 Hz controller, a new command every 0.2 s, so a policy trained on them is expected to keep up with about that rate, which matters for the quantization result at the end of the page.

Before OpenVLA, learned policies came in two kinds. Imitation-learning policies such as Diffusion Policy are trained from scratch on tens to hundreds of demonstrations of one task in one scene; they can be precise, but everything they know about objects comes from those demonstrations. Generalist policies such as RT-1-X (35M parameters) and Octo (93M) are trained on data pooled from many robots, but they are still small transformers trained from scratch on robot data alone. A vision-language model has instead been pretrained on Internet-scale images and text, so it already associates a photo of a pot with the word "pot" and with "the metal one on the left". A vision-language-action model (VLA) is such a model fine-tuned to output robot commands, so that the policy can use what the model learned from the Internet.

How OpenVLA works: three parts and one token sequence

OpenVLA is the Prismatic-7B vision-language model with no new modules. Prismatic has three parts, drawn in the top half of Figure 1.

The vision encoder runs two pretrained vision transformers on the same 224×224 image. Each cuts the image into 14×14-pixel patches, a 16×16 grid of 256 patches, and outputs one vector per patch (the ViT explainer covers how a transformer reads patches). SigLIP SO400M was trained to match images with their captions, the CLIP objective with a sigmoid loss, and outputs 1152 numbers per patch. DINOv2 ViT-L was trained on images alone by self-distillation (the method of DINO, scaled up) and outputs 1024. For each patch the two vectors are concatenated into one vector of 2176 numbers, so the encoder's output for one image is a 256 × 2176 matrix. The paper sizes the two encoders together at 600M parameters and calls their outputs semantic and spatial features. A plausible reason for the split: a caption rarely changes when a cup moves 2 cm, so caption-matching training has little reason to keep exact positions, while DINOv2's image-only objective does. Prismatic's authors found that adding DINOv2 features improves spatial reasoning, which a controller needs.

The projector is a small MLP that maps each 2176-number patch vector to 4096 numbers, the width of a Llama 2 token embedding. This turns the image into 256 image tokens that the language model processes exactly as it processes embedded words; this construction is the one LLaVA popularized. The paper calls the projector a 2-layer MLP. The fused projector in the released code has three linear layers (2176 → 8704 → 4096 → 4096) with GELU activations between them.

The language model is Llama 2 7B, a 32-layer decoder-only transformer from the LLaMA family. It predicts the next token at every position. OpenVLA changes nothing in it except the meaning of 256 rows of its vocabulary, described in the next section.

The bottom half of Figure 1 shows one training example as Llama sees it: a start token, the 256 image tokens, the prompt "In: What action should the robot take to put the eggplant in the pot?\nOut:", seven action tokens and an end token, a little under 300 positions in total; hovering a segment shows whether it carries a loss.

Why Prismatic and not a better-known model? The authors fine-tuned three open vision-language models into policies on BridgeData V2. LLaVA and IDEFICS-1 were comparable on single-object tasks, and LLaVA was 35% better in absolute success rate on five language-grounding tasks, where the scene holds several objects and the instruction names one. Prismatic beat LLaVA by roughly 10% absolute on both kinds of task, which the authors attribute to the fused SigLIP + DINOv2 features. Input resolution stayed at 224×224: a 384×384 version showed no difference in their evaluations and took 3 times longer to train, because the number of image tokens grows with the square of the side length.

How a continuous action becomes seven tokens

Llama 2 outputs a probability distribution over the 32,000 entries of its vocabulary, so it cannot output the number 0.0100 directly. OpenVLA follows RT-2 and discretizes each action dimension separately into 256 bins, which the released code does in three steps.

First, each of the six motion dimensions is normalized with two statistics of the training data: its 1st percentile $q_{01}$ and its 99th percentile $q_{99}$, computed separately for every dataset in the mix. The map sends $q_{01}$ to −1 and $q_{99}$ to +1 and clips everything outside:

The gripper value is left as it is, 0 or 1, which already lies in [−1, 1]. Second, 256 evenly spaced edges cut [−1, 1] into 255 equal bins, and $u$ is replaced by the index of its bin (the end of this section explains why this still gives 256 token ids). Third, bin index $d$, which the code numbers 1 to 256, becomes token id

so the action vocabulary is the last 256 ids of Llama's 32,000, and a larger action value gives a smaller id. Decoding runs the steps backwards: id to index, index to the centre of its bin, centre back through (1).

A worked example with the real BridgeData V2 statistics from the released checkpoint: for the x-axis, $q_{01} = -0.02873$ and $q_{99} = +0.02831$. Take a command of $a = +0.010$, 1 cm along x:

Decoding token 31,826 gives back the centre of bin 174, which un-normalizes to 0.010081: an error of 0.08 mm. Each bin is $0.05704 / 255$ = 0.224 mm wide, so any command between the two percentiles comes back within 0.11 mm. The full action for this step, with small moves on the other axes and the gripper open, is the seven ids shown in Figure 1's action cells.

Why the 1st and 99th percentiles and not the smallest and largest values ever recorded, the bounds RT-2 used? The recorded x-axis motion in BridgeData V2 runs from −0.401 to +0.417, more than 14 times the width of the percentile band. With min-max bounds each bin would be 3.2 mm wide, and the middle 98% of all recorded commands would share about 18 of the 255 bins, so the model would have about 18 distinct values for nearly every x-axis motion it ever makes. Yaw is worse: its recorded extremes are ±6.28 rad, a full turn, while 98% of yaw commands lie within ±0.21 rad, a band that min-max bounds would cover with 8 bins. With percentile bounds the band gets all 255 bins, and the cost is that the 2% of commands outside it are clipped to the end bins.

At the slider's ends, past the percentile band, the readout marks the value clipped: every x-axis command above 0.0283 becomes the same token, and decoding returns 0.0283 or close to it. The gripper shows a side effect of skipping normalization: 0 and 1 land on tokens 31,872 and 31,744, and the lower half of its bins is never used.

Where do 256 free token ids come from? A tokenizer can normally be extended with new tokens, but the Llama tokenizer reserves only 100 special tokens for additions, fewer than 256. OpenVLA, again following RT-2, overwrites the 256 least-used tokens, which in Llama's vocabulary are the last 256. Their embedding rows and output-layer rows are retrained along with everything else, and the text pieces they used to encode can no longer be produced.

In the released code, np.linspace(-1, 1, 256) makes 256 bin edges, which bound 255 intervals. np.digitize returns 1 to 256, and index 256 is reached only by a value of exactly +1, which after clipping means any value at or above the 99th percentile. The model still uses 256 ids, with clipped-high values on their own id; decoding maps index 256 to the centre of the last interval.

Training: next-token prediction on 970k robot episodes

Training uses Llama's ordinary next-token cross-entropy, with every position except the action tokens masked out. For one example with image $o$, instruction $\ell$ and action tokens $t_1, \dots, t_7$, the loss is

and the released code adds the same term for the end token, so the model also learns to stop after seven. The image tokens and the prompt get no loss: the model is never trained to predict the instruction, only to produce the right command after reading it. The gradient reaches every weight, including both vision encoders. One training example in code:

# one OpenVLA training example (shapes for one image; batch omitted)
img, instr, a = sample()              # img 224x224x3, a: 7 floats
p_sig  = siglip(img)                  # [256, 1152]  16x16 patches of 14 px
p_dino = dinov2(img)                  # [256, 1024]  same patches
patch  = concat([p_sig, p_dino], -1)  # [256, 2176]
vis    = projector(patch)             # [256, 4096]  Llama embedding width

u   = clip(2 * (a[:6] - q01) / (q99 - q01) - 1, -1, 1)   # gripper a[6] kept as is
u   = concat([u, a[6:]])                                  # 7 values in [-1, 1]
d   = digitize(u, linspace(-1, 1, 256))                   # 7 ints in 1..256
act = 32000 - d                                           # 7 ids in 31744..31999

ids = tok("<s>In: What action should the robot take to " + instr + "?\nOut: ")
seq = [embed(ids[:1]), vis, embed(ids[1:]), embed(act), embed("</s>")]
logits = llama2_7b(seq)                    # next-token logits at every position
loss = cross_entropy(logits, labels)       # labels = -100 except act and </s>
loss.backward(); adamw.step()              # every weight gets a gradient

The data starts from Open X-Embodiment, which at the time held more than 70 robot datasets and more than 2M trajectories from 22 robots. The authors kept only manipulation datasets with at least one third-person camera and single-arm end-effector control, so that every example has the same kind of image and the same seven-number action, and weighted the remaining datasets with Octo's mixture, which down-weights less diverse datasets. They also added DROID, a newer large dataset, at a 10% weight, saw its action-token accuracy stay low, and removed it for the final third of training. They removed the first step of every BridgeData V2 episode, whose recorded action is all zeros; a model trained with those steps learned to output all-zero commands and froze during evaluation. After these steps 970k episodes remain.

Three choices in the recipe differ from standard vision-language training, and the authors made each from smaller BridgeData V2 experiments before the full run.

• The vision encoder is trained, not frozen. Freezing is the usual choice for vision-language models, and Prismatic's own experiments found it better; the intuition is that a frozen encoder keeps the features of its Internet-scale pretraining. For control the opposite held: at an earlier stage of the project, fine-tuning the encoder reached 80.0% average success on BridgeData V2 tasks against 46.7% with it frozen (Appendix D.3). The authors hypothesize that the pretrained features lack the fine spatial detail needed to place a gripper precisely.
• 27 epochs. Language and vision-language models usually see their data once or twice. OpenVLA's real-robot success kept improving until training action-token accuracy, the fraction of action tokens where the model's most likely token is the correct bin, passed 95%. The final run made 27 passes over the data.
• A fixed learning rate of 2e-5. The authors swept it over several orders of magnitude and kept 2e-5, the value Prismatic used for its own training, with no warmup, which they found gave no benefit.

The final run used 64 A100 GPUs for 14 days, 21,500 A100-hours, with a batch size of 2048. That cost is paid once; the fine-tuning recipes below start from the released checkpoint.

Running the policy: seven forward passes per command

At test time the prompt is built the same way, the image goes through the encoders once, and Llama generates seven tokens, one forward pass each, reusing the cached keys and values of earlier positions. The seven ids are decoded with the 1st and 99th percentiles of a dataset chosen by name, for example BridgeData V2 for a WidowX in a similar scene:

# inference: one command from one camera frame
ids  = llama2_7b.generate(prompt_with_image, max_new_tokens=7)  # greedy
d    = 32000 - ids                        # 1..256
c    = clip(d - 1, 0, 254)                # index of a bin interval
u    = bin_centers[c]                     # 255 centres in (-1, 1)
a    = where(mask, (u + 1) / 2 * (q99 - q01) + q01, u)  # mask False for gripper
robot.send(a)                             # q01/q99 of the chosen dataset

Because the statistics are per dataset, the command scale belongs to one robot and controller in the training mix; a new robot needs fine-tuning (and its own statistics). In bfloat16, 2 bytes per parameter, the model needs 15 GB of GPU memory and runs at about 6 Hz on one RTX 4090, without compilation, speculative decoding or other speed-ups. The authors also release a server that runs the model on a remote GPU and streams commands to the robot.

How OpenVLA compares with RT-2-X, Octo and RT-1-X

The out-of-the-box evaluation used two robots that earlier generalist policies had been tested on, with every policy run from the same initial robot and object states. The first is the WidowX from BridgeData V2: 17 tasks with 10 trials each, 170 rollouts per policy. The tasks are grouped by what they change relative to the training data: visual (unseen backgrounds, distractor objects, new object appearances), motion (unseen object positions and orientations), physical (unseen object sizes and shapes), semantic (unseen target objects, instructions and concepts from the Internet) and language grounding (several objects, one of them named in the instruction). The second is the Google mobile manipulator from the RT-1 and RT-2 papers: 12 tasks with 5 trials each, 60 rollouts.

On BridgeData V2, OpenVLA averages 70.6% ± 3.2% success against RT-2-X's 50.6% ± 3.5% (the ± is the standard error over 170 rollouts). Octo reaches 20.0% and RT-1-X 18.5%; the paper reports both failing to manipulate the right object when distractors are present and sometimes waving the arm around aimlessly. OpenVLA leads RT-2-X in every category except semantic generalization, where RT-2-X scores 38.8% against 36.3%. On the Google robot the two are comparable, 85.0% ± 4.6% against 78.3% ± 5.4% with overlapping error bars: OpenVLA leads in-distribution (88.0% against 72.0%) and they tie at 82.9% out of distribution. RT-1-X and Octo score 33.3% and 26.7%. Over all 29 tasks on both robots OpenVLA leads RT-2-X by 16.5 points, the abstract's headline, with a seventh of the parameters.

The paper credits the gap to three factors. OpenVLA's curated dataset is larger, 970k episodes against about 350k for RT-2-X. It was cleaned more carefully: RT-2-X was trained with the all-zero first steps of BridgeData V2 left in, and the authors report that it often freezes on Bridge tasks unless the way it is queried is changed. And OpenVLA's encoder adds DINOv2's spatial features to SigLIP's semantic ones. The semantic category goes the other way for a different reason. RT-2-X was co-fine-tuned on robot data and Internet image-text data together, which keeps more of its pretraining knowledge about unseen objects and concepts; OpenVLA was fine-tuned on robot data only.

Fine-tuning OpenVLA to a new robot

A lab whose robot or scene is not in the training mix usually cannot use the out-of-the-box model as is, so the second question is how well OpenVLA adapts. The test uses Franka Emika Panda arms in two setups: Franka-Tabletop, a table-mounted arm with a 5 Hz controller, and Franka-DROID, the setup of the DROID dataset with a 15 Hz controller. Seven tasks have 10 to 150 demonstrations each, and every policy is evaluated on 129 rollouts. OpenVLA is fully fine-tuned on each task's demonstrations, which takes 8 A100s for 5 to 15 hours per task.

The comparisons: Diffusion Policy trained from scratch on the same demonstrations, a version of it restricted to OpenVLA's inputs and outputs, Octo fine-tuned on the same data (RT-2-X's API does not allow fine-tuning), and an ablation called OpenVLA (scratch), which fine-tunes the Prismatic model directly with no robot pretraining. Select the fine-tune suite in Figure 3 and step through the tasks.

No policy wins every task. Diffusion Policy wins the two narrow single-instruction tasks, "put carrot in bowl" (66.7% against OpenVLA's 60.0%) and "pour corn into pot" (93.3% against 53.3%). The pretrained generalists win the tasks with several objects in the scene and an instruction naming one of them: on "move <object> onto plate" OpenVLA scores 69.4% and Diffusion Policy 19.4%. OpenVLA has the best average on both setups, 67.2% against Diffusion Policy's 48.5% on Franka-Tabletop and 58.3% against 35.0% on Franka-DROID, and it is the only policy with at least 50% on every task. The abstract summarizes the comparison as a 20.4% margin over Diffusion Policy. OpenVLA (scratch) averages 43.4% on Franka-Tabletop, 23.8 points below the pretrained model, which measures what the 970k episodes of robot pretraining contribute.

On the tasks Diffusion Policy wins, the paper describes its trajectories as smoother and more precise. Diffusion Policy predicts a chunk of future actions at once (8 actions at 5 Hz, executing the first 3 before predicting again) from two frames of history plus the robot's proprioceptive state, and smooths across chunks; OpenVLA predicts one action at a time from one frame. The authors name action chunking and temporal smoothing as likely ways to give OpenVLA the same dexterity.

In simulation the margins are smaller. On the four LIBERO task suites (Appendix E), fine-tuned OpenVLA averages 76.5% against 75.1% for fine-tuned Octo and 72.4% for Diffusion Policy. The authors attribute the narrower gap to OpenVLA's pretraining on real-robot data only, with no simulation data.

LoRA and other cheaper ways to fine-tune OpenVLA

Full fine-tuning a 7B model needs more memory than one GPU has. The paper compares six fine-tuning strategies on Franka-Tabletop tasks (33 rollouts each), recording GPU memory at batch size 16. These experiments, and the quantization experiments in the next section, used a smaller OpenVLA variant: a SigLIP-only encoder, pretrained on Octo's Open X-Embodiment mixture instead of the 970k-episode one.

LoRA freezes every pretrained weight matrix $W$ and trains a low-rank correction beside it:

For one of Llama's 4096 × 4096 attention projections at rank $r = 32$, $B$ and $A$ hold 2 × 4096 × 32 = 262,144 trainable numbers against 16,777,216 frozen ones, 1.6%. OpenVLA applies LoRA to every linear layer of the model, including the vision encoder's; in total 97.6M of the 7,188.1M parameters train, 1.4%. The frozen weights need no gradients and no optimizer state, which saves most of the memory.

The six strategies from the paper's Table 1:

• Full fine-tuning: 69.7% ± 7.2%, all 7,188.1M parameters, 163.3 GB, sharded across 2 GPUs with FSDP.
• LoRA, r = 32: 68.2% ± 7.5%, 97.6M parameters, 59.7 GB on one GPU.
• LoRA, r = 64: 68.2% ± 7.8%, 195.2M parameters, 60.5 GB. Doubling the rank changes nothing measurable.
• Sandwich (vision encoder, token embeddings and last layer): 62.1% ± 7.9%, 914.2M, 64.0 GB.
• Frozen vision (everything except the encoder): 47.0% ± 6.9%, 6,760.4M, 156.2 GB on 2 GPUs.
• Last layer only: 30.3% ± 6.1%, 465.1M, 51.4 GB.

Figure 4 plots the same rows as success against memory. LoRA at rank 32 is within error of full fine-tuning at 37% of its memory (59.7 GB against 163.3 GB), on one GPU instead of two, and the paper recommends it as the default. The two strategies that leave the vision encoder untouched, last layer only and frozen vision, perform poorly; last layer only, at 30.3%, scores below fine-tuning the Prismatic model with no robot pretraining at all (43.4% on the same Franka-Tabletop tasks in the previous section). Sandwich does better because it trains the encoder, at almost ten times LoRA's trainable parameters. A LoRA fine-tune takes 10 to 15 hours on one A100, which the paper counts as an 8× reduction in compute against full fine-tuning.

Quantization: why int8 lost rollouts and int4 did not

To run on smaller GPUs, the weights can be stored at lower precision. The paper serves the model with the 8-bit method of LLM.int8() and the 4-bit method used by QLoRA, and runs 8 BridgeData V2 tasks with 80 rollouts per precision (Table 2, the inference view of Figure 4):

• bfloat16: 71.3% ± 4.8%, 16.8 GB.
• int8: 58.1% ± 5.1%, 10.2 GB.
• int4: 71.9% ± 4.7%, 7.0 GB.

4-bit matches bfloat16 in less than half the memory. 8-bit, which keeps more precision than 4-bit, loses 13 points because its extra quantization operations slow inference on most GPUs the paper measured; 4-bit pays the same kind of overhead but moves a quarter of bfloat16's bytes from memory per token, which compensates for it. On the A5000 used for these rollouts, 8-bit ran at 1.2 Hz and 4-bit at 3 Hz. The BridgeData V2 demonstrations were recorded with a 5 Hz non-blocking controller, one that keeps the arm moving while the next command is computed.

Figure 5 draws the timing. The top lane is the training data, a command every 0.2 s. The middle lane is the model: each bar is one inference, from the frame it reads (amber dot) to the command it emits (bright tick). The bottom lane is the arm. At 1.2 Hz a command arrives every 0.83 s, 4.2 times the interval in the training data, and the arm has kept moving for 0.83 s since the frame behind that command was taken. In the paper's words, 1.2 Hz "significantly changes the system dynamics compared to the training dataset"; at 3 Hz, 4-bit is closer to the dynamics seen during data collection.

The paper tests this explanation two ways. Offline, on training data, 8-bit and 4-bit predict action tokens with accuracy comparable to bfloat16, so the 8-bit weights are not producing worse tokens. And with blocking control, where each command is fully executed before the next frame is taken, speed stops mattering: 8-bit scores 74.4% ± 4.9%, bfloat16 70.0% ± 5.1% and 4-bit 68.8% ± 5.2% (Appendix D.4, Table 11). Switch Figure 5 to blocking to see the arm wait.

Limitations the paper names

One camera frame in, nothing else. OpenVLA reads a single image. It takes no history of earlier frames, no proprioception (the robot's own joint and gripper state) and no second camera, all of which many robot setups provide. The authors suggest that vision-language models pretrained on interleaved image and text data could make multi-image input a fine-tuning change.

Throughput. About 6 Hz on an RTX 4090 suits 5 Hz tabletop control. An ALOHA bimanual setup runs at 50 Hz. The paper names action chunking, which emits several future commands per inference as Diffusion Policy does, and speculative decoding, where a small model drafts tokens and the large one checks them, as possible remedies.

Reliability. On the tested tasks OpenVLA typically succeeds less than 90% of the time; its averages in this paper are 70.6% on BridgeData V2 and 85.0% on the Google robot.

Open design questions. The authors list the size of the base vision-language model, whether co-training on robot and Internet data together (as RT-2-X did) helps, and which visual features are best, as questions their compute budget left unanswered.