RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

Encode a robot action as a short string of integers, and a vision-language model can output it.

RT-2 fine-tunes a vision-language model, trained on web images and text, to answer "what action should the robot take?" with eight numbers. Because the same weights keep training on web data, the robot follows instructions its own demonstrations never covered, such as "move coke can to taylor swift".

The method has four parts: a fixed encoding of actions as integers, a choice of which existing tokens carry those integers, a mask at decoding time, and a training mix that keeps the web data in. The results come from about 6,000 evaluation trajectories, and they separate what transferred from the web (recognizing objects, symbols and people) from what did not (new motions).

A robot policy trained only on robot demonstrations can only connect words to things those demonstrations contain. RT-1, the policy RT-2 builds on, is a 35M-parameter transformer trained on demonstrations collected "with 13 robots over 17 months in an office kitchen environment". Ask it to move a can next to a picture of a celebrity, or to pick the healthy snack, and nothing in its training connects those words to anything in the camera image, because no demonstration involved one. A vision-language model (VLM) trained on a billion web image-text pairs can name the celebrity in a photo and say which snack is healthy, but it outputs text, and a robot arm needs motor commands.

The earlier way to combine the two was a pipeline. A language model such as SayCan or PaLM-E wrote a plan in words ("pick the apple"), and a separate low-level policy turned each step into motion. The low-level policy was still trained only on robot data, so whatever the language model understood had to fit through a short instruction string drawn from the policy's small vocabulary of skills.

RT-2 removes the separate policy. It takes a pretrained VLM (PaLI-X at 5B or 55B parameters, or PaLM-E at 12B) and fine-tunes that same network to output the robot's action as a short string of numbers, in the slot where it would normally write an answer. Both backbones turn the camera image into a sequence of embedding vectors with a Vision Transformer (ViT-22B in PaLI-X, ViT-4B in PaLM-E) and process them together with the prompt's tokens. The paper calls the result a vision-language-action (VLA) model. Every weight that encodes "what a strawberry looks like" is now also on the path that computes where the gripper moves. Four design choices make this work, and the page takes them in order:

1. an encoding of each action as eight integers between 0 and 255, taken from RT-1;
2. a choice of which existing vocabulary tokens stand for those integers in each backbone;
3. a decoding-time mask so that an action prompt can only produce action tokens;
4. co-fine-tuning, which keeps the original web data in every training batch alongside the robot data.

How RT-2 turns a robot action into text

The robot is a 7-DoF mobile manipulator: an arm with seven joints on a wheeled base. At each control step the policy outputs eight values. Six describe how the gripper should move relative to where it is now: three position changes $\Delta\text{pos}_{x,y,z}$ and three rotation changes $\Delta\text{rot}_{x,y,z}$. The seventh is how far the gripper is open. The eighth is a discrete terminate command that the policy issues when it judges the task complete. The paper writes the target in this order:

A language model emits tokens, so each continuous value has to become one of a finite set of symbols. RT-2 reuses RT-1's scheme: the range of each continuous variable is cut into 256 equal-width bins, and the value is replaced by the index of the bin it falls in, an integer from 0 to 255. If a variable lives in $[\ell, h]$, the bin of a value $a$ is

Neither paper lists the bounds $\ell$ and $h$, so take an illustrative $\pm 0.1$ m for a position change. A bin is then 0.2 / 256 = 0.78 mm wide. A desired move of +0.04 m lands at 256 × 0.14 / 0.2 = 179.2, so bin 179. Decoding bin 179 to its centre gives +0.040234 m, an error of 0.23 mm; no value is ever more than half a bin, 0.39 mm, from its bin centre. The terminate command is discrete and is not binned. The paper does not give its integer coding, and every action string it prints starts with 1, including strings from the middle of a task.

BINS = 256

def to_bin(a, lo, hi):            # one continuous action dimension
    t = (a - lo) / (hi - lo)      # position inside the variable's bounds, 0..1
    return min(int(t * BINS), BINS - 1)

def from_bin(b, lo, hi):          # assumption: decode to the bin centre
    return lo + (b + 0.5) * (hi - lo) / BINS

# with illustrative bounds of -0.1 m .. +0.1 m:
to_bin(0.04, -0.1, 0.1)           # -> 179
from_bin(179, -0.1, 0.1)          # -> 0.040234 (0.23 mm off)

Join the eight integers with spaces and the action is a short text answer. Section 4.4 of the paper gives one: 1 128 124 136 121 158 111 255. Section 3.2 also prints an example, 1 128 91 241 5 101 127, which has only seven numbers although the same paragraph defines eight; the Provenance panel below records the slip. Move any slider in Figure 1 and watch its bin index, the assembled string, and the round trip back to a number for the dimension you moved.

Why discretize instead of predicting eight floats with a regression head? RT-1 tried the alternative. Replacing its 256-bin output with a single Gaussian per dimension cut its seen-task success from 97% to 68% and its unseen-task success from 76% to 43%. Demonstrations often contain two good ways to do something (go left of the can or right of it); a softmax over bins can put probability on both, and a Gaussian has to put its mean between them. For RT-2 there is a second reason: integers are text, so the VLM's existing output layer and loss work unchanged.

Training then looks like any other VQA (visual question answering) example. The input is the camera image and a prompt in the paper's exact format, "Q: what action should the robot take to [task instruction]? A:", and the target is the eight-integer string. The loss is the model's usual next-token cross-entropy over those eight tokens:

In (3), $t_i$ is the token for the $i$-th integer of the demonstrated action and $t_{<i}$ are the tokens before it. The appendix states that this next-token objective "corresponds to the behavior cloning loss in robot learning": behaviour cloning is supervised learning that makes the policy assign high probability to whatever action the human demonstrator took in that situation. The gradient of (3) reaches every weight in the VLM, the vision encoder included.

# one training example (behaviour cloning written as text)
image  = camera_frame                      # what the robot saw
prompt = "Q: what action should the robot take to pick rxbar chocolate? A:"
target = "1 128 124 136 121 158 111 255"   # integers from the paper's Sec. 4.4
#         term dx  dy  dz  drx dry drz grip

# the VLM's ordinary next-token loss over the 8 target tokens
loss = -sum(log p(target[i] | image, prompt, target[:i]) for i in range(8))
loss.backward()     # gradients reach every weight of the VLM

Why 256 bins are enough: closed-loop control

Rounding errors do not pile up over a long motion, because RT-2 runs in closed loop: after each action the robot takes a new camera image, and the next action is computed from where the gripper actually is. An error from step 3 shows up in the image at step 4 and gets corrected there.

Figure 2 simulates this in the table plane. The gripper starts at the lower left and should reach the target at the upper right in 30 steps. Each step asks for the move toward the target, clipped to ±2 cm per axis, rounds it to one of $n$ bins, and executes it with an arm whose moves come out too long or too short by a fixed fraction per axis (up to about 24% in this simulation) plus about 1.4 mm of random jitter per step. The teal path re-reads its position each step. The amber path plans all 30 commands at the start from a perfect model and never looks again.

At 256 bins the closed-loop path ends about 2.4 mm from the target with the default noise, an error set by the arm's jitter, since the bin width is only 0.16 mm here. The open-loop path, given the same rounded commands, ends about 22 mm away because the arm's gain error compounds over 30 steps. Dropping to 16 or 32 bins barely changes either number. At 2 bins every command is a full ±1 cm jump, so the closed-loop gripper chatters around the target and stops about 1 cm away. At 1 bin every command decodes to zero and nothing moves. Rounding stops being the largest error somewhere below 16 bins, and 256 is far past that point.

Which vocabulary tokens carry the action bins

The string 1 128 124 ... only works as a training target if each of the 256 bin values maps to a single token the model can emit in one decoding step. RT-2 reserves 256 tokens for this, and the two backbones need different choices because their tokenizers differ.

PaLI-X is the easy case. In its tokenizer, "integers up to 1000 each have a unique token", so bin 128 is written with the token that already means the number 128. Nothing is added to the vocabulary and nothing is overwritten.

PaLM-E "does not provide this convenient representation of numbers", so RT-2 overwrites "the 256 least frequently used tokens" to stand for the 256 bins. The target for PaLM-E is then a string of those rare tokens; the paper's description of the output, "a string of numbers/least frequently used tokens representing a robot action", covers both backbones. Whatever subword a rare token spelled before, after fine-tuning the model treats it as one action bin. The paper calls this "a form of symbol tuning", a method from Wei et al. (2023) that fine-tunes a model on examples whose natural-language labels ("positive", "negative") are replaced by arbitrary symbols ("foo", "bar"), so the model has to learn the mapping from the examples instead of from what the label words mean. The rarest-256 choice is RT-2's own. The reasoning behind it, which the paper does not measure, is that tokens which almost never occur in training text are the ones whose old meanings cost the least to give up.

Constrained decoding: only action tokens on action prompts

The fine-tuned model still answers ordinary visual questions in words, and RT-2 keeps it that way on purpose. That creates a problem at the robot: nothing in the weights stops the model from writing "the" where an action integer should go, and a robot cannot execute "the". RT-2 fixes this at decoding time. When the prompt is a robot-action prompt, it "constrain[s] its output vocabulary via only sampling valid action tokens"; on vision-language prompts the full vocabulary stays available.

The standard way to implement such a constraint is to set the logit of every token outside the allowed set $A$ to $-\infty$ before the softmax. The remaining probabilities are then renormalized over the allowed tokens only:

Figure 4 uses twelve illustrative logits. Without the mask, "the" takes 0.269 of the probability, bin 128 takes 0.099, and all the action tokens shown together hold 0.400. With the mask, every word token drops to zero and each action token's probability is divided by 0.400, so bin 128 rises to 0.247. The ranking among the action tokens does not change; the mask only removes the illegal options. Because the mask is applied at each of the eight decoding steps, every action RT-2 sends to the robot is well formed, even when the model is unsure which bin to pick.

Co-fine-tuning: why the web data stays in the batch

The obvious way to adapt the VLM is to fine-tune it on robot demonstrations alone. RT-2 instead co-fine-tunes: each training batch mixes robot examples with the VLM's original web data, mostly WebLI (about 10B image-text pairs in 109 languages, filtered to the 1B best-matching pairs) plus captioning and VQA datasets. The paper raises the sampling weight of the robot data so that it makes up "about 50% of the training mixture" for RT-2-PaLI-X and about 66% for RT-2-PaLM-E. With the 55B model's batch size of 2048, that is roughly 1,024 robot examples and 1,024 web examples per step, for 80K steps at learning rate 1e-3. The 5B PaLI-X model uses the same learning rate and batch size for 270K steps; PaLM-E-12B uses learning rate 4e-4, batch size 512, for 1M steps.

The paper's explanation for mixing is forgetting: "keeping the original data around the fine-tuning part of training, allows the model to not forget its previous concepts learned during the VLM training." A gradient step on robot data alone moves the shared weights toward whatever predicts action integers, and nothing in that step keeps them good at answering questions about images. With web examples in the same batch, each step follows the gradient of the summed loss, so the update also has to keep the captioning and VQA loss low.

Table 8 tests this with RT-2-PaLI-X at two sizes and three training regimes, scored on the six unseen generalization splits. A 5B model trained from scratch on robot data, with no VLM pretraining, averages 9%. Fine-tuning the pretrained 5B model reaches 42%, and co-fine-tuning it reaches 44%. At 55B, fine-tuning gives 52% and co-fine-tuning 63%. The authors skipped the 55B from-scratch run because the 5B one had done so badly.

Pretraining accounts for most of the gain (9% to 42% at 5B). Co-fine-tuning adds 2 points at 5B and 11 points at 55B. Switch Figure 5 to the background splits: at 5B, co-fine-tuning scores 67% and 29% on easy and hard backgrounds, against 79% and 50% for plain fine-tuning, and its higher average comes mostly from easy unseen objects (60% against 24%). At 55B co-fine-tuning is at least as good on every split. The 55B co-fine-tuned average is 63 in Table 8, while Table 6 lists the same model with identical per-split numbers and an average of 62; the plain mean of the six splits is 62.3.

Serving a 55B model to a robot

RT-1, at 35M parameters, targeted a control rate of 3 Hz. A 55B-parameter model does not fit on "the standard desktop-style machines or on-robot GPUs commonly used for real-time robot control", and the paper describes its model as the largest used for direct closed-loop robot control by over an order of magnitude. RT-2 runs in "a multi-TPU cloud service" (TPUs are Google's machine-learning accelerators) that the robot queries over the network once per step.

With that setup the 55B model runs at 1 to 3 Hz and the 5B model at about 5 Hz. At 1 Hz the whole budget for one step is 1,000 ms: upload the image, run the vision encoder and eight decoding steps, return eight integers, move the arm. A pick-and-place that takes 20 seconds therefore gets between 20 and 60 decisions from the 55B model. The same service can serve several robots at once.

# on the TPU cloud service: one request returns one action
def act(image, prompt):
    ids = []
    for i in range(8):                       # 8 decoding steps
        logits = vla(image, prompt, ids)
        logits[NOT_ACTION_IDS] = -inf        # constrained decoding, eq. (4)
        ids.append(sample(logits))
    return ids

# on the robot: closed loop, one network round trip per step
while True:
    ids = service.act(camera.read(), prompt)
    if is_terminate(ids[0]):                 # integer coding not published
        break
    robot.apply(decode_bins(ids[1:]))        # 1-3 Hz for 55B, ~5 Hz for 5B

How well RT-2 generalizes to new objects, backgrounds and rooms

The first evaluation asks how the policy does on tasks it was trained on and on scenes it was not. The seen-task suite is RT-1's, over 200 instructions such as picking, knocking over, standing upright, moving near, and drawer tasks, with object placement, time of day and robot position still varying. The unseen suite has over 280 tasks, mostly pick-and-place, along three axes, each split into easy and hard:

• unseen objects, where the hard split uses "harder-to-grasp and more unique objects (such as toys)";
• unseen backgrounds, where the hard split adds more varied backgrounds and novel objects;
• unseen environments, where the easy split is a kitchen sink and the hard split is "a more visually distinct office desk environment with monitors and accessories".

All four baselines are trained on the same robot data. RT-1 is the previous state of the art. VC-1 and R3M are pretrained visual representations, each feeding an RT-1 backbone. MOO uses a VLM to mark the object of interest with a coloured pixel in the image and passes that to an RT-1-style policy, so the VLM helps perception but its representations are not trained into the policy.

On seen tasks RT-1 and both RT-2 models are close: RT-1 92%, RT-2-PaLI-X-55B 91%, RT-2-PaLM-E-12B 93%, with MOO at 75%, VC-1 at 63% and R3M at 45%. On the unseen average both RT-2 variants score 62%, against 32% for RT-1 and 35% for MOO, and 12% and 10% for R3M and VC-1. The paper summarizes this as about 2x over the next two baselines (RT-1 and MOO) and about 6x over the other two.

The gap over RT-1 is not the same everywhere. For PaLI-X and PaLM-E respectively, the margins are +39 and +53 points on easy objects, +19 and +33 on hard objects, +25 and +4 on easy backgrounds, +39 and +62 on hard backgrounds, +37 and +10 on easy environments, and +21 and +19 on hard environments. The largest margins are on hard backgrounds and easy objects. R3M and VC-1 score 0 on the easy environment and 2 and 0 on the hard one: a pretrained image representation without language pretraining did not carry these policies into a new room.

The two RT-2 variants reach the same average by different routes. The paper observes that the PaLM-E version "seems to perform better" on the harder versions of the generalization scenarios and worse on the easier ones, and gives no reason. The split is not clean column by column: PaLM-E also leads on easy objects, 84 to 70, and trails slightly on hard environments, 33 to 35. A footnote to Table 6 adds that PaLM-E's original pretraining data includes robot images used for high-level planning questions, which can resemble the evaluation scenes, though none of those examples contains low-level actions.

A separate check uses the open-source Language-Table simulator from Lynch et al. (2022), a tabletop pushing benchmark with a different robot. There a smaller PaLI 3B model writes each action as two integers from -10 to +10 (a 2D move of the end effector), runs at about 5 Hz, and scores 90 ± 10 against 72 to 77 for BC-Zero, RT-1 and LAVA.

Emergent skills: symbols, reasoning and people

The paper calls a capability emergent when it comes from the web pretraining rather than from the robot data: "they emerge by transferring Internet-scale pretraining". The robot data covers a fixed set of motions. Appendix B introduces it as "one of seven skills" and then lists eight: pick object, move object near object, place object upright, knock object over, open drawer, close drawer, place object into receptacle, and pick object from receptacle and place it on the counter. None of those demonstrations mentions numbers, logos, or people.

The quantitative test has three categories, with instructions taken from the paper:

• symbol understanding, such as "move apple to 3" or "push coke can on top of heart";
• reasoning, covering math ("move X near the sum of two plus one"), logos ("move cup to google"), nutrition ("get me a healthy snack"), and colour or language ("mueve la manzana al vaso verde");
• person recognition, covering celebrities ("move coke can to taylor swift") and CelebA-style attributes ("move coke can to the person with glasses").

The motions are the robot's existing picking and moving skills. The new part is which object or location the motion is aimed at, and that requires recognizing the digit 3, the Google logo, or a face. The two RT-2 models are compared with RT-1 and VC-1, which the paper calls the top two baselines from the previous evaluation (on the Table 6 unseen average MOO had scored higher than VC-1, 35 to 10). To reduce variance the evaluation is A/B style: all four models run one after another under the same conditions.

Averaged over the three categories, RT-2-PaLI-X scores 60% and RT-1 17%, the "more than 3x" in Section 4.2; the appendix phrases the same table as "2x to 3x better than RT-1". By category the multiple ranges from about 2.6x (person recognition, 53 against 20) through 2.9x (reasoning, 46 against 16) to 5.1x (symbol understanding, 82 against 16). The 12B PaLM-E model averages 40%. It beats PaLI-X on math, 35 to 25, and on logos, 56 to 52, and ties it on CelebA at 53. The paper attributes the math result to PaLM-E's pre-training mixture, which makes it "more capable at math calculation than the mostly visually pre-trained PaLI-X".

The paper also shows qualitative cases that no table scores: "put strawberry into the correct bowl", where the robot has to infer that the strawberry goes with the other fruit, "pick up the bag about to fall off the table", and, in the abstract, using a rock as "an improvised hammer".

Chain-of-thought: a plan before the action

Chain-of-thought prompting gets a language model to write intermediate reasoning before its answer. RT-2 tries the robot version. The authors take RT-2-PaLM-E and fine-tune it "for just a few hundred gradient steps" on data with an extra "Plan" step: a short sentence describing what the robot is about to do, placed before the action integers. The paper's example of the format is "Instruction: I'm hungry. Plan: pick rxbar chocolate. Action: 1 128 124 136 121 158 111 255."

At inference the model writes the plan first and then decodes the action with the plan already in its context, so the eight integers are conditioned on a sentence the model wrote itself. The plans in Figure 9 of the paper read like the instructions in the robot data ("pick 7up can", "move green can near green rice chip bag"): a request such as "Bring me a drink" is first translated into the kind of command the robot data contains, then acted on.

The evidence for this result is qualitative. The paper says the CoT model "is able to answer more sophisticated commands" and calls the result "some initial evidence". It reports no benchmark number comparing the CoT model with the standard one.

Limits, and what RT-2 adds to earlier work

The paper lists three limitations. First, the robot "does not acquire any ability to perform new motions". Web data changes what the robot recognizes and where it aims a motion, but the motions themselves are the ones in the robot data. Appendix G lists failure cases: pushing a banana whose "center-of-mass is far from where the robot makes contact", a pen that "rolls off the table", grasping objects by a specific part "such as the handle", new motions like "wiping with a towel", dexterous ones like "folding a towel", and reasoning that needs "multiple layers of indirection".

Second, the compute cost. At 1 to 3 Hz from a TPU service, RT-2 suits slow tabletop manipulation, and the paper expects real-time inference to become "a major bottleneck" for tasks that need high-frequency control. It suggests quantization and distillation as ways to run such models faster or on cheaper hardware. Third, few VLMs were available to build on. The authors hope for more open models, citing LLaVA, and for proprietary ones to open their fine-tuning APIs, which would be enough to build a VLA.

Most individual pieces predate the paper. The 256-bin encoding is RT-1's. PaLI-X and PaLM-E already read images and wrote text. Masking a decoder's output to a valid set is a standard technique for structured generation. RT-2 contributes the combination: the bins written as ordinary tokens of a large pretrained VLM, trained together with the web data, so one network answers questions about an image and moves the arm. Later work addressed the third limitation and revisited the action output. OpenVLA, an open 7B model, keeps discrete action tokens and reports 16.5 points higher absolute task success across 29 tasks than the 55B RT-2-X, an RT-2 model trained on about 350k robot trajectories. π₀ keeps a pretrained backbone but generates continuous actions with a separate flow-matching action expert, and runs robots at up to 50 Hz.