π0: A Vision-Language-Action Flow Model for General Robot Control

A vision-language model that outputs a robot’s next 50 actions by refining random noise.

π₀ adds a 300M-parameter action expert to the 3B PaliGemma vision-language model and trains both with flow matching on over 10,000 hours of robot data from seven robot configurations. Each decision takes ten small steps from Gaussian noise to a chunk of 50 joint commands, in 73 ms on an RTX 4090.

Most learned robot policies are trained for one task on one robot. π₀ is one pre-trained model for seven robot configurations. Prompted directly, it folds shirts and busses tables; fine-tuned on 5 to 100-plus hours of task data, it folds laundry from a bin, packs eggs and assembles boxes.

Language and vision models became general by pre-training one large network on web-scale data and then fine-tuning it for each use. Robotics has three obstacles to copying that recipe. Robot data cannot be scraped; someone has to teleoperate a robot to collect it. Robots differ in body: a single 6-joint arm and a two-armed mobile base have different numbers of motors to command. And the output is a stream of continuous motor commands at 20 to 50 per second, where a language model emits one discrete token at a time.

Physical Intelligence addressed the first obstacle by collecting over 10,000 hours of teleoperated data across 7 robot configurations and 68 tasks, plus a 9.1% share of open-source robot datasets. The paper answers the other two with a few design choices, and this page takes them in order: a shared, zero-padded action vector for every robot; predicting a chunk of 50 actions per model call; generating that chunk with flow matching instead of predicting one answer; a timestep schedule tilted toward high noise; a second set of transformer weights (the action expert) that shares attention with a pre-trained vision-language model; and an attention mask that lets the expensive part of each decision run once.

What goes in and what comes out

At robot time $t$ the model sees an observation $\mathbf{o}_t$ made of three parts: two or three camera images $\mathbf{I}^1_t, \dots, \mathbf{I}^n_t$, a language command $\ell_t$ such as "bus the table", and the robot's proprioceptive state $\mathbf{q}_t$, the vector of current joint angles and gripper positions. It outputs an action chunk, the next $H = 50$ actions:

Each action $\mathbf{a}_{t'}$ is a vector of joint and gripper targets. The seven robot configurations need different lengths for it. A single UR5e arm with a parallel gripper has 7 dimensions; a Franka has 8; the two-armed setups (bimanual UR5e, bimanual Trossen, bimanual ARX or AgileX) have 14; the Mobile ALOHA-style robots add 2 dimensions for a wheeled base that can drive and turn but not slide sideways, for 16 action dimensions; the Mobile Fibocom adds 3 for a base that can also slide sideways, for 17. Their state vectors stay at 14, since the base adds commands but no joint angles.

π₀ gives every robot the same vector width, 18, which the paper chose "to accommodate two 6-DoF arms, 2 grippers, a mobile base, and a vertically actuated torso" (DoF is degrees of freedom, one per independently driven joint). Robots with fewer dimensions are zero-padded, and robots with fewer than three cameras have the missing image slots masked out. Every chunk is therefore a $50 \times 18$ array, 900 numbers. For a Franka, 8 of the 18 columns carry real commands and the other 10 are zeros the model learns to leave alone. The figure shows the padding for each robot; no robot in the list fills all 18 slots.

Action chunks: 50 actions per model call

The simplest policy maps one observation to one action and repeats. At 50 Hz that means one forward pass every 20 ms, and one call to π₀ (image encoders, the observation pass and ten action steps) takes 73 ms on an RTX 4090. A one-action policy of this size would fall behind by a factor of 3.65 on every step.

ACT (Zhao et al., 2023) made action chunking the standard workaround: predict the next $H$ actions in one call and execute them one per control tick. π₀ predicts $H = 50$ but does not run the whole chunk. On the 50 Hz robots it executes the first 25 actions (0.5 s) and then calls the model again from a fresh observation; on the 20 Hz UR5e and Franka it executes 16 (0.8 s). The remaining 25 or 34 actions are discarded. At 50 Hz with 73 ms per call, the GPU is busy 14.6% of the time.

In the figure, the top rows show each predicted chunk, solid where it is executed and dashed where it is thrown away. The bottom row shows the inference calls at their true 73 ms width. Drag the replanning slider down and the calls move closer together; below 4 actions per call at 50 Hz (80 ms apart) the next call is due before the previous one finishes, and the blocks overlap. Set $H = 1$ to see the policy with no chunk at all.

ACT also proposed temporal ensembling: call the model every tick and average the overlapping predictions for the current action. π₀'s authors tried it early on, found that it hurt performance, and execute each chunk open-loop instead: once a chunk is chosen, no new observation changes the actions executed from it.

Chunking changes what the network has to model. The target is no longer one action given the observation but the joint distribution of 50 future actions, a distribution over 900 continuous numbers. It is also multimodal, meaning it has several separate high-probability regions: a sock can be grasped from the left or from the right, and both are correct. Earlier vision-language-action models such as RT-2 and OpenVLA discretize each action dimension into 256 bins and emit them as tokens, one action per call. π₀ uses flow matching, a continuous generative model from the diffusion family, to sample all 900 numbers at once.

How flow matching generates an action chunk

Flow matching (Lipman et al., 2022) trains a network to output a velocity: a direction and speed for every point in space and every time $\tau$ between 0 and 1. Integrating that velocity carries a sample of Gaussian noise at $\tau = 0$ to a sample of data at $\tau = 1$. Generation is solving an ordinary differential equation, with no random noise added along the way.

Training needs a target velocity, and flow matching gets one by fixing a path between each noise sample and each data sample. π₀ uses the straight line. Draw noise $\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ with the same $50 \times 18$ shape as the clean chunk, and define the noisy chunk at time $\tau$ as

At $\tau = 0$ this is pure noise and at $\tau = 1$ it is the clean chunk. This is the reverse of the DDPM convention, where $t = 0$ is clean data and the largest $t$ is noise; generation here walks $\tau$ upward from 0 to 1. Differentiating (1) with respect to $\tau$, the $\tau\,\mathbf{A}_t$ term gives $\mathbf{A}_t$ and the $(1-\tau)\boldsymbol{\epsilon}$ term gives $-\boldsymbol{\epsilon}$, so the velocity along the line is the same at every $\tau$:

Take one number of the chunk, say the gripper command at some future step, with clean value 0.8. Draw $\epsilon = -1.2$ and $\tau = 0.3$. The noisy input, the target velocity, and the point reached by moving at that velocity for the remaining 0.7 units of time are

The last line lands on the clean value. The figure below shows the same construction for one dimension across all 50 future steps.

The network $\mathbf{v}_\theta$ sees the noisy chunk and the observation, never the clean chunk or the noise separately, and is trained with a squared error against (2):

In (3), $p(\mathbf{A}_t \mid \mathbf{o}_t)$ is the demonstration data and $q(\mathbf{A}_t^{\tau} \mid \mathbf{A}_t)$ is the distribution of noisy chunks that (1) produces from a given clean one. The target in (3) depends on the particular $\boldsymbol{\epsilon}$ drawn, which the network cannot see: many different (clean chunk, noise) pairs pass through the same noisy point. A squared-error loss is minimized by the average of its targets, so the best possible network outputs the average velocity over every pair whose line passes through that point. Solving (1) for the noise, $\boldsymbol{\epsilon}$ equals $\mathbf{A}_t^{\tau} - \tau\mathbf{A}_t$ divided by $1-\tau$; substituting that into (2) and averaging gives the optimal velocity in closed form:

The optimal velocity points from the current point toward the model's best guess of the clean chunk, scaled by the time left. Lipman et al. proved that training on the per-pair target of (3) has the same gradient as training on this averaged field directly, which is why the simple loss produces a working sampler. The training step, with shapes:

# One pi0 training example (batch dimension omitted).
# o_t: up to 3 images (224x224), a prompt, the state q_t (18 numbers)
A    = actions[t : t + 50]           # clean chunk, shape (50, 18)
x    = sample_beta(1.5, 1)           # x in [0, 1]
tau  = 0.999 * (1 - x)               # eq (5): tau in [0, 0.999], mostly small
eps  = randn(50, 18)                 # Gaussian noise, same shape as A

A_tau = tau * A + (1 - tau) * eps    # eq (1): a point on the straight line
u     = A - eps                      # eq (2): the target velocity
v     = model(images, prompt, q_t, A_tau, tau)   # shape (50, 18)

loss = mean((v - u) ** 2)            # eq (3), over all 900 numbers
loss.backward()                      # gradients reach the action expert AND
opt.step()                           # the PaliGemma backbone (not frozen)

At inference π₀ starts from $\mathbf{A}^0_t \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ and takes forward Euler steps along the learned velocity:

with $\delta = 0.1$, so ten steps from $\tau = 0$ to $\tau = 1$. There is no noise injected between steps and no guidance scale. Each individual training line is straight, but the averaged field $\mathbf{v}^*$ is not constant along a sampling trajectory, so the trajectories bend, and Euler steps with a large $\delta$ cut corners. The next section shows where the bending happens. For comparison, Diffusion Policy (Chi et al., 2023) ran 100 DDPM steps in its simulation benchmarks and 16 DDIM steps on its real robots, so the step-count gap to π₀ is not quite ten-to-one in practice.

Why not predict the average action

A regression policy trained with squared error on demonstrations learns $\mathbb{E}[\mathbf{A}_t \mid \mathbf{o}_t]$, the average of the demonstrated chunks. When the demonstrations split between two good options, the average can be a bad one. Suppose half the demonstrations move the gripper left of a cup and half move it right: the average moves it into the cup.

The figure runs this case in two dimensions with the exact optimal velocity $\mathbf{v}^*$ for a two-cluster target, which is what a perfectly trained network would output. Press Play to watch 56 noise samples integrate from $\tau = 0$ to 1. Early on, every sample heads toward the middle, because at high noise the best guess of the clean action is the overall average; as $\tau$ grows, the noisy point carries enough information to tell which cluster it belongs to, and the trajectories bend apart. Then drag the step count to $N = 1$. A single Euler step from $\tau = 0$ evaluates the field only where it points at the average, so all 56 samples land in the cup, exactly on the regression answer. At $N = 2$, 12 of the 56 still end in the cup; at $N = 10$ none do.

The paper does not report a sweep over step counts; ten is its setting for all experiments. The toy shows why one step is not enough, and the inference section shows what each step costs.

Which noise levels to train on

Each training example draws one $\tau$, and the distribution it is drawn from decides which noise levels the network practices most. The original flow-matching papers draw $\tau$ uniformly on $[0, 1]$. Stable Diffusion 3 (Esser et al., 2024) draws from a logit-normal that concentrates on the middle. π₀'s appendix quotes their reasoning: at low noise (high $\tau$) the model only has to learn something close to the identity, since the noisy input is nearly the answer; at high noise (low $\tau$) it only has to learn the mean of the data, which $\mathbf{v}^*$ above shows is all the input can tell it. Both ends are easy for images, so SD3 trains mostly in between.

π₀ disagrees about the high-noise end, and says so as a hypothesis. For images, the mean image given a text label is easy to predict. For robots, the mean action given an observation, $\mathbb{E}[\mathbf{A}_t \mid \mathbf{o}_t]$, is "a much harder problem", because the observation constrains the action far more tightly than a caption constrains an image: the model has to read exactly where the cup is in three camera views to get the mean right. So π₀ puts most of its training on low $\tau$, with the density

A Beta(1.5, 1) density is proportional to $x^{1/2}$ on $[0, 1]$, rising toward $x = 1$. The substitution $x = (s - \tau)/s$ maps $x = 1$ to $\tau = 0$ and $x = 0$ to $\tau = s$, so $p(\tau)$ peaks at $\tau = 0$ with density 1.5 (against 1.0 for uniform) and falls to zero at $\tau = 0.999$. In numbers: 64.7% of draws land in the noisier half $\tau < 0.5$, 14.6% below 0.1, and 3.1% above 0.9, against 50%, 10% and 10% for uniform.

Nothing above $s$ is ever drawn. The paper's reason: those timesteps "are not needed so long as the integration step δ is greater than 1 − s". An Euler step starting at or below $s$ with $\delta > 0.001$ jumps past the untrained sliver to the end, so the model never has to evaluate there. With $s = 0.999$ that holds for any step count up to 1,000; π₀'s 10 steps satisfy it with a wide margin. Toggle the three schedules in the figure and slide $s$ to see how much of the interval each one covers.

The paper reports no ablation of this schedule against uniform or logit-normal sampling on actions, so the evidence for it is the hypothesis and the overall results. Someone reproducing π₀ with weaker numbers has a candidate parameter to vary here.

How the action expert attaches to PaliGemma

π₀'s backbone is PaliGemma, a 3B-parameter vision-language model: a SigLIP image encoder (a vision transformer trained contrastively on image-text pairs, like CLIP) that turns each image into tokens, followed by the Gemma 2B language model. Starting from it gives π₀ a network that already maps "the red mug" to the red mug in an image. π₀ has to add a velocity network for 900 continuous numbers to this transformer without damaging what it already knows, and without paying for 3B parameters on each of the ten Euler steps.

π₀ runs one transformer with two sets of weights. The images and the language prompt pass through PaliGemma's weights. The state $\mathbf{q}_t$ and the 50 noisy action tokens, inputs PaliGemma never saw in pre-training, pass through a second, smaller set the paper calls the action expert, initialized from scratch. At every one of the 18 layers each token uses its own set's query, key and value projections, output projection and MLP. The two sets interact in exactly one place: the self-attention step, where the queries of all tokens score against the keys of all tokens (subject to the mask in the next section). An action token can attend to an image token's keys and values, but no action token is ever multiplied by PaliGemma's MLP weights.

Sharing attention puts one constraint on the two sets. Their queries and keys have to live in the same space, so both sides use the same head layout: 8 query heads and 1 key-value head of size 256 (Gemma uses multi-query attention). Everything else can differ. The backbone has width 2048 and MLP size 16,384; the action expert has width 1024 and MLP size 4096. Counting weight matrices per layer (query, key, value, output, and Gemma's three-matrix gated MLP) gives 110.1M for a backbone layer and 17.3M for an expert layer, so 1.98B and 311M over 18 layers. The 311M is the paper's "∼300M"; the backbone figure plus PaliGemma's token embeddings and image encoder makes up its 3B.

Two reasons favor separate weights over pushing action tokens through PaliGemma's own layers. The paper reports that the separate set improved performance over a single shared set. And the size split matches the workload: the backbone runs once per decision, the expert ten times, so shrinking the expert shrinks the repeated cost. The paper says the architecture builds on Transfusion (Zhou et al., 2024), which trains one transformer with a cross-entropy loss on discrete tokens and a flow-matching loss on continuous ones, and that adding the second weight set makes it analogous to a mixture of experts with two experts. Unlike a learned mixture of experts, the choice of expert is fixed by token type. π₀ has no text loss; the only loss is (3) on the action outputs, and it updates both weight sets.

The action expert is also where $\tau$ enters. Each noisy action $\mathbf{a}^{\tau}_{t'}$ (an 18-vector) becomes a token through a small MLP:

$W_1$ lifts the action to the expert width $w = 1024$, $\phi(\tau)$ is a sinusoidal encoding of the scalar $\tau$ with the same width (the sine-and-cosine scheme the original Transformer used for positions), $W_2$ maps the concatenated 2048 numbers back to 1024, and $W_3$ is a final $1024 \times 1024$ layer. All 50 action tokens get the same $\phi(\tau)$, so every token carries the current noise level into the layers. At the output, a linear layer maps each action token's final 1024-vector back to 18 numbers, giving the $50 \times 18$ velocity.

The blockwise causal mask and the KV cache

The input sequence has three blocks. With the openpi release's settings (three image slots at $224 \times 224$ pixels, 256 tokens each, and a 48-token prompt budget) they are: block 1, the 768 image tokens and up to 48 language tokens; block 2, the single state token $\mathbf{q}_t$; block 3, the 50 noisy action tokens. That is 867 tokens at most. Within a block, attention runs in both directions. Across blocks it is causal: a token can attend to its own block and earlier blocks, never later ones.

The paper gives a reason for each boundary. Block 1 holds exactly the kind of input PaliGemma was trained on, and stopping it from attending to the new state and action tokens keeps its computation free of inputs PaliGemma never saw, which the paper says minimizes distribution shift from pre-training. The state gets its own block because it does not change during the ten Euler steps; since it cannot attend to the action block, its keys and values are identical on every step and can be cached. The action block attends to everything, and within it all 50 action tokens see each other, so the chunk is generated jointly.

The caching follows from the mask. Because nothing in blocks 1 and 2 depends on the noisy actions, their keys and values at every layer are the same on all ten Euler steps. π₀ computes them once per decision and stores them; each Euler step then runs only the 50 action tokens through the action expert, attending to the stored prefix. With one key-value head, the stored prefix is 18 layers × 2 (keys and values) × 256 numbers per token, about 15 MB in 16-bit floats for 817 tokens. Without the cache, each Euler step would rerun the full sequence through both weight sets.

What one decision costs

The paper's Table I times one decision with three camera images on an NVIDIA GeForce RTX 4090. The image encoders take 14 ms. The observation forward pass, which fills the cache, takes 32 ms. The ten Euler steps take 27 ms together, 2.7 ms each. That is 73 ms on the robot's own computer. The mobile robots ran inference off-board over Wi-Fi, which adds 13 ms for 86 ms. The full loop, with the cached prefix:

# One pi0 decision on a 50 Hz robot (RTX 4090 timings from Table I).
kv = model.prefix(images, prompt, q_t)   # 14 ms encoders + 32 ms forward pass,
                                         # keys/values of 817 tokens cached
A = randn(50, 18)                        # tau = 0: pure noise
for k in range(10):                      # delta = 0.1
    tau = 0.1 * k
    v = model.actions(A, tau, kv)        # 50 action tokens only, ~2.7 ms
    A = A + 0.1 * v                      # eq (4), forward Euler
execute(A[:25])                          # 0.5 s open-loop, then decide again

The figure rescales Table I to other step counts and adds a second bar that is not in the paper: our estimate of the same decision if every Euler step reran the 32 ms observation pass, 14 + 10 × 34.7 = 361 ms, about 4.9 times longer. At the paper's 10 steps the cached prefix (46 ms) still costs more than all the Euler steps together (27 ms), so halving the step count would save about 13.5 ms. The flow steps overtake the prefix only past 17 steps.

With the cache, one decision uses 73 of the 500 ms between calls on a 50 Hz robot. The paper notes that quantization and other optimizations could cut this further but does not report them.

Training data and the n^0.43 weighting

Pre-training uses 903M timesteps of Physical Intelligence's own data, where a timestep is one training example (an observation and its 50-action chunk): 106M from single-arm robots and 797M from dual-arm robots, over 68 tasks. A "task" in this paper is broad; "bussing" covers every dish, cup, utensil and piece of trash the robot meets. Open-source data makes up 9.1% of the training mixture: the "OXE Magic Soup" subset of Open X-Embodiment defined by the OpenVLA authors, plus Bridge v2 and DROID. Those datasets run at 2 to 10 Hz with one or two cameras, but cover many more objects and scenes.

The task-robot combinations differ greatly in size; the paper names laundry folding as over-represented. Sampling every timestep equally would spend most training on the largest combinations; sampling every combination equally would repeat the smallest ones many times over. π₀ weights each combination by $n^{0.43}$, where $n$ is its number of samples, so the share of training drawn from combination $i$ is

The exponent compresses ratios. A combination with 100 times the data of another is sampled $100^{0.43} \approx 7.2$ times as often; at 10 times the data, $10^{0.43} \approx 2.7$ times. In the figure's twelve schematic combinations, the largest holds 38.9% of the raw data and gets 20.3% of the samples at $\alpha = 0.43$. Slide $\alpha$ to 1 to recover the raw shares and to 0 to sample every combination equally.

After pre-training comes post-training: fine-tuning on a smaller, high-quality dataset for one downstream task. The paper explains the split this way: the diverse but lower-quality pre-training data "allows the model to recover from mistakes and handle highly varied situations", and the post-training data "teaches the model to perform the task well" with a consistent, fluent strategy. The simplest tasks needed 5 hours of post-training data and the most complex used 100 hours or more.

What the experiments show

Tasks are scored by progress rubrics averaged over 10 trials: bussing scores the fraction of objects put in the right bin, toast scores a point per slice lifted and per slice plated, and so on. The paper runs four sets of experiments.

The base model, without fine-tuning. Five tasks present in pre-training: shirt folding, bussing easy (7 objects), bussing hard (12), grocery bagging (7 items) and toast out of a toaster. Baselines were trained on the same mixture: OpenVLA (7B parameters), Octo (93M, with a diffusion action head), and π₀-small, a 470M-parameter model with no VLM initialization that encodes language with DistilBERT and uses a DiT-style action expert. Because OpenVLA and Octo could not be trained as long, the paper also reports a "compute parity" π₀ trained for 160k steps instead of 700k. Across all tasks and comparisons the parity version outperforms every baseline, and the full model is best by a large margin, near perfect on shirt folding and easy bussing. The paper attributes OpenVLA's low scores to its autoregressive discretization, which does not support action chunks; an OpenVLA trained only on UR5e data did better but stayed far below π₀.

Following language. On bussing, table setting and grocery bagging, the paper compares a flat prompt ("bag the groceries") with step-by-step commands from an expert human and from a high-level VLM policy, an autonomous planner analogous to SayCan. π₀ follows commands markedly more accurately than π₀-small, improves significantly with the human's commands, and improves to a lesser degree with the planner's. Per the paper, "due to π₀-small's limited language following ability, overall it does not gain with the addition of a high-level expert." π₀-small is also much smaller, so model size and VLM initialization are confounded in this comparison.

New dexterous tasks. Five tasks absent from pre-training, in tiers by how far they are from it: stacking bowls and folding towels (easy), putting Tupperware in a microwave (medium), replacing a paper towel roll and packing items into a drawer with the Franka (hard). Baselines are OpenVLA and Octo fine-tuned from their public checkpoints, ACT and Diffusion Policy trained only on each task, and π₀ trained from scratch. π₀ fine-tuned from pre-training generally outperforms the others, though not on every setting: with 5 hours of Tupperware data it performs similarly to the baselines, while the 1-hour version is significantly better. The strongest baselines were the ones trained from scratch on the task. Pre-training helped most on the tasks closest to the pre-training data, and the pre-trained π₀ was frequently better than π₀ from scratch, sometimes by as much as 2×.

Long multi-stage tasks. Laundry folding on a static and on the mobile robot, unloading a dryer, bussing a real lunch table, building a box from flat cardboard, packing a to-go box and packing eggs, each taking 5 to 20 minutes. The paper could not solve these with other methods, so it compares π₀ only with its own ablations: pre-trained and fine-tuned, pre-trained only ("out-of-box"), and trained on the fine-tuning data alone ("scratch"). The full recipe reaches more than 50% of the maximum score on every task and performs best across the board, and on many of the hardest tasks pre-training produces a very large improvement over training from scratch.

The discussion section lists what is not settled. The experiments "do not yet provide a comprehensive understanding of how the pre-training datasets should be composed"; not all tasks work reliably; it is unclear how much data a task needs to reach near-perfect performance; and whether the approach extends to autonomous driving, navigation or legged locomotion is left to future work.