Visual Instruction Tuning

One linear layer lets a language model read an image as 256 tokens, and text-only GPT-4 writes the conversations that teach it to answer.

LLaVA connects a frozen CLIP vision encoder to the Vicuna language model through a trainable projection matrix, trains that connection in two stages, and scores the result with GPT-4 as the judge.

How an image turns into tokens, which tokens carry loss, what each training stage updates, and what the paper's 85.1 and 92.53 measure.

Instruction tuning is fine-tuning a pretrained language model on pairs of (instruction, response) so that at test time it does what a written instruction asks instead of continuing the text. FLAN did it with 60-odd NLP tasks rephrased as instructions, InstructGPT with human-written demonstrations, and by spring 2023 the cheap recipe was machine-written data: Alpaca fine-tuned LLaMA-7B on 52K demonstrations that text-davinci-003 wrote from 175 seed examples, and Vicuna fine-tuned LLaMA-13B on about 70K conversations that users had shared from ChatGPT, all of it text in and text out.

Vision-language models of the same season had the opposite gap. BLIP-2 and OpenFlamingo could look at a picture, but asked "what is unusual about this image?" they returned a caption ("a man is sitting on the back of a yellow cab"), because captioning was what their training data taught. There was no multimodal equivalent of Alpaca's 52K instruction pairs, and hand-writing one is slow and ill-defined.

LLaVA (Large Language and Vision Assistant) is three things built in order. A data pipeline in which text-only GPT-4 writes 158K image-grounded conversations from captions and bounding boxes, never seeing a pixel. A model, trained in two stages on that data, that puts a single trainable matrix between a frozen CLIP vision encoder and Vicuna-13B, so that an image enters the language model as 256 ordinary-looking tokens. And an evaluation in which GPT-4 grades the model's answers against reference answers, giving the paper's headline relative score of 85.1. Fine-tuned on ScienceQA instead, the same model reaches 90.92% accuracy alone and 92.53% with GPT-4 arbitrating disagreements, a new state of the art at the time.

The page follows that order: the data, the architecture as tensor shapes (equation (1)), the training sequence and loss (equations (2) and (3)), the two stages and what receives gradient in each, then the two evaluations and what their numbers mean.

Writing image conversations without the image

The obvious way to turn an image-caption pair into an instruction sample is to prepend a question: "Describe the image briefly" followed by the caption as the answer. The paper does exactly this for its first training stage, sampling the question from a list of eleven paraphrases ("Describe the image concisely.", "Give a short and clear explanation of the subsequent image.", and so on, Appendix Table 11). Such samples teach nothing about counting, spatial relations, or reasoning, because the caption never contained any of that.

For richer data the authors used GPT-4, which in March 2023 accepted only text. So the image is handed over as a symbolic representation: the five human-written COCO captions for that image, plus every annotated object as a line category: [x1, y1, x2, y2], corners given as fractions of the image width and height. Figure 1 draws the paper's own example (Appendix Table 14): three people, two backpacks, two suitcases, a bicycle, and four cars, one of which spans most of the frame. Click a box and read the four numbers GPT-4 read.

From those numbers GPT-4 can infer what a caption omits. The big car box runs from x = 0.261 to 0.787 and the three person boxes sit inside its horizontal span with centers at x ≈ 0.73, 0.66, and 0.47, so "one person closer to the left side of the vehicle, another in the middle, the third on the right" is arithmetic on the box list. A backpack at [0.384, 0.696, 0.485, 0.914] is low in the frame and left of the car's center, which becomes "one located near the left rear wheel" in the generated description. The captions carry the activity and setting (packing an SUV for a trip in an underground garage), and the box list carries the layout.

Each image gets three kinds of response, generated by separate prompts. Conversation: a multi-turn exchange in which a person asks about object types, counts, actions, locations, and relative positions, restricted to questions with definite answers. Detailed description: a paragraph written in response to one of sixteen "describe this image in detail" paraphrases. Complex reasoning: a question whose answer needs steps beyond what is visible ("What challenges do these people face?"), answered with reasoning. For each kind the authors hand-wrote a few seed examples; those seeds are the only human annotation in the pipeline and are pasted into the GPT-4 prompt as in-context demonstrations (Appendix Tables 13, 15, 16).

The released set, LLaVA-Instruct-158K, has 58K conversations, 23K detailed descriptions, and 77K complex-reasoning samples over about 80K unique COCO images. An early ablation had ChatGPT write the same data; GPT-4's output was consistently better, notably on spatial reasoning, so GPT-4 wrote the release.

An image becomes 256 tokens

The model has three parts: a vision encoder $g$, a projection matrix $\mathbf{W}$, and a language model $f_\phi$ with parameters $\phi$. The encoder is CLIP ViT-L/14, the vision half of a model trained to match images to their captions, and it is used as a frozen feature extractor. The language model is Vicuna-13B. The only new parameters are in $\mathbf{W}$:

Read (1) as shapes. The image $\mathbf{X}_{\mathtt{v}}$ is resized and center-cropped to 224 × 224 pixels. ViT-L/14 cuts it into 14 × 14-pixel patches, 16 by 16 of them, so 256 patches, plus one CLS token (a summary position CLIP uses for its own pooled output), and runs a 24-layer Transformer of width 1024 over those 257 positions. The grid feature $\mathbf{Z}_{\mathtt{v}}$ is the output of layer 23, the layer before the last one, with the CLS position dropped: a 256 × 1024 matrix, one 1024-vector per patch. $\mathbf{W}$ is a linear map from 1024 to the language model's embedding width, 5120 for Vicuna-13B, applied to each row. That gives $\mathbf{H}_{\mathtt{v}}$, 256 × 5120: 256 vectors that have the same shape as a word embedding, one per patch, in patch order.

# one image through equation (1), Vicuna-13B sizes
x_v   = preprocess(img)                    # [3, 224, 224]
out   = clip_vit_l14(x_v, output_hidden_states=True)
z_v   = out.hidden_states[-2][1:]          # layer 23 of 24, CLS dropped
                                           #   -> [256, 1024]
h_v   = W(z_v)                             # nn.Linear(1024, 5120)
                                           #   -> [256, 5120]

Why the layer before the last? CLIP's last layer feeds the contrastive objective, which compares one pooled vector per image against a caption embedding, so the authors' hypothesis is that it summarizes the image as a whole while layer 23 keeps more local detail. The evidence is the ScienceQA ablation: 90.92% accuracy with layer 23 against 89.96% with layer 24. In code this is hidden_states[-2] with the first position sliced off.

Why a single matrix and not a heavier connector? Flamingo inserts gated cross-attention layers into the language model, and BLIP-2 puts a Querying Transformer with 32 learned query tokens in front of it. Both change what the language model computes. A linear projection changes nothing: the 256 image vectors are inserted into the input sequence exactly where word embeddings would go, and every layer of Vicuna treats them as tokens. $\mathbf{W}$ has 1024 × 5120 weights plus 5120 bias terms, 5,248,000 numbers, which trains in hours; the paper says the lightness let them iterate on data quickly and names the heavier connectors as future work.

Figure 2 is (1) as a data flow. Pick a patch; the same index is a row of $\mathbf{Z}_{\mathtt{v}}$, a row of $\mathbf{H}_{\mathtt{v}}$, and a slot in the token sequence. The layer buttons switch the tap point in the CLIP stack and show the ablation number for each.

The two markers are new vocabulary entries the code adds to the tokenizer; between them sit 256 placeholder tokens whose embeddings are overwritten with the rows of $\mathbf{H}_{\mathtt{v}}$ at runtime. An image therefore costs 258 of Vicuna's 2048 positions.

The training sequence and where the loss lands

A training sample is an image $\mathbf{X}_{\mathtt{v}}$ and a conversation of $T$ turns $(\mathbf{X}_{\mathtt{q}}^1, \mathbf{X}_{\mathtt{a}}^1, \dots, \mathbf{X}_{\mathtt{q}}^T, \mathbf{X}_{\mathtt{a}}^T)$. The paper packs it into one token sequence in Vicuna's chat format (its Table 2): a system message, then alternating ### Human: and ### Assistant: turns, with ### doubling as the stop marker. The system message is Vicuna v0's: "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions." The image goes into the first human turn only, and its position relative to the first question is a coin flip:

The random order is cheap augmentation: the model sees the image both before and after a question, so it cannot learn to rely on one layout. Later turns carry only text; the image is already in the context. Training uses the language model's ordinary next-token objective, restricted to the assistant's tokens. For a sequence of length $L$ the probability of the answers is

and the loss is its negative log, summed over the positions $i$ that hold an answer token or the stop marker after an answer. $\theta$ is whatever is trainable in the current stage. The conditioning set of each factor is the image plus every instruction and answer token before position $i$, which is just causal attention over the packed sequence; the paper writes $\mathbf{X}_{\mathtt{v}}$ out explicitly to stress that every answer in every turn is grounded on the image.

"Restricted to the assistant's tokens" is implemented as label masking. The code copies the token ids as labels and overwrites every position belonging to the system message, the image, the markers, or a human turn with IGNORE_INDEX = -100, which the cross-entropy skips. Those tokens are still in the input, so the model conditions on them; it is never asked to predict them. A concrete count, with words standing in for tokens and the Table 14 conversation as the sample: the system message is 23 words, the first question 9, the first answer 9. With the 256 image tokens, the markers, and the role labels the single-turn sequence has 302 positions, and the loss touches 10 of them: the 9 answer words and the stop marker.

# the sequence for one single-turn sample (words stand in for tokens)
text   = system + "### Human: " + "<im_patch>" * 256 + question + "\n" \
       + "### Assistant: " + answer + "\n###"
ids    = tokenize(text)                    # [L]
emb    = embed(ids)                        # [L, 5120]
emb[img_slots] = h_v                       # 256 rows replaced by H_v
labels = ids.clone()
labels[not answer_or_stop] = -100          # IGNORE_INDEX: no loss here
logits = vicuna(emb)                       # [L, 32000]
loss   = cross_entropy(logits[:-1], labels[1:])   # equation (3)

Figure 3 lays out that sequence. Drag the prediction cursor along the answer: the token under it is predicted from everything to its left, underlined, and the image block is always inside that span. Add turns to see the later questions join the context with no second copy of the image, and flip the first turn's order to see the two layouts that equation (2) chooses between.

Multi-turn samples are the conversation type only; detailed description and complex reasoning are single-turn. During stage 2 the three types are sampled uniformly, so a batch of 32 mixes short exchanges and paragraph answers.

Two stages: the projector, then the whole model

At the start, $\mathbf{W}$ is random, so the 256 image vectors are noise to Vicuna. The paper's reason for a separate first stage is to align the image features with the word embeddings while the language model, frozen, keeps its pretrained knowledge intact; the ablation that removes the first stage costs 5.11 points on ScienceQA (85.81 vs 90.92), the largest drop in Table 8.

Stage 1, feature alignment. The data is CC3M (Conceptual Captions, 3 million web image-caption pairs) filtered down to 595K. The filter is a coverage rule: extract noun phrases from every caption, drop phrases seen fewer than 3 times, then walk the phrases from rarest to commonest, adding their captions to the pool and capping any phrase at 100 randomly chosen captions. Each pair becomes a single-turn sample by the naive expansion above: one of the eleven brief-description questions, the original caption as the answer. Trainable parameters: $\theta = \{\mathbf{W}\}$. The vision tower and Vicuna are frozen. One epoch, learning rate 2e-3, batch 128 (8 A100s × 16), about 4,650 optimizer steps, roughly 4 hours. The paper describes the result as a visual tokenizer compatible with the frozen language model: after stage 1, $\mathbf{W}$ maps CLIP patches to vectors Vicuna can caption from.

Stage 2, end-to-end fine-tuning. The data is LLaVA-Instruct-158K. Trainable parameters: $\theta = \{\mathbf{W}, \phi\}$, the projector and every language-model weight. The vision tower stays frozen. Three epochs, learning rate 2e-5, batch 32 (8 × 4), about 4,900 steps per epoch, roughly 10 hours. Both stages use Adam with no weight decay, a cosine schedule with 3% warmup, bf16, gradient checkpointing, and FSDP (the 13B model sharded across the 8 GPUs). The learning rate drops by 100× between stages because stage 1 moves 5.2 million fresh parameters and stage 2 moves 13 billion pretrained ones.

The stages differ in what the backward pass touches. In stage 1 the loss gradient still has to travel through all 40 Vicuna layers to reach $\mathbf{W}$, because $\mathbf{W}$ sits at the input; the layers' activations are differentiated through, but their weights, having requires_grad = False, accumulate nothing. The vision tower runs under no_grad in both stages, so no gradient ever reaches CLIP. Figure 4 animates that: pulses leave the loss, cross the language model (dashed when it is frozen), and stop at $\mathbf{W}$.

loss.backward()
# stage 1: only W.grad exists. CLIP ran under no_grad; Vicuna's weights
#          are frozen, so the gradient passes through its activations
#          on the way to W and updates nothing there.
# stage 2: W.grad and every Vicuna weight. CLIP still gets nothing.
opt.step()

One stage-2 step, concretely. Draw 32 samples; each is an image and a conversation. Run CLIP on the 32 images (32 × 256 × 1024), multiply by $\mathbf{W}$ (32 × 256 × 5120), and build 32 packed sequences, padded to the longest, at most 2048 positions. Vicuna returns logits of shape 32 × $L$ × 32,000. Cross-entropy against the shifted, masked labels gives one scalar. Backward populates gradients for $\mathbf{W}$ and all of Vicuna; Adam updates both. Repeat about 14,800 times across three epochs.

GPT-4 as judge: what 85.1 measures

There was no benchmark for "follows visual instructions well", so the paper builds two, and scores both with Vicuna's method: GPT-4 as the grader. The protocol has three steps, and GPT-4 sees only text in every one of them. First, for each test image, write a reference answer with text-only GPT-4 that is given the ground-truth captions and boxes in place of the image; the paper calls this an approximate upper bound. Second, have the candidate model answer the same question from the actual image. Third, give a text-only GPT-4 judge the captions and boxes, the question, and both answers in one prompt, and ask for two scores from 1 to 10 for helpfulness, relevance, accuracy, and level of detail, plus an explanation. The reported number is a relative score: the mean judge score of the candidate divided by the mean judge score of the reference, times 100. If the reference averaged 8.0 out of 10 and the candidate 6.8, the relative score would be 85.0 (illustrative numbers; the paper reports only the ratios).

LLaVA-Bench (COCO) is 30 COCO validation images with the three question types generated by the data pipeline, 90 questions. Its purpose is the data ablation in Table 4. With all 158K samples the relative score is 85.1 overall: 83.1 on conversation, 75.3 on detailed description, 96.5 on complex reasoning. Training on conversation data alone gives 73.8; adding 5% of the detail data and 10% of the reasoning data lifts that to 80.5, and the full set to 85.1, with the gains showing on the conversation questions too. The model after stage 1 only, with no instruction tuning at all, scores 21.5; stage 1 taught it only to produce a caption.

LLaVA-Bench (In-the-Wild) is 24 images that are nothing like COCO (memes, paintings, sketches, indoor and outdoor scenes) with 60 questions and hand-written detailed descriptions. Here LLaVA scores 67.3 (± 2.0 over three runs) against 38.1 for BLIP-2 and 19.1 for OpenFlamingo, and 81.7 on the complex-reasoning questions. The paper writes those gaps as "+29%" and "+48%"; they are differences of 29.2 and 48.2 relative-score points.

Three properties of the score to keep in mind when reading those bars. The reference is not a ceiling: text-only GPT-4 reading captions can miss what the captions omit, and a candidate can in principle beat it, which is why complex reasoning reaches 96.5 on COCO. A judge score is a rating, not a probability; the ± 0.3 spread when the same LLaVA answers are re-judged three times (Table 5, last row) is the noise floor of the grader itself, separate from the ± 2.0 from re-sampling the model. And the judge is the same model that wrote the reference answers, so it may favor answers in its own style; the paper does not test for that.

The In-the-Wild set is built to expose failures, and the paper reports two. Asked the name of a restaurant from a photo of a ramen bowl, the model needs knowledge of the chain's branding, which is beyond 158K COCO conversations. Asked whether a fridge contains strawberry-flavored yogurt when it contains yogurt and strawberries, the model says yes. The authors describe the second as perceiving the image as a bag of patches: the objects are all detected, and the relation between them is lost.

ScienceQA: 90.92 alone, 92.53 with a judge

ScienceQA is 21,208 multiple-choice science questions with a text or image context, split 12,726 / 4,241 / 4,241 for train, validation, and test, each answer annotated with a lecture and an explanation. For this benchmark the model is fine-tuned in stage 2 on ScienceQA instead of the chat data: the question and its context are the instruction, the reasoning and the answer are the response, single-turn. It trains for 12 epochs, reads layer-23 CLIP features, and is asked to write the reasoning first and then the answer. That configuration scores 90.92% on the test set. The prior state of the art, MM-CoT Large, scored 91.68.

Then the paper adds text-only GPT-4. Prompted with two in-context examples and no image, GPT-4 alone scores 82.69, and it fails a large share of questions by reporting that it lacks the image. Two combinations follow. Complement: use LLaVA's answer whenever GPT-4 declines; 90.97, indistinguishable from LLaVA alone. Judge: whenever the two disagree, prompt GPT-4 again with the question and both candidate answers and ask it to choose; 92.53, above LLaVA alone in every one of the eight categories and above MM-CoT Large on average (though not on natural-science or text-context questions, where MM-CoT Large keeps 95.91 and 95.26). Some image-context questions do not actually need the image, and on those the judge corrects LLaVA's mistakes. Figure 6 has every row of Table 7.

The ablations (Table 8) each change one thing from the 90.92 configuration. Last-layer CLIP features: 89.96. Answer before reasoning: 89.77; the paper also reports that reasoning-first reaches 89.77 by epoch 6 while answer-first needs 12 to get there, so writing the reasoning first mostly speeds convergence. No stage 1: 85.81. Vicuna-7B in place of 13B: 89.84. Of the four, only skipping the projector pre-training costs more than about a point.

Limits and what came next

The paper's own list: the bag-of-patches failure above; 224 × 224 input, which cannot read a yogurt brand or a menu; knowledge and multilingual coverage limited to what Vicuna and 158K COCO conversations contain; and hallucination, discussed in the broader-impact appendix. The training data itself inherits GPT-4's errors, since no human checked the 158K samples, and it inherits COCO's object vocabulary of 80 categories for the boxes.

Six months later the same authors released LLaVA-1.5, and its changes line up with this list: the linear $\mathbf{W}$ became a two-layer MLP with a GELU between, the vision tower became CLIP ViT-L/14 at 336 pixels (24 × 24 = 576 image tokens), and the instruction data grew to 665K by adding academic VQA datasets with short-answer formatting. The recipe otherwise stayed: a frozen CLIP tower, a small projector pre-trained alone, then the projector and the language model fine-tuned together on instruction data. That recipe is what "visual instruction tuning" came to mean, and it is the architecture of many open vision-language models that followed, including robot policies such as OpenVLA, which pass a projector's output into a LLaMA-family model the same way.