LLaVA: Visual Instruction Tuning

One linear layer lets a language model read an image as 256 tokens, and text-only GPT-4 writes the conversations that teach it to answer.

LLaVA connects a frozen CLIP vision encoder to the Vicuna language model through one trainable projection matrix, trains it in two stages on 595K captions and then 158K GPT-4-written conversations, and grades the result with GPT-4 as the judge.

The tensor shapes from pixels to tokens, the positions that carry loss, the weights each stage updates, and what the paper's 85.1 and 92.53 measure.

Instruction tuning fine-tunes a pretrained language model on (instruction, response) pairs, so that at test time it carries out a written request instead of continuing the text. FLAN did it with 60-odd NLP datasets rephrased as instructions, InstructGPT with human-written demonstrations, and by spring 2023 the cheap route was machine-written data. Alpaca fine-tuned LLaMA-7B on 52K demonstrations that text-davinci-003 wrote from 175 human seed examples, and Vicuna fine-tuned LLaMA-13B on about 70K conversations that users had shared from ChatGPT, text in and text out like the rest.

Vision-language models of the same season had the opposite gap. The paper shows a photo of a man ironing clothes on the back of a yellow taxi and asks "What is unusual about this image?". BLIP-2 answers "a man is sitting on the back of a yellow cab" (Table 3), a caption, because captions were what its training data paired with images. No multimodal counterpart to Alpaca's 52K pairs existed, and writing one by hand is slow.

LLaVA (Large Language and Vision Assistant) has three parts, and the page takes them in this order:

• Data. Text-only GPT-4 writes 158K image-grounded conversations from each image's captions and bounding boxes. It never receives the pixels.
• Model. A frozen CLIP vision encoder, one trainable matrix, and Vicuna-13B. The image enters the language model as 256 vectors that sit where word embeddings would. Training runs in two stages: first the matrix alone, then the matrix and the language model together.
• Evaluation. GPT-4 rates the model's answers against reference answers, which gives the paper's headline relative score of 85.1. Fine-tuned on the ScienceQA benchmark instead, the same model reaches 90.92% accuracy alone and 92.53% with GPT-4 settling disagreements, a new best at the time.

How text-only GPT-4 wrote the training data

The obvious way to turn an image-caption pair into an instruction sample is to put a request in front of the caption: the instruction is "Describe the image concisely." and the response is the caption. The paper does this for its first training stage, drawing the request from a list of eleven paraphrases ("Give a short and clear explanation of the subsequent image." is another; Appendix Table 11). A sample built this way cannot teach counting, spatial relations or reasoning, because a one-line caption contains none of them.

For richer data the authors used GPT-4, which in early 2023, like ChatGPT, accepted only text. Each COCO image is therefore handed over as a symbolic representation in two blocks of plain text. The first is the five human-written COCO captions for the image. The second has one line per annotated object, in the form category: [x1, y1, x2, y2], where the numbers are the box's corners as fractions of the image width and height (0 is the left or top edge, 1 the right or bottom). Figure 1 draws the paper's own example from Appendix Table 14: three people, two backpacks, two suitcases, a bicycle and four cars, 12 boxes in all. Click a box to find its line in the prompt.

The captions supply the activity and the setting (people packing an SUV in an underground garage). The box list supplies the layout, and GPT-4's generated description in Table 14 uses it. The large car box spans x = 0.261 to 0.787, and the three person boxes have centers at x ≈ 0.47, 0.66 and 0.73, all inside that span; the description places "one person standing closer to the left side of the vehicle, another person in the middle, and the third person on the right side". A backpack at [0.384, 0.696, 0.485, 0.914] sits low in the frame and left of the car's center, and becomes "one located near the left rear wheel".

Each image gets three kinds of response, each from its own prompt:

• Conversation: a multi-turn exchange in which a person asks about object types, counts, actions, locations and relative positions, restricted to questions with definite answers.
• Detailed description: a paragraph written in response to one of sixteen "describe this image in detail" paraphrases (Appendix Table 12).
• Complex reasoning: a question whose answer needs steps beyond what is visible ("What challenges do these people face?"), answered with the reasoning.

For each kind the authors hand-wrote a few seed examples, which go into the GPT-4 prompt as in-context demonstrations (Appendix Tables 13, 15, 16). Those seeds are the only human annotation in the pipeline. The released set, LLaVA-Instruct-158K, has 58K conversations, 23K detailed descriptions and 77K complex-reasoning samples over about 80K unique COCO images. In an early ablation ChatGPT wrote the same kind of data; GPT-4's output was consistently better, notably on spatial reasoning, so GPT-4 wrote the release.

How LLaVA turns an image into 256 tokens

The model has three parts: a vision encoder $g$, a projection matrix $\mathbf{W}$, and a language model $f_\phi$ with parameters $\phi$. The encoder is CLIP ViT-L/14, the image half of a model trained to match images with their captions, used here as a frozen feature extractor; its architecture is the Vision Transformer, which runs a Transformer over a grid of image patches. The language model is Vicuna-13B. The only new parameters are in $\mathbf{W}$:

Follow one image through equation (1). CLIP's preprocessor resizes and center-crops the image $\mathbf{X}_{\mathtt{v}}$ to 224 × 224 pixels. ViT-L/14 cuts that into 14 × 14-pixel patches, 16 per side, so 256 patches, and adds one CLS token, an extra position whose final output CLIP uses as its summary of the whole image. A 24-layer Transformer of width 1024 runs over those 257 positions. The grid feature $\mathbf{Z}_{\mathtt{v}}$ is the output of layer 23, the one before the last, with the CLS position dropped: a 256 × 1024 matrix, one 1024-number vector per patch. $\mathbf{W}$ maps 1024 numbers to the language model's embedding width, 5120 for Vicuna-13B, and is applied to each row separately. The result, $\mathbf{H}_{\mathtt{v}}$, is 256 × 5120: 256 vectors with the shape of a word embedding, one per patch, in patch order (left to right, top to bottom).

# one image through equation (1), Vicuna-13B sizes
x_v = preprocess(img)                  # [3, 224, 224]
out = clip_vit_l14(x_v, output_hidden_states=True)
z_v = out.hidden_states[-2][1:]        # layer 23 of 24, CLS dropped -> [256, 1024]
h_v = W(z_v)                           # nn.Linear(1024, 5120)       -> [256, 5120]

Why the layer before the last? The paper ran both. With layer 23 the ScienceQA model scores 90.92%, with layer 24 it scores 89.96%. The authors' hypothesis is that the last layer focuses on global and abstract properties of the image, while the layer before it keeps more localized detail that helps with specific questions. It fits how CLIP is trained, since only the last layer's CLS output is compared with the caption embedding, but the paper offers it as a hypothesis and tests it with that one ablation. In code it is hidden_states[-2] with the first position sliced off.

Why a single matrix, when heavier connectors existed? Flamingo inserts gated cross-attention layers into the language model, and BLIP-2 puts a Querying Transformer with 32 learned query tokens in front of it; both change what the language model computes. With a linear projection, the 256 image vectors are placed in the input sequence where word embeddings would go, and every Vicuna layer processes them with the same weights it uses for words. $\mathbf{W}$ has 1024 × 5120 weights plus 5120 biases, 5,248,000 numbers in all, about 0.04% of the 13.3B-parameter model, and trains in hours. The paper chose it so that experiments on the data could iterate quickly, and lists the heavier connectors as future work.

In Figure 2, pick a patch: the same index is a row of $\mathbf{Z}_{\mathtt{v}}$, a row of $\mathbf{H}_{\mathtt{v}}$, and a slot in the token sequence. The layer buttons move the tap point in the CLIP stack and show the ScienceQA accuracy for each.

The two markers are vocabulary entries the code adds to the tokenizer. Between them sit 256 placeholder tokens, and at runtime their embeddings are overwritten with the rows of $\mathbf{H}_{\mathtt{v}}$. An image therefore costs 258 of Vicuna's 2048 positions, about an eighth of the context window.

The training sequence and where the loss lands

A training sample is an image $\mathbf{X}_{\mathtt{v}}$ and a conversation of $T$ turns, each turn a question $\mathbf{X}_{\mathtt{q}}^t$ and an answer $\mathbf{X}_{\mathtt{a}}^t$. The paper packs it into one token sequence in Vicuna's chat format (Table 2): a system message, then alternating ### Human: and ### Assistant: turns, with ### also serving as the stop marker after each answer. The system message is Vicuna v0's: "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions." The image goes into the first human turn only, and whether it comes before or after the first question is chosen at random:

The random order is a cheap augmentation: the model sees the image both before and after a question, so it cannot come to depend on one layout. Later turns carry text only, since the image is already in the context. Training uses the language model's ordinary next-token objective, applied only to the assistant's tokens. For a target answer sequence of length $L$ the paper writes the probability as

and the loss is its negative log: a sum of $-\log p_\theta(x_i \mid \ldots)$ over the answer tokens and the stop marker after each answer. $\theta$ is whatever is trainable in the current stage. Each factor conditions on the image and on every instruction and answer token before position $i$, which is what causal attention over the packed sequence gives anyway. The paper writes $\mathbf{X}_{\mathtt{v}}$ out explicitly to stress that every answer, in every turn, is grounded on the image, and it leaves the system message and earlier stop markers out of (3) for readability.

"Only the assistant's tokens" is implemented as label masking. The code copies the token ids as labels and overwrites each position that belongs to the system message, the image, the image markers or a human turn with IGNORE_INDEX = -100, a value the cross-entropy skips. Those tokens stay in the input, so the model conditions on them, but it is never asked to predict them. A worked count, with words standing in for tokens and the first question of the Table 14 conversation as the sample: the system message is 23 words, the question ("What type of vehicle is featured in the image?") 9, and the answer ("The image features a black sport utility vehicle (SUV).") 9. Add the 256 image tokens and the role labels and the single-turn sequence has 302 positions. The loss uses 10 of them, the 9 answer words and the stop marker, following the green tokens of the paper's Table 2.

# one single-turn sample (words stand in for tokens)
text   = system + "### Human: " + "<im_patch>" * 256 + question + "\n" \
       + "### Assistant: " + answer + "\n###"
ids    = tokenize(text)                 # [N]
emb    = embed(ids)                     # [N, 5120]
emb[img_slots] = h_v                    # the 256 placeholder rows become H_v
labels = ids.clone()
labels[not answer_or_stop] = -100       # IGNORE_INDEX: no loss at these positions
logits = vicuna(emb)                    # [N, 32000]
loss   = cross_entropy(logits[:-1], labels[1:])   # -log of equation (3)

Figure 3 lays that sequence out. Drag the prediction cursor along the answer: the token under it is predicted from everything to its left, which is underlined, and the image block is always inside that span. Add turns and the later questions join the context with no second copy of the image; flip the first turn's order to see the two layouts equation (2) chooses between.

Only the conversation type is multi-turn; detailed description and complex reasoning are single-turn. In the second training stage the three types are sampled uniformly, so a batch of 32 mixes short exchanges with paragraph-length answers.

Two training stages and what each one updates

At the start $\mathbf{W}$ is random, so the 256 image vectors mean nothing to Vicuna. Why not fine-tune everything on the instruction data from there? The paper first aligns the image features with Vicuna's word embeddings while Vicuna stays frozen, which it describes as aligning the two while preserving the language model's pretrained knowledge. Skipping that first stage costs 5.11 points on ScienceQA (85.81 vs 90.92), the largest drop among the paper's ablations.

Stage 1, feature alignment. The data is CC3M (Conceptual Captions, about 3 million web image-caption pairs) filtered down to 595K by a coverage rule. The filter extracts noun phrases from every caption with spaCy, drops phrases seen fewer than 3 times, then walks the remaining phrases from rarest to most common and adds captions containing each one to the pool, capping any phrase at 100 randomly chosen captions. A phrase seen 40 times keeps all 40 captions and a phrase seen 50,000 times adds at most 100 more, so the 595K cover many concepts without being dominated by the most common ones. Each pair becomes one single-turn sample of the naive kind:

### Human: <image> Describe the image concisely.
### Assistant: <the original CC3M caption>
###

Trainable parameters: $\theta = \{\mathbf{W}\}$, with the vision tower and Vicuna frozen. One epoch, learning rate 2e-3, batch 128 (8 A100s × 16 each), so 595K / 128 ≈ 4,650 optimizer steps, within 4 hours. The paper describes the result as a visual tokenizer compatible with the frozen language model: after stage 1, $\mathbf{W}$ maps CLIP patches to vectors that Vicuna can write a caption from.

Stage 2, end-to-end fine-tuning. The data is LLaVA-Instruct-158K. Trainable parameters: $\theta = \{\mathbf{W}, \phi\}$, the projector and every language-model weight; the vision tower stays frozen. Three epochs, learning rate 2e-5, batch 32 (8 × 4), so 158K / 32 ≈ 4,940 steps per epoch and about 14,800 in total, within 10 hours. The learning rate is 100 times smaller than in stage 1 because stage 1 moves 5.2 million freshly initialized parameters and stage 2 moves 13 billion pretrained ones.

Both stages use Adam with no weight decay, a cosine learning-rate schedule with 3% warmup, bf16 arithmetic, and a maximum sequence length of 2048. Memory is handled differently. In stage 1 only $\mathbf{W}$ has gradients and optimizer state, so each GPU holds a full copy of the frozen model and the training command adds only gradient checkpointing. In stage 2 every Vicuna weight needs a gradient and Adam's two running averages. The usual mixed-precision accounting from the ZeRO paper puts that at about 16 bytes per parameter, roughly 16 × 13B ≈ 208 GB, far more than one A100's 40 or 80 GB, so stage 2 adds FSDP (fully sharded data parallel), which splits weights, gradients and optimizer state across the 8 GPUs.

The stages differ in what the backward pass touches. In stage 1 the gradient of the loss still has to travel back through all 40 Vicuna layers to reach $\mathbf{W}$, because $\mathbf{W}$ sits at the input. The backward pass differentiates through those layers' activations, but their weights have requires_grad = False, so nothing accumulates in them. The vision tower runs without gradient tracking in both stages, so no gradient ever reaches CLIP. In Figure 4 the pulses leave the loss, cross the language model (dashed when it is frozen) and stop at $\mathbf{W}$.

loss.backward()
# stage 1: only W.grad is filled. CLIP ran without gradient tracking, and
#          Vicuna's weights have requires_grad = False, so the backward pass
#          goes through Vicuna's activations to reach W and stores nothing there.
# stage 2: W.grad and a gradient for every Vicuna weight. CLIP still gets none.
opt.step()

One stage-2 step, concretely. Draw 32 samples, each an image and a conversation. Run CLIP on the 32 images (32 × 256 × 1024), multiply by $\mathbf{W}$ (32 × 256 × 5120), and build 32 packed sequences padded to the longest, at most 2048 positions. Vicuna returns logits of shape 32 × $N$ × 32,000, where $N$ is the padded length. Cross-entropy against the shifted, masked labels gives one scalar. The backward pass fills gradients for $\mathbf{W}$ and all of Vicuna, and Adam updates both. Stage 2 repeats this about 14,800 times.

What the 85.1 relative score measures

No benchmark existed for "follows visual instructions well", so the paper builds two and scores both the way Vicuna was scored, with GPT-4 as the grader. The protocol has three steps, and GPT-4 sees only text in each:

1. For each test question, text-only GPT-4 writes a reference answer from the ground-truth captions and boxes in place of the image. The paper calls this an approximate theoretical upper bound.
2. The candidate model answers the same question from the actual image.
3. A text-only GPT-4 judge gets the captions and boxes, the question and both answers in one prompt, and returns two scores from 1 to 10 for helpfulness, relevance, accuracy and level of detail, plus an explanation.

The reported number is a relative score: the candidate's mean judge score divided by the reference's mean judge score, times 100. If the reference averaged 8.0 out of 10 and the candidate 6.8, the relative score would be 6.8 / 8.0 × 100 = 85.0 (illustrative numbers; the paper reports only the ratios). Figure 5 builds such a score from sixty rated questions and reruns it two ways.

After a few presses of each button the two lanes separate. In the figure's model, re-judging leaves almost every rating where it was and occasionally moves one by a point, so the score moves by a few tenths. Re-sampling gives every question a new answer that can be better or worse, so the score moves by about two points. The paper measured both: LLaVA's In-the-Wild score over three inference runs is 67.3 ± 2.0, and the same LLaVA answers judged three times give 66.7 ± 0.3. The figure also shows a reference answer losing to the candidate on some questions. The reference is an approximation, and text-only GPT-4 reading captions can miss what the captions omit. On the COCO set described next, LLaVA's complex-reasoning relative score is 96.5: averaged over those questions, the judge rated LLaVA's answers almost as highly as the reference answers. The ratings are 1-to-10 quality scores, so a relative score of 67.3 means LLaVA's answers were rated, on average, about two-thirds as highly as the reference answers; it is not a percentage of questions answered correctly.

LLaVA-Bench (COCO) is 30 COCO validation images with the three question types from the data pipeline, 90 questions in all. It exists for the data ablation in Table 4. With all 158K training samples the relative score is 85.1 overall: 83.1 on conversation, 75.3 on detailed description, 96.5 on complex reasoning. Training on the conversation data alone gives 73.8. Adding 5% of the detail data and 10% of the reasoning data lifts that to 80.5, and the conversation questions themselves go from 76.5 to 81.0, so data of the other two types helps the conversation answers too. Detail and complex data without conversation gives 81.9. The model after stage 1 only, with no instruction tuning, scores 21.5, since stage 1 trained it to answer every request with a caption.

LLaVA-Bench (In-the-Wild) is 24 images unlike COCO (indoor and outdoor scenes, memes, paintings, sketches) with 60 questions and a hand-written detailed description per image. Here LLaVA scores 67.3 against 38.1 for BLIP-2 and 19.1 for OpenFlamingo, and 81.7 ± 1.8 on the complex-reasoning questions. The paper writes those gaps as "+29%" and "+48%"; they are differences of 29.2 and 48.2 relative-score points. Figure 6 has both tables.

The judge is the same model that wrote the reference answers, and the paper does not test whether it favors answers written in its own style. Its appendix says only that the robustness of GPT-4 evaluation across situations is a subject for future work.

The In-the-Wild set was built to expose failures, and the paper reports two. Asked the name of the restaurant in a photo of a ramen bowl (it is ICHIRAN), the model needs broad knowledge and multilingual understanding, which 158K COCO conversations do not supply. Asked whether a fridge holds strawberry-flavored yogurt when it holds yogurt and strawberries, LLaVA says yes. The authors describe this as perceiving the image as a "bag of patches": both objects are detected, and the relation between them is lost.

ScienceQA: 90.92 alone, 92.53 with a GPT-4 judge

ScienceQA is 21,208 multiple-choice science questions, each with a text or image context, split 12,726 / 4,241 / 4,241 into train, validation and test, with each answer annotated with a lecture and an explanation. For this benchmark stage 2 trains on ScienceQA instead of the chat data, as single-turn samples: the question and its context are the instruction, the reasoning and the answer are the response. The best configuration reads layer-23 CLIP features, writes the reasoning before the answer, as in chain-of-thought prompting, and trains for 12 epochs. It scores 90.92% on the test set. The prior state of the art, MM-CoT Large, scored 91.68.

Then the paper adds text-only GPT-4. Prompted with two in-context examples and no image, GPT-4 alone scores 82.69% (GPT-3.5 scored 75.17), and on a substantial number of questions it fails by reporting that it lacks the image or plot. Two ways of combining the models follow. In the complement scheme, LLaVA's answer is used whenever GPT-4 declines; that gives 90.97, almost the same as LLaVA alone. In the judge scheme, written out below, GPT-4 is prompted again whenever the two disagree, with the question and both answers, and asked for its own final answer:

# ScienceQA, "GPT-4 as the judge" (Sec. 5.2); answers are option letters
a_llava = llava(question, context, image)     # reasoning, then the answer
a_gpt   = gpt4_two_shot(question, context)    # text only, no image
if a_gpt == a_llava:
    final = a_llava
else:                                         # they disagree: ask GPT-4 again
    final = gpt4(question, context, a_llava, a_gpt)   # "your own final answer"

The judge scheme scores 92.53, a new state of the art. It beats LLaVA alone in every one of the eight question categories and beats MM-CoT Large on the average, though not on natural-science or text-context questions, where MM-CoT Large keeps 95.91 and 95.26 against the judge's 91.56 and 90.62. A text-only model helps on image-context questions because some of them do not need the image to be answered, and on those the GPT-4 judge corrects some of LLaVA's errors. The authors write that, to their knowledge, this is the first time GPT-4 is used for model ensembling. Figure 7 has every row of Table 7 and the ablations of Table 8.

Each ablation in Table 8 changes one thing from the 90.92 configuration. Last-layer CLIP features: 89.96. Answer before reasoning: 89.77. The paper's text adds that reasoning-first reaches 89.77 by epoch 6 while answer-first needs 12 epochs to get there, that 24 epochs do not help, and concludes that the order mainly speeds up convergence; Table 8's 1.15-point gap at 12 epochs says the order also changes the final accuracy (see the Provenance panel). No stage-1 pre-training: 85.81, the only change that costs more than about a point. Vicuna-7B in place of 13B: 89.84.

Limits, and what LLaVA-1.5 changed

The paper names these limits: the bag-of-patches failure above; fine detail such as a yogurt brand, which the paper says needs higher-resolution input (LLaVA sees a 224 × 224 crop); knowledge and multilingual coverage bounded by what Vicuna and 158K COCO conversations contain; and hallucination, outputs not grounded in the input, which the broader-impact appendix discusses. The training data also inherits GPT-4's errors, since apart from the seed examples no human wrote or checked the 158K samples, and its box lists cover only COCO's 80 object categories.

Six months later the same authors released LLaVA-1.5, and its changes map onto that list. The linear $\mathbf{W}$ became a two-layer MLP with a GELU between the layers, the vision tower became CLIP ViT-L/14 at 336 pixels (24 × 24 = 576 image tokens), and the instruction data grew to 665K samples by adding academic VQA datasets with prompts that ask for short answers. The rest of the recipe stayed: a frozen CLIP tower, a small projector pre-trained alone, then the projector and the language model fine-tuned together on instruction data. Many later open vision-language models follow the same layout, including robot policies such as OpenVLA, which feed a projector's output into a LLaMA-family model the same way.