CLIP: Learning Transferable Visual Models From Natural Language Supervision

Train on 400 million web images and their captions, and any list of class names becomes an image classifier.

OpenAI trained an image encoder and a text encoder to pick out which caption in a batch belongs to which image. At test time the text encoder turns class names into classifier weights, and with no ImageNet training images the result matches a supervised ResNet-50 on ImageNet.

A supervised ResNet-101 and zero-shot CLIP both score 76.2% on ImageNet. On ImageNet Sketch, drawings of the same 1,000 classes, the ResNet-101 scores 25.2% and CLIP scores 60.2%.

CLIP (Contrastive Language-Image Pre-training) is a pair of networks trained together. An image encoder maps a picture to a vector, a text encoder maps a sentence to a vector of the same length, and training makes the two vectors point the same way when the sentence is the picture's caption. The training data is 400 million (image, caption) pairs collected from the web, with no human labeling.

After training, CLIP classifies images into categories it was never trained on. You write each category as a short sentence ("A photo of a golden retriever."), embed the sentences once, embed the image, and pick the sentence whose vector is closest. On ImageNet this zero-shot classifier reaches 76.2% top-1 accuracy (the true class scores highest on 76.2% of test images), the accuracy of the original ResNet-50, which was trained on ImageNet's 1.28 million labeled images; CLIP uses none of them. The paper benchmarks this zero-shot transfer on over 30 existing computer-vision datasets.

The page goes in this order: why fixed label sets were the bottleneck, which training task the authors picked and why, the shared embedding space, the contrastive loss with a worked 6-pair example, the learned temperature, the full-size training step with shapes, zero-shot classification and prompts, the robustness results, and the limitations.

Why fixed label sets limit image classifiers

A standard image classifier is trained on a fixed list of categories, such as ImageNet's 1,000. Each training image carries an integer, and the network's last layer has one output per integer. The integer carries no meaning inside the model: class 207 is "golden retriever" only in a lookup file outside it. The loss treats a golden retriever labeled as class 208 (Labrador retriever) exactly as wrong as one labeled as a toaster, and the model cannot output a category that was not in the list. To add "satellite photo of farmland", someone collects and labels examples and the model is retrained.

"A maltese puppy sitting in a teacup" names a breed, an age, an object and a pose, and the same words appear in millions of other captions, so a model reading them can relate this image to every other image described with "puppy" or "teacup". People also write captions without being paid to. CLIP tests whether that noisier signal, collected at a scale no labeling project can reach, trains a better visual model than clean labels.

Earlier image-text work had used MS-COCO and Visual Genome (about 100,000 photos each) or YFCC100M, which shrinks to 15 million photos once images without natural-language titles or descriptions are removed. The authors built a new dataset instead. They searched the web for (image, text) pairs whose text contains one of 500,000 queries (every word that occurs at least 100 times in English Wikipedia, plus high-mutual-information bigrams, the titles of Wikipedia articles above a search-volume threshold, and WordNet synsets), kept up to 20,000 pairs per query, and ended with 400 million pairs. They call it WIT, for WebImageText. Its total word count is similar to WebText, the corpus GPT-2 was trained on.

Choosing the training task

Given pairs, the obvious objective is captioning: feed the image to a network and train it to generate the caption one word at a time. The authors started there, with an image CNN and a text transformer trained from scratch, and measured how quickly each objective learns zero-shot ImageNet classification per image processed (the paper's Figure 2). Three objectives, same data:

• Transformer language model. Generate the caption word by word. The text model has 63 million parameters and already uses twice the compute of its ResNet-50 image encoder.
• Bag of words. Predict which words appear in the caption, ignoring their order. This learns 3x faster than the transformer language model.
• Contrastive. Embed images and captions separately and predict which caption in the batch goes with which image. Starting from the same bag-of-words text encoding and swapping only the objective gives a further 4x.

So the contrastive objective reaches a given zero-shot accuracy with about 12 times fewer images than word-by-word captioning. Both predictive objectives try to reproduce the exact words that came with an image, and web text around an image varies widely: a photo of a dog might be captioned with the dog's name, a joke, or the camera settings. Picking the right caption out of a batch only requires the image and text embeddings to carry enough information to tell this caption from the others in the batch. The authors cite earlier results that pointed the same way: contrastive objectives learned better image representations than their predictive equivalents, and generative image models needed over an order of magnitude more compute for the same quality.

Two encoders, one embedding space

CLIP has two networks, trained from scratch (no ImageNet initialization, no pre-trained language model):

• An image encoder, either a modified ResNet or a Vision Transformer (ViT). The ResNet variants replace the final global average pooling with a single layer of transformer-style attention pooling; the ViTs add one extra layer normalization before the transformer.
• A text encoder, a Transformer with GPT-2's modifications. The base size has 63 million parameters, 12 layers, width 512 and 8 attention heads. It reads lower-cased byte pair encoding (BPE) tokens, subword pieces from a 49,152-entry vocabulary, with sequences capped at 76 tokens and bracketed by start and end tokens.

Each encoder's output goes through one learned linear projection into a shared space of dimension $d_e$ (512 for ViT-B/32, 768 for ViT-L/14). The authors dropped the nonlinear projection head that image-only contrastive methods such as SimCLR used, because they saw no difference in training efficiency.

The text encoder uses causal (masked) self-attention, the kind a language model uses, where each token attends only to itself and the tokens before it. Under that mask only the last token, the end-of-text token [EOS], has seen the whole caption, and CLIP takes the caption's feature from that token's top-layer activation, layer-normalized and then projected. The authors kept the mask so that a pre-trained language model could initialize the encoder later, or language modeling could be added as a second objective; the paper leaves both to future work.

Both projected vectors are then divided by their length ($L_2$-normalized), which places every embedding on the unit sphere. For unit vectors the dot product equals the cosine of the angle between them, so the comparison between image $i$ and caption $j$ is

where $\hat{\mathbf{I}}_i$ and $\hat{\mathbf{T}}_j$ are the normalized embeddings. A value near 1 means the vectors point the same way, near 0 that they are unrelated, and negative that they point apart. Normalizing removes length from the comparison, so a caption cannot score well against every image by having a long vector. Training should rotate each image vector toward its own caption's vector and away from the others.

In the figure, images (amber) and captions (teal) sit on a circle, a 2D stand-in for the 512-dimensional sphere. Pick an image and read the spokes: each spoke's brightness is the cosine to one caption, and the brightest one goes to the image's own caption.

How the CLIP contrastive loss works

Take a batch of $N$ pairs. Embed and normalize all $N$ images and all $N$ captions, then compute (1) for every image against every caption to fill an $N \times N$ grid $S$. The $N$ diagonal cells $S_{ii}$ are the pairs that occurred together on the web. The other $N^2 - N$ cells pair an image with some other image's caption. Those are the negatives, and nobody had to label them: any caption from elsewhere in the batch is a wrong answer for image $i$. Each image gets one positive and $N - 1$ negatives per step.

The loss treats each row as a classification problem. Row $i$ holds image $i$'s scores against the $N$ captions, and the correct class is caption $i$. A softmax turns the row into probabilities (exponentiate each score, divide by the row's sum), and cross-entropy charges $-\log$ of the probability on the correct caption. The columns get the same treatment: column $j$ holds caption $j$'s scores against all images, and the correct class is image $j$. The loss averages the two directions:

$\tau$ is a temperature that divides every score before the softmax; the next section covers it. For now use CLIP's starting value, $\tau = 0.07$, so every cosine is multiplied by $1/0.07 \approx 14.3$. The paper does not number this equation; (2) is its Figure 3 pseudocode written out. The row term trains image-to-text matching, the direction zero-shot classification uses; the column term trains text-to-image matching, the direction a text search over images uses. Both read the same grid, so the second direction costs one more softmax and no extra encoder work.

A worked example with six pairs

Take a batch of six pairs: photos of a dog, a cat, a beach, a bike, a pizza and a guitar, each with its own caption. Early in training the cosines are small and nearly flat. The dog image's row, against the six captions in that order, might be

Multiplied by 14.3 these become the logits, the scores that go into the softmax, $[2.29, 1.71, 0.29, 0.71, 0.43, 0.57]$, and the softmax gives probabilities 0.446 on the dog caption and 0.252 on the cat caption, with the rest between 0.060 and 0.093. The row contributes $-\log 0.446 = 0.807$ to the image-to-text loss. For comparison, if all six cosines were equal the softmax would be uniform and every row would cost $\ln 6 = 1.792$, the loss of guessing. Averaged over all six rows and all six columns, this early batch has $\mathcal{L} = 0.890$.

The gradient shows where training pushes. For softmax followed by cross-entropy, the derivative of the row's loss with respect to logit $j$ is $p_j - y_j$, the predicted probability minus the target (1 on the diagonal, 0 elsewhere). For the dog row that is $-0.554$ on the dog caption, which gradient descent raises, and $+0.252$ on the cat caption, the largest push down among the negatives, then $+0.093$ on the bike caption and less on the rest. Each negative is pushed down in proportion to the probability it currently takes, so the captions most easily confused with the right one (cat, for a dog photo) get the most correction. Through (1), those logit gradients become gradients on the two embedding vectors and flow back into both encoders.

After training the same row might read $[0.90, 0.55, 0.08, 0.20, 0.10, 0.12]$. The logits are $[12.86, 7.86, 1.14, 2.86, 1.43, 1.71]$, the dog caption gets probability 0.993, and the row costs 0.0068. The cat caption still has the highest cosine among the negatives, 0.55, but a gap of 0.35 in cosine is 5 in logits and a factor of $e^5 \approx 148$ in probability. The whole trained batch has $\mathcal{L} = 0.0020$.

The figure holds both grids. Switch between the four views to see the raw cosines, the row softmax, the column softmax, and the gradient; switch the stage to compare early and trained.

The objective is older than CLIP. It is the multi-class $N$-pair loss from metric learning (Sohn, 2016), popularized for representation learning as InfoNCE (Oord et al., 2018), and applied to medical images paired with radiology reports by ConVIRT (Zhang et al., 2020). CLIP simplifies ConVIRT's recipe: training from scratch, a linear projection instead of a nonlinear one, no sampling of single sentences from the text, and a random square crop of the resized image as the only data augmentation. The authors write that with 400 million pairs, over-fitting is not a major concern.

The learned temperature

Dividing every cosine by $\tau$ before the softmax is the same as multiplying by a scale $1/\tau$. The code stores the log of that scale as one learned parameter, t in the paper's pseudocode and logit_scale in model.py:

Storing the log keeps the scale positive whatever value gradient descent gives $t$. The initial value is $t = \log(1/0.07) \approx 2.66$, a scale of about 14.3, a setting the authors took from Wu et al. (2018). The paper clips $t$ so the scale never exceeds 100, which means $\tau \ge 0.01$, and says the clip was necessary to prevent training instability. A small $\tau$ spreads the logits apart, so the softmax concentrates on the top score. A large $\tau$ squeezes them together, so the softmax approaches uniform.

A scale is needed at all because of (1). Cosines are bounded, so with scale 1 the logits of a row span at most 2 units. Suppose the correct caption sits at cosine 1 and all 32,767 other captions of a full batch sit at cosine 0. With scale 1, the correct caption's probability is $e^{1}/(e^{1} + 32{,}767\,e^{0}) \approx 8.3 \times 10^{-5}$ and the row costs 9.40, barely below the chance value $\ln 32{,}768 = 10.40$. With scale 14.3 the same geometry gives probability 0.980 and a loss of 0.020.

Figure 3 shows one image's softmax over five captions as you drag $\tau$. Switch the batch to 32,768 captions to add the 32,763 other captions of a full batch, each at an illustrative cosine of 0.05, as one lumped bar. At the initial $\tau = 0.07$ the dog caption gets 0.663 among five captions but only 0.00125 among 32,768; it needs $\tau$ near the floor of 0.01 to take 0.998.

Figure 4 trains $t$ along with the embeddings on a toy: six pairs, each embedding a free angle on a circle instead of an encoder output, trained with Adam on (2). Press Play and compare the three temperature settings. With $\tau = 1$ fixed, the pairs line up and spread evenly around the circle, yet the loss stops at 1.03; even perfect alignment cannot push a softmax far from uniform when the logits span only 2 units. With the scale fixed at 14.3 the loss falls to about 0.003. With $t$ learned, the scale first drops, because a sharp softmax over misaligned pairs puts high probability on wrong captions and the gradient lowers it; once the pairs line up, raising the scale lowers the loss further with no limit, so the scale climbs until the clamp at 100 stops it, and the loss stops falling at the same step.

One training step at full size

Now the real sizes, for a ViT-B/32 CLIP (the base-size Vision Transformer with 32 x 32-pixel patches, so a 224 x 224 image becomes a 7 x 7 grid of 49 patches) with $d_e = 512$ and the paper's batch of $N = 32{,}768$ pairs:

1. Encode. The ViT maps 32,768 images to features, and the text Transformer maps 32,768 captions of up to 76 tokens to the activation at each caption's [EOS].
2. Project and normalize. Two linear maps give $\mathbf{I}, \mathbf{T} \in \mathbb{R}^{32768 \times 512}$, and every one of the 65,536 rows is scaled to unit length.
3. Score. $S = \hat{\mathbf{I}}\hat{\mathbf{T}}^{\top}$ has $32{,}768^2 \approx 1.07 \times 10^{9}$ cells: 32,768 correct pairs on the diagonal and 1,073,709,056 wrong ones off it. Multiply by $e^{t}$.
4. Loss. The labels are $[0, 1, \dots, 32767]$. Cross-entropy along the rows, cross-entropy down the columns, average. At initialization, when every cosine is close to 0, the loss is close to $\ln 32{,}768 = 10.40$.
5. Backward. The gradient reaches both encoders, both projections and $t$.

In the paper's notation the forward pass and loss are these lines:

# CLIP core, from the paper's Figure 3 (numpy-like)
I_f = image_encoder(I)            # [n, d_i]   images in a batch
T_f = text_encoder(T)             # [n, d_t]   their captions

# project into the shared space, then L2-normalize to unit length
I_e = l2_normalize(I_f @ W_i, axis=1)   # [n, d_e]
T_e = l2_normalize(T_f @ W_t, axis=1)   # [n, d_e]

# every image against every caption: an n x n grid of scaled cosines
logits = (I_e @ T_e.T) * np.exp(t)      # t = log(1/tau), learned

# the right answer for row i (and for column i) is i: the diagonal
labels = np.arange(n)
loss_i = cross_entropy(logits, labels, axis=0)   # caption -> image
loss_t = cross_entropy(logits, labels, axis=1)   # image -> caption
loss   = (loss_i + loss_t) / 2

In loss_i the softmax runs over axis=0, down each column, so it is the caption-to-image term; loss_t runs along rows and is image-to-caption. The grid alone is large. In 32-bit floats it takes about 4.3 GB, so no single GPU builds it: the paper shards the similarity computation, with each GPU computing only the pairwise similarities its local batch of embeddings needs. If the batch were split evenly over 256 GPUs, a GPU would hold 128 images and 128 captions and compute their 128 rows and 128 columns of the grid against the full batch.

The authors trained eight CLIP models: five ResNets (RN50, RN101, and RN50x4, RN50x16, RN50x64, scaled EfficientNet-style to about 4, 16 and 64 times the compute of RN50) and three ViTs (ViT-B/32, ViT-B/16, ViT-L/14). All ran 32 epochs over the 400 million pairs, about 12.8 billion pairs seen, which at 32,768 per batch is 390,625 steps. Training used Adam with decoupled weight decay on all weights that are not gains or biases, a cosine learning-rate schedule, mixed precision, gradient checkpointing, half-precision Adam statistics, and half-precision stochastically rounded text-encoder weights. RN50x64 took 18 days on 592 V100 GPUs; ViT-L/14 took 12 days on 256 V100s, then one more epoch at 336-pixel resolution produced ViT-L/14@336px, the best model and the one the paper means by "CLIP" unless it says otherwise.

Zero-shot classification: the text encoder writes the classifier

A trained CLIP scores how well any image matches any text. To classify ImageNet with it, write each of the 1,000 class names into a sentence, "A photo of a {label}.", and run the 1,000 sentences through the frozen text encoder. The 1,000 normalized outputs form a matrix $W \in \mathbb{R}^{1000 \times d_e}$. Embed the test image once to get $\hat{\mathbf{x}}$; its cosines against the rows of $W$, scaled and passed through a softmax, give the class probabilities

and the prediction is the largest. No gradient step and no labeled image is involved. The paper describes (4) as a multinomial logistic regression classifier with L2-normalized inputs, L2-normalized weights, no bias, and temperature scaling. In that reading the image encoder is the backbone and the text encoder is a hypernetwork, a network whose output is another network's weights: it writes one weight vector per class from the class's description. A new task costs one sentence per class, embedded once and cached for every later prediction.

The same reading describes pre-training. Each training step is a classification task with 32,768 classes, each defined by one caption, and one example per class, the matched image; a zero-shot task differs only in that you write the class captions.

# zero-shot ImageNet: no training, no labeled images
classes = ["tabby cat", "golden retriever", ...]     # 1,000 names
prompts = [f"A photo of a {c}." for c in classes]
W = l2_normalize(text_encoder(prompts), axis=1)       # [1000, d_e]

x = l2_normalize(image_encoder(image))                # [d_e]
logits = 100 * (x @ W.T)                              # cosine to each class
pred = classes[logits.argmax()]                       # highest cosine wins

The 100 in the code is the multiplier the official prompt-engineering notebook uses; any positive multiplier gives the same argmax and changes only the probabilities. On ImageNet the best model reaches 76.2% top-1 and 95% top-5 (the true class is among the five highest scores), which matches Inception-V4's top-5. The previous zero-shot result on ImageNet, Visual N-Grams (Li et al., 2017), was 11.5%. The authors note the comparison is loose: CLIP used a 10x larger dataset and a vision model with about 100x more compute per prediction. A CLIP ResNet-50 trained on the YFCC100M data that Visual N-Grams used matched its reported ImageNet number within one V100 GPU-day.

Across 27 datasets, zero-shot CLIP beats a fully supervised logistic-regression classifier fitted on ResNet-50 features on 16, ImageNet included. The spread is wide: it wins by 28.9 points on Stanford Cars and 22.5 on Food101, and loses by 37.1 on EuroSAT satellite images and 34.0 on KITTI distance estimation. Against few-shot learning on its own features, zero-shot CLIP matches the average of a 4-shot logistic regression, one trained on 4 labeled examples per class, so a class name written as text is worth about four labeled images per class on average.

Prompts and prompt ensembles

The class names that datasets ship with were not written for a text encoder. Most WIT captions are full sentences, so a bare word like dog is an unusual input. Wrapping every label in "A photo of a {label}." closes part of that gap and raises ImageNet accuracy by 1.3%.

Wording also resolves ambiguity. ImageNet has both construction cranes and cranes that fly. In the Oxford-IIIT Pet dataset, "boxer" means a dog breed, but to a text encoder reading the word alone it could as well mean an athlete. The authors wrote per-dataset templates: "A photo of a {label}, a type of pet." for Oxford Pets, a type of food for Food101, a type of aircraft for FGVC Aircraft, "a satellite photo of a {label}." for satellite datasets, and quotation marks around the text or number to be read for OCR (optical character recognition) datasets.

The second technique is ensembling. Build one class vector from several templates ("A photo of a big {label}.", "A photo of a small {label}.") by averaging their embeddings, not their predicted probabilities. The average is one vector per class, so once cached, an ensemble of 80 templates costs the same at test time as a single template. The official notebook normalizes each template's embedding, averages, and renormalizes:

# one class vector from many prompts (order as in the official notebook)
texts = [tpl.format(name) for tpl in templates]   # e.g. 80 templates
e = text_encoder(texts)                           # [80, d_e]
e = e / e.norm(dim=-1, keepdim=True)              # each prompt to unit length
w = e.mean(dim=0)                                 # average in embedding space
w = w / w.norm()                                  # renormalize: one vector

On ImageNet, the 80-template ensemble adds 3.5% over the single default prompt, and the two techniques together add almost 5% with no change to the trained weights. Averaged over 36 datasets they add almost 5 points, about the gain the paper's Figure 4 shows from using 4 times more compute with plain class names. The figure below sketches the mechanism in 2D with made-up angles: a boxer photo, three breed classes, and the three ways of writing the class text.

Robustness to distribution shift

ImageNet models are usually tested on the ImageNet validation set, which is drawn from the same distribution as its training set. Several datasets ask what happens when the images change and the classes stay: ImageNetV2 (a re-collection following the original process), ImageNet Sketch (drawings), ImageNet-R (renditions such as paintings, cartoons and toys), ObjectNet (objects photographed at odd angles and on unusual backgrounds), ImageNet-A (natural images that ResNet-50 classifiers get wrong), and the video datasets Youtube-BB and ImageNet-Vid. The paper reports that a ResNet-101 makes 5 times as many mistakes on these seven shifted datasets as on the ImageNet validation set.

The paper measures robustness following Taori et al. (2020). Across many ImageNet-trained models, accuracy under distribution shift rises predictably with ImageNet accuracy (linearly, once both accuracies are logit-transformed), along a trend well below the diagonal $y = x$ where a model would lose nothing. Effective robustness is how far a model sits above that trend: extra shifted-dataset accuracy beyond what its ImageNet accuracy predicts. Zero-shot CLIP models sit far above it and close the gap between ImageNet accuracy and shifted accuracy by up to 75%.

The cleanest comparison fixes ImageNet accuracy. A ResNet-101 and zero-shot CLIP both score 76.2% on ImageNet, and the figure shows them on each shifted dataset. The differences run from 5.8 points on ImageNetV2, which was built to match ImageNet, through 35.0 on Sketch, 39.7 on ObjectNet and 51.2 on ImageNet-R, to 74.4 on ImageNet-A. The ImageNet-A number exaggerates the gap: that dataset was built by deleting images that ResNet-50 classifiers predict correctly, so a ResNet-101 fails on it partly by construction.

The authors test whether the robustness comes from zero-shot CLIP never having trained on the ImageNet distribution. Fitting an L2-regularized logistic-regression classifier on CLIP features using the ImageNet training set raises ImageNet accuracy by 9.2 points, to 85.4%, which ties the 2018 state of the art. Average accuracy on the shifted datasets slightly decreases. ImageNetV2, the dataset built to follow ImageNet's own process, is the only one that improves significantly, which the paper reads as the gain from supervised adaptation being concentrated around the ImageNet distribution. ImageNet-R drops by 4.7 points, ObjectNet by 3.8, ImageNet Sketch by 2.8 and ImageNet-A by 1.9.

Limitations and what came after

The paper's own list of limitations, with its numbers:

• Zero-shot CLIP is on average competitive with a linear classifier on ResNet-50 features, which is well below the state of the art on most datasets. The authors estimate that about a 1000x increase in compute would be needed for zero-shot CLIP to reach overall state-of-the-art performance, which they call infeasible on current hardware.
• Compared with task-specific models, it is weak at fine-grained classification (models of cars, species of flowers, variants of aircraft) and at abstract, systematic tasks such as counting the objects in an image. The Stanford Cars win above is against the linear classifier on ResNet-50 features, not against specialized models.
• On tasks unlikely to appear in its pre-training data, such as classifying the distance to the nearest car in a photo, its performance can be near random.
• It generalizes poorly to images unlike its training data. It reaches only 88% on handwritten MNIST digits, where logistic regression on raw pixels does better; a nearest-neighbor search of the pre-training data finds almost no images that resemble MNIST.
• It can only choose among the texts it is given. It does not generate captions.

Because WIT was collected from the web, some evaluation images may appear in it. The authors ran a duplicate detector over 35 evaluation datasets and found a median overlap of 2.2% and an average of 3.2%. Overall accuracy rarely shifted by more than 0.1%; the largest estimated gain was 0.6%, on Birdsnap, which has 12.1% overlap.

Later systems, outside the scope of the CLIP paper, reuse its encoders as components. DALL-E 2 (Ramesh et al., 2022) generates a CLIP image embedding from the text with a learned prior and decodes it into an image. Stable Diffusion v1 conditions its image generator on a frozen CLIP ViT-L/14 text encoder; the latent diffusion paper it builds on trained its own text encoder from scratch. LLaVA feeds features from a frozen CLIP ViT-L/14 image encoder into a language model, which gives CLIP's image representation a text decoder.