Multimodal AI papers explained
Multimodal models learn across different kinds of input, such as images, text, and audio. The central design choice is how to connect those representations: align them in a shared space, bridge pretrained models, or train a sequence model to produce text from another modality.
The four papers here take four concrete approaches: align images and text in a shared space, connect a pretrained language model to visual inputs, instruction-tune a model on image conversations, and transcribe speech. Read the three below in order to move from matching images with text, to answering with visual context, to transcribing and translating speech.
Start here
All Multimodal explainers
6 papers · newest explainers first
MultimodalLive
BLIP-2
A small trained module turns an image into 32 vectors that a frozen language model reads as its prompt.
MultimodalLive
wav2vec 2.0
Pretrain on 53,000 hours of untranscribed speech by hiding short spans of the audio and training the model to pick each hidden frame's true discrete code out of 100 distractors; ten minutes of transcripts then reach 4.8% WER.
MultimodalLive
LLaVA (Visual Instruction Tuning)
One linear layer lets a language model read an image as 256 tokens, and text-only GPT-4 writes the conversations that teach it to answer.
MultimodalLive
Flamingo
Join a frozen vision model to a frozen language model with a few trained layers, and it learns new image tasks from examples in its prompt.
MultimodalLive
Whisper
One plain Transformer, trained on 680,000 hours of noisy web transcripts, transcribes new audio without fine-tuning.
MultimodalLive
CLIP
Train on 400 million web images and their captions, and any list of class names becomes an image classifier.
