Multimodal AI papers explained
Multimodal models learn across different kinds of input, such as images, text, and audio. The central design choice is how to connect those representations: align them in a shared space, bridge pretrained models, or train a sequence model to produce text from another modality.
The four papers here take four concrete approaches: align images and text in a shared space, connect a pretrained language model to visual inputs, instruction-tune a model on image conversations, and transcribe speech. Read the three below in order to move from matching images with text, to answering with visual context, to transcribing and translating speech.
Start here
All Multimodal explainers
5 papers · newest explainers first
MultimodalLive
BLIP-2
A small trained module turns an image into 32 vectors that a frozen language model reads as its prompt.
MultimodalLive
LLaVA (Visual Instruction Tuning)
Map CLIP’s image patches into a language model’s word space with one linear layer, then train on 158K conversations that text-only GPT-4 wrote from captions and boxes.
MultimodalLive
Flamingo
Connect a frozen vision model to a frozen language model with a trained bridge, and it learns new tasks from a few examples.
MultimodalLive
Whisper
One plain Transformer, trained on 680,000 hours of noisy web transcripts, transcribes new audio without fine-tuning.
MultimodalLive
CLIP
Train on 400 million web images and their captions, and any list of class names becomes an image classifier.
