Multimodal AI papers explained

Multimodal models learn across different kinds of input, such as images, text, and audio. The central design choice is how to connect those representations: align them in a shared space, bridge pretrained models, or train a sequence model to produce text from another modality.

These three papers offer three concrete approaches. Read them in this order to move from matching images with text, to answering with visual context, to transcribing and translating speech.

Start here

  1. CLIP

    Learn how contrastive training aligns images and captions in a shared representation.

  2. Flamingo

    Explore the bridge that lets a language model condition on visual inputs.

  3. Whisper

    Follow audio features through an encoder-decoder model trained for speech tasks.

All Multimodal explainers

3 papers · newest explainers first

MultimodalLive
Flamingo
Connect a frozen vision model to a frozen language model with a trained bridge, and it learns new tasks from a few examples.
Read the explainer
MultimodalLive
Whisper
Train on 680,000 hours of messy web audio and transcribe anything, zero-shot.
Read the explainer
MultimodalLive
CLIP
Teaching images and text to share a space.
Read the explainer