The papers worth understanding

Train a diffusion model block-by-block, not end-to-end.

Self-play and search teach one network, from the rules alone, to master chess, shogi, and Go.

Exact attention, with far fewer trips to slow GPU memory.

Plan ahead in a game without ever being told its rules, by learning a model that predicts only what planning needs.

Fit a small network to a scene’s photos, and render it from any new camera angle.

Q-learning plus a convnet learns seven games from raw pixels.

Store a scene as millions of tiny 3D Gaussians and a GPU renders new views of it in real time, no neural network in the loop.

Re-center and re-scale each layer’s inputs, and deep networks train far faster.

Steer a diffusion model by amplifying what the label adds, with no separate classifier.

Run the diffusion in a compressed latent, not on pixels.

Make every NLP task the same task: text in, text out.

Train on 680,000 hours of messy web audio and transcribe anything, zero-shot.

The Transformer, from scratch.

Skip the hand-written reward: teach an agent by comparing short clips of its behavior. The root of RLHF.

Average each node with its neighbors, and a handful of labels classify the whole graph.

A big model ranks the wrong answers too; that ranking teaches a small one to generalize.

Train a small model far too long, so it is cheap to run.

Loss falls as a power law in model size, data, and compute.

Noising data is a smooth diffusion you can run backward to make new data.

One promptable model that cuts out any object, zero-shot.

Hold everything fixed but depth: a uniform stack of tiny 3×3 filters, 16–19 layers deep, that reset how convnets are built.

Reuse a trained DDPM but sample 10–50× faster, deterministically, by skipping most of the steps.

Generate robot actions by denoising: a diffusion model writes the next move and handles multimodal demonstrations cleanly.

Connect a frozen vision model to a frozen language model with a trained bridge, and it learns new tasks from a few examples.

Align a model with human feedback, and 1.3B beats 175B.

Normalize each example across its own features, not across the batch, so it needs no batch and steadies RNNs.

A model too big for one GPU trains anyway, by splitting each layer’s matrix multiplies across many.

Add the input back, and 152-layer networks finally train.

One LSTM reads a sentence into one fixed vector; another decodes the translation from it. Reverse the source and it trains.

A vision-language-action robot model that denoises a 50-step action chunk by flow matching.

For SGD, weight decay equals L2 regularization. For Adam it does not, and decoupling them lets Adam generalize like SGD.

The attention mechanism, born in a translator: let the decoder soft-search the source instead of one fixed vector.

Hide words, read both directions, fine-tune for anything.

The same diffusion model, built four ways: VAE, DDPM, score matching, and SDEs.

Lay out the KV cache like operating-system pages: 96% of memory holds real data, throughput jumps 2-4×.

Bound how much the policy changes each step, measured in KL, and the update can be large without collapsing.

The U-shaped net that segments cells from a few dozen images.

Data parallelism copies the whole optimizer onto every GPU; partition those copies instead and a trillion-parameter model fits.

Two networks play a forgery game until the fakes pass.

Learn the gradient of the data density at many noise levels, then walk noise into images with annealed Langevin dynamics.

Let an ODE solver be the network, and train it with constant memory.

Route each token to one expert, so parameters scale to a trillion while the compute spent per token stays flat.

Cut an image into patches and a plain Transformer does the rest.

The adaptive optimizer that trains almost everything.

A 7B language model that writes the robot’s next move, with open weights and a cheap fine-tune.

Clip the update so you can reuse each batch without blowing up.

Encode token position by rotating queries and keys, so attention reads out relative distance.

A few outlier channels wreck 8-bit quantization; scale the problem into the weights, which quantize cleanly.

A 7B model that runs like a giant, by fixing attention.

Encode a robot action as a short string of integers, and a vision-language model can output it.

Predict masked video patches in a learned representation space, then plan robot actions in that same space.

Make a large model reason by showing it the steps.

Why generative models still work even without being told the noise level.

A field-free voxel that reads where a surface crosses, so open, hollow, and tangled 3D shapes all fit.

Train a 1000-layer net in 7GB: recompute activations instead of storing them.

Predict the embeddings of masked image blocks, not their pixels.

Share one set of keys and values across attention heads. The KV cache shrinks H-fold and decoding gets much faster.

Give a language model an open book to answer from.

Fine-tune giant models by training almost nothing.

Distill a diffusion model to 1-4 steps and keep its diversity, by mixing consistency and score distillation.

Cut transformer training memory 5× with sequence parallelism and selective recompute.

Learn video by predicting the features of masked spatio-temporal tubes, no pixels and no labels.

Probe any point in space and time to rebuild a video in moving 3D.

Restrict A to a scalar and the SSM becomes a matrix you can multiply on a GPU. 2-8× faster than Mamba.

Distill a black-box LLM like GPT-4 by aligning an open proxy to it, then learning from the proxy’s full distribution.

Vector search that recomputes embeddings instead of storing 173 GB of them.

Compress any vector to a few bits per number, online, near the information-theoretic floor.

Alignment without a reward model.

An agent that trains inside its own dream.

Sequence modeling without attention.

An autoencoder you can sample from. (VAE)

How noise becomes images.

Why scale changed everything. (GPT-3)

The compute-optimal scaling law.

Teaching images and text to share a space.

A simpler road to generative models.