Deep learning training and optimization papers explained

Training a model means choosing both how to update its parameters and how to fit the computation into available hardware. Optimizers, normalization, parameter-efficient adaptation, and distributed systems solve different parts of that problem. These papers show where the improvements come from and what they cost.

Begin with an optimizer, learn how a small trainable update can adapt a frozen model, then add quantization. The rest of the collection covers scaling decisions and techniques for reducing memory use during large training runs.

Start here

  1. Adam

    Understand how moving averages of gradients set adaptive parameter updates.

  2. LoRA

    See how low-rank updates make adaptation possible with fewer trainable parameters.

  3. QLoRA

    Combine low-rank adapters with a quantized base model to reduce memory use.

All Training explainers

13 papers · newest explainers first

TrainingLive
QLoRA
Finetune a 65B model on one 48 GB GPU: freeze it in 4-bit, train a small adapter through it, and lose no quality.
Read the explainer
TrainingLive
ZeRO
Data parallelism copies the whole optimizer onto every GPU; partition those copies instead and a trillion-parameter model fits.
Read the explainer
TrainingLive
Knowledge Distillation
A big model ranks the wrong answers too; that ranking teaches a small one to generalize.
Read the explainer
TrainingLive
Megatron-LM
A model too big for one GPU trains anyway, by splitting each layer’s matrix multiplies across many.
Read the explainer
TrainingLive
AdamW (Decoupled Weight Decay)
For SGD, weight decay equals L2 regularization. For Adam it does not, and decoupling them lets Adam generalize like SGD.
Read the explainer
TrainingLive
Layer Normalization
Normalize each example across its own features, not across the batch, so it needs no batch and steadies RNNs.
Read the explainer
TrainingLive
Batch Normalization
Re-center and re-scale each layer’s inputs, and deep networks train far faster.
Read the explainer
TrainingLive
Reducing Activation Recomputation
Cut transformer training memory 5× with sequence parallelism and selective recompute.
Read the explainer
TrainingLive
Gradient Checkpointing
Train a 1000-layer net in 7GB: recompute activations instead of storing them.
Read the explainer
TrainingLive
Scaling Laws for Neural Language Models
Loss falls as a power law in model size, data, and compute.
Read the explainer
TrainingLive
Adam
The adaptive optimizer that trains almost everything.
Read the explainer
TrainingLive
Chinchilla
The compute-optimal scaling law.
Read the explainer
TrainingLive
LoRA
Fine-tune giant models by training almost nothing.
Read the explainer