Deep learning training and optimization papers explained
Training a model means choosing both how to update its parameters and how to fit the computation into available hardware. Optimizers, normalization, parameter-efficient adaptation, and distributed systems solve different parts of that problem. These papers show where the improvements come from and what they cost.
Begin with an optimizer, learn how a small trainable update can adapt a frozen model, then add quantization. The rest of the collection covers scaling decisions and techniques for reducing memory use during large training runs.
Start here
All Training explainers
15 papers · newest explainers first
TrainingLive
μP (Tensor Programs V)
Scale each layer’s initialization and learning rate with width, and the best hyperparameters of a small model also work for a large one.
TrainingLive
Lottery Ticket Hypothesis
Prune a trained network, reset the survivors to their original initial values, and the 10% that remain train as fast and as accurately as the whole.
TrainingLive
QLoRA
Finetune a 65B model on one 48 GB GPU: freeze it in 4-bit, train a small adapter through it, and lose no quality.
TrainingLive
ZeRO
Data parallelism keeps a full copy of the training state on every GPU. ZeRO gives each GPU one slice and fetches the rest when a step needs it.
TrainingLive
Knowledge Distillation
A big model’s probabilities for the wrong answers teach a small model how to generalize.
TrainingLive
Megatron-LM
A model too big for one GPU trains anyway, by splitting each layer’s matrix multiplies across many.
TrainingLive
AdamW (Decoupled Weight Decay)
For SGD, weight decay equals L2 regularization. For Adam it does not, and decoupling them lets Adam generalize like SGD.
TrainingLive
Layer Normalization
Normalize each example across its own features, not across the batch, so it needs no batch and steadies RNNs.
TrainingLive
Batch Normalization
Standardize every layer’s inputs over each mini-batch, and an ImageNet classifier reaches its original accuracy in 14 times fewer training steps.
TrainingLive
Reducing Activation Recomputation
Cut activation memory 5× and recover over 90% of the time lost to recomputation, with sequence parallelism and selective recompute.
TrainingLive
Gradient Checkpointing
Store a few activations and recompute the rest: square-root memory for one extra forward pass.
TrainingLive
Scaling Laws for Neural Language Models
Test loss falls as a power law in model size, data and compute, so small training runs predict large ones.
TrainingLive
Adam
Two running averages give every weight its own learning rate.
TrainingLive
Chinchilla
At a fixed training budget, a smaller model trained on more text reaches a lower loss.
TrainingLive
LoRA
Fine-tune a large model by freezing its weights and training a low-rank update beside them.
