Deep learning training and optimization papers explained
Training a model means choosing both how to update its parameters and how to fit the computation into available hardware. Optimizers, normalization, parameter-efficient adaptation, and distributed systems solve different parts of that problem. These papers show where the improvements come from and what they cost.
Begin with an optimizer, learn how a small trainable update can adapt a frozen model, then add quantization. The rest of the collection covers scaling decisions and techniques for reducing memory use during large training runs.
Start here
All Training explainers
16 papers · newest explainers first
TrainingLive
μP (Tensor Programs V)
Scale each layer’s initialization and learning rate with width, and the best hyperparameters of a small model also work for a large one.
TrainingLive
Lottery Ticket Hypothesis
Prune a trained network, reset the survivors to their original initial values, and the 10% that remain train as fast and as accurately as the whole.
TrainingLive
Dropout
Switch off half the network at random on every training example, and it overfits far less.
TrainingLive
QLoRA
Finetune a 65B model on one 48 GB GPU: store the frozen model in 4 bits and train a small 16-bit adapter through it.
TrainingLive
ZeRO
Data parallelism keeps a full copy of the training state on every GPU. ZeRO gives each GPU one slice and fetches the rest when a step needs it.
TrainingLive
Knowledge Distillation
A big model’s probabilities for the wrong answers teach a small model how to generalize.
TrainingLive
Megatron-LM
A transformer too big for one GPU trains anyway if each layer’s matrix multiplies are split across several GPUs.
TrainingLive
AdamW (Decoupled Weight Decay)
For SGD, weight decay equals L2 regularization. For Adam it does not, so AdamW applies the decay to the weights directly.
TrainingLive
Layer Normalization
Normalize each example across its own features, not across the batch, so it works at batch size 1 and in recurrent networks.
TrainingLive
Batch Normalization
Standardize every layer’s inputs over each mini-batch, and an ImageNet classifier reaches its original accuracy in 14 times fewer training steps.
TrainingLive
Reducing Activation Recomputation
Cut activation memory 5× and recover over 90% of the time lost to recomputation, with sequence parallelism and selective recompute.
TrainingLive
Gradient Checkpointing
Store a few activations and recompute the rest: square-root memory for one extra forward pass.
TrainingLive
Scaling Laws for Neural Language Models
Test loss falls as a power law in model size, data and compute, so small training runs predict large ones.
TrainingLive
Adam
Two running averages give every weight its own learning rate.
TrainingLive
Chinchilla
At a fixed training budget, a smaller model trained on more text reaches a lower loss.
TrainingLive
LoRA
Fine-tune a large model by freezing its weights and training a low-rank update beside them.
