Transformer and neural network architecture papers explained
An architecture determines how information moves through a network. Attention connects distant tokens directly; recurrent and state space models compress the past into a state. Comparing these mechanisms reveals the trade-offs between expressive power, memory, and computation.
Start with the Transformer, examine how attention can run more efficiently without changing its result, then compare an alternative built around a selective state. Other explainers cover positional information, sparse experts, graphs, and earlier sequence models.
Start here
Attention Is All You Need
Build self-attention and the Transformer from embeddings to the encoder and decoder.
FlashAttention
See how organizing memory access makes exact attention more efficient.
Mamba
Compare attention with a state space model that selectively remembers its input.
All Architecture explainers
13 papers · newest explainers first
