Reinforcement learning and alignment papers explained
In reinforcement learning, an agent learns from the consequences of its actions. The hard part is assigning credit across time while the policy itself changes the data it sees. Alignment methods add another question: how should human preferences become a learning signal?
Begin with value learning, move to direct policy optimization, then see how preferences guide a language model. The wider collection explores planning, exploration, world models, and learning actions for robots.
Start here
Playing Atari with Deep RL (DQN)
Learn how rewards and next-state values train an agent from game pixels.
Proximal Policy Optimization
Understand policy updates and why limiting their size helps training.
InstructGPT
Follow demonstrations and human comparisons into reward modeling and RLHF.
All RL & alignment explainers
17 papers · newest explainers first
RL & alignmentLive
Decision Transformer
Tell a GPT the score you want and it picks the actions that earn it, with no value function and no Bellman backup.
RL & alignmentLive
Curiosity-Critic
Reward an agent only for the prediction error its world model can still reduce, and curiosity stops chasing random noise.
RL & alignmentLive
A3C (Asynchronous Advantage Actor-Critic)
Run many copies of an agent in parallel, and deep RL trains stably on a CPU, no replay buffer needed.
RL & alignmentLive
Soft Actor-Critic
Reward an agent for succeeding and for staying random, and off-policy control gets both sample-efficient and stable.
RL & alignmentLive
Trust Region Policy Optimization
Bound how much the policy changes each step, measured in KL, and the update can be large without collapsing.
RL & alignmentLive
AlphaZero
Self-play and search teach one network, from the rules alone, to master chess, shogi, and Go.
RL & alignmentLive
Deep RL from Human Preferences
Skip the hand-written reward: teach an agent by comparing short clips of its behavior. The root of RLHF.
RL & alignmentLive
MuZero
Plan ahead in a game without ever being told its rules, by learning a model that predicts only what planning needs.
RL & alignmentLive
Diffusion Policy
Generate robot actions by denoising: a diffusion model writes the next move and handles multimodal demonstrations cleanly.
RL & alignmentLive
OpenVLA
A 7B language model that writes the robot’s next move, with open weights and a cheap fine-tune.
RL & alignmentLive
RT-2
Encode a robot action as a short string of integers, and a vision-language model can output it.
RL & alignmentLive
π₀ (pi-zero)
A vision-language-action robot model that denoises a 50-step action chunk by flow matching.
RL & alignmentLive
Playing Atari with Deep RL (DQN)
Q-learning plus a convnet learns seven games from raw pixels.
RL & alignmentLive
InstructGPT
Align a model with human feedback, and 1.3B beats 175B.
RL & alignmentLive
Proximal Policy Optimization
Clip the update so you can reuse each batch without blowing up.
RL & alignmentLive
World Models
An agent that trains inside its own dream.
RL & alignmentLive
Direct Preference Optimization
Alignment without a reward model.
