Reinforcement learning and alignment papers explained
In reinforcement learning, an agent learns from the consequences of its actions. The hard part is assigning credit across time while the policy itself changes the data it sees. Alignment methods add another question: how should human preferences become a learning signal?
Begin with value learning, move to direct policy optimization, then see how preferences guide a language model. The wider collection explores planning, exploration, world models, and learning actions for robots.
Start here
Playing Atari with Deep RL (DQN)
Learn how rewards and next-state values train an agent from game pixels.
Proximal Policy Optimization
Understand policy updates and why limiting their size helps training.
InstructGPT
Follow demonstrations and human comparisons into reward modeling and RLHF.
All RL & alignment explainers
20 papers · newest explainers first
RL & alignmentLive
DreamerV3
An agent learns a model of its environment, practices inside it, and one fixed set of hyperparameters works on over 150 tasks.
RL & alignmentLive
DDPG (Deep Deterministic Policy Gradient)
A network outputs one action per state and is trained along the gradient of a learned Q-function, so deep Q-learning works for continuous actions such as joint torques.
RL & alignmentLive
Constitutional AI
A short list of written principles replaces the human labels that mark harmful answers.
RL & alignmentLive
Decision Transformer
Label each logged step with the reward that followed it, train a GPT on that, then prompt it with the score you want. No value function, no Bellman backup.
RL & alignmentLive
Curiosity-Critic
Reward an agent only for the prediction error its world model can still reduce, and curiosity stops chasing random noise.
RL & alignmentLive
A3C (Asynchronous Advantage Actor-Critic)
Run many copies of an agent in parallel, and deep RL trains stably on a CPU, no replay buffer needed.
RL & alignmentLive
Soft Actor-Critic
Reward an agent for succeeding and for staying random, and off-policy control gets both sample-efficient and stable.
RL & alignmentLive
Trust Region Policy Optimization
Bound how much the policy changes each step, measured in KL, and the update can be large without collapsing.
RL & alignmentLive
AlphaZero
Self-play and search teach one network, from the rules alone, to master chess, shogi, and Go.
RL & alignmentLive
Deep RL from Human Preferences
An agent learns a task from a person picking the better of two short clips of its behavior, with no hand-written reward.
RL & alignmentLive
MuZero
Plan ahead in a game without ever being told its rules, by learning a model that predicts only what planning needs.
RL & alignmentLive
Diffusion Policy
A robot policy that generates a short sequence of future actions by denoising random noise.
RL & alignmentLive
OpenVLA
A 7B vision-language model that outputs a robot arm’s next move as seven tokens.
RL & alignmentLive
RT-2
Encode a robot action as a short string of integers, and a vision-language model can output it.
RL & alignmentLive
π₀ (pi-zero)
A vision-language model that outputs a robot’s next 50 actions by refining random noise.
RL & alignmentLive
Playing Atari with Deep RL (DQN)
Q-learning, a convnet and a replay memory learn seven Atari games from raw pixels and the score.
RL & alignmentLive
InstructGPT
Fine-tuned on human feedback, a 1.3B model's answers are preferred to the 175B GPT-3's.
RL & alignmentLive
Proximal Policy Optimization
Clipping the probability ratio lets a policy train on each batch of experience for several epochs.
RL & alignmentLive
World Models
An agent learns a model of its game, then trains inside that model.
RL & alignmentLive
Direct Preference Optimization
Tune a language model on preference pairs with one classification loss, no reward model and no RL.
