Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov · 2017

PPO. A clipped-surrogate policy-gradient method that balances stability and simplicity — the default RL algorithm behind RLHF and most modern agents.

What you'll get

Outline: a plain-English breakdown of the paper's core idea, prerequisites, and the concepts you'll need to implement it.
Exercises: five to ten hands-on tasks, each with a concept card, a prompt, a starter code stub, and a collapsible reference solution.
Runnable notebook: a single .ipynb you can download and open in Jupyter or VS Code to work through every exercise.
Extensions: suggested follow-up experiments so you don't stop at a faithful reimplementation.

We're building visual explainers for every paper in the library — so you can understand the core ideas before you write a single line of code.

Architecture diagrams

Layered, annotated diagrams of the model or system — every component mapped to the paper section that defines it.

Attention heatmaps

Step through attention patterns token-by-token and see which parts of the input the model focuses on.

Training curves

Live-rendered loss and metric plots from reference runs — so you know what convergence should look like.

Concept maps

Prerequisite graphs that show exactly which ideas feed into the paper — and where they appear in the workbook exercises.