Learn · landmark papers, lesson by lesson
Understand research papersby building them.
Interactive walkthroughs of the papers behind modern AI. Animated visuals, live Python you can edit and run, and explanations written for real understanding.
Animated visuals
Every core idea drawn out step by step.
Live Python
Edit and run real code right in your browser.
Exercises
Hints and worked solutions to check yourself.
Attention Is All You Need
Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin
The landmark 2017 paper that introduced the Transformer architecture, replacing recurrence and convolutions with pure attention mechanisms - the foundation of GPT, BERT, and virtually all modern large language models.
- Lessons
- 14
- Access
- All lessons free. Sign in from lesson 6.
BERT: Pre-training of Deep Bidirectional Transformers
Devlin, Chang, Lee, Toutanova
Google's breakthrough pre-training approach that learns deeply bidirectional representations by jointly conditioning on both left and right context in all layers.
- Lessons
- 13
- Access
- All lessons free. Sign in from lesson 6.
Deep Residual Learning for Image Recognition
He, Zhang, Ren, Sun
The paper that introduced residual connections (skip connections), enabling training of very deep neural networks (100+ layers) and winning ImageNet 2015 by a large margin.
- Lessons
- 13
- Access
- Lessons 1-5 open, then membership.
Generative Adversarial Nets
Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville, Bengio
The paper that introduced GANs - a framework where two networks compete: a generator creating fake data and a discriminator trying to detect fakes. The foundation of modern image generation.
- Lessons
- 14
- Access
- All lessons free. Sign in from lesson 6.
Coming soon to the reading room.
29 in production- Language2022
Training Language Models to Follow Instructions with Human Feedback
Ouyang, Wu, Jiang, Almeida, et al.
InstructGPT and RLHF: supervised fine-tuning, a learned reward model, and PPO - the alignment pipeline behind ChatGPT.
RLHF · Alignment Coming soon - Language2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, Wang, Schuurmans, Bosma, et al.
Showing worked steps in the prompt unlocks multi-step reasoning in large models - a simple idea that reshaped how we use LLMs.
LLMs · Reasoning Coming soon - Generative2022
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, Blattmann, Lorenz, Esser, Ommer
Run diffusion in a compressed latent space and condition it on text - the architecture behind Stable Diffusion.
Diffusion · Text-to-image Coming soon - Language2021
LoRA: Low-Rank Adaptation of Large Language Models
Hu, Shen, Wallis, Allen-Zhu, et al.
Freeze the model and train tiny low-rank update matrices instead - fine-tuning huge models on a single GPU with almost no quality loss.
Fine-tuning · Efficiency Coming soon - Multimodal2021
Learning Transferable Visual Models From Natural Language Supervision
Radford, Kim, Hallacy, Ramesh, et al.
CLIP: train images and captions into one shared space with a contrastive loss, giving zero-shot image classification and the glue for text-to-image.
Contrastive · Vision-language Coming soon - Science & graphs2021
Highly Accurate Protein Structure Prediction with AlphaFold
Jumper, Evans, Pritzel, Green, et al.
AlphaFold 2 predicts 3D protein structure from sequence at near-experimental accuracy, solving a 50-year-old grand challenge in biology.
Biology · Structure Coming soon - Language2020
Language Models are Few-Shot Learners
Brown, Mann, Ryder, Subbiah, et al.
GPT-3 and the discovery of in-context learning: a 175B-parameter model that solves new tasks from a handful of examples in the prompt.
LLMs · Scaling Coming soon - Language2020
Scaling Laws for Neural Language Models
Kaplan, McCandlish, Henighan, Brown, et al.
Loss falls as a smooth power law in model size, data and compute - the paper that turned model training into a predictable engineering budget.
LLMs · Scaling Coming soon - Language2020
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, Perez, Piktus, Petroni, et al.
RAG: pair a generator with a document retriever so answers are grounded in real sources - the pattern behind most LLM products today.
RAG · Retrieval Coming soon - Vision2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, Beyer, Kolesnikov, Weissenborn, et al.
ViT: slice an image into patches, treat them like tokens, and a plain Transformer matches the best CNNs.
Transformers · Vision Coming soon - Vision2020
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Mildenhall, Srinivasan, Tancik, Barron, et al.
A small MLP that stores a whole 3D scene, rendered with classic volume rendering to produce photoreal new viewpoints.
3D · Rendering Coming soon - Generative2020
Denoising Diffusion Probabilistic Models
Ho, Jain, Abbeel
Learn to reverse a gradual noising process and you can generate images from pure noise - the paper that made diffusion models work.
Diffusion Coming soon - Language2018
Improving Language Understanding by Generative Pre-Training
Radford, Narasimhan, Salimans, Sutskever
The original GPT: pre-train a Transformer decoder on raw text, then fine-tune it. The recipe every modern language model grew from.
LLMs · Pre-training Coming soon - Reinforcement learning2017
Proximal Policy Optimization Algorithms
Schulman, Wolski, Dhariwal, Radford, Klimov
A clipped objective that keeps each policy update small and safe - simple, stable, and the default RL algorithm, including for RLHF.
Policy gradients Coming soon - Vision2016
You Only Look Once: Unified, Real-Time Object Detection
Redmon, Divvala, Girshick, Farhadi
Detection as a single regression pass over the image - fast enough for real-time video and a template for a decade of detectors.
Object detection Coming soon - Reinforcement learning2016
Mastering the Game of Go with Deep Neural Networks and Tree Search
Silver, Huang, Maddison, Guez, et al.
Policy and value networks guiding Monte Carlo tree search - the system that beat a world champion at Go a decade early.
Deep RL · Search Coming soon - Science & graphs2016
Semi-Supervised Classification with Graph Convolutional Networks
Kipf, Welling
The GCN layer: each node averages its neighbours' features, bringing deep learning to graphs, molecules and social networks.
Graph neural networks Coming soon - Vision2015
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, Fischer, Brox
The encoder-decoder with skip connections that segments images pixel by pixel from very little data - later the backbone of diffusion models.
Segmentation · Medical Coming soon - Foundations2015
Batch Normalization: Accelerating Deep Network Training
Ioffe, Szegedy
Normalize each layer's activations per mini-batch to train faster with higher learning rates - standard in almost every deep CNN.
Training · Normalization Coming soon - Language2014
Sequence to Sequence Learning with Neural Networks
Sutskever, Vinyals, Le
The encoder-decoder LSTM that mapped whole sentences to whole sentences, launching neural machine translation and the seq2seq pattern still used everywhere.
RNNs · Translation Coming soon - Language2014
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, Cho, Bengio
The first attention mechanism: let the decoder look back over every source word instead of squeezing a sentence into one fixed vector.
Attention · Translation Coming soon - Foundations2014
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Srivastava, Hinton, Krizhevsky, Sutskever, Salakhutdinov
Randomly silence neurons during training so the network cannot rely on any one of them - a one-line fix for overfitting.
Regularization Coming soon - Foundations2014
Adam: A Method for Stochastic Optimization
Kingma, Ba
Adaptive per-parameter learning rates from running estimates of the gradient's mean and variance - the default optimizer for deep learning.
Optimization Coming soon - Language2013
Efficient Estimation of Word Representations in Vector Space
Mikolov, Chen, Corrado, Dean
word2vec - skip-gram and CBOW - showing that simple models trained on huge text produce word vectors where king - man + woman lands near queen.
Embeddings · NLP Coming soon - Generative2013
Auto-Encoding Variational Bayes
Kingma, Welling
The variational autoencoder and the reparameterization trick - learning a smooth latent space you can sample new data from.
VAEs · Probabilistic Coming soon - Reinforcement learning2013
Playing Atari with Deep Reinforcement Learning
Mnih, Kavukcuoglu, Silver, Graves, et al.
DQN: a single network learns to play Atari games from raw pixels, using experience replay to make Q-learning stable.
Deep RL · Q-learning Coming soon - Vision2012
ImageNet Classification with Deep Convolutional Neural Networks
Krizhevsky, Sutskever, Hinton
AlexNet: GPUs, ReLUs and dropout crushed ImageNet 2012 and kicked off the deep learning era.
Computer Vision · CNNs Coming soon - Vision1998
Gradient-Based Learning Applied to Document Recognition
LeCun, Bottou, Bengio, Haffner
LeNet-5 and the convolutional network as we know it - convolutions, pooling and backprop, reading handwritten digits on real bank cheques.
Computer Vision · CNNs Coming soon - Foundations1997
Long Short-Term Memory
Hochreiter, Schmidhuber
Gated memory cells that fix the vanishing gradient, letting recurrent networks remember over long sequences - the workhorse of sequence modeling for 20 years.
RNNs · Sequence models Coming soon
Start free. Go deep when you're ready.
Papers marked Free open every lesson; you just sign in from lesson 6. On every course, the first Implement task is free and the rest come with membership.
- 01
Lessons 1-5
Anyone
Start reading right away. No account needed.
- 02
Lesson 6 & Implement task 1
Free account
Sign in to keep reading and try your first graded coding task.
- 03
Lesson 7+ & tasks 2+
Researcher membership
The full course plus every graded implementation task.