AI Roadmaps
← All roadmaps
IntermediateRoadmap 11 · 11 steps · 3h 28m of video

Reinforcement Learning

Agents that learn by trial and error: from Bellman equations and Q-learning to PPO, AlphaGo, and the RL behind reasoning models.

Before this: Deep Learning. Fine-Tuning & Reasoning Models pairs well at the end.

  1. 18 min
    Step 1

    Reinforcement learning, the essentials

    Rewards, policies, and the learning loop in plain language.

    Reinforcement Learning: Essential Concepts

    StatQuest with Josh Starmer

  2. 18 min
    Step 2

    The RL framework, by the book

    States, actions, returns, and how the pieces fit together.

    Reinforcement Learning, by the Book

    Mutual Information

    openai/spinningup★ 12k
  3. 22 min
    Step 3

    Bellman equations and dynamic programming

    The recursion every value-based method builds on.

    Bellman Equations, Dynamic Programming, Generalized Policy Iteration | Reinforcement Learning Part 2

    Mutual Information

  4. 12 min
    Step 4

    Q-learning

    Learn action values from experience without a model.

    Q-learning - Explained!

    CodeEmporium

  5. 11 min
    Step 5

    Deep Q-learning

    Replace the Q-table with a neural network.

    Deep Q-Learning - Combining Neural Networks and Reinforcement Learning

    deeplizard

    huggingface/deep-rl-class★ 5.1k
  6. 29 min
    Step 6

    Policy gradients

    Optimize the policy directly instead of values.

    Policy Gradient Methods | Reinforcement Learning Part 6

    Mutual Information

  7. 31 min
    Step 7

    PPO, step by step

    The stable default policy-gradient algorithm, explained simply.

    Simply Explaining Proximal Policy Optimization (PPO) | Deep Reinforcement Learning

    Johnny Code

    DLR-RM/stable-baselines3★ 14k
  8. 13 min
    Step 8

    Get started with Stable Baselines3

    Train your first agent on a Gymnasium environment in Python.

    Getting Started with Stable Baselines3 - Lowest Learning Curve Python Reinforcement Learning Library

    Johnny Code

    Farama-Foundation/Gymnasium★ 12.8k
  9. 8 min
    Step 9

    AlphaGo

    How RL plus tree search beat a Go champion.

    Reinforcement Learning: AlphaGo

    Graphics in 5 Minutes

  10. 22 min
    Step 10

    PPO for language models

    How PPO adapts to training LLMs with feedback.

    Proximal Policy Optimization (PPO) for LLMs Explained Intuitively

    Julia Turc

    huggingface/trl★ 19.8k
  11. 24 min
    Step 11

    GRPO and reasoning RL

    The simpler RL recipe behind DeepSeek-R1-style reasoning.

    GRPO: The RL Trick Behind DeepSeek-R1's Reasoning

    Jia-Bin Huang

Build it

Watching isn't learning. Pick one and put it on GitHub.

Where to go next