AI Roadmaps
← All roadmaps
IntermediateRoadmap 04 · 15 steps · 22h 04m of video

LLMs from Scratch

Understand and build GPT: tokenizers, attention, transformers, GPT-2, Llama, mixture of experts.

Before this: Deep Learning.

  1. 8 min
    Step 1

    LLMs in 8 minutes

    The whole idea before the details.

    Large Language Models explained briefly

    3Blue1Brown

  2. 1h 00m
    Step 2

    Intro to large language models

    What they are, how they're trained, where they're going.

    [1hr Talk] Intro to Large Language Models

    Andrej Karpathy

  3. 27 min
    Step 3

    Transformers, visually

    The architecture behind every modern LLM.

    Transformers, the tech behind LLMs | Deep Learning Chapter 5

    3Blue1Brown

  4. 26 min
    Step 4

    Attention, step by step

    The one mechanism everything depends on.

    Attention in transformers, step-by-step | Deep Learning Chapter 6

    3Blue1Brown

  5. 23 min
    Step 5

    How LLMs store facts

    What the MLP layers are doing between attention blocks.

    How might LLMs store facts | Deep Learning Chapter 7

    3Blue1Brown

  6. 2h 14m
    Step 6

    Build a tokenizer

    Many LLM quirks are tokenizer quirks. Build BPE yourself.

    Let's build the GPT Tokenizer

    Andrej Karpathy

    karpathy/minbpe★ 11.1k
  7. 1h 56m
    Step 7

    Build GPT from scratch

    Code a transformer and watch it learn Shakespeare.

    Let's build GPT: from scratch, in code, spelled out.

    Andrej Karpathy

    karpathy/nanoGPT★ 65k
  8. 2h 45m
    Step 8

    LLM coding workshop

    Build, pretrain, and fine-tune an LLM in one 3-hour session.

    Building LLMs from the Ground Up: A 3-hour Coding Workshop

    Sebastian Raschka

    rasbt/LLMs-from-scratch★ 108.7k
  9. 4h 01m
    Step 9

    Reproduce GPT-2

    Real training: data, mixed precision, multi-GPU, and evals.

    Let's reproduce GPT-2 (124M)

    Andrej Karpathy

    karpathy/llm.c★ 31.5k
  10. 3h 31m
    Step 10

    From base model to ChatGPT

    Pretraining, fine-tuning, and RL, end to end.

    Deep Dive into LLMs like ChatGPT

    Andrej Karpathy

    karpathy/nanochat★ 60.7k
  11. 11 min
    Step 11

    Rotary position embeddings

    How modern models encode position.

    Rotary Positional Embeddings: Combining Absolute and Relative

    Efficient NLP

  12. 3h 04m
    Step 12

    Code Llama from scratch

    RMSNorm, RoPE, grouped-query attention, and the KV cache in PyTorch.

    Coding LLaMA 2 from scratch in PyTorch - KV Cache, Grouped Query Attention, Rotary PE, RMSNorm

    Umar Jamil

    naklecha/llama3-from-scratch★ 15.2k
  13. 20 min
    Step 13

    Mixture of experts

    How models get huge without getting slow.

    A Visual Guide to Mixture of Experts (MoE) in LLMs

    Maarten Grootendorst

  14. 18 min
    Step 14

    Multi-head latent attention

    DeepSeek's trick for shrinking the KV cache.

    How DeepSeek Rewrote the Transformer [MLA]

    Welch Labs

  15. 1h 19m
    Step 15

    A university course on building LLMs

    Stanford CS336, lecture 1. Continue with the full course if you want depth.

    Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 1: Overview and Tokenization

    Stanford Online

    stanford-cs336/assignment1-basics★ 3.2k

Build it

Watching isn't learning. Pick one and put it on GitHub.

Where to go next