LLMs from Scratch
Understand and build GPT: tokenizers, attention, transformers, GPT-2, Llama, mixture of experts.
Before this: Deep Learning.
-
8 min
Step 1LLMs in 8 minutes
The whole idea before the details.
Large Language Models explained briefly3Blue1Brown
-
1h 00m
Step 2Intro to large language models
What they are, how they're trained, where they're going.
[1hr Talk] Intro to Large Language ModelsAndrej Karpathy
-
27 min
Step 3Transformers, visually
The architecture behind every modern LLM.
Transformers, the tech behind LLMs | Deep Learning Chapter 53Blue1Brown
-
26 min
Step 4Attention, step by step
The one mechanism everything depends on.
Attention in transformers, step-by-step | Deep Learning Chapter 63Blue1Brown
-
23 min
Step 5How LLMs store facts
What the MLP layers are doing between attention blocks.
How might LLMs store facts | Deep Learning Chapter 73Blue1Brown
-
2h 14m
Step 6Build a tokenizer
Many LLM quirks are tokenizer quirks. Build BPE yourself.
Let's build the GPT TokenizerAndrej Karpathy
karpathy/minbpe★ 11.1k -
1h 56m
Step 7Build GPT from scratch
Code a transformer and watch it learn Shakespeare.
Let's build GPT: from scratch, in code, spelled out.Andrej Karpathy
karpathy/nanoGPT★ 65k -
2h 45m
Step 8LLM coding workshop
Build, pretrain, and fine-tune an LLM in one 3-hour session.
Building LLMs from the Ground Up: A 3-hour Coding WorkshopSebastian Raschka
rasbt/LLMs-from-scratch★ 108.7k -
4h 01m
Step 9Reproduce GPT-2
Real training: data, mixed precision, multi-GPU, and evals.
Let's reproduce GPT-2 (124M)Andrej Karpathy
karpathy/llm.c★ 31.5k -
3h 31m
Step 10From base model to ChatGPT
Pretraining, fine-tuning, and RL, end to end.
Deep Dive into LLMs like ChatGPTAndrej Karpathy
karpathy/nanochat★ 60.7k -
11 min
Step 11Rotary position embeddings
How modern models encode position.
Rotary Positional Embeddings: Combining Absolute and RelativeEfficient NLP
-
3h 04m
Step 12Code Llama from scratch
RMSNorm, RoPE, grouped-query attention, and the KV cache in PyTorch.
Coding LLaMA 2 from scratch in PyTorch - KV Cache, Grouped Query Attention, Rotary PE, RMSNormUmar Jamil
naklecha/llama3-from-scratch★ 15.2k -
20 min
Step 13Mixture of experts
How models get huge without getting slow.
A Visual Guide to Mixture of Experts (MoE) in LLMsMaarten Grootendorst
-
18 min
Step 14Multi-head latent attention
DeepSeek's trick for shrinking the KV cache.
How DeepSeek Rewrote the Transformer [MLA]Welch Labs
-
1h 19m
Step 15A university course on building LLMs
Stanford CS336, lecture 1. Continue with the full course if you want depth.
Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 1: Overview and TokenizationStanford Online
stanford-cs336/assignment1-basics★ 3.2k
Build it
Watching isn't learning. Pick one and put it on GitHub.
- Train nanoGPT on text you care about (your notes, song lyrics, a codebase) and sample from it.
- Swap GPT-2's components for RoPE, RMSNorm, and SwiGLU one by one and compare loss on TinyStories.
- Train your own BPE tokenizer and compare its token counts with GPT-2's on English, code, and another language.