Track
From Scratch
Goal: Build tiny nets, tokenizers, and a GPT by hand, in code, instead of only watching the architecture explained.
Prereqs: Deep Learning and Python for ML.
Status: done
Work through the steps in order. Bold links open YouTube.
| Step | Concept | YouTube | Read |
|---|---|---|---|
| 1 | Autograd engine from scratch | Karpathy — The spelled-out intro to neural networks and backpropagation: building micrograd | |
| 2 | A net from raw NumPy, no autograd | Samson Zhang — Building a neural network FROM SCRATCH (no Tensorflow/Pytorch, just numpy & math) | |
| 3 | MLP language model from scratch | Karpathy — Building makemore Part 2: MLP | |
| 4 | A deeper sequence model from scratch | Karpathy — Building makemore Part 5: Building a WaveNet | |
| 5 | BPE tokenizer from scratch | minbpe — minimal byte-pair encoding implementation | |
| 6 | Train a GPT yourself | nanoGPT — the simplest repo for training a GPT | |
| 7 | A GPT in raw C/CUDA | llm.c — training GPT-2 without PyTorch or Python |
Found a broken link or an unclear step? Report a problem with this track.