GPU Programming & CUDA
Understand why GPUs are fast and write your own kernels: CUDA basics, memory, tiled matmul, Triton, and FlashAttention.
Before this: Deep Learning (PyTorch and attention basics). C or C++ basics help. Needs an NVIDIA GPU or a free Colab GPU.
-
29 min
Step 1CUDA clicks in 30 minutes
Threads, blocks, and grids from a first principles view.
Give Me 30 min, I'll Make CUDA Click ForeverZachary Huang
-
13 min
Step 2Write your first CUDA kernel
Launch a kernel and see the grid, block, and thread layout.
Write Your First CUDA Kernel in 15 Minutes (Threads, Blocks, Grid Explained)Faradawn Yang
-
8 min
Step 3GPU memory hierarchy
Registers, shared memory, and global memory, and what each costs.
Memory Hierarchy | GPU Programming | Episode 6Simon Oz
-
6 min
Step 4Memory coalescing
A small access-pattern change gives a large speedup.
4.5x Faster CUDA C with just Two Variable Changes || Episode 3: Memory CoalescingTushar Gautam
-
9 min
Step 5Tiled matrix multiplication
Use shared memory to cut slow global reads.
Must Know Technique in GPU Computing | Episode 4: Tiled Matrix Multiplication in CUDA CTushar Gautam
siboehm/SGEMM_CUDA★ 1.4k -
56 min
Step 6Profile CUDA kernels in PyTorch
Find where time goes before you optimize.
Lecture 1 How to profile CUDA kernels in PyTorchGPU MODE
gpu-mode/lectures★ 7.1k -
9 min
Step 7Custom PyTorch operators
Hook your own CUDA or C++ code into PyTorch.
Lightning Talk: Extending PyTorch with Custom Python/C++/CUDA Operators - Richard Zou, MetaPyTorch Foundation
-
1h 22m
Step 8Practitioner's guide to Triton
Write fast kernels in Python-like code.
Lecture 14: Practitioners Guide to TritonGPU MODE
triton-lang/triton★ 20.6k -
16 min
Step 9A fast softmax kernel
Fuse steps to avoid extra trips through memory.
How to write a fast Softmax kernelSimon Oz
-
12 min
Step 10How FlashAttention works
Tile attention so the big matrix never hits slow memory.
How FlashAttention Accelerates Generative AI RevolutionJia-Bin Huang
Dao-AILab/flash-attention★ 25.3k -
7h 38m
Step 11FlashAttention from first principles in Triton
Derive it and code the forward and backward pass.
Flash Attention derived and coded from first principles with Triton (Python)Umar Jamil
hkproj/triton-flash-attention★ 279
Build it
Watching isn't learning. Pick one and put it on GitHub.
- Write a vector add and a naive matmul in CUDA, then add tiling and chart the speedup.
- Implement a fused softmax in Triton and benchmark it against PyTorch.
- Profile one PyTorch model layer, find its bottleneck, and write up what you changed.