AI Roadmaps
← All roadmaps
AdvancedRoadmap 20 · 11 steps · 11h 37m of video

GPU Programming & CUDA

Understand why GPUs are fast and write your own kernels: CUDA basics, memory, tiled matmul, Triton, and FlashAttention.

Before this: Deep Learning (PyTorch and attention basics). C or C++ basics help. Needs an NVIDIA GPU or a free Colab GPU.

  1. 29 min
    Step 1

    CUDA clicks in 30 minutes

    Threads, blocks, and grids from a first principles view.

    Give Me 30 min, I'll Make CUDA Click Forever

    Zachary Huang

  2. 13 min
    Step 2

    Write your first CUDA kernel

    Launch a kernel and see the grid, block, and thread layout.

    Write Your First CUDA Kernel in 15 Minutes (Threads, Blocks, Grid Explained)

    Faradawn Yang

  3. 8 min
    Step 3

    GPU memory hierarchy

    Registers, shared memory, and global memory, and what each costs.

    Memory Hierarchy | GPU Programming | Episode 6

    Simon Oz

  4. 6 min
    Step 4

    Memory coalescing

    A small access-pattern change gives a large speedup.

    4.5x Faster CUDA C with just Two Variable Changes || Episode 3: Memory Coalescing

    Tushar Gautam

  5. 9 min
    Step 5

    Tiled matrix multiplication

    Use shared memory to cut slow global reads.

    Must Know Technique in GPU Computing | Episode 4: Tiled Matrix Multiplication in CUDA C

    Tushar Gautam

    siboehm/SGEMM_CUDA★ 1.4k
  6. 56 min
    Step 6

    Profile CUDA kernels in PyTorch

    Find where time goes before you optimize.

    Lecture 1 How to profile CUDA kernels in PyTorch

    GPU MODE

    gpu-mode/lectures★ 7.1k
  7. 9 min
    Step 7

    Custom PyTorch operators

    Hook your own CUDA or C++ code into PyTorch.

    Lightning Talk: Extending PyTorch with Custom Python/C++/CUDA Operators - Richard Zou, Meta

    PyTorch Foundation

  8. 1h 22m
    Step 8

    Practitioner's guide to Triton

    Write fast kernels in Python-like code.

    Lecture 14: Practitioners Guide to Triton

    GPU MODE

    triton-lang/triton★ 20.6k
  9. 16 min
    Step 9

    A fast softmax kernel

    Fuse steps to avoid extra trips through memory.

    How to write a fast Softmax kernel

    Simon Oz

  10. 12 min
    Step 10

    How FlashAttention works

    Tile attention so the big matrix never hits slow memory.

    How FlashAttention Accelerates Generative AI Revolution

    Jia-Bin Huang

    Dao-AILab/flash-attention★ 25.3k
  11. 7h 38m
    Step 11

    FlashAttention from first principles in Triton

    Derive it and code the forward and backward pass.

    Flash Attention derived and coded from first principles with Triton (Python)

    Umar Jamil

    hkproj/triton-flash-attention★ 279

Build it

Watching isn't learning. Pick one and put it on GitHub.

Browse all roadmaps →