AI Roadmaps
← All roadmaps
AdvancedRoadmap 35 · 13 steps · 6h 59m of video

Mixture of Experts & Efficient LLM Architectures

See how modern LLMs grow without getting slower: sparse mixture of experts, cheaper attention, linear attention, state space models, and hybrids.

Before this: LLMs from Scratch. Run, Serve & Scale AI helps for the parallelism step.

  1. 8 min
    Step 1

    What mixture of experts is

    Many expert networks, with only a few active per token.

    What is Mixture of Experts?

    IBM Technology

  2. 32 min
    Step 2

    MoE, visually explained

    Routers, top-k gating, and load balancing, drawn step by step.

    Mixture of Experts (MoE), Visually Explained

    Jia-Bin Huang

  3. 34 min
    Step 3

    Switch Transformers

    Top-1 routing and a load-balancing loss that scaled MoE far.

    Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

    Yannic Kilcher

  4. 34 min
    Step 4

    Code token routing

    Hugging Face walkthrough of a router in code, with a Colab notebook.

    MoE Token Routing Explained: How Mixture of Experts Works (with Code)

    Hugging Face

  5. 1h 26m
    Step 5

    Mixtral, line by line

    Umar Jamil explains Mixtral's sparse experts and sliding-window attention in code.

    Mistral / Mixtral Explained: Sliding Window Attention, Sparse Mixture of Experts, Rolling Buffer

    Umar Jamil

    hkproj/mistral-src-commented★ 30
  6. 1h 22m
    Step 6

    MoE at scale

    Stanford CS336 lecture on training and routing in real MoE models.

    Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 4: Mixture of experts

    Stanford Online

  7. 20 min
    Step 7

    Expert parallelism

    Split experts, layers, and data across GPUs for inference.

    LLM Inference Optimization #2: Tensor, Data & Expert Parallelism (TP, DP, EP, MoE)

    Faradawn Yang

  8. 7 min
    Step 8

    Multi-query and grouped-query attention

    Share keys and values across heads to shrink the KV cache.

    Multi-Head Attention (MHA), Multi-Query Attention (MQA), Grouped Query Attention (GQA) Explained

    DataMListic

  9. 29 min
    Step 9

    From GQA to latent and sparse attention

    How MLA and DeepSeek's sparse attention cut attention cost further.

    How Attention Got So Efficient [GQA/MLA/DSA]

    Jia-Bin Huang

  10. 35 min
    Step 10

    Linear attention

    Drop softmax to make attention scale linearly and run like an RNN.

    Beyond Softmax — Linear Attention

    Jia-Bin Huang

  11. 32 min
    Step 11

    Mamba and state space models

    A recurrent model with input-dependent state updates and linear-time sequence cost.

    MAMBA from Scratch: Neural Nets Better and Faster than Transformers

    Algorithmic Simplicity

    state-spaces/mamba★ 19k
  12. 9 min
    Step 12

    Hybrid Mamba-Transformer models

    NVIDIA's Hymba mixes attention and Mamba heads in one small model.

    Hymba by NVIDIA: A Hybrid Mamba-Transformer SOTA Small LM

    AI Papers Academy

  13. 11 min
    Step 13

    DeepSeek-V3, putting it together

    How MoE, MLA, and training tricks combine in one frontier model.

    How Did They Do It? DeepSeek V3 and R1 Explained

    No Hype AI

Build it

Watching isn't learning. Pick one and put it on GitHub.

Browse all roadmaps →