Mixture of Experts & Efficient LLM Architectures
See how modern LLMs grow without getting slower: sparse mixture of experts, cheaper attention, linear attention, state space models, and hybrids.
Before this: LLMs from Scratch. Run, Serve & Scale AI helps for the parallelism step.
-
8 min
Step 1What mixture of experts is
Many expert networks, with only a few active per token.
What is Mixture of Experts?IBM Technology
-
32 min
Step 2MoE, visually explained
Routers, top-k gating, and load balancing, drawn step by step.
Mixture of Experts (MoE), Visually ExplainedJia-Bin Huang
-
34 min
Step 3Switch Transformers
Top-1 routing and a load-balancing loss that scaled MoE far.
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient SparsityYannic Kilcher
-
34 min
Step 4Code token routing
Hugging Face walkthrough of a router in code, with a Colab notebook.
MoE Token Routing Explained: How Mixture of Experts Works (with Code)Hugging Face
-
1h 26m
Step 5Mixtral, line by line
Umar Jamil explains Mixtral's sparse experts and sliding-window attention in code.
Mistral / Mixtral Explained: Sliding Window Attention, Sparse Mixture of Experts, Rolling BufferUmar Jamil
hkproj/mistral-src-commented★ 30 -
1h 22m
Step 6MoE at scale
Stanford CS336 lecture on training and routing in real MoE models.
Stanford CS336 Language Modeling from Scratch | Spring 2025 | Lecture 4: Mixture of expertsStanford Online
-
20 min
Step 7Expert parallelism
Split experts, layers, and data across GPUs for inference.
LLM Inference Optimization #2: Tensor, Data & Expert Parallelism (TP, DP, EP, MoE)Faradawn Yang
-
7 min
Step 8Multi-query and grouped-query attention
Share keys and values across heads to shrink the KV cache.
Multi-Head Attention (MHA), Multi-Query Attention (MQA), Grouped Query Attention (GQA) ExplainedDataMListic
-
29 min
Step 9From GQA to latent and sparse attention
How MLA and DeepSeek's sparse attention cut attention cost further.
How Attention Got So Efficient [GQA/MLA/DSA]Jia-Bin Huang
-
35 min
Step 10Linear attention
Drop softmax to make attention scale linearly and run like an RNN.
Beyond Softmax — Linear AttentionJia-Bin Huang
-
32 min
Step 11Mamba and state space models
A recurrent model with input-dependent state updates and linear-time sequence cost.
MAMBA from Scratch: Neural Nets Better and Faster than TransformersAlgorithmic Simplicity
state-spaces/mamba★ 19k -
9 min
Step 12Hybrid Mamba-Transformer models
NVIDIA's Hymba mixes attention and Mamba heads in one small model.
Hymba by NVIDIA: A Hybrid Mamba-Transformer SOTA Small LMAI Papers Academy
-
11 min
Step 13DeepSeek-V3, putting it together
How MoE, MLA, and training tricks combine in one frontier model.
How Did They Do It? DeepSeek V3 and R1 ExplainedNo Hype AI
Build it
Watching isn't learning. Pick one and put it on GitHub.
- Add a top-2 MoE feed-forward layer to a small GPT and compare loss per compute with the dense version.
- Log how many tokens each expert receives during training, then add a load-balancing loss and show the change.
- Measure KV-cache memory and generation speed for MHA versus GQA in a small model at several context lengths.