Speech & Audio AI
How machines hear and speak: waveforms, spectrograms, speech recognition with Whisper, text-to-speech, audio codecs, and voice agents.
Before this: Deep Learning. LLMs from Scratch helps for the later steps.
-
20 min
Step 1The Fourier transform
Why any sound is a sum of frequencies.
But what is the Fourier Transform? A visual introduction.3Blue1Brown
-
25 min
Step 2Understanding audio signals
Sampling, waveforms, and how audio becomes model input.
Understanding Audio Signals for Machine LearningValerio Velardo - The Sound of AI
-
31 min
Step 3Mel spectrograms
The image-like input most speech models actually use.
Mel Spectrograms Explained EasilyValerio Velardo - The Sound of AI
huggingface/audio-transformers-course★ 537 -
6 min
Step 4MFCCs
Classic compact audio features, and why they mattered.
Mel Frequency Cepstral Coefficients (MFCC) ExplainedDataMListic
-
20 min
Step 5CTC loss
Train speech recognition without frame-level alignments.
Connectionist Temporal Classification (CTC) ExplainedDataMListic
-
15 min
Step 6wav2vec 2.0
Self-supervised pretraining for speech representations.
Wav2vec2 A Framework for Self-Supervised Learning of Speech Representations - Paper ExplainedDataMListic
-
1h 03m
Step 7Whisper
Large-scale weak supervision gives robust open speech recognition.
OpenAI Whisper: Robust Speech Recognition via Large-Scale Weak Supervision | Paper and CodeAleksa Gordić - The AI Epiphany
openai/whisper★ 113.2k -
27 min
Step 8Modern text-to-speech
How VITS-style models turn text into natural voices.
IAP 2025 Building Sound Lecture 12: Modern TTS — VITS and moreGridspace
-
26 min
Step 9Neural audio codecs
Compress audio into tokens that language models can handle.
Neural Audio Codec - Encodec and SoundStreamEgor Shvecov
-
15 min
Step 10Full-duplex voice models
Moshi: a voice model that listens and talks at once.
Moshi The Talking AISam Witteveen
kyutai-labs/moshi★ 11.6k
Build it
Watching isn't learning. Pick one and put it on GitHub.
- Transcribe a podcast episode with Whisper and measure word error rate on a 5-minute hand-corrected sample.
- Plot waveform, spectrogram, and mel spectrogram for three sounds and write what changes between them.
- Build a small voice assistant: Whisper in, an LLM in the middle, a local TTS voice out.