AI Roadmaps
← All roadmaps
IntermediateRoadmap 12 · 10 steps · 4h 07m of video

Speech & Audio AI

How machines hear and speak: waveforms, spectrograms, speech recognition with Whisper, text-to-speech, audio codecs, and voice agents.

Before this: Deep Learning. LLMs from Scratch helps for the later steps.

  1. 20 min
    Step 1

    The Fourier transform

    Why any sound is a sum of frequencies.

    But what is the Fourier Transform? A visual introduction.

    3Blue1Brown

  2. 25 min
    Step 2

    Understanding audio signals

    Sampling, waveforms, and how audio becomes model input.

    Understanding Audio Signals for Machine Learning

    Valerio Velardo - The Sound of AI

  3. 31 min
    Step 3

    Mel spectrograms

    The image-like input most speech models actually use.

    Mel Spectrograms Explained Easily

    Valerio Velardo - The Sound of AI

    huggingface/audio-transformers-course★ 537
  4. 6 min
    Step 4

    MFCCs

    Classic compact audio features, and why they mattered.

    Mel Frequency Cepstral Coefficients (MFCC) Explained

    DataMListic

  5. 20 min
    Step 5

    CTC loss

    Train speech recognition without frame-level alignments.

    Connectionist Temporal Classification (CTC) Explained

    DataMListic

  6. 15 min
    Step 6

    wav2vec 2.0

    Self-supervised pretraining for speech representations.

    Wav2vec2 A Framework for Self-Supervised Learning of Speech Representations - Paper Explained

    DataMListic

  7. 1h 03m
    Step 7

    Whisper

    Large-scale weak supervision gives robust open speech recognition.

    OpenAI Whisper: Robust Speech Recognition via Large-Scale Weak Supervision | Paper and Code

    Aleksa Gordić - The AI Epiphany

    openai/whisper★ 113.2k
  8. 27 min
    Step 8

    Modern text-to-speech

    How VITS-style models turn text into natural voices.

    IAP 2025 Building Sound Lecture 12: Modern TTS — VITS and more

    Gridspace

  9. 26 min
    Step 9

    Neural audio codecs

    Compress audio into tokens that language models can handle.

    Neural Audio Codec - Encodec and SoundStream

    Egor Shvecov

  10. 15 min
    Step 10

    Full-duplex voice models

    Moshi: a voice model that listens and talks at once.

    Moshi The Talking AI

    Sam Witteveen

    kyutai-labs/moshi★ 11.6k

Build it

Watching isn't learning. Pick one and put it on GitHub.

Browse all roadmaps →