AI Roadmaps
← All roadmaps
IntermediateRoadmap 21 · 15 steps · 4h 28m of video

Data Engineering for AI

Move, store, clean, and version the data that models learn from: SQL, Parquet, DuckDB, pipelines, labeling, and LLM-scale dataset curation.

Before this: Python & Math for AI (Python and pandas basics). Machine Learning helps for the feature and labeling steps.

  1. 14 min
    Step 1

    What data engineers do

    How raw data becomes something analysts and models can use.

    How Data Engineering Works

    AltexSoft

  2. 18 min
    Step 2

    SQL basics

    SELECT, WHERE, GROUP BY, and JOIN: the language of data.

    Learn Basic SQL in 15 Minutes | Business Intelligence For Beginners | SQL Tutorial For Beginners 1/3

    Adam Finer - Learn BI

  3. 7 min
    Step 3

    ETL vs ELT

    Two ways to build pipelines, and why AI teams care.

    ETL vs ELT: Powering Data Pipelines for AI & Analytics

    IBM Technology

  4. 16 min
    Step 4

    Warehouse, lake, and lakehouse

    Where data lives, and the trade-offs of each design.

    Data Warehouse vs Data Lake vs Data Lakehouse | ETL, OLAP vs OLTP

    codebasics

  5. 11 min
    Step 5

    The Parquet file format

    Columnar files make big datasets small and fast to read.

    Parquet File Format - Explained to a 5 Year Old!

    Data Mozart

  6. 11 min
    Step 6

    Query files with DuckDB

    Run fast SQL on local CSV and Parquet with no server.

    DuckDB Tutorial For Beginners In 12 min

    MotherDuck

    duckdb/duckdb★ 43.2k
  7. 1h 00m
    Step 7

    Fast dataframes with Polars

    A quicker, lazier alternative to pandas for big tables.

    Polars Crash Course - Modern Data Frames in Python

    NeuralNine

    pola-rs/polars★ 40.4k
  8. 12 min
    Step 8

    What Apache Spark is

    How data gets split across many machines when one is not enough.

    What is Spark? (Visual Explanation)

    Data with Baraa

  9. 13 min
    Step 9

    Schedule pipelines with Airflow

    DAGs, tasks, and schedules for jobs that run every day.

    Learn Apache Airflow in 10 Minutes | High-Paying Skills for Data Engineers

    Darshil Parmar

    apache/airflow★ 47.6k
  10. 12 min
    Step 10

    Validate data with Pandera

    Catch bad rows before they silently break training.

    How to Use Pandas With Pandera to Validate Your Data in Python

    ArjanCodes

    unionai-oss/pandera★ 4.5k
  11. 13 min
    Step 11

    Version data with DVC

    Track datasets like code so experiments stay reproducible.

    Versioning Data with DVC (Hands-On Tutorial!)

    DVCorg

    treeverse/dvc★ 16k
  12. 4 min
    Step 12

    Feature stores

    Serve the same features in training and in production.

    What is a Feature Store for Machine Learning?

    Hopsworks

  13. 20 min
    Step 13

    Label data with Label Studio

    Annotate text and images to build your own training set.

    Label Studio: The Easiest Way To Annotate Your Datasets

    NeuralNine

    HumanSignal/label-studio★ 28.7k
  14. 27 min
    Step 14

    Find near-duplicates with MinHash

    Shingling and LSH spot repeated documents at scale.

    Locality Sensitive Hashing (LSH) for Search with Shingling + MinHashing (Python)

    James Briggs

    ekzhu/datasketch★ 3k
  15. 29 min
    Step 15

    How an LLM dataset is built

    FineWeb's filtering and deduplication of web data, step by step.

    How to Create an LLM Dataset | FineWeb Overview

    Hugging Face

    huggingface/datatrove★ 3.4k

Build it

Watching isn't learning. Pick one and put it on GitHub.

Browse all roadmaps →