Data Engineering for AI
Move, store, clean, and version the data that models learn from: SQL, Parquet, DuckDB, pipelines, labeling, and LLM-scale dataset curation.
Before this: Python & Math for AI (Python and pandas basics). Machine Learning helps for the feature and labeling steps.
-
14 min
Step 1What data engineers do
How raw data becomes something analysts and models can use.
How Data Engineering WorksAltexSoft
-
18 min
Step 2SQL basics
SELECT, WHERE, GROUP BY, and JOIN: the language of data.
Learn Basic SQL in 15 Minutes | Business Intelligence For Beginners | SQL Tutorial For Beginners 1/3Adam Finer - Learn BI
-
7 min
Step 3ETL vs ELT
Two ways to build pipelines, and why AI teams care.
ETL vs ELT: Powering Data Pipelines for AI & AnalyticsIBM Technology
-
16 min
Step 4Warehouse, lake, and lakehouse
Where data lives, and the trade-offs of each design.
Data Warehouse vs Data Lake vs Data Lakehouse | ETL, OLAP vs OLTPcodebasics
-
11 min
Step 5The Parquet file format
Columnar files make big datasets small and fast to read.
Parquet File Format - Explained to a 5 Year Old!Data Mozart
-
11 min
Step 6Query files with DuckDB
Run fast SQL on local CSV and Parquet with no server.
DuckDB Tutorial For Beginners In 12 minMotherDuck
duckdb/duckdb★ 43.2k -
1h 00m
Step 7Fast dataframes with Polars
A quicker, lazier alternative to pandas for big tables.
Polars Crash Course - Modern Data Frames in PythonNeuralNine
pola-rs/polars★ 40.4k -
12 min
Step 8What Apache Spark is
How data gets split across many machines when one is not enough.
What is Spark? (Visual Explanation)Data with Baraa
-
13 min
Step 9Schedule pipelines with Airflow
DAGs, tasks, and schedules for jobs that run every day.
Learn Apache Airflow in 10 Minutes | High-Paying Skills for Data EngineersDarshil Parmar
apache/airflow★ 47.6k -
12 min
Step 10Validate data with Pandera
Catch bad rows before they silently break training.
How to Use Pandas With Pandera to Validate Your Data in PythonArjanCodes
unionai-oss/pandera★ 4.5k -
13 min
Step 11Version data with DVC
Track datasets like code so experiments stay reproducible.
Versioning Data with DVC (Hands-On Tutorial!)DVCorg
treeverse/dvc★ 16k -
4 min
Step 12Feature stores
Serve the same features in training and in production.
What is a Feature Store for Machine Learning?Hopsworks
-
20 min
Step 13Label data with Label Studio
Annotate text and images to build your own training set.
Label Studio: The Easiest Way To Annotate Your DatasetsNeuralNine
HumanSignal/label-studio★ 28.7k -
27 min
Step 14Find near-duplicates with MinHash
Shingling and LSH spot repeated documents at scale.
Locality Sensitive Hashing (LSH) for Search with Shingling + MinHashing (Python)James Briggs
ekzhu/datasketch★ 3k -
29 min
Step 15How an LLM dataset is built
FineWeb's filtering and deduplication of web data, step by step.
How to Create an LLM Dataset | FineWeb OverviewHugging Face
huggingface/datatrove★ 3.4k
Build it
Watching isn't learning. Pick one and put it on GitHub.
- Turn a public CSV dataset into partitioned Parquet files and answer five questions about it with DuckDB SQL.
- Write an Airflow DAG that downloads a daily dataset, validates it with Pandera, and saves a versioned copy with DVC.
- Deduplicate a few thousand scraped web pages with MinHash and report how many near-duplicates you removed.