Track

Data-Centric AI

Goal: Learn how to improve models by improving data, from data-centric development and data cascades through datasheets, labeling, active learning, weak supervision, label-error detection, synthetic data, deduplication, and web-scale filtering.

Prereqs: ML Basics. LLMs helps for pretraining-data filtering.

Status: done

Work through the steps in order. Bold links open YouTube.

Step	Concept	YouTube	Read
1	Model-centric to data-centric AI	A Chat with Andrew on MLOps: From Model-centric to Data-centric AI	Ng — Data-Centric AI Development
2	Data cascades		Sambasivan et al. 2021 — “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
3	Datasheets for datasets		Gebru et al. 2018 — Datasheets for Datasets
4	Dataset composition bias		Buolamwini & Gebru 2018 — Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification
5	Data statements for NLP		Bender & Friedman 2018 — Data Statements for Natural Language Processing
6	Active learning for labeling		Settles 2009 — Active Learning Literature Survey
7	Weak supervision and Snorkel	Ratner — Programmatically Building Training Data with Snorkel	Ratner et al. 2017 — Snorkel: Rapid Training Data Creation with Weak Supervision
8	Confident learning and label errors	Cleanlab: AI to Find and Fix Errors in ML Datasets	Northcutt, Jiang & Chuang 2021 — Confident Learning: Estimating Uncertainty in Dataset Labels
9	Pervasive label errors in test sets		Northcutt, Athalye & Mueller 2021 — Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks
10	Synthetic tabular data		Xu et al. 2019 — Modeling Tabular data using Conditional GAN
11	Deduplicating training data		Lee et al. 2022 — Deduplicating Training Data Makes Language Models Better
12	Filtering web-scale pretraining data	How to Create an LLM Dataset: FineWeb Overview	Penedo et al. 2024 — The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

Found a broken link or an unclear step? Report a problem with this track.