Synthetic Data & Data-Centric AI
Improve models by improving their data: find label errors, label less, augment, generate synthetic tables and LLM training sets, and know when synthetic data backfires.
Before this: Machine Learning. Deep Learning helps for the GAN and VAE step; Fine-Tuning & Reasoning Models helps for the LLM steps.
-
26 min
Step 1From model-centric to data-centric AI
Andrew Ng's case for fixing data quality before tuning the model.
Data centric AI development From Big Data to Good Data Andrew NgDatabricks
-
1h 13m
Step 2Find label errors
MIT lecture on confident learning: spot mislabeled examples automatically.
Lecture 2: Label ErrorsIntroduction to Data-Centric AI
cleanlab/cleanlab★ 11.8k -
1h 02m
Step 3Dataset creation and curation
MIT lecture on sourcing, labeling, and cleaning a dataset.
Lecture 3: Dataset Creation and CurationIntroduction to Data-Centric AI
-
6 min
Step 4Active learning
Let the model choose which examples are worth labeling next.
Active (Machine) Learning - ComputerphileComputerphile
-
30 min
Step 5Weak supervision
Combine noisy heuristic rules into training labels with Snorkel.
Hand in hand with weak supervision using snorkel - Szymon WojciechowskiPyData
-
10 min
Step 6Data augmentation
Flips, crops, and color shifts that stretch a small image dataset.
C4W2L10 Data AugmentationDeepLearningAI
-
7 min
Step 7What synthetic data is
Generated data that keeps real data's patterns, and where it helps.
What is Synthetic Data? No, It's Not "Fake" DataIBM Technology
-
9 min
Step 8Synthetic tables with SDV
Generate single- and multi-table data with copulas and GANs.
Synthetic Data Generation with SDVCeADAR Ireland
sdv-dev/SDV★ 3.6k -
35 min
Step 9CTGAN and TVAE in Python
The theory behind two tabular generators, then code using both.
Synthetic Data Generation with Generative Models in PythonNeuralNine
-
3 min
Step 10Synthetic datasets for LLM fine-tuning
Turn your documents into QA and reasoning pairs with a CLI.
How to Create Synthetic Datasets for Fine-Tuning LlamaMeta Developers
meta-llama/synthetic-data-kit★ 1.7k -
19 min
Step 11Curating data for LLM post-training
Lessons from building the OpenThoughts reasoning dataset.
Data and Environment Curation for Post-Training LLMs — Mahesh Sathiamoorthy, Bespoke LabsAI Engineer
-
18 min
Step 12Model collapse
What can go wrong when models train on model-generated data.
What Happens When All Training Data is AI Generated?Mutual Information
Build it
Watching isn't learning. Pick one and put it on GitHub.
- Run cleanlab on a public text classification dataset, review the 50 most suspicious labels by hand, and report how many were truly wrong.
- Generate a synthetic copy of a tabular dataset with SDV, then compare a model trained on synthetic data with one trained on real data.
- Create 200 QA pairs from your own notes with synthetic-data-kit, filter out the bad ones, and publish the dataset on Hugging Face.