Bayesian ML & Probabilistic Programming
Reason with uncertainty: priors and posteriors, MCMC, PyMC, hierarchical models, Gaussian processes, variational inference, and Bayesian deep learning.
Before this: Machine Learning and Math for ML, Deeper (probability distributions, MLE, the multivariate Gaussian, KL divergence).
-
13 min
Step 1Probabilities of probabilities
Treat an unknown rate as something you can be uncertain about.
Binomial distributions | Probabilities of probabilities, part 13Blue1Brown
-
1h 37m
Step 2Bayesian updating by counting
McElreath builds a posterior from counted ways the data could occur.
Statistical Rethinking 2023 - 02 - The Garden of Forking DataRichard McElreath
-
18 min
Step 3Conjugate priors
Update a Beta prior with data in closed form.
Bayesian Updates and Conjugate PriorsSteve Brunton
-
18 min
Step 4MLE vs MAP
How adding a prior turns maximum likelihood into MAP.
What are Maximum Likelihood (ML) and Maximum a posteriori (MAP)? ("Best explanation on YouTube")Iain Explains Signals, Systems, and Digital Comms
-
16 min
Step 5Bayesian linear regression
Regression with a distribution over weights, not one answer.
Bayesian Linear Regression : Data Science Conceptsritvikmath
-
12 min
Step 6Markov chain Monte Carlo
Sample a posterior known only up to its normalizing constant.
Markov Chain Monte Carlo (MCMC) : Data Science Conceptsritvikmath
-
18 min
Step 7Metropolis-Hastings
The classic accept-or-reject MCMC sampler.
Metropolis - Hastings : Data Science Conceptsritvikmath
-
32 min
Step 8Hamiltonian Monte Carlo
Use gradients to make big, efficient sampler moves.
The intuition behind the Hamiltonian Monte Carlo algorithmBen Lambert
-
1h 27m
Step 9Bayesian modeling with PyMC
Write a model in Python and let PyMC sample it.
Chris Fonnesbeck - Probabilistic Python: An Introduction to Bayesian Modeling with PyMCPyData
pymc-devs/pymc★ 9.9k -
11 min
Step 10Posterior predictive checks
Simulate fake data from the model to see if it fits.
What is a posterior predictive check and why is it useful?Ben Lambert
arviz-devs/arviz★ 1.9k -
13 min
Step 11Hierarchical models
Share strength across groups with partial pooling.
Statistical Gold Nuggets | Bayesian Hierarchical ModelsVery Normal
-
24 min
Step 12Gaussian processes
A prior over whole functions, with error bars for free.
Gaussian ProcessesMutual Information
cornellius-gp/gpytorch★ 3.9k -
8 min
Step 13Bayesian optimization
Tune expensive black boxes with a GP and an acquisition function.
Bayesian OptimizationDataMListic
bayesian-optimization/BayesianOptimization★ 8.7k -
25 min
Step 14Variational inference and the ELBO
Turn inference into optimization when sampling is too slow.
Variational Inference | Evidence Lower Bound (ELBO) | Intuition & VisualizationMachine Learning & Simulation
pyro-ppl/numpyro★ 2.8k -
31 min
Step 15Bayesian neural networks
Get uncertainty estimates out of deep networks.
Uncertainty estimation and Bayesian Neural Networks - Marcin MożejkoPyData
Build it
Watching isn't learning. Pick one and put it on GitHub.
- Estimate a coin's bias three ways (grid, conjugate Beta update, and PyMC) and plot the three posteriors together.
- Fit a hierarchical PyMC model to per-group data (e.g. ratings by restaurant) and compare it with no pooling and full pooling.
- Use Bayesian optimization to tune two hyperparameters of a scikit-learn model and compare it with random search.