Math for ML, Deeper
The next layer of math behind ML papers: change of basis, SVD, Jacobians, Hessians, convexity, Lagrange multipliers, Gaussians, and KL divergence.
Before this: Python & Math for AI (vectors, matrices, derivatives, Bayes' theorem, the normal distribution).
-
10 min
Step 1Span and basis vectors
What it means for vectors to build a whole space.
Linear combinations, span, and basis vectors | Chapter 2, Essence of linear algebra3Blue1Brown
-
14 min
Step 2Dot products and duality
Why a dot product is secretly a projection and a linear map.
Dot products and duality | Chapter 9, Essence of linear algebra3Blue1Brown
-
13 min
Step 3Change of basis
Describe the same vector in different coordinate systems.
Change of basis | Chapter 13, Essence of linear algebra3Blue1Brown
-
16 min
Step 4Singular value decomposition
Any matrix factors into rotate or reflect, scale, then rotate or reflect.
SVD Visualized, Singular Value Decomposition explained | SEE Matrix , Chapter 3 #SoME2Visual Kernel
fastai/numerical-linear-algebra★ 11k -
10 min
Step 5The multivariable chain rule
The rule backpropagation applies over and over.
Multivariable chain ruleKhan Academy
-
6 min
Step 6The Jacobian matrix
All partial derivatives of a vector function in one matrix.
The Jacobian matrixKhan Academy
-
6 min
Step 7The Hessian matrix
Second derivatives describe curvature around a point.
The Hessian matrix | Multivariable calculus | Khan AcademyKhan Academy
-
22 min
Step 8Taylor series
Approximate a function near a point with polynomials.
Taylor series | Chapter 11, Essence of calculus3Blue1Brown
-
15 min
Step 9Convexity
Why every local minimum of a convex function is also a global one.
Convexity 101 [Optimization Bootcamp]Steve Brunton
cvxpy/cvxpy★ 6.4k -
13 min
Step 10Lagrange multipliers
Optimize a function while obeying a constraint.
Understanding Lagrange Multipliers VisuallySerpentine Integral
-
11 min
Step 11The covariance matrix
How features vary together, summed up in one matrix.
The Covariance Matrix : Data Science Basicsritvikmath
-
7 min
Step 12The multivariate Gaussian
The bell curve in many dimensions, shaped by covariance.
Multivariate Normal (Gaussian) Distribution ExplainedDataMListic
-
31 min
Step 13The central limit theorem
Why sums of random things look normal.
But what is the Central Limit Theorem?3Blue1Brown
-
11 min
Step 14Entropy, cross-entropy, and KL divergence
The information theory behind the most common loss function.
A Short Introduction to Entropy, Cross-Entropy and KL-DivergenceAurélien Géron
Build it
Watching isn't learning. Pick one and put it on GitHub.
- Compress a photo with NumPy's SVD at several ranks and plot quality against storage size.
- Check your hand-derived Jacobian of a small two-layer network against autograd in PyTorch.
- Write a notebook that computes entropy, cross-entropy, and KL divergence for two distributions and shows KL is not symmetric.