Self-Driving Cars & Autonomous Systems
Follow the self-driving stack from sensors and localization through perception, prediction, and planning to end-to-end models and simulation.
Before this: Deep Learning and Computer Vision. Robotics & Embodied AI helps for the planning and imitation steps.
-
25 min
Step 1Introduction to self-driving
Waabi's overview of the problem and the classic autonomy stack.
[01] Introduction to Self-Driving (Waabi CVPR 24 Tutorial on Self-Driving Cars)Waabi
-
28 min
Step 2Sensors and hardware
Cameras, lidar, radar, and compute, and what each sees.
[02] Hardware and Sensors (Waabi CVPR 24 Tutorial on Self-Driving Cars)Waabi
-
11 min
Step 3Kalman filters
Fuse noisy measurements with a motion model to track state.
Visually Explained: Kalman FiltersVisually Explained
-
27 min
Step 4Localization
Work out where the car is on the map, to centimeters.
Localization (Waabi CVPR 23 Tutorial on Self-Driving Cars)Waabi
-
36 min
Step 5HD maps
What goes into driving maps and how they are built.
[10] Mapping (Waabi CVPR 24 Tutorial on Self-Driving Cars)Waabi
-
26 min
Step 6Perception
Detect and track cars and pedestrians in 3D from sensor data.
Perception (Waabi CVPR 23 Tutorial on Self-Driving Cars)Waabi
-
57 min
Step 7Bird's-eye-view 3D perception
MIT talk on efficient 3D perception, including camera and lidar BEV fusion.
Efficient 3D Perception for Autonomous Vehicles Zhijian Liu MITxLAB for Safe Autonomous Systems
-
54 min
Step 8Motion forecasting
Predict where other road users will move next.
Motion Forecasting (Waabi CVPR 23 Tutorial on Self-Driving Cars)Waabi
-
52 min
Step 9Motion planning and control
Choose a safe trajectory, then follow it with steering and throttle.
Motion Planning and Control (Waabi CVPR 23 Tutorial on Self-Driving Cars)Waabi
-
28 min
Step 10Imitation learning for driving
Tübingen lecture on learning to drive from human demonstrations.
Self-Driving Cars - Lecture 2.3 (Imitation Learning: Imitation Learning)Tübingen Machine Learning
-
42 min
Step 11End-to-end driving
Wayve's CEO on replacing the hand-built stack with one learned model.
How End-to-End Learning Created Autonomous Driving 2.0: Wayve CEO Alex KendallSequoia Capital
-
53 min
Step 12Simulation
Test driving software in simulated worlds before it reaches the road.
[08] Simulation (Waabi CVPR 24 Tutorial on Self-Driving Cars)Waabi
carla-simulator/carla★ 14.7k -
41 min
Step 13How openpilot controls a real car
comma's engineer on sending steering and speed commands to stock cars.
How Do We Control The Car? | COMMA_CONcomma
commaai/openpilot★ 64.2k
Build it
Watching isn't learning. Pick one and put it on GitHub.
- Implement a 1D and then 2D Kalman filter that tracks a simulated car from noisy GPS readings.
- Drive a car along waypoints in the CARLA simulator with a PID controller and plot the tracking error.
- Train a small behavior-cloning model in CARLA or a 2D driving sim and report how often it leaves the lane.