Track
Eval Harnesses
Goal: Unit tests for models, SWE-bench-style harnesses, regression evals.
Prereqs: LLMs.
Status: done
Work through the steps in order. Bold links open YouTube.
| Step | Concept | YouTube | Read |
|---|---|---|---|
| 1 | Why evals matter | Hamel Husain — AI Evaluations Clearly Explained in 50 Minutes | |
| 2 | What benchmarks measure and miss | AI Benchmarks Explained — MMLU, SWE-bench, and more | A Survey on Evaluation of Large Language Models |
| 3 | Benchmark contamination | Benchmark Data Contamination of Large Language Models: A Survey | |
| 4 | Functional-correctness code evals | Chen et al. 2021 — Evaluating Large Language Models Trained on Code (HumanEval) | |
| 5 | Repo-level agent evals | How SWE-bench Changed the Way We Test AI Coders | Jimenez et al. 2023 — SWE-bench: Can Language Models Resolve Real-World GitHub Issues? |
| 6 | LLM-as-judge for open-ended tasks | Zheng et al. 2023 — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena | |
| 7 | Running an open eval harness | EleutherAI — lm-evaluation-harness | |
| 8 | Writing evals for your own app | OpenAI — evals |
Found a broken link or an unclear step? Report a problem with this track.