Evaluation & Benchmarking for LLMs
Measure language models and LLM apps properly: classic metrics, public benchmarks and leaderboards, LLM judges, error analysis, and open-source eval tools.
Before this: AI Engineering. LLMs from Scratch helps for the perplexity step.
-
6 min
Step 1What LLM benchmarks are
Standard tests for comparing models, and what they leave out.
What are Large Language Model (LLM) Benchmarks?IBM Technology
-
2 min
Step 2Perplexity
How surprised a language model is by text, the basic training metric.
What is perplexity?Hugging Face
-
5 min
Step 3BLEU
Score generated text by n-gram overlap with references, and its limits.
What is the BLEU metric?Hugging Face
-
31 min
Step 4Read and run benchmarks
How MMLU-style scores are computed, then run them yourself.
What Do LLM Benchmarks Actually Tell Us? (+ How to Run Your Own)Adam Lucek
EleutherAI/lm-evaluation-harness★ 14.5k -
9 min
Step 5How Arena ranks models
Pairwise human votes turned into scores with Bradley-Terry.
Behind LMArena's leaderboard: understanding AI model performanceArena
-
6 min
Step 6LLM as a judge
Use one model to grade another's outputs, and where that goes wrong.
LLM as a Judge: Scaling AI Evaluation StrategiesIBM Technology
-
1h 49m
Step 7LLM evaluation, the full lecture
Stanford lecture on metrics, judge biases, factuality, agents, and benchmarks.
Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 8 - LLM EvaluationStanford Online
-
12 min
Step 8Error analysis
Read real outputs and group failures before writing any metric.
Error Analysis: The Highest ROI Technique In AI EngineeringHamel Husain
-
55 min
Step 9Set up evals systematically
Unit tests, human review, and aligned LLM judges for an app.
How to Systematically Setup LLM Evals (Metrics, Unit Tests, LLM-as-a-Judge)Dave Ebbelaar
-
12 min
Step 10Evaluate RAG
Context precision, recall, and faithfulness, measured with RAGAS.
RAG Evaluation: Precision, Recall, Faithfulness, RAGAS Explained ClearlyLogical Lenses
vibrantlabsai/ragas★ 16.4k -
9 min
Step 11Test prompts with promptfoo
Compare models on your prompts with assertions in a YAML file.
Start with PromptFoo in under 10 min.Jason Koo
promptfoo/promptfoo★ 27.5k -
43 min
Step 12Evals with Inspect
Its creator walks through the UK AISI's open-source eval framework.
Inspect - A LLM Eval Framework Used by Anthropic, DeepMind, Grok and More.Hamel Husain
UKGovernmentBEIS/inspect_ai★ 3.3k
Build it
Watching isn't learning. Pick one and put it on GitHub.
- Run two small open models on three lm-evaluation-harness tasks and write up where their scores differ and why.
- Hand-label 50 outputs from an LLM app, build an LLM judge for the most common failure, and report how often it agrees with you.
- Build a 20-question RAG test set and track RAGAS faithfulness and context recall as you change chunk size.