AI Roadmaps
← All roadmaps
IntermediateRoadmap 30 · 12 steps · 5h 00m of video

Evaluation & Benchmarking for LLMs

Measure language models and LLM apps properly: classic metrics, public benchmarks and leaderboards, LLM judges, error analysis, and open-source eval tools.

Before this: AI Engineering. LLMs from Scratch helps for the perplexity step.

  1. 6 min
    Step 1

    What LLM benchmarks are

    Standard tests for comparing models, and what they leave out.

    What are Large Language Model (LLM) Benchmarks?

    IBM Technology

  2. 2 min
    Step 2

    Perplexity

    How surprised a language model is by text, the basic training metric.

    What is perplexity?

    Hugging Face

  3. 5 min
    Step 3

    BLEU

    Score generated text by n-gram overlap with references, and its limits.

    What is the BLEU metric?

    Hugging Face

  4. 31 min
    Step 4

    Read and run benchmarks

    How MMLU-style scores are computed, then run them yourself.

    What Do LLM Benchmarks Actually Tell Us? (+ How to Run Your Own)

    Adam Lucek

    EleutherAI/lm-evaluation-harness★ 14.5k
  5. 9 min
    Step 5

    How Arena ranks models

    Pairwise human votes turned into scores with Bradley-Terry.

    Behind LMArena's leaderboard: understanding AI model performance

    Arena

  6. 6 min
    Step 6

    LLM as a judge

    Use one model to grade another's outputs, and where that goes wrong.

    LLM as a Judge: Scaling AI Evaluation Strategies

    IBM Technology

  7. 1h 49m
    Step 7

    LLM evaluation, the full lecture

    Stanford lecture on metrics, judge biases, factuality, agents, and benchmarks.

    Stanford CME295 Transformers & LLMs | Autumn 2025 | Lecture 8 - LLM Evaluation

    Stanford Online

  8. 12 min
    Step 8

    Error analysis

    Read real outputs and group failures before writing any metric.

    Error Analysis: The Highest ROI Technique In AI Engineering

    Hamel Husain

  9. 55 min
    Step 9

    Set up evals systematically

    Unit tests, human review, and aligned LLM judges for an app.

    How to Systematically Setup LLM Evals (Metrics, Unit Tests, LLM-as-a-Judge)

    Dave Ebbelaar

  10. 12 min
    Step 10

    Evaluate RAG

    Context precision, recall, and faithfulness, measured with RAGAS.

    RAG Evaluation: Precision, Recall, Faithfulness, RAGAS Explained Clearly

    Logical Lenses

    vibrantlabsai/ragas★ 16.4k
  11. 9 min
    Step 11

    Test prompts with promptfoo

    Compare models on your prompts with assertions in a YAML file.

    Start with PromptFoo in under 10 min.

    Jason Koo

    promptfoo/promptfoo★ 27.5k
  12. 43 min
    Step 12

    Evals with Inspect

    Its creator walks through the UK AISI's open-source eval framework.

    Inspect - A LLM Eval Framework Used by Anthropic, DeepMind, Grok and More.

    Hamel Husain

    UKGovernmentBEIS/inspect_ai★ 3.3k

Build it

Watching isn't learning. Pick one and put it on GitHub.

Browse all roadmaps →