Learn · Intermediate · Members

Checking and evaluations

Picture an exam written after the answers are handed in. Every student gets full marks, because the questions were chosen to fit what they wrote. Nobody learned anything, and nobody can say who knew the subject. Checking work done with Claude can go wrong the same way. If the test is written after the work, or the one who did the work also marks it, a pass tells you very little. This chapter turns the order around: the questions come first, the marker is someone else, and the marking is tried on wrong answers before it is trusted.

This chapter teaches how to know that work done with Claude is right. You test a check before you trust it, write the cases that define success before the work, and hand the review to a second agent. Last, you learn how a claim gets its own test and results file.

Lessons

• Lesson 1A check that can fail
• Lesson 2Write the evals first
• Lesson 3A separate reviewer
• Lesson 4Our own test behind a claim

Further resources

• Hamel Husain: Your AI Product Needs Evals: how unit tests, human review and model review fit together, taught through one real product.
• Arize: How to build LLM-as-a-judge evaluators: when plain code can check an answer, and when a model has to judge it.
• Anthropic: Demystifying evals for AI agents: the words for tasks, graders and trials, and the kinds of check a coding agent needs.