module 06 / 08
Evals
Measure model behavior so quality improves for reasons you can explain.
the point
Without evals, AI product work becomes vibes. Evals turn errors into a system you can improve.
start here
go deeper
understanding check
You should be able to…
- write examples that represent real product failures
- separate unit evals, human review, and production monitoring
- use error analysis to choose the next change
- know when an eval is being gamed
prove it by building
Build an eval set for an LLM feature
intermediate · typescript or python, json fixtures, model API, simple report output
make
a repeatable eval harness with examples, expected behavior, and scoring notes
definition of done
compare two prompts or models and explain the regression you would ship or reject