AI evaluation harness
A repeatable test suite that scores model outputs against fixed cases.
AI
Definition
An evaluation harness runs a fixed set of inputs through a model or pipeline and scores the outputs with deterministic checks, model-graded rubrics or human review.
It is the regression test suite of an AI system.
Why it matters
Without one, every prompt or model change is an unmeasured bet, and quality regressions surface as customer complaints.
How to apply it
- Build the harness from real production failures.
- Gate deployments on it in CI.
- Track cost and latency alongside quality.