Eval (AI evaluation)
What is Eval (AI evaluation)?
An eval is a structured test suite that scores an AI agent or prompt's behavior against a set of example inputs and expected outcomes, run before shipping a change and continuously in production, to catch a quality regression the way a unit test catches a code bug. Evals typically score accuracy, tone, tool-call correctness, and guardrail adherence, since a small prompt or model change can shift behavior in ways plain manual testing easily misses.
More detail
Evals commonly combine automated scoring (does the output match an expected answer or pattern) with a second, separate model acting as a judge for qualities like tone or helpfulness that are harder to check with exact matching.
Running evals continuously against live production traffic — not just before a release — is how a team catches a quality drift caused by a model provider's own silent update, not just their own code changes.
Frequently asked
- How is an eval different from a unit test?
- A unit test checks deterministic code behavior against an exact expected output; an eval scores probabilistic AI outputs against example cases using a mix of automated matching and model-based judging, since the same prompt can produce slightly different wording each run.
- Why run evals in production, not just before release?
- An underlying model provider can update their model without an application's own code changing, so continuous production evals catch a resulting quality shift that pre-release testing against a fixed model version wouldn't reveal.
See also
Build it on Orbit
Voice, messaging, email, video, and AI agents on one platform and one pay-as-you-go bill. Start free — no credit card required.