Most teams evaluating an AI agent run one kind of test and call it a program: a demo conversation, a latency measurement, or a rubric scored on a week's traffic. Each of those is one lens, and a single lens produces a number that survives exactly until production introduces the traffic the lens never saw. An evaluation framework is the three lenses run as one methodology — and a gate that decides what a passing score is allowed to change.
This post assembles the framework. It is deliberately a map: the rubric categories are grounded in the existing AI voice agent QA post and the failure catalog; the benchmark categories each have a dedicated post underneath them; and the measurement layer that turns the framework into a buying decision is the how-to-benchmark guide, which is the feature ground this framework operates against. What is new here is the assembly — the order the lenses run in, what each one cannot see, and the gate that connects them.
The evaluation rubric, grounded in tests that already exist
A rubric you design in a workshop is a hypothesis about what can go wrong. A rubric grounded in a failure catalog is a checklist of what already has. Build the second kind.
The failure catalog enumerates six ways an agent breaks — over-wide credentials, deterministic gates retried forever, flaky upstreams, unpinned prompts, runaway cost, and actions with no attribution. Each of those is a rubric category, because each is a class of conversation a generic "was the answer helpful" score will pass right up until it produces an incident. The QA post adds the four scoring surfaces that cut across those modes: outcome completion (did the task finish), resolution quality (resolved or merely routed), policy and disclosure (explicit pass/fail, not folded into tone), and conversation hygiene (greeting, dead air, handoff summary).
Assembled, the rubric has a working structure:
- Outcome rubrics, one fail-condition per discrete task. Booking confirms a time, date, and name recited back; support resolves or escalates with a reference. Every task in the outcome space gets its own condition — a shared politeness rubric is not coverage.
- Policy and disclosure checks, scored as pass/fail. Recording notice, identity verification before account detail, refund terms stated as written. These correspond to the catalog's compliance modes, and they fail loudly only because they are not averaged into a friendliness score.
- Configuration-drift checks. The rubric set must include the conversations a prompt edit or knowledge update can regress — the catalog's mode 4. A rubric that measures today's behavior but cannot be replayed against tomorrow's candidate configuration measures nothing about the change.
- Cost and loop guards as evaluated behavior, not just settings. A runaway tool loop (catalog mode 5) appears in conversation scoring as repeated invocations and unbounded turn counts. Score for it; do not rely on the ceiling sitting at its default.
The stopping rule comes from the QA post: coverage is a property of the rubric set, not of the sample size. When production surfaces a failure no rubric would catch, the framework's answer is a new rubric — that is how the evaluation grows, and it is the discipline that separates a framework from a dashboard.
The benchmark categories
Benchmarks divide into three categories by what they measure, and a framework needs all three because each one's blind spot is a different lens's strength.
Latency and turn performance
An agent that resolves well at 700 ms per turn in a demo can regress into awkward dead air under production concurrency, and no content rubric notices. Latency is its own benchmark category: per-stage turn time, measured under load, with a p95 you can hold a vendor to. Devotel Orbit publishes a per-stage methodology and a ~1.1-second p50 internal SLO on the latency benchmark page — the framework treats any agent platform that cannot show a p95 turn time under load as unmeasured in this category, and the pricing guide explains why that number is a cost lever, not just a quality one.
Quality scoring
The rubric above, run over every conversation rather than a sample. On Orbit this is the Quality hub, per-agent scorecards, and the Evaluations surface scoring voice and inbox together with an LLM judge — the per-intent decomposition the benchmarking guide demands is the shipped default view, not a custom report. Quality scoring is the category buyers most often mistake for the whole framework: a perfect QA score says the conversations you scored met the rubric as configured, and nothing about the conversations the rubric does not reach.
Roleplay and simulation training
The third category evaluates before traffic exists. Practice Studio rehearses agents against an AI playing the customer, scored 0–100 against a supervisor-authored rubric, with no PSTN or SMS traffic generated. In the framework its role is specific: it is the only lens that can measure a scenario you have never seen in production — the regulated disclosure, the angry-customer de-escalation, the edge case your call mix has not produced yet this quarter. Simulation is also the remediation loop for human agents in a hybrid org: the evaluation flags the gap, practice closes it, and the leaderboard reflects the recovery.
Three lenses: black-box, simulation, dial-plan
The categories above say what to measure. The lenses say from where — and mixing them up is how a team ends up trusting a score that was never measuring what they thought.
Black-box evaluation scores the agent from outside: conversations in, outcomes out, no access to the prompt, the retrieval, or the tool wiring. It is the only lens a prospective buyer fully controls, which is why the benchmark sheet runs entirely on it — the four metrics (deflection, first-contact resolution, fallback-to-human split by reason, hallucination containment) are all black-box constructs, computable on any vendor from per-conversation records. Its blind spot is diagnosis: a black box tells you that the booking intent regressed, never why.
Simulation evaluation replaces production with saved cases. It is the lens that answers the black box's "why" — replay the failing conversation against a prompt candidate, a knowledge update, or a configuration change, and compare scores before the change touches live traffic. Orbit grounds this lens in two shipped mechanisms: regression suites that carry a pinned version id, so a replay runs against a frozen prompt rather than whatever is live, and a candidate-version override that scores the whole corpus against the proposed change before promotion. Practice Studio is the human-facing twin of the same lens. Simulation's blind spot is coverage in the other direction: it knows only the cases you saved, so a simulation suite that never grows past its original corpus certifies yesterday's failure modes against tomorrow's traffic.
Dial-plan evaluation inspects the telephony path itself — the route the call takes, the carrier hops between the agent and the handset, the codecs and dead-air behavior that content scores sit on top of. A conversation rubric can pass while the path degrades; only this lens sees it. Orbit's voice pipeline terminates over Devotel's own wholesale softswitch rather than a resold aggregator hop, so the dial-plan lens has one fewer unmeasured intermediary to audit — and a pilot that runs on a demo path instead of the production route is not a dial-plan evaluation of anything you will operate.
The framework rule: a score means nothing until you can name which lens produced it, and each lens's blind spot is another lens's whole job. Black-box says that, simulation says why, dial-plan says whether the path allowed it at all.
The gate: what a passing score is allowed to change
The framework's last piece is the one most evaluation programs skip. Lenses and rubrics produce scores; something has to decide what a score is allowed to do.
On Orbit that decision is a promotion gate, and it runs on the simulation lens by construction: a prompt or configuration candidate replays against the pinned regression corpus, scores arrive before exposure, and promotion or rollback is the same single operation on an immutable version. A rubric failure routes into remediation — a new rubric for the coverage gap, a practice scenario for the human-agent variant — instead of into a coaching session or a silent prompt edit. The gate is what turns "we measure quality" into "quality findings change what ships."
The complementary rule holds for buying: the benchmark guide's operability section treats the gate itself as a vendor criterion — per-intent export, a published or measurable p95 under load, rubric customization, configurable handoff summaries, versioned knowledge sources. A vendor that can decompose the aggregate but cannot gate a change on the decomposition has given you the diagnosis lens without the remediation path.
Frequently asked questions
What are the three ways to evaluate an AI agent?
Black-box, simulation, and dial-plan. Black-box scores conversations from the outside and is the only lens a buyer fully controls. Simulation replays saved failing cases against a candidate prompt or configuration before exposure, which is what answers "why" and gates promotion. Dial-plan evaluation inspects the carrier path the content scores sit on. Each lens has a blind spot the other two cover, so a framework runs all three and names which lens produced every score.
How is AI-agent evaluation different from benchmarking?
Benchmarking is one application of the framework — the black-box lens run as a vendor comparison, usually during a two-week pilot, against a fixed question set and per-intent denominators. Evaluation is the larger program: the rubric set, all three lenses, and the promotion gate that connects scores to changes. The how-to-benchmark guide covers the buying application in detail.
Can an AI agent pass evaluation and still fail in production?
Yes, in two ways. A pass means the scored conversations met the rubric as configured — coverage gaps (an outcome no rubric measures) and single-lens scores (a content rubric that cannot see the telephony path) both survive a "passing" program. The framework's answer is structural: grow the rubric set from every production failure, run all three lenses, and gate promotion on simulation replay so a finding changes what ships before customers meet it.
How does Orbit expose the evaluation surface?
The Quality hub, per-agent scorecards, and Evaluations surface run the black-box and quality-scoring layers across voice and inbox with per-intent decomposition on every paid tier. Simulation is grounded in pinned-version regression replay and Practice Studio's scored roleplay. Latency is published with a per-stage methodology on the latency benchmark page, and the dial-plan lens audits one fewer hop because voice terminates on Devotel's own softswitch.
Where to go next
Form the rubric from the QA post and the failure catalog — the categories are already written. Run the buying application with the benchmark guide's four metrics and two-week sheet. Keep the third category honest with Practice Studio scenarios for the cases your traffic has not produced. The framework is the assembly, and the gate is the part that makes it more than measurement.