You would not let a human agent answer calls without quality assurance, so an AI voice agent gets the same treatment — and the playbook carries over less than most teams expect. The sampling math, the rubric, and the follow-through loop all change shape the moment the thing you are evaluating is a machine.
What actually changes when the agent is not a person
A human-agent QA program works around scarcity. Listeners are expensive, so the program samples a few calls per agent per week, scores them on a fixed form, and calibrates the listeners monthly so scores stay comparable. Everything in that design answers to one constraint: a person has to listen to each call.
None of those constraints survives an AI-agent program, and four things change as a result:
- Sampling goes from selective to comprehensive. A human-AI hybrid contact center still samples human calls because listening is expensive. An AI agent's conversations are transcripts, and a language model scores transcripts cheaply — so the question stops being "which 2% do we check" and becomes "did we configure the rubric to cover every outcome we care about." Coverage shifts from a staffing problem to a rubric-design problem.
- The failure modes are different. Humans drift slowly and predictably: a weak greeting, a skipped disclosure, a bad week. AI agents fail probabilistically — a prompt edit regresses one intent, a knowledge-base update stops surfacing the policy the agent quotes, a tool call starts returning empty. QA for AI agents is closer to regression testing than to coaching.
- The unit of evaluation is the conversation, not the shift. Human QA averages scores across an agent's week. For an AI agent every interaction is scoreable, so evaluation becomes a per-conversation verdict tied to the exact prompt and configuration version that produced it.
- Remediation closes the loop differently. Coaching a person takes weeks and decays. For an AI agent, the fix is a prompt, knowledge, or configuration change — which means the QA loop has to feed a controlled promotion path, not a one-on-one meeting.
What evaluation coverage means in practice
Coverage is a property of your rubric set, not of your sample size. The working definition: for every outcome your agent can produce, some rubric exists that measures it.
An inbound voice agent that books appointments and answers support questions produces a finite outcome space: the booking flow either completed or did not; the support question either resolved or escalated; the caller either got a policy-compliant answer or did not. Coverage means the rubric set spans that space — a fail-condition rubric for each discrete task, a resolution rubric for support, explicit checks for the disclosures your compliance posture requires, and a systemic check for the conversation-level behaviors (verbatim greeting, dead-air handling, clean handoff) that cut across tasks.
A 100% sample scored against a rubric that only measures politeness has worse coverage than a 1% sample scored against a rubric that measures the whole outcome space. When you find a failure in production that no rubric would have caught, the fix is a new rubric — that is how coverage grows, and it is the stopping point of the having-a-metric discussion: coverage is a checklist you verify outcome by outcome, not a percentage you raise.
How the Orbit quality cluster runs the workflow
Devotel Orbit ships the whole loop inside the dashboard's Quality area, one suite with five surfaces:
- The quality hub — the cross-channel scorecard rollup for inbox and voice together. LLM-judged outcomes are scored against your configured rubrics, on every paid tier, with no add-on SKU.
- Agents — per-agent scorecards with drill-down, so every conversation is one click from the aggregate. Sorting the agent list by score tells you which agent or intent needs attention first.
- Evaluations — the record of scored interactions, which is where a supervisor verifies the rubric actually fires on the failures it was designed to catch before any tuning.
- Leaderboard — gamification across agents: points, leaderboards, and badges fed by QA scorecards, handled volume, and CSAT, with a weighted scorecard for tuning what counts. In a hybrid org this keeps human and AI agents on one comparable scale.
- Practice — the remediation loop. A low-scoring conversation becomes a replayable scenario; prompt candidates and configuration changes are exercised against saved failing conversations before they touch production traffic.
The point of the cluster is that coverage, scoring, and remediation live in one place rather than three tools with exports between them. Scores can also flow into workforce management alongside human-agent adherence data, so a supervisor sees one operational picture instead of a human WFM system plus a separate AI dashboard.
What this does not do: evaluation does not guarantee anything. A passing eval tells you the scored conversations met the rubric as configured. The capability is evaluate-only — the assurance comes from your sample counting as the whole outcome space and from the promotion gate that practice provides, not from the scoring itself.
An example rubric for inbound support and bookers
Take an inbound agent that handles support questions and books appointments — a common single-intent deployment. A starter rubric set usually spans four categories:
- Outcome completion. A fail-condition rubric per discrete task. For the booker: "appointment confirmed with a time, date, and customer name recited back" — anything missing recitation fails. For support: "the caller's question was answered or the conversation reached a clean escalation with a ticket reference."
- Resolution quality. Did the agent resolve, or did it route? Scored separately because a transferred booking attempt and a completed one both count as "handled" if you do not split them.
- Policy and disclosure. The disclosures your program requires — recording notice, identity verification before account detail, quiet-hours statements in outbound — scored as explicit pass/fail checks rather than folded into a tone score.
- Conversation hygiene. Greeting delivered as configured, no dead-air stretches, no repeated questions, and a handoff that summarizes when it escalates. These cut across every intent and catch regressions that outcome rubrics alone miss.
Weight the categories deliberately: a fast booker that skips verification is not a good agent. Then track pass rates by intent, not just aggregate — an aggregate figure hides a broken booking flow behind high support scores.
Frequently asked questions
How many calls should an AI-agent QA program evaluate?
All of them, once rubrics exist. Because transcripts are scored by a model rather than a human listener, the cost of a full sample is small; the constraint that forces sampling on human programs is a staffing constraint, not a statistical one. Sampling still makes sense for human-AI hybrid queues that include human agents.
What makes an AI agent fail an evaluation versus a human agent?
Humans fail on behavior patterns — a skipped disclosure, a weak close. AI agents fail on configuration — a prompt that regressed an intent, a knowledge source that no longer returns the right policy, an empty tool response. AI QA is regression testing with a rubric, which is why failing conversations should feed a practice/promotion loop rather than a coaching session.
How does Orbit score evaluations?
Orbit's quality suite scores conversations against the rubrics you configure using an LLM judge, across inbox and voice together, included in every paid tier. The scored output is per-conversation verdicts with drill-down, aggregated into the per-agent scorecards that back the hub view and the leaderboard.
Does a passing evaluation guarantee production quality?
No — evaluation is evidence, not a guarantee. A pass means the scored conversations met the rubric as it was configured. The confidence comes from coverage (rubrics spanning the outcome space) and from gating prompt changes through a practice loop that exercises the candidate against saved failing conversations before promotion.
Where to go next
The voice AI agents overview covers the pillar this QA loop protects, and the explainer on LLM orchestration describes the runtime layer that rubrics constrain. Start with the rubric set above, get coverage on the outcome space, and route every regression found in production into a new rubric plus a practice scenario — that is the whole pattern.