Short answer: Every AI-voice vendor ships a quality check: a dashboard score, a CSAT readout, a hallucination spot-check. The comparison a buyer actually needs is narrower: which checks catch the sad paths, the failure modes the average-score dashboard passes right up until production traffic finds them. Six checks separate an eval-guardrail program from a generic quality score: a rubric grounded in a failure catalog, outcome-based scoring instead of tone scoring, simulation against saved failing conversations, a production-sampling policy rather than a one-off QA batch, tenant-authored guardrails with per-rule analytics, and a promotion gate that turns scores into a rollout decision. Devotel Orbit ships all six as named dashboard and API surfaces; the vendor matrix below grades the field against them.
1. Why generic quality checks miss the sad paths
A generic AI-voice quality check answers one question: was the average recent call acceptable? It produces a number, such as an average helpfulness score, a sentiment readout, or a sampled-listen opinion, and the number is honest about the traffic it saw. The sad paths are the traffic it did not see: the caller the agent misroutes, the policy the agent invents under a confident tone, the handoff that lands without a summary, the deterministic gate the agent retries forever, the call that completes the wrong task completely politely. Each of these passes an average-tone score, because tone is not what failed.
The AI-agent failure catalog names these modes directly: over-wide credentials, deterministic gates retried forever, flaky upstreams, unpinned prompts, runaway cost, and actions with no attribution. A quality program that cannot name the failure mode it caught is a score, not an evaluation. The playbook below turns that into a comparison: for each check, the question to ask every vendor, and the shipped surface a grading buyer should expect to see.
2. The six checks that define eval guardrails
Failure-catalog-grounded rubric. A rubric designed in a workshop is a hypothesis. A rubric grounded in a failure catalog is a checklist of what already broke. The comparison question is not "do you score quality", since everyone does. The question is "does your rubric have a category for each known failure mode, or does it fold policy and completion into a single score?"
Outcome-based scoring. Resolution quality separates "the task completed" from "the call was polite." The comparison question: does the score distinguish outcome completion and policy compliance as explicit pass/fail categories, or does it average them into tone?
Simulation against saved sad paths. A failing conversation is a test asset. The comparison question: can the tenant replay a saved failing conversation against the next agent version before it ships, or does regression testing mean another production sample?
Production-sampling policy. One-off QA batches go stale the day traffic shifts. The comparison question: is production evaluation a governed, ongoing sampling policy the tenant controls, or a manual QA exercise?
Tenant-authored guardrails with per-rule analytics. Built-in scanners are table stakes. The comparison question: can the tenant write its own policy rules and see per-rule hit analytics proving enforcement, or is the tenant limited to the vendor's fixed set?
Promotion gate. Scores that cannot stop a rollout are telemetry. The comparison question: does a passing or failing score gate deployment, and can the tenant wire the quality verdict into what the next agent version is allowed to change?
3. How Devotel Orbit ships each check
Rubric and outcomes. The AI voice agent QA post lays out the scoring surfaces (outcome completion, resolution quality, policy and disclosure as explicit pass/fail, and conversation hygiene) grounded in the failure catalog's six modes. The Voice Agent Quality Index (VAQI) doc defines the per-call 0–100 quality verdict the tenant reads, with the rubric categories visible instead of folded into a single blended number.
Simulation and the eval suite. Saved failing conversations replay against the next version through the evaluation lifecycle: the quality-evaluation-lifecycle concept doc documents the flow from production sampling through review, and the AI-agent evaluation framework post assembles the three lenses of live scoring, simulation against saved failing cases, and dial-plan checks into the one methodology the gate grades.
Sampling as a policy. Production evaluation on Orbit is a per-agent sampling policy the tenant manages, not a QA batch someone remembers to run. The how-to-benchmark guide is the measurement ground, and the sampling policy keeps the current agent under continuous evaluation as traffic shifts.
Tenant-authored guardrails. The custom guardrail DSL announce post ships tenant-authored rules beyond the built-in scanners, and the guardrail-analytics view shows hits per agent and per rule. The tenant writes the policy; the platform enforces it; the analytics prove enforcement.
The gate. The evaluation framework post's closing move is the gate: a score the rollout respects. The guardrails accuracy-training comparison post grades the adjacent levers (grounding citations, guardrails, model presets, prompt templates), and the promotion gate is what turns those levers' scores into a deploy decision instead of a report.
4. The vendor field against the six checks
Grading vendors here is a disclosure exercise, and the honest pattern across the field is that generic checks pass two of six and stop. Built-in scanner suites are common; outcome-explicit rubrics, replayable sad-path suites, and a gate are rare. Enterprise platforms (Kore.ai, Cognigy, PolyAI, Ada) ship partial versions of several cells, and the sibling Orbit vs Kore.ai, Orbit vs Cognigy, and Orbit vs PolyAI posts carry the per-vendor disclosure this matrix joins. The voice-AI specialists' own head-to-heads live on the Vapi vs Retell framework post.
| Check | Voice-AI specialists (Vapi, Retell, Bland, Synthflow) | Enterprise platforms (Kore.ai, Cognigy, PolyAI, Ada) | Devotel Orbit |
|---|---|---|---|
| Rubric grounded in a published failure catalog | No | Partial | Yes |
| Outcome-based scoring (completion and policy explicit, not averaged into tone) | Partial | Partial | Yes |
| Simulation against saved failing conversations | Partial | Partial | Yes |
| Production-sampling policy (not a one-off QA batch) | No | Partial | Yes |
| Tenant-authored guardrails with per-rule analytics | No | Partial | Yes |
| Promotion gate wired to the quality verdict | No | Partial | Yes |
Where each check wins. The specialists' honest concession mirrors the accuracy-matrix concession: they lead on open model marketplaces and speech naturalness, and a buyer whose core requirement is that marketplace should weigh them first. The enterprise platforms ship the partial pattern of scanner sets plus some rules, some analytics, and some gating, which is why their cells read Partial rather than No. On the six sad-path checks themselves, the shipped Orbit surfaces are the reference, and the cross-linked posts above carry the per-cell argument rather than this matrix restating it.
5. How to run this comparison on your own traffic
Ask each finalist three questions in the demo. First: show a real failing conversation from production and replay it against the candidate version. If the vendor cannot, simulation is a slide, not a surface. Second: open the rubric and ask which failure-catalog mode each category catches. If policy is a tone deduction, the sad paths are unmonitored. Third: ask what a failing score stops. If the answer is "we review it weekly," there is no gate. The evaluation framework post is the extended rubric; the QA post is the scoring detail; the VAQI doc is the verdict format. Six checks, three questions, and the vendor field sorts itself.
Frequently asked questions
What is the difference between an eval guardrail and a generic AI-voice quality check?
A generic quality check scores the average recent call as a blended number over traffic it sampled. An eval guardrail is a check targeted at a named failure mode: an explicit rubric category, a replayable sad-path suite, a sampling policy that runs continuously, and a gate that stops a failing rollout. The generic check describes traffic; the guardrail catches the failure class the average hides.
Why does the sad-path framing matter more than average-call quality?
Because production finds the sad paths first. The failure catalog enumerates the modes that pass an average-tone score, among them misroutes, confident policy invention, summary-less handoffs, and deterministic-gate retries, and each one is invisible to a blended score. A quality program that cannot name the mode it caught passes the same traffic the incident finds.
Which Orbit surfaces ship the six checks?
The VAQI per-call verdict and rubric categories (VAQI doc), the evaluation lifecycle from sampling through review (lifecycle doc), the tenant-authored guardrail DSL with per-rule analytics, the per-agent production-sampling policy, and the promotion gate the evaluation framework defines. The QA post and the evaluation framework post carry the scoring and methodology detail.
Where do specialists or enterprise platforms genuinely beat this matrix?
Specialists genuinely lead on open model-marketplace choice and speech-model naturalness; enterprise platforms ship partial versions of most cells. The concession is the same one the accuracy-training comparison makes: on the six sad-path checks themselves, Orbit ships the readiest guardrail surface.
How does this comparison join the existing vendor posts?
The enterprise cells join the Orbit vs Kore.ai, Orbit vs Cognigy, and Orbit vs PolyAI disclosures; the specialist cells join the Vapi vs Retell framework post. This matrix is a cross-vendor join over those published cells, not a new claim. The six sad-path checks are the rubric the prior posts' cells already grade against.