A P50 latency number is the most reassuring figure in voice operations and the least truthful one. It tells you the middle session felt fine while the two failure modes callers actually complain about, the agent that talks over them and the agent that leaves a hole between turns, average themselves out of sight. The Voice Agent Quality Index (VAQI) exists because of that blind spot: it is the composite live-traffic read on your AI voice agents, and it is deliberately more than one time series.
Why a composite index beats raw latency percentiles
Three failure families share the latency space in a voice agent:
- Latency itself. Time-to-first-byte (how fast the agent starts speaking after the caller's turn ends) and the turn gap (the dead air between turns). VAQI carries both as averages and p95s, so one bad session cannot hide inside the mean.
- Turn-taking. Whether the conversation is a real exchange at all. A latency percentile cannot tell you that callers abandon or that the agent answers in monologues; turns-per-session and turn-gap trends can.
- Barge-in. The share of turns the caller interrupts, and the share of those interruptions the agent yields to cleanly. A caller who cannot interrupt experiences the agent as a talking wall, and no latency figure ever catches it.
A generic quality check that samples P50 latency alone grades the first family and files the other two under "soft stuff." A composite index keeps all three on one denominator, every completed session in the window you selected, so a degradation in any family moves the same read. That is the entire argument for reading VAQI instead of a percentile: the index is the smallest set of numbers that still separates "the bot takes forever" from "the bot never listens," two complaints that arrive as identical CSAT slides.
What the index composes
The Voice → Voice Agent Quality page (route /voice/vaqi, visible to owner, admin, and developer roles) aggregates the per-session quality records every completed AI voice session emits, over a window you pick: 24 hours, 7 days, or 30 days. The index headline cards are:
- Sessions. The denominator. Check it before trusting any average beneath it; a three-session window is noise, not a baseline.
- Latency. TTFB (average + p95) and turn gap (average + p95), with the dashboard warning markers at p95 over 1,200 ms and 1,500 ms respectively. Beneath them sits the pipeline split across average STT, LLM, and TTS latency, which tells you which stage moved when a latency card flashes.
- Turn-taking. Average session duration and average turns per session: is the agent actually conversing.
- Barge-in. The interruption rate and the clean-yield rate on those interruptions. High interruption with low yield is the talking-over failure; the same rate with high yield is healthy callers probing and correcting.
The full field list and the exact dashboard thresholds are documented per endpoint in the VAQI reference. What matters to the operator reading this post is that one window covers all three families at once, so a change in caller experience always has a mechanical name.
Bounding a window and comparing a model preset
The window selector is part of the page URL (?days=1, 7, or 30), so a link lands on a specific window; share it into an incident channel and everyone reads the same denominator. The discipline is:
- 24h for incident triage: "is the regression still moving?"
- 7d for the operating baseline (the default).
- 30d for trend review and setting thresholds; one bad afternoon should not move a thirty-day mean.
Comparing two configurations means bounding windows around a dated change. Say you moved an agent to a cheaper model preset. Read the 24h card before the swap and note the daily breakdown row; swap; then read the same card after. The daily table gives you per-day session counts, turn gap, and TTFB, so the step change is anchored to the deployment day rather than drift. The model-preset guide walks the same before/after read for the four curated presets (Balanced, High Intelligence, Ultra Fast, and Cost Saver), and the QA playbook covers the complementary rubric side. Use the preset picker to bound the experiment; use VAQI to grade it.
A worked sample: degrade barge-in after a TTS swap
The remediation path runs backward from the component that moved. Suppose a tenant swaps to a cheaper TTS voice on Monday. By Wednesday CSAT is sliding, and the index shows:
- Sessions at roughly 1,900 for the window. The aggregate is trustworthy.
- Barge-in rate up from 12% to 19%. Callers are interrupting more.
- Barge-in success down from 91% to 70%. The agent is no longer yielding when interrupted.
- Component cards pin the cause: average TTS latency doubled while STT and LLM held flat. The cheaper voice renders slowly enough that the agent keeps talking through the caller's interruption instead of stopping.
The remediation options, in the order a tenant-owned control panel presents them, are: roll back the TTS change; keep the voice but shorten end-of-turn sensitivity or the first response length so there is less speech to talk through; or fall back to a stricter model preset whose card bundles a faster voice. The index then verifies the fix on the next session batch. All three options are console-level configuration; nothing here touches carriers or routing.
When the index falls but the guardrails hold
Guardrails and VAQI measure two different things, and a falling index does not say which one moved. VAQI measures session mechanics: latency, turn-taking, interruption behavior. The eval guardrails comparison covers the policy side with tenant-authored rules about what the agent may say, enforced at runtime. When VAQI falls while guardrail firing rates stay green, you have a model or configuration choice, not a safety regression; the agent is slow or rude, not non-compliant. When guardrails fire while VAQI holds steady, you have a policy violation with healthy mechanics. Diagnose the falling number by family, and you stop re-prompting blind every time a chart moves.
Frequently asked questions
Why not just alert on P50 latency?
P50 averages away the tail. Callers remember the slow sessions and the overlap failures, both of which a median flattens into "fine." VAQI reports p95 for TTFB and turn gap and treats barge-in as a first-class family, so the tail the median hides becomes the number you alert on.
What is the difference between VAQI and an eval?
An eval is an offline correctness gate. It replays recorded or scripted conversations against a rubric and answers "did the agent say the right thing." VAQI is computed from live sessions and answers "does the agent feel fast and polite to real callers right now." A program needs both: evals gate correctness before promotion; VAQI watches mechanics in production.
Which window should I use?
Use 24h for triage, 7d for the weekly operations read (the default), and 30d for baselining thresholds. The narrowest window with enough sessions to make averages meaningful wins; check the Sessions card before trusting anything below it.
Where does per-session VAQI data come from?
Every completed AI voice session emits one quality record with duration, turn count, TTFB, turn gaps, the STT/LLM/TTS pipeline split, and barge-in behavior. VAQI rolls those records up over your selected window. The dashboard page refreshes the rollup every 30 seconds, so an open page on the 24h window works as a live wallboard during a change window.
What are the dashboard warning thresholds?
The page highlights TTFB when its p95 exceeds 1,200 ms and the turn gap when its p95 exceeds 1,500 ms. Those are default markers. The right thresholds for your use case come from your own 30d baseline, set on p95 rather than the average, and they remain tenant-owned just like the other quality thresholds on the page.
How do I compare two model presets fairly?
Bound the comparison with the window: read 24h before the swap, swap, then read again. The daily breakdown anchors the step change to the deployment day, and both presets grade against the same three families instead of whatever metric happened to move first.