How to Reduce Latency in AI Voice Agents — A Stage-by-Stage Latency Budget

An AI voice agent's response latency is the sum of four stages — speech-to-text (STT), the language model (LLM), text-to-speech (TTS), and the network and telephony path that carries the audio — and you reduce it by attacking each stage on its own, then overlapping them so they run concurrently instead of one after another. The largest, most reliable wins come from streaming every stage (never waiting for a complete result before starting the next), endpointing faster, caching the system prompt, tiering the model to the task, and putting the media path physically close to the caller; together these pull a naive ~2.5-second turn down into the ~1-second range where a phone call stops feeling like a walkie-talkie.

This guide breaks the latency budget down stage by stage, gives the concrete technique that cuts each one, and shows how Orbit by Devotel's voice architecture — a self-hosted media edge, fully streamed STT → LLM → TTS, per-agent model tiering, and speculative execution — is engineered against a published ~1.1-second p50 turn target. Every number here is sourced so you can verify it rather than take it on faith.

Why turn latency is the whole game

Humans are exquisitely sensitive to conversational timing. A widely cited cross-linguistic study of ten languages (Stivers et al., Proceedings of the National Academy of Sciences, 2009) found the gap between conversational turns clusters tightly around 200 milliseconds — faster than a person can consciously plan a reply, which is why a half-second of dead air on a phone call already reads as a hesitation, and a full second reads as "did the line drop?"

That is the bar an AI voice agent is measured against. A chatbot can take two seconds to render a reply and nobody notices; a voice agent that takes two seconds to start speaking sounds broken. So the target is not "fast" in the abstract — it is time-to-first-audio per turn, the interval from the moment the caller stops talking to the moment they hear the agent begin its answer. Everything below is about shrinking that one interval.

The latency budget, stage by stage

A single conversational turn passes through five measurable stages between "caller stops talking" and "caller hears the reply." The table below shows what each stage does, a typical industry contribution, and the per-stage target Orbit publishes on its voice-agent latency page.

Add those naively, in series, and a poorly built agent easily spends 2.5–3 seconds per turn. The trick is that most of these stages do not have to run in series. Once you stream, the LLM can be generating while STT is still finalising, and TTS can be synthesising the first sentence while the LLM is still writing the second. Orbit sums its per-stage targets to 1300 ms and then collapses roughly 200 ms of that through parallel overlap, landing a p50 turn target of 1100 ms and a p95 target of 1500 ms — published openly, with the caveat that measured production percentiles republish there as post-migration telemetry lands.

Nine techniques that cut each stage

Here is the checklist, ordered by impact. Each maps to one or more stages in the budget above.

1. Stream every stage end to end. This is the single highest-leverage change. Never wait for a complete STT transcript before prompting the LLM, and never wait for a complete LLM reply before synthesising audio. Feed partial transcripts into the model and feed the model's first sentence into TTS the moment a sentence boundary appears. Streaming is what turns a serial 2.5-second pipeline into an overlapped ~1-second one.
2. Endpoint faster — and smarter. Endpointing (deciding the caller is done speaking) is often the largest single stage. A fixed silence timeout trades latency for interruptions: too short and you cut people off, too long and every turn pays the penalty. Tune the silence window per agent, and prefer semantic endpointing that also reads the transcript ("I'd like to book a…" is clearly unfinished) over pure voice-activity detection.
3. Cache the system prompt and pre-warm turn one. A voice agent's system prompt and tool definitions are large and identical on every turn. Prompt caching keeps them pinned so the model does not re-read them each time, cutting time-to-first-token substantially. The first turn is the worst offender — a cold cache — so pre-warm it before the caller finishes their opening sentence.
4. Tier the model to the task. Not every turn needs your most capable model. Route simple, high-frequency intents (yes/no confirmations, menu routing, FAQ lookups) to a faster classifier-tier model and reserve the frontier reasoning model for turns that genuinely need it. A smaller model can shave 100–200 ms off time-to-first-token.
5. Fire the LLM speculatively on interim transcripts. Rather than waiting for the final transcript, start generating on a stable interim transcript so the model is already streaming tokens by the time STT emits its final result. If the final transcript matches, you have hidden the entire STT-final stage; if it diverges, you discard and re-fire.
6. Put the media path at the edge. Every 1000 km between the caller and your media server adds roughly 10 ms of one-way latency — and it is paid on every audio packet, both directions, for the whole call. Terminating and processing media close to the caller, rather than backhauling it across a continent, removes a floor you can never optimise away in software.
7. Choose a low-first-byte streaming TTS with a warm failover. The metric that matters for TTS is time-to-first-byte, not total synthesis time, because you play audio as it streams. Pick a synthesiser built for streaming, and keep a second provider warmed as failover so a slow first byte from one vendor does not stall the turn.
8. Chunk on sentence boundaries. Hand the TTS the first complete sentence as soon as it is available instead of waiting for the whole reply. Good boundary detection lets audio start playing while the model is still writing, which is where most of the parallel-overlap saving comes from.
9. Measure per stage and hold an SLO. You cannot cut what you cannot see. Emit a histogram per stage, per model, and set a turn-latency SLO so a regression in any one stage — a slower model, a mis-tuned endpoint, a distant caller — surfaces before customers feel it.

How Orbit's architecture minimises it

Orbit's AI voice agents are built directly on the techniques above rather than bolting a voice-AI layer onto a third-party telephony stack. The specifics, all shipped today:

• A self-hosted media edge. Orbit runs voice, video, and SIP on its own media plane rather than reselling a third-party media cloud, so the audio path — jitter buffering, transcoding, RTP transport — is under Orbit's control end to end. That is the "media edge" lever from technique 6: fewer hops between caller and processing, and a floor the platform owns instead of inherits.
• Fully streamed STT → LLM → TTS. Deepgram Nova-3 handles streaming STT, Anthropic's Claude Sonnet generates over a streaming /chat/stream endpoint, and Cartesia Sonic synthesises audio with an ElevenLabs Flash v2.5 failover armed. A streaming sentence-chunker overlaps the LLM and TTS stages so audio starts playing before the model has finished writing.
• Per-agent model tiering. Claude Sonnet is the production default for reasoning-heavy turns; a faster classifier-tier model (Claude Haiku) can be selected per agent to trim 100–200 ms off simple intents. The tier is a dashboard setting, not a re-integration.
• Prompt caching and cold-turn pre-warm. The system prompt and tool registry are pinned for the session, and an optional pre-warm heats the cache before the first turn so the opening reply does not pay a cold-start penalty.
• Speculative-on-partial. Orbit pre-fires the LLM on a stable interim transcript so the model is already streaming when STT emits its final result — technique 5, tunable per agent for teams that would rather keep a deterministic cold floor.
• A per-agent latency budget you set. Each agent carries a configurable time-to-first-token target (a soft observability signal) and a hard per-turn ceiling, so SLA-driven tenants can hold voice and text agents to the same clock.
• Measurement in the open. Every turn emits a per-stage, per-model histogram, and Orbit publishes the full E1–E5 budget and methodology on its latency benchmark page — described there as "the budget the production wire path is engineered against."

Because outbound calls terminate over Devotel's own wholesale softswitch rather than a resold aggregator hop, the telephony leg of the budget is also carried on infrastructure Orbit operates rather than rents.

How the numbers compare

Latency claims in this category are notoriously optimistic, so the honest comparison separates independently measured p50s from vendor-marketed figures. The measured column below comes from a common third-party test harness (tested.media, March 2026); the marketed column is each vendor's own published best case, several of which measure slower when tested.

Two honest caveats. First, Orbit's 1100 ms is a published internal SLO target, not a marketing best case — it is a conservative sum-of-budget figure, and Orbit's live measured percentiles are re-baselining after a media-plane migration and republish on the benchmark page as post-migration telemetry lands. Second, a p50 without a p95 hides the tail: an agent that averages 700 ms but spikes to 1.5 s on tool calls feels worse than one that holds a steady 1.1 s. Ask any vendor — Orbit included — for the p95 and the test conditions (region, model, prompt length, tool use), not just the headline p50.

Frequently asked questions

What is a good latency target for an AI voice agent?

Aim for time-to-first-audio under about one second at p50, and watch your p95 as closely as your p50. Human conversation turns over in roughly 200 ms (Stivers et al., 2009), so sub-second is where an agent stops sounding laggy; below ~800 ms it starts to feel genuinely conversational. Chasing a marketed sub-500 ms number matters far less than holding a consistent sub-second turn with a tight tail.

Which stage adds the most latency?

Usually endpointing (detecting the caller has finished) and LLM time-to-first-token. Endpointing is a direct trade-off against interruptions — shorten the silence window and you cut people off — so semantic endpointing that reads the transcript beats a blunt timeout. LLM time-to-first-token is dominated by prompt size and cold caches, which is why prompt caching and pre-warming pay off so heavily.

Does streaming really matter that much?

Yes — it is the single biggest lever. Without streaming, the stages run in series and the turn is the sum of all of them. With streaming, the LLM generates while STT finalises and TTS synthesises while the LLM writes, so the turn approaches the length of the longest single stage plus a little overlap rather than the sum. That is the difference between a ~2.5-second turn and a ~1-second one on the same components.

How does the network and telephony path affect latency?

It sets a floor you cannot optimise away in software. Round-trip time scales with distance — roughly 10 ms per 1000 km each way — and it is paid on every audio packet for the entire call, so a caller far from your media server pays it continuously. Terminating media close to the caller (a media edge) and keeping the telephony path on infrastructure you operate is how you keep that floor low.

How does Orbit keep voice-agent latency down?

Orbit streams STT, the LLM, and TTS end to end on a self-hosted media edge, tiers the model per agent (Claude Sonnet for reasoning, Claude Haiku for simple intents), caches and pre-warms the prompt, and speculatively fires the LLM on interim transcripts. It targets a 1100 ms p50 turn, lets you set a per-agent latency budget, and publishes its full per-stage methodology on the latency benchmark page. See the AI voice agents overview, the cloud phone system, and voice pricing for the surrounding platform, or compare directly against Vapi, Retell, and Twilio.

The takeaway

Reducing AI voice-agent latency is not one optimisation — it is a budget you manage across four stages, and the biggest single win is refusing to run them in series. Stream everything, endpoint intelligently, cache and pre-warm the prompt, tier the model, and put the media path next to the caller, and a naive multi-second turn collapses toward the one-second mark where a conversation feels natural. Whatever platform you build on, insist on a published per-stage budget and a p95 alongside the p50 — the vendors confident enough to show their methodology in the open, as Orbit does on its latency benchmark page, are the ones whose numbers you can actually trust.

Published 14 July 2026.