A voice AI agent is software that holds a real phone conversation with a person. It listens, understands what the caller wants, decides what to do, speaks back, and can hang up having actually resolved the call. Three things always run underneath it: a speech-to-text model, a language model, and a text-to-speech model, wired into a telephony stream. Every production voice agent in market is that pipeline, whether it is built on Orbit, Vapi, Retell, or a custom stack. This page introduces the pipeline stage by stage, explains the parts that decide and act, shows how an Orbit agent carries a real call end to end, and closes with the build-and-evaluate sequence a team repeats.
It assumes no prior voice-AI exposure. If you want the marketing-level definition first, start with What is Voice AI?; this page is the technical introduction it points to.
The pipeline: speech-to-text, then the model, then text-to-speech
Every agent, on any platform, turns caller audio into model input, gets a text answer back, and turns that answer into audio again. The same three stages run in the same order each turn:
- Speech-to-text (STT / ASR). Caller audio streams into a speech model that emits partial transcripts the moment words finish, so the turn can start before the caller has finished their sentence. Accuracy under real phone codecs and background noise decides turn quality, before the model runs at all.
- The model (LLM). A language model receives the running transcript plus the agent's own prompt (instructions, knowledge source, and the tools it may invoke) and picks the next response or action. Everything on this page about prompt and tools is about giving this stage less to guess.
- Text-to-speech (TTS). The model's answer streams into a speech-synthesis model that renders audio the caller hears, usually while the sentence is still forming. Streaming matters here too: waiting for the full sentence before speaking is the engineering equivalent of a pause.
Barge-in closes the loop: if the caller speaks over the agent mid-reply, the speech-to-text stage keeps listening and the pipeline treats it as the start of the agent's next turn, rather than ignoring the interruption. Low turn latency, the time from the caller's last word to the agent's first, is what makes this feel like a conversation rather than a walkie-talkie. The voice-agent latency benchmark names the budget Orbit measures each stage against.
The parts that decide and act
Two prompt-side parts decide everything the pipeline says and does, and they are both configured before the first call, not learned at runtime.
The system prompt. Standing instructions that tell the model who it is, what the call is for, what tone to hold, and, critically, what it must not do and when it must hand off to a human. The whole agent's personality and safety envelope lives in this text; vague prompts sound fine in a demo and cause the wrong escalation on the hundredth real call.
Tools (function calls). The structured capabilities the model may invoke mid-turn: look up an order, check appointment availability, book a slot, open a ticket, mark a number do-not-call. Each tool carries a name, a description, and a typed parameter schema; when the model fills it, the agent runtime executes the call and feeds the result back as an observed fact. This is how an agent does things rather than just talk, and it is what separates a production agent from a speaking demo.
Walkthrough: how an Orbit agent answers a call
- Answer. A call arrives over Orbit's voice network (Devotel's carrier-of-record softswitch) and routes to the AI agent assigned to the dialed number, same as any inbound contact.
- Listen. Caller audio streams into the speech-to-text stage; partial transcripts accumulate.
- Decide. The transcript meets the system prompt plus the agent's tool list; the model responds or invokes a tool.
- Act. The runtime executes the named tool, for example checking the caller's open orders, and hands the result back as an observed fact.
- Speak. The final composed answer synthesizes to audio; barge-in stays armed throughout, so an interruption cleanly starts a new turn.
- Hand off if needed. On a handoff rule (low confidence, a restricted topic, or an explicit caller request) the call transfers to a human queue with the live transcript attached, so the person picks up with full context rather than a fresh hello.
An Orbit agent is configured in the dashboard once: prompt, tools, and handoff rules. That same agent then serves inbound voice and every messaging channel from one account, which is the "configure once" property the Devotel Orbit Voice AI Agents product page lists among its capabilities. Because termination is Devotel-owned rather than resold, latency and caller-identity signing (STIR/SHAKEN) stay under the same owner as the pipeline itself.
How to build and evaluate your first agent, step by step
Define the scope
Pick one narrow call type the agent owns completely, such as appointment status or order lookup, before broadening.
Write the system prompt
Name the agent's role, the task, the tone, the forbidden topics, and the exact handoff conditions.
Wire the tools
Give it only the specific capabilities that call type needs, each narrowly scoped and typed.
Assign the number
Route a dedicated inbound number to the agent from the Orbit dashboard so it answers live traffic.
Evaluate against real calls
Score recorded conversations for resolution, wrong-tool invocations, and premature handoffs; tighten the prompt and tool list, then rerun.
Ship, then expand
Only after evals stay green; each new call type gets its own narrow pass through the same sequence.
Frequently asked questions
What is a voice AI agent?
One system prompt, one tool list, and one handoff policy on top of the speech-to-text → model → text-to-speech pipeline every production agent runs. The combination answers a real call, reasons through it, acts where tools allow, and hands off when it should not continue.
How does a voice AI agent differ from an IVR?
An IVR navigates a fixed tree by keypresses or keywords; a voice AI agent holds a free-form conversation, invokes tools, and hands off with context, so callers say what they need rather than hunting through a menu. The AI agents vs traditional IVR guide compares them directly.
Which models should I choose?
Pick per stage: a speech-to-text model tuned for phone audio, a reasoning model that matches task complexity, and a natural text-to-speech voice. Orbit ships a curated default for each and exposes alternatives per agent where a workload needs a different trade.
What do I need before my first agent?
A narrow call type to own, standing instructions naming the task and handoff conditions, and only the tools that call type needs. Then point a dedicated inbound number at the agent and run an evaluation loop scoring real recordings before expansion.
Can the agent hand off to a human?
Yes. A handoff rule (low confidence, restricted topic, explicit request) transfers the call to a human queue with the live transcript attached, so no caller context is lost and the human picks up mid-conversation rather than restarting it.
How fast does it have to respond?
Low turn latency is the difference between a conversation and a walkie-talkie. Several seconds of dead air breaks a call, so any serious voice-AI provider states per-stage timing publicly. Orbit publishes its per-stage voice-agent latency budget on the voice AI agents page.
Sources and further reading
- Vapi: Introduction: the instructional shape this introduction deliberately mirrors. The pipeline named, then a real walkthrough, then a build sequence.
- What is Voice AI?: the broader definitional framing this page assumes.
- Devotel Orbit Voice AI Agents: the product page the Orbit-side walkthrough references, including the per-stage latency benchmark and the Orbit-vs-Vapi-vs-Retell AI comparison.
- Glossary: AI voice agent: the term-level definition of the speech pipeline stages.
Published 19 September 2026. Part of the Orbit resources library: foundational guides for teams building on communications infrastructure.