When an AI agent calls a paid API, the failures that cost you money cluster into six modes, and each one has a distinct recovery: scope the credential, fix a deterministic gate rather than retrying it, make upstream flakiness retryable, gate prompt promotion on regression replay, set a per-conversation cost ceiling, and refuse tool calls that carry no delegation chain. This post is the catalog; each mode links to the Devotel Orbit docs page that holds the contract detail, so it stays a map rather than a duplicate.
If the orchestration framing is new, start with What is LLM orchestration, and where does it live in your stack?. The capabilities this catalog leans on — scoped token exchange, pinned regression replay, agent versions — shipped in the updates covered by Scoped agent credentials, pinned regression replay, and the Agent ROI guide.
The failure matrix
| # | Aspect | Symptom | Root cause | Recovery step |
|---|---|---|---|---|
| 1 | Credential scope | An agent reaches a system it was never meant to touch, or a spill over-bills a downstream API | The agent ran on an ambient session token wider than the task | Exchange the token for a shorter-lived credential narrowed to exactly the tools the task needs |
| 2 | Deterministic gates | The same tool call fails the same way on every retry | A fixed gate — malformed input, an unapproved template, channel setup left incomplete | Fix the gate; a retry re-runs the identical check and fails identically |
| 3 | Upstream flakiness | A tool call fails once, succeeds on a later attempt, leaves ambiguous state between | The upstream endpoint timed out or returned a transient 5xx | Retry with an idempotency key, so a retry replays the original result instead of executing twice |
| 4 | Prompt regression | A prompt update degrades the behavior a customer notices | The change was promoted with no evidence it still passed the saved corpus | Pin the regression suite to a version and score the candidate before promotion |
| 5 | Runaway cost | One conversation loops tool calls and burns the tenant's AI budget | No per-conversation cost ceiling, or the ceiling never moved from its default | Set the conversation cost ceiling before the agent goes live, not after the first surprise |
| 6 | No attribution | An audit cannot say which agent acted or for whom | The tool call carried no delegation chain | Route the call through token exchange so the credential itself names agent and sponsor |
Mode 1 — the over-wide credential
The most expensive failures are usually authorization problems, not model problems. An agent that runs on whatever session token it happened to hold can reach systems it has no business in, and when it spills, the blast radius is everything that token could do.
Devotel Orbit's agent runtime speaks OAuth 2.0 Token Exchange (RFC 8693) at POST /agents/:id/token-exchange. The agent presents its existing token plus its actor token and receives a shorter-lived credential narrowed to the task — never wider than it already held. The exchanged token carries a signed delegation chain naming the agent and the human sponsor behind it. Recovery for this mode is preventative: scope every paid downstream call before it runs.
Mode 2 — the deterministic gate, retried
Two failure classes look identical at first — a send that fails — and only one of them is retry-safe. A malformed recipient value, a template that never reached approval, a channel whose setup was never finished: these fail the same way forever, no matter how many times you re-send. The distinguishing question is whether the gate is deterministic. If it is, the recovery is to fix the gate, and the queue-side equivalents are the message stuck in queued page. For the full symptom-to-fix decode of each error class, see a flow execution that failed — its retry-versus-fix table is the mode in miniature.
Mode 3 — the flaky upstream
The other class is the upstream that failed once and would succeed on a second attempt. The risk there is not the failure — it is the duplicate: retry a send without an idempotency key and you can charge a customer twice or deliver the same message two times. Orbit's creation endpoints accept an Idempotency-Key header; a retry with the same key replays the original result instead of executing again. The contract lives in the idempotency and safe retries concept page, and the companion Idempotency and safe retries for CPaaS APIs post walks the failure scenarios end to end.
Mode 4 — the unpinned prompt
Prompt regressions are the failure mode that disguises itself as success: the agent still answers, the answers are subtly worse, and nothing threw an error. The root cause is almost always promotion without evidence — a prompt change went live without being replayed against the cases that matter.
Saved regression tests can carry a pinned_version_id, so a replay runs each conversation against a frozen prompt instead of whatever is live, and a run can override the pin with a candidate version id to score the whole corpus before anything ships. Because every snapshot is immutable, promotion and rollback are the same operation: the recovery for a regression that got through is POST /agents/:id/versions/:vid/promote on the last good version — a one-call rollback, recorded as its own auditable entry. The full contract is Agent versions; the staged ladder beyond it is the AI agent rollout pipeline.
Mode 5 — the unlimited loop
A tool-calling agent without a ceiling is an agent with a blank check. One conversation that loops — a retrieval step that never resolves, a tool that keeps being re-invoked — will spend whatever it is allowed to spend. Orbit agents carry a per-conversation AI cost cap that stops a runaway conversation before it becomes a billing incident; the cap is a tenant-owned control, and it is useless if it sits at its default while the agent is live. The knobs and their behavior are on the agent cost controls page, and the margin side — what a conversation should be worth — is the Agent ROI attribution guide, summarized in LLM spend cost governance for FinOps.
Mode 6 — the unattributable action
The audit failure is the one you discover after the incident. A tool call succeeded, something downstream changed, and no record answers which agent did it on whose behalf. Token exchange closes this mode the same way it closes mode 1: a credential minted without an attribution chain is refused, so every executed action carries its own proof. If your compliance posture treats AI agents as regulated actors, the map of what to document is in the EU AI Act readiness page — the controls described there are tenant-configured and tenant-owned.
Reading the matrix in production
The six modes are not independent. A delegation chain (mode 6) is what makes an over-wide credential (mode 1) visible; a version pin (mode 4) is what makes a regression recoverable in one call; a cost ceiling (mode 5) is what turns a looping tool call from a billing incident into a stopped conversation of bounded size. Operators running agents in front of customers should treat the matrix as a pre-flight checklist: scoped credentials on every paid call, an idempotency key on every retryable send, a pinned regression suite gating every promotion, and a ceiling on every conversation.