Every AI-agent vendor ships a guardrail configuration screen. You can write the PII rule, the topic denylist, the citation requirement, apply the policy to an agent, and get a satisfying green checkmark back. What most vendors do not ship is the other half of the loop: evidence that the rule you configured is actually executing against live traffic, at the rate you expect, on the agents you intended. Configuration is a promise; the firing signal is the fulfillment of that promise.
Devotel Orbit ships the monitoring half as a first-class surface: Agents → Guardrail Effectiveness. This guide is the operator's walkthrough — what the surface measures, how to read its panels, the diagnostic patterns worth memorizing, and the full loop from authoring a policy to proving it works.
The observability gap: configured versus firing
A guardrail policy goes through three states, and most dashboards only let you see the first two:
- Authored — the policy exists in the library. It has rules, a version, a name.
- Applied — the policy is attached to one or more agents. A toggle says so.
- Firing — on live conversations, the policy's rules are actually matching and acting: refusing a response, escalating to a human, capping spend, logging a low-confidence grounding.
The gap between state 2 and state 3 is where guardrail programs fail in production. A policy can be applied to every agent and still never fire, because nobody sends the traffic pattern it guards against, the match text was authored against the wrong string, or the agent was rebuilt and the policy attachment silently dropped in the new version. The applied toggle is green in every one of those failure modes. The only way to close the gap is an effectiveness metric, observed on live traffic, that names the firing count — not another configuration screen.
What the surface ships
Open Agents → Guardrail Effectiveness. The page's own description states the loop it closes:
> Monitor whether your AI agents' guardrails are catching violations on live traffic — refusal and block rates, violation trends over time, and per-policy effectiveness across your fleet.
The same aggregation is available to scripts and exports through the API at GET /api/v1/agents/guardrail-analytics — the dashboard is a rendering of that rollup, so anything you read on the page is reproducible from the API for a report or an export pipeline. Access is restricted to owner and admin roles; a member who opens the page sees a permission notice rather than an empty dashboard.
The metrics panel
The rollup decomposes every agent turn into an outcome, and the metrics panel is those outcomes read four ways.
Refusal rate — the share of turns the guardrail layer refused outright. This is the primary firing signal: a denial-rule matched and the response was stopped before it reached the caller.
Block rate — the hard-stop share: refusals plus cost-capped turns. Cost capping is the guardrail for budget, and it fires like any other rule, so it belongs in the blocked fraction.
Escalation rate — turns routed to a human rather than refused. An escalation is a firing too — the policy decided the agent should not handle this turn — and it reads separately from refusals because an over-weighted escalation share usually means the agent's scope is drawn too tight, not that the policy is too aggressive.
Violation trend — daily refusal, escalation, error, and cost-capped counts plotted over the look-back window you pick: 7, 14, 30, 60, or 90 days. The trend is what separates a one-day surge (an agent redeploy, a traffic-quality shift) from a sustained baseline level.
Per-policy breakdown — every applied policy with its firing counts and rates across all the agents carrying it. This is the attribution layer: you can answer "which of my policies fire" and "which policy on which agent," not just "guardrails as a class fired some amount."
There is also a grounding-confidence panel: among turns where the grounding check ran, the mean confidence score and the share scored below the low-confidence threshold. That panel watches a different failure mode — not policy violations, but factual drift — and a rising low-confidence share reads as "the knowledge base is going stale" before the refusal rate moves at all.
How to read the dashboard
Three readings cover most of what operators come to check.
Silent policies. A policy with zero firings over the window, on agents that see real traffic, is not automatically a problem — a perfect fraud-denylist will ideally fire zero times. But a silence reading is a question: is the absence of firings the absence of violations, or the absence of detection? The worked example below is the procedure for telling those apart.
Over-firing policies. A refusal rate double the fleet baseline on one agent is the false-positive profile. Do not read firings as automatically healthy: if your refund-terms denylist fires on two of every hundred order-status calls, the policy text is matching utterances it was never meant to touch. Page the exact firing counts by policy, find the one carrying the rate, then narrow its rule text — the dashboard tells you which policy owns the over-firing, so the fix is a policy edit rather than a blind re-prompt.
When a policy change is the fix. A spike in the trend that survives a window change (7-day agrees with 30-day) and traces back to a single policy version is a candidate for revision. Change the policy, republish the version, and watch the same panel again: if the firing rate returns to baseline within the window, the revision was the fix. The trend panel is the regression-test output for this loop.
The tenant-owned loop
Everything described above answers to a control you own end to end:
- Configure in Studio — author the policy in the reusable policy library, then apply it to agents through the per-agent safety configuration. Each application pins a policy version, so the firing signal below is attributable to a version you named.
- Verify on live traffic — open Guardrail Effectiveness, pick the window, page the per-policy breakdown. You are looking for the version you just applied showing a firing count consistent with what it guards: a citation-enforcement rule on a busy support agent should fire; a fraud denylist should mostly not.
- Adjust — revise the policy text, the threshold, or the application scope, and publish a new version. The next observation window grades the revision on live traffic, not on a synthetic test set.
The custom guardrail DSL announcement covers the authoring surface; the evaluation framework covers pre-promotion replay; this surface covers the live-traffic verification leg. The loop closes when all three are in routine use.
Worked example: the PII-redaction policy that went silent
Scenario: your tenant carries a PII-redaction policy — inline masking for credit-card and national-ID patterns — applied to six support agents. The dashboard window is 30 days, and the policy row shows zero firings on all six, all month. The applied toggle is green everywhere. Two hypotheses compete:
Hypothesis A: nothing to redact. Legitimate silence. Nobody speaks a card number to the support line, so the redactor never fires. This is the policy's ideal state when the violations are genuinely absent.
Hypothesis B: the matcher never matches. The rule was authored against a pattern that does not survive how people actually speak a card number — "four one one one, space, five five five five" does not match a digits-contiguous regex. The policy is applied, the violations arrive daily, and the firing count stays at zero because the pattern text is wrong for the traffic.
The diagnostic procedure runs entirely on tenant-owned surfaces:
- Check the traffic actually contains the pattern. Listen to a handful of recent transcripts on the suspected agents (the conversation record is your own — you hold the audit side of your own traffic). If card numbers appear on calls that show zero redaction firings, Hypothesis A is eliminated.
- Isolate the matcher against a known-bad utterance. In the policy editor, evaluate the rule text against a verbatim card-number utterance from step 1 as a test case. If the test case passes the matcher but live traffic does not, the matcher fails on the actual spoken formatting — rewrite the pattern for spoken-formatted digits, publish a new version.
- Re-observe. Stay on the same window; the new version's first firing should appear inside the next day of live traffic. The per-policy row now names the new version, and the firing count moves off zero for the first time in the month.
The entire diagnosis was possible because the surface attributes firings by policy and by version. An aggregate "guardrails: healthy" score would have hidden the silence behind the other five firing policies; the version stamp is what let you trust the re-observation. For the failure-mode taxonomy this example slots into, see the AI agent failure catalog.
Frequently asked questions
How much traffic do I need before the dashboard means anything? The counts are meaningful from the first turn — a firing count of zero over ten thousand turns says something; zero over eleven says almost nothing. Read any rate under a few hundred turns as directional only, and give a new policy at least one full day of ordinary traffic before concluding silence from it.
Can I attribute a firing to a specific policy when an agent carries several? Yes. The per-policy breakdown reports each applied policy with its own counts and rates across every agent carrying that policy, and the version stamp distinguishes a firing on the text you shipped last week from the text you shipped this morning. What the rollup deliberately does not do is attribute a firing to an individual conversation — that level stays in the conversation record; the dashboard is the fleet-level rollup.
Can I export the data or cite it in a report? The dashboard is a rendering of GET /api/v1/agents/guardrail-analytics, so anything on the page is available programmatically for the same owner/admin scope that can view the page. Pull the rollup on a cadence, land it in your warehouse, and the trend panel becomes a reportable control: "our published guardrail policy fired N times this quarter, rate R." The analytics export guide covers the general export pattern; the audit trail on the policy side follows the discipline in the audit-log deep link guide.
Does effective-monitoring change what my agents will do? No. The surface is read-only: it observes the firing signal that the policies you already applied generate. Changing what fires means changing the policy itself — the loop above, not this dashboard.
---
The monitor half of the guardrail loop had been the missing leg of most vendors' story — configure, trust, never verify. Devotel Orbit closes it on one page. Configure the policy, watch it fire, adjust it when the numbers say to, and the guardrail stops being a promise and starts being a measurement.