Quick answer
Devotel Orbit's champion-challenger canary rollout surface promotes an AI agent's candidate version — the challenger — against the live version — the champion — by widening the challenger's share of real traffic through a percentage ladder. The default ladder is 5 → 25 → 50 → 100, and you can set your own percent steps when you start the rollout. Every step is gated by a quality scorecard — rubric pass rate, average sentiment, p50 and p95 response latency, thumbs-down rate, and error rate. A clean, sufficiently sampled stage advances the candidate to the next percentage; any regression rolls the candidate back off traffic entirely. One rollout per agent is active at a time, and its state persists with your organization's agent settings, so it survives promote/rollback of the live config.
How it differs from the classic A/B experiment
Orbit has two distinct online-experimentation surfaces for agents, and they answer different questions.
The classic A/B experiment ties two prompt versions (variant A and variant B) to a single conversion metric — reply received, goal completion, handoff avoided, and so on — and routes a configurable percentage of inbound conversations to variant B with sticky per-contact assignment. It computes per-variant stats and chi-square significance, and then stops: ending the experiment stores an optional winner stamp but never auto-promotes. Promoting the winner is a separate, deliberate call the operator makes. That operator-pull-only shape is intentional: one metric says whether B beats A, not whether B is safe for everyone.
The canary rollout is the complement: it is the promotion mechanism, not the measurement. A canary widens the challenger's live-traffic share step by step while a multi-metric scorecard watches every stage — so instead of asking "did variant B win on conversions?", it asks "is the candidate healthy at 5%? at 25%? at 50%?" and advances or rolls back on that evidence. The experiment decides whether a prompt deserves promotion; the canary is how it gets promoted without a big-bang cutover.
How to run a canary rollout end to end
The walkthrough below uses the dashboard, but every step maps to a public endpoint under /agents/:id/canary-rollout, so the same flow works from CI runbooks.
- Pick the challenger. Under the agent's versions page, choose the prompt version you want to promote — any version except the one currently live. Start the rollout on that agent. If this agent already has an active rollout, the start is refused — cancel or roll back the existing one first rather than orphaning its bookkeeping. Accepting the default ladder gives you
5 → 25 → 50 → 100; you can pass your own percentages (1–100, ascending) plus a minimum per-stage sample size, and choose whether a warn-level regression should roll back immediately or hold its stage. - Watch the scorecard. At each stage, the rollout compares the candidate's last seven days of live traffic against the equal-length window before it — the same windowing the quality-scorecard dashboard already shows, so a canary decision can never disagree with what you see there. The comparison reports each metric's baseline, current, delta, and severity, and refuses to judge a stage with too little traffic rather than advancing on noise.
- Check now. The evaluate action runs one scoring pass: if every metric is clean at a sufficient sample, the candidate advances to the next percentage; if any metric regresses, it is pulled off traffic and the rollout is marked rolled back; on the final stage, a clean pass completes the rollout. Unscored or insufficient data holds at the current stage with the reason surfaced in the history.
- Promote on completion. A completed rollout means the candidate proved out at full traffic — it does not silently rewrite your live config. Promote the candidate onto the agent from the versions page when you're ready; the canary gave you the evidence, the promotion is still your call.
- Keep a kill switch. At any point you can pull the candidate off traffic immediately regardless of the scorecard, or cancel a rollout started by mistake. The two differ only in the audit record: rolled back means it shifted traffic and got pulled; cancelled means it never should have.
Only one rollout per agent is active at a time, and the record is stored with your organization's agent settings rather than on any prompt-version row — so it persists across promote/rollback of the live config.
When to use which promotion mechanism
Orbit ships three complementary gates; choose per the shape of the risk:
- Canary rollout — when the candidate is already chosen and the question is safe live promotion. Use it for model swaps, prompt reweights, and tool-config changes where a staged cutover with automatic rollback beats a flip.
- Classic A/B experiment — when two contenders are still fighting for the job and the question is which one wins a single conversion metric. Run the experiment first; the canary promotes whatever wins.
- Regression pinning — the pre-promotion gate: replay a saved test corpus against a frozen version id before live traffic ever sees the candidate, as in the scoped-credentials and regression-pinning guide. Cheap and offline; it cannot tell you how live traffic behaves, which is exactly what the canary adds.
A healthy promotion pipeline runs all three: pin and replay offline, pick the winner with an experiment if you had two, then let the canary stage the cutover with the scorecard watching.
Frequently asked questions
Is the evaluate pass a scheduled job?
No. The "Check now" action in the dashboard drives advance/rollback — there is no background scheduler ticking the rollout today. A scheduled ticker is tracked separately, and the endpoint it will call already exists; until it ships, an operator (or your CI runbook) makes each pass explicitly, which also means every advance and rollback is attributable to whoever triggered it.
What counts as "enough traffic" for a stage?
The rollout compares a minimum sample you set when starting it (or the scorecard's default) against the current stage's actual conversation count. Below the floor, the decision is always hold with the insufficient-samples reason — the candidate never advances on noise.
Can two people start rollouts on the same agent?
No — the start is refused with a conflict when a rollout is already active for that agent. Cancel or roll back the existing rollout first; the persisted record keeps one rollout per agent exactly because the ladder, the stage index, and the history belong to a single promotion attempt.
Does a completed rollout promote the candidate automatically?
No. Completing the final stage means the candidate proved itself at full traffic; promoting it onto the live agent remains a deliberate versions-page action, the same deliberate promote the classic experiment requires.