Designing Reliable CPaaS Webhooks: Delivery Logs, Retries, and Testing on Devotel Orbit

A webhook integration has two sides, and most articles about "reliable webhooks" only cover half of yours. The sender side — the schedule, the dead-letter queue, the retry timing — runs on our infrastructure, and the comparison of the two retry architectures we publish is in Webhook Retry Logic: Framework Retries vs Delivery-Guaranteed Retries. This guide is the receiver side: what Devotel Orbit emits, where you watch it travel, the four patterns your endpoint must implement, a working receiver in Node.js, and the test loop you run before the endpoint ever takes production traffic. Everything referenced here is already shipped — the Webhook Tester, request and delivery logs, and event sources and sinks all live under Developer in the dashboard, and every page linked below is in the current docs.

1. What Orbit actually emits

Orbit dispatches 260 event types across messaging, voice, contacts, campaigns, agents, flows, and account activity — the generated catalog is Webhook Events, and a subscription validator rejects any name not in it. An integration usually starts with a small slice. Subscribe to exactly the events you consume; ["*"] is a debugging tool, not a steady state.

Every delivery is one HTTP POST with a JSON envelope: a stable id (evt_...), a type, a created_at timestamp, and a data payload. Inbound events from different carriers also carry a normalized block so one consumer handles SMS, WhatsApp, and RCS inbound with one code path — documented in Normalized inbound event envelope.

2. The surface you build it on

Three surfaces in the dashboard, plus the docs, cover everything between "Orbit emitted it" and "your endpoint processed it."

Request logs and delivery logs. Under Developer → Request logs you see every delivery attempt per endpoint: the event id, the attempt number in the retry schedule, the status code your endpoint returned (or the timeout), and the captured request and response headers. This is the dashboard for "did it get there" questions. The wire-level anatomy, including how headers are captured per attempt, is in Webhook event payloads, and the inspection-plus-replay workflow is in Inspect webhook deliveries and replay failures.

Event sources and sinks. Developer → Event sources / Event sinks is the fan-out layer: an event source defines what to subscribe to, and a sink defines where matching events land. A webhook endpoint is one sink type; the same flow that retries HTTP deliveries and dead-letters exhausted events feeds these sinks, so replay and inspection apply to your fan-out too. The model is documented in Webhook fan-out and event sinks.

Webhook tester. Developer → Webhook tester sends a signed test event to your endpoint without going through production traffic — the loop covered in section 5, and in Test webhooks with the Webhook Tester.

3. The four receiver-side patterns

Orbit delivers at least once, on a published schedule: one attempt plus nine retries, exponential backoff starting at 30 seconds with jitter, roughly 4.3 hours total. Anything still failing then sits in the dead-letter queue for 7 days, replayable individually or in bulk. That published contract is what each pattern below turns into effectively-once processing on your side.

Verify the signature — before anything else. Every delivery carries an X-Devotel-Signature header in the form t=<unix_timestamp>,v1=<hmac_hex>. Compute HMAC-SHA256 of <timestamp>.<raw_body> with your endpoint's signing secret and compare in constant time. Reject on failure with a 4xx, which Orbit classifies as proven-dead — no retries burned on a request you never trusted. The same verification covers every retry and every dead-letter replay, because all three share the same signing format. Multi-language receivers are in Verify webhook signatures; failure triage is in Troubleshooting signature failures.

Persist processed event ids — deduplicate on `evt_...`. At-least-once delivery means a slow 200 response produces a redelivery; a dead-letter replay can arrive days after the original. Keep every processed event id for at least 7 days — the retry window plus the DLQ retention — and drop any redelivery whose id you have already seen. This is the step that converts "at least once" into "exactly once as far as your database can tell."

Answer the retry schedule deliberately. Return a 2xx within 30 seconds. Push slow work — database writes, fan-outs, third-party calls — to your own queue and acknowledge immediately. Reserve 5xx for genuinely retryable failure, and know the two sharp semantics your status codes carry: a 401, 403, 404, or 410 means proven-dead and skips all retries straight to the DLQ, and a 410 Gone answered to a specific event type permanently skips retries for that type. The lifecycle codes are cataloged in the troubleshooting guide.

Own a dead-letter drain. The 7-day DLQ is the final safety net, not a bug drawer. Replay from the dashboard — individually or in bulk — after you fix the endpoint fault. If you size your receiver for your own retries inside the 30-second window (pause between attempts, retry transient 5xx from upstream dependencies), remember the platform's schedule runs above yours: backoff is already exponential on our side, so exponential inside a single attempt just burns your window. The full schedule is in Webhooks overview.

A worked example: Node.js receiver

The Orbit Node SDK (@devotel-orbit/node) ships Orbit.webhooks.constructEvent, which verifies the signature and parses the envelope in one call, throwing a typed OrbitWebhookSignatureError on failure. This receiver handles the two delivery outcomes for outbound messages; the dedup step is intentionally first-class.

import express from "express";
import { Orbit, OrbitWebhookSignatureError } from "@devotel-orbit/node";

const app = express();

// Raw body, mounted ONLY on the webhook route — verify over raw bytes.
app.post("/webhooks/orbit", express.raw({ type: "application/json" }), async (req, res) => {
  let event;
  try {
    event = Orbit.webhooks.constructEvent(
      req.body.toString("utf8"),
      req.headers["x-devotel-signature"] as string,
      process.env.ORBIT_WEBHOOK_SECRET!,
    );
  } catch (err) {
    if (err instanceof OrbitWebhookSignatureError) {
      // Proven-dead per Orbit's classification: bad signature burns no retries.
      return res.status(400).json({ error: err.message });
    }
    throw err;
  }

  // Idempotency: persist evt ids for >= 7 days; replay-safe.
  if (await alreadyProcessed(event.id)) {
    return res.json({ received: true, duplicate: true });
  }

  switch (event.type) {
    case "message.delivered":
      await recordDelivery(event.id, {
        messageId: event.data.message_id,
        channel: event.data.channel,
        deliveredAt: event.data.timestamp,
      });
      break;
    case "message.failed":
      await recordFailure(event.id, {
        messageId: event.data.message_id,
        reason: event.data.error_code ?? "provider_failure",
      });
      break;
    default:
      // Unsubscribed-to events shouldn't arrive; acknowledge anyway.
      break;
  }

  await markProcessed(event.id);
  res.json({ received: true });
});

Two production notes on this shape. First, recordDelivery/recordFailure here are database upserts keyed on your own idempotency material (message_id + terminal status), so a replay rewrites the same row rather than appending — persist and dedup are two halves of the same mechanism. Second, in real deployments acknowledge the webhook before the queue insert when the downstream work is slow: res.json({ received: true }) first, then enqueue. The sender's 30-second timeout does not care how busy your database is.

4. Monitoring delivery gaps

Watching failure rates on your /webhooks/* route is not the same as monitoring webhook health. Three concrete signals close the loop.

Wire an alert on DLQ accumulation. A growing dead-letter queue is the earliest sign of receiver drift — an SSL renewal nobody applied, a config rollback, a DNS change. Orbit auto-disables an endpoint after 50 consecutive failures and notifies the org admin; treat that notification as an alarm, not a newsletter. Deliberate alerting on DLQ growth catches the drift long before the disablement does it for you.

Reconcile event arrival against message send volume. If you send 10,000 SMS and process 9,400 message.delivered + message.failed events, the missing six hundred are a gap to investigate in the delivery log — endpoint disabled, timeout pattern, or subscription mismatch (delivered subscribed, failed not). The per-attempt inspection page groups these by status code, so shape-first: are the failures timeout, 5xx, or proven-dead 4xx?

Log replays yourself. Dashboard replays are a manual remediation path. When you replay a DLQ event, note it in your own audit log with the evt_... id so your dedup window and dashboards explain the second arrival. The receiver in section 3 already logs duplicate: true — that line in your own logs is the difference between "duplicate ever happened" and "we know exactly which replay not to worry about."

5. Build-test loop with the Webhook Tester

Run this loop on every event type before any production send touches the endpoint, under Developer → Webhook tester:

1. Send the signed test event from the tester to your running endpoint, pointed at your development environment. Expected: 2xx, one row in your dedup store.
2. Replay the same event id and confirm the receiver returns 2xx with duplicate: true and no second side effect. This exercises the idempotency contract, not the network.
3. Break the signature — change one character of the signing secret in your environment, resend, and confirm the endpoint returns a 4xx immediately. The tester shows you the same headers the production dispatcher sends, so this is literally the validation that runs in production.
4. Simulate the failure taxonomy. Temporarily make your handler return 503, then 410, and watch the delivery log classify them: the first burns the retry schedule, the second dead-letters instantly. Fix the endpoint, then replay the DLQ entry and confirm the duplicate-drop path takes it.

Only after all four pass does the endpoint deserve the subscription. The tester's full walkthrough is in Test webhooks with the Webhook Tester.

Frequently asked questions

What webhook events does Devotel Orbit emit?

260 event types across messaging (message.delivered, message.failed, message.received), voice lifecycle (call.initiated through call.completed and recording.completed), contacts, campaigns, agents, flows, and account activity, plus inbound events normalized across carriers via the normalized envelope block. The canonical catalog is Webhook Events, and the SDK's typed WebhookEventType union mirrors it — TypeScript receivers get narrowing per switch arm for free.

How does Orbit retry failed webhook deliveries?

One initial attempt plus nine retries on a published exponential schedule starting at 30 seconds with up to 20% jitter — roughly 4.3 hours from first attempt to dead-letter queue. Proven-dead responses (401, 403, 404, 410) skip retries and DLQ immediately; 410 Gone also permanently skips retries for one event type. DLQ entries stay replayable for 7 days, individually or in bulk.

How do I verify Orbit webhook signatures?

Compute HMAC-SHA256 of <timestamp>.<raw_body> with your endpoint's signing secret and constant-time-compare against v1= in the X-Devotel-Signature header. In Node.js, Orbit.webhooks.constructEvent(rawBody, signatureHeader, secret) from @devotel-orbit/node does both verification and parsing, and throws a typed OrbitWebhookSignatureError so your catch clause can answer 400 immediately. Express receivers: mount the raw body parser on the webhook route only.

Do I need idempotency if Orbit guarantees delivery?

Yes — the guarantee is at-least-once, which means duplicates are part of the contract, not an incident. Persist processed envelope ids (evt_...) for at least 7 days — the full retry window plus the DLQ retention period — and drop any delivery whose id you have already seen. Replays from the dead-letter queue reuse the original event id, so the same dedup step covers retries and replays.

How do I test my webhook endpoint before going live?

Use the Webhook Tester under Developer in the dashboard: send a signed test event, replay the same event id to exercise your dedup, corrupt the secret to confirm 4xx handling, then simulate 503 and 410 responses and watch the delivery log classify them. Replay the resulting DLQ entry last. Section 5 above turns those four passes into a fixed pre-ship checklist.