Event streaming to Kafka and batch HTTP — a Devotel Orbit guide

Quick answer: Webhooks deliver one event per HTTP call. When your consumer is Kafka, a SIEM, or a data pipeline, a fan-out of per-event receiver endpoints is the wrong shape. Devotel Orbit's Developer → Event sinks page streams the same platform event taxonomy your webhooks receive (message.*, call.*, and the rest of the catalog) into your own Kafka topic or a batch HTTP collector. This guide covers when to choose each transport, how to configure one, and what the delivered records look like.

Why a sink when webhooks already exist

A webhook endpoint answers "call me back for each event." A sink answers "push the event stream into my data plane." The difference matters at three places:

• A per-event HTTP receiver per event type is one more public endpoint your team has to host, secure, and keep available. A sink replaces it with a connection to infrastructure you already run.
• Kafka consumers expect ordered partitions, not HTTP requests. Reassembling a webhook fan-out back into a stream costs you a relay service in the middle.
• Throughput. An HTTP batch collector receives arrays of events per request — up to 1,000 envelopes per POST — instead of one connection per event.

If a single HTTPS endpoint behind a retry policy already does the job, keep the webhook. Sinks are for teams whose consumer is a stream, not an endpoint. For designing the receiver side when a webhook is the right call, see designing reliable CPaaS webhooks.

What Devotel Orbit ships

Open Developer → Event sinks in the dashboard. Two sink kinds ship, configured the same way and carrying the same event taxonomy your webhook subscriptions use:

• Kafka sink — bootstrap brokers (one host:port per line, up to 16), a topic name, an optional partition-key path, and SASL authentication (plain, scram-sha-256, or scram-sha-512).
• HTTP batch sink — an HTTPS collector URL and an optional max batch size (1–1,000 events per POST body, defaulting to 100). Orbit POSTs a JSON array of envelopes to the collector as batches fill.

Both kinds take an event filter — one pattern per line: an exact event type (message.sent), a prefix glob (message.*), or * / blank to receive everything. The filter uses the same semantics as a webhook endpoint's event list, so a sink and a webhook subscribed to message.* see the same stream.

Both kinds are also available as API endpoints — GET and PATCH /api/v1/developer/event-sinks/:kind where :kind is kafka or http_batch — if you manage configuration from code. The dashboard and the API read and write the same configuration.

Choosing a transport

• Choose the Kafka sink when your consumer is already a Kafka consumer: an analytics pipeline, a SIEM ingest path, a Flink or ksqlDB job, or a microservice that subscribes to topics. You get per-key ordering and your consumer group controls the read position.
• Choose the HTTP batch sink when the receiver is an HTTPS endpoint you own and the integration is request-shaped: a collector that validates, transforms, and forwards, or a queue behind an HTTP front door. You manage one endpoint regardless of event volume, and batching keeps connection counts flat.

Neither replaces the other's job. Teams that run both usually send the full stream to Kafka for analytics and a filtered subset (for example call.completed) to an HTTP collector that drives an operational workflow.

Partitioning on the Kafka sink

Ordering is the reason to put events through Kafka at all, and ordering holds only within a partition key. Orbit derives each record's partition key deterministically:

1. An explicit partition-key path you configured — a dot-path into the event envelope, for example data.contact_id.
2. If that path is unset or resolves to nothing, the first non-empty value among data.contact_id, data.conversation_id, data.message_id, data.call_id, data.to, then the event id.
3. If none resolve, a stable hash of the serialized event.

The practical result: every event for one contact — or one conversation, depending on where your identifiers live — lands on the same partition in publish order. Leave the partition-key path blank for the default, or set it when your consumers key on a different field (for example data.conversation_id when conversation ordering is the guarantee you build on).

The record value is the platform event envelope serialized as JSON — the same event body a webhook delivers — so one parser serves both the webhook path and the sink path.

Credentials

Each sink kind has one secret:

• Kafka sink: the SASL password, paired with the SASL username and mechanism you set.
• HTTP batch sink: a bearer token. Orbit sends it as the Authorization: Bearer header on every batch POST.

Secrets are encrypted at rest, and the dashboard never displays a stored secret back — it shows only whether one is configured. Type a new value to replace it; leave the field blank to keep the current one. Removing a stored secret is an explicit, confirmed action, and it stops delivery until you set a new one.

Collector URLs on the HTTP batch sink accept HTTPS only, and URLs pointing at private or link-local addresses are rejected — the same validation the webhook URL field applies.

Watching delivery health

An enabled sink reports its own health in the dashboard. The Enabled / Failing / Disabled status chip reflects the last delivery attempt, and the health line under it shows when the last attempt ran and how many events it delivered. A failing sink surfaces the error from the last attempt in place, so the first diagnostic step — wrong broker port, expired token, collector returning non-2xx — happens on the page, not in a log query.

The Enabled toggle is distinct from saving configuration: turn a sink off and the configuration stays intact, but no events are delivered until you turn it back on.

Set up your first sink

1. Open Developer → Event sinks.
2. Pick the Kafka or HTTP batch tab.
3. For Kafka: enter your bootstrap brokers (one host:port per line), the topic name, and — if your brokers require it — the SASL username, mechanism, and password. For HTTP batch: enter the collector URL and, if you want a bound smaller than the default, a max batch size.
4. Set an event filter if the sink should see a subset — otherwise blank delivers the full taxonomy.
5. Save, flip Enabled on, and watch the status chip. Generate a test event (send yourself a message) and confirm the delivery-health line shows the event count you expect.

If the chip reads Failing, the error text on the page names what the delivery attempt hit — broker unreachable, authentication refused, or a collector response you need to fix on your side.

Frequently asked questions

Do event sinks replace webhooks?

No. Webhooks stay the right delivery for request-shaped integrations — one HTTPS endpoint that handles a handful of event types. Sinks serve consumers that are streams (Kafka) or batch collectors, where per-event HTTP calls are the wrong shape. Both run side by side on the same event taxonomy.

What does one delivered record contain?

Each record carries the platform event envelope — event type, timestamp, id, and the event data — serialized as JSON and delivered as UTF-8 bytes. On Kafka, static headers you configured ride along as record headers; on the HTTP batch sink, the POST body is a JSON array of envelopes.

When does the HTTP batch sink flush a batch?

The delivery worker flushes at the configured max batch size (1–1,000 events). Leaving it unset uses the default of 100 events per POST.

How do Kafka records get partitioned?

By a deterministic partition key: your configured partition-key path if set, otherwise the first non-empty among the event's contact, conversation, message, call, and recipient identifiers. Every event for a given contact lands on the same partition in publish order, which is the ordering guarantee downstream consumers rely on.

How are credentials stored?

Encrypted at rest, and never returned by any read — the API and the dashboard report only whether a secret is configured. Replacing a secret means typing a new value; removing one is a separate, confirmed action that halts delivery until a new secret is set.

Is the same configuration available through the API?

Yes. GET /api/v1/developer/event-sinks/:kind returns the current config (with a credentials_configured flag, never the secret), and PATCH /api/v1/developer/event-sinks/:kind merges your changes. :kind is kafka or http_batch.