BroadNet
HTTP API & webhooksWebhooks

Webhook delivery and reliability: retries, ordering, dead-letter

Updated 6 min readStandards-based
Direct answer

A webhook is a server-to-server HTTP callback: the messaging platform POSTs an event to the consumer's URL. Reliable delivery is at-least-once, so producers retry failed deliveries with exponential backoff and jitter, dead-letter after a cap, and consumers must be idempotent because duplicates and out-of-order events are expected.

DiagramHow an A2P message reaches the phone
  1. Your applicationSends over the REST API or SMPP
  2. BroadNet platformValidates, routes and bills each message
  3. Operator SMSCDirect operator connection
  4. Subscriber handsetThe message is delivered
The delivery receipt (DLR) travels back along the same path to your webhook
§01

Webhook architecture

A webhook inverts the usual client/server roles: instead of the application polling the messaging platform for updates, the platform POSTs an event to a URL the application registered — a delivery-status change, an inbound message, an account event. This is the push half of the async model whose pull half is polling GET /messages/{id}. The consumer's endpoint is an ordinary HTTPS server that must accept a POST, do the minimum needed to durably record the event, and return a 2xx quickly. A common and important pattern is to acknowledge on receipt and process asynchronously: the endpoint writes the raw event to a queue or log, returns 200, and a separate worker does the real work — so slow business logic never causes the producer to time out and retry.

A delivery-status webhook POST to the consumerhttp
POST /webhooks/dlr HTTP/1.1
Host: app.example
Content-Type: application/json
Webhook-Id: 8b1a...
Webhook-Timestamp: 1752916522
Webhook-Signature: v1,3f2b...

{
  "event": "message.delivered",
  "message_id": "01HZY8Q3M6",
  "status": "delivered",
  "occurred_at": "2026-07-19T09:15:25Z"
}
§02

Delivery guarantees: at-least-once

Distributed delivery cannot be simultaneously exactly-once and available, so nearly every production webhook system chooses at-least-once: the producer keeps retrying until it sees a success acknowledgement, accepting that a consumer may occasionally receive the same event more than once. The alternative extremes are worse — at-most-once drops events on the first failure, and true exactly-once requires distributed coordination the HTTP transport does not provide. At-least-once pushes one obligation onto the consumer: it must be idempotent, deduplicating on a stable event identifier so a redelivered event is harmless. The acknowledgement contract is simple: any 2xx means 'stop retrying'; anything else (or a timeout) means 'retry later'.

§03

Retries with exponential backoff and jitter

When a delivery fails or times out, the producer retries on a schedule that grows exponentially — a base delay doubled each attempt up to a ceiling — so a briefly unavailable endpoint is not hammered. Crucially, the delay includes random jitter: without it, a fleet of events that all failed during the same outage would retry in lockstep and produce a synchronised thundering herd the moment the endpoint recovers. Full jitter (a random value between zero and the current backoff bound) spreads that load smoothly. Retries continue until the event succeeds or a total attempt or time cap is reached, after which the event is dead-lettered rather than retried forever.

Exponential backoff with full jittertext
base = 2s ; factor = 2 ; cap = 3600s
attempt n bound = min(cap, base * factor^(n-1))
delay(n)        = random(0, bound)      # full jitter

n=1  bound  2s    delay in [0, 2s]
n=2  bound  4s    delay in [0, 4s]
n=3  bound  8s    delay in [0, 8s]
n=4  bound 16s    delay in [0, 16s]
...  until success or attempt/time cap -> dead-letter
§04

Timeouts and ordering

A producer sets a short per-delivery timeout — commonly a few seconds — and treats a timeout the same as a failure: the connection is abandoned and the event is scheduled for retry. This is the mechanical reason consumers should ack fast and process asynchronously. Ordering is the harder guarantee. Because retries and parallel delivery interleave, a naive webhook stream is only best-effort ordered: a retried earlier event can arrive after a later one. Systems that need order either deliver per-key in sequence (all events for one message id on a single ordered partition, blocking on failure) or, more commonly, make the payload self-ordering — each event carries a timestamp and a monotonic sequence so the consumer can discard a stale update rather than depend on arrival order.

Ordering strategies for webhook streams
StrategyHowCost
Best-effortDeliver as available, retry independentlyEvents can arrive out of order
Per-key orderedSerialise all events for one id; block on failureHead-of-line blocking, lower throughput
Self-ordering payloadCarry timestamp + sequence; consumer discards staleConsumer logic, but no blocking
§05

Dead-letter and endpoint health

An endpoint that stays down must not consume retry capacity forever. After the retry policy is exhausted, the event is moved to a dead-letter queue — a durable store of undeliverable events that an operator can inspect and replay once the endpoint is fixed — rather than silently dropped. To protect both sides while an endpoint is unhealthy, producers track per-endpoint delivery health and apply circuit-breaking: after a sustained run of failures the circuit opens and new events are parked or fast-failed instead of being attempted, then a half-open probe periodically tests recovery before normal delivery resumes. This stops a dead consumer from generating a backlog of doomed attempts and lets a recovering one ramp back up gracefully. BroadNet's concrete webhook and retry behaviour is documented for developers.

§06

Building an idempotent consumer

Because at-least-once delivery guarantees duplicates, the consumer — not the producer — owns exactly-once effects. The pattern is a dedup store keyed on a stable, producer-supplied event identifier (a Webhook-Id or event id, never the payload hash alone). On receipt the consumer checks whether it has already processed that id; if so it acknowledges with 2xx and does nothing else; if not, it records the id and applies the effect in the same transaction so a crash cannot apply the effect without recording the id. Combined with self-ordering payloads, this makes a redelivered or out-of-order event a no-op, which is what turns an at-least-once transport into exactly-once business outcomes.

Idempotent consumer (pseudocode)text
on POST(event):
  verify_signature(event)          # reject if invalid
  id = event.headers['Webhook-Id']
  begin transaction:
    if seen(id): return 200        # duplicate -> no-op
    if event.sequence <= last_seq(event.message_id):
        mark_seen(id); return 200  # stale -> discard
    apply_effect(event)
    mark_seen(id); set_last_seq(...)
  commit
  return 200
FAQ

Frequently asked

What delivery guarantee do webhooks provide?

At-least-once. The producer retries until it sees a 2xx acknowledgement, so a consumer may receive the same event more than once. Exactly-once over plain HTTP would require distributed coordination the transport does not provide, so the consumer is expected to be idempotent and deduplicate on a stable event id.

Why add jitter to webhook retry backoff?

Without jitter, every event that failed during one outage retries in lockstep when the endpoint recovers, creating a synchronised thundering herd that can knock it down again. Full jitter — a random delay between zero and the current backoff bound — spreads the retries smoothly so the recovering endpoint sees a gradual ramp.

How should a webhook endpoint respond?

Quickly and with a 2xx. The recommended pattern is to durably record the raw event, return 200 immediately, and process asynchronously in a separate worker. That way slow business logic never causes the producer's delivery timeout to fire and trigger an unnecessary retry.

Are webhook events delivered in order?

By default only best-effort — retries and parallel delivery can interleave, so an earlier event may arrive after a later one. To get order, either serialise per key (at the cost of head-of-line blocking) or make the payload self-ordering with a timestamp and sequence number so the consumer discards stale updates.

What is a dead-letter queue for webhooks?

A durable store for events that could not be delivered after the retry policy was exhausted. Instead of dropping them silently, the producer parks them so an operator can inspect the failures and replay the events once the consumer endpoint is fixed.

Why apply circuit-breaking to webhook endpoints?

A persistently failing endpoint would otherwise consume retry capacity with doomed attempts and build a backlog. Circuit-breaking opens after sustained failures, parking or fast-failing new events, then uses a half-open probe to test recovery before resuming normal delivery — protecting both the producer and the recovering consumer.

References

Source specifications

The published standards this page is grounded in. Opens in a new tab.

BNMaintained by the BroadNet engineering team·Carrier-grade messaging since 2003
Put it on the wire

Build with a real SMPP bind and REST API.

Request test access with real test keys: send to your own verified number first, then move to production across 200+ countries once your sender identity and routes are approved.