Designing Webhooks Developers Can Trust

Every webhook system works perfectly in the demo. Then a consumer's server goes down for forty minutes, your delivery worker gives up after three tries, and a customer discovers a week later that 200 orders never synced. Webhooks are the most trust-sensitive surface of your API, and most teams design them like a fire-and-forget HTTP call.

Webhooks are distributed systems in disguise

A webhook is your server making an HTTP request to someone else's server, over the public internet, to a machine you do not control, run by a team you have never met. That is a distributed systems problem wearing a convenience feature's clothes. Every classic failure mode applies: the receiver is down, the receiver is slow, the receiver returns 200 and then crashes before persisting, DNS flakes, TLS certificates expire, a load balancer times out mid-request.

The teams that get this right stop thinking "we send a POST when something happens" and start thinking "we operate a delivery pipeline with explicit guarantees." That shift changes every design decision that follows. It is the same mental move we describe in our guide to event-driven architecture: the messaging layer does not remove complexity, it relocates it, and you have to decide where it lands.

Promise at-least-once, design for duplicates

You cannot deliver a webhook exactly once. Nobody can. The receiver might process your request and die before responding, and from your side that looks identical to a failure. So you retry, and now they have the event twice.

The honest contract is at-least-once delivery, stated loudly in your docs. That contract forces two requirements:

• Every event gets a globally unique, stable ID. Something like evt_9f3b2c81 that never changes across redeliveries. Put it in the payload and in a header.
• Consumers must deduplicate on that ID. Their handler checks a store of processed IDs before acting. A unique constraint on event_id in their database is the simplest correct implementation.

Design your payloads to make idempotent handling easy. An event that says invoice.paid with the full invoice state is safe to process twice. An event that says "increment the balance by 50" is a bug generator. Ship facts, not deltas.

{
  "id": "evt_9f3b2c81",
  "type": "invoice.paid",
  "created": "2026-07-04T09:14:22Z",
  "api_version": "2026-06-01",
  "data": {
    "object": {
      "id": "inv_442",
      "status": "paid",
      "amount": 4900,
      "currency": "eur"
    }
  }
}

Include an api_version field. Webhook payloads are an API surface, and they will evolve. Versioning them from day one costs nothing. Retrofitting versioning after fifty integrators depend on your field names costs weeks.

Retries that respect the receiver

The retry policy is where webhook systems earn or lose their reputation. Three retries over five minutes is not a retry policy, it is a coin flip. Real outages last hours. Deploys go wrong, databases fail over, certificates expire on a Friday night.

A schedule that works in practice: exponential backoff starting around one minute, doubling each attempt, capped at several hours between attempts, continuing for roughly 24 to 72 hours total. That covers the overnight outage nobody noticed until morning. Add jitter to every delay so a thousand failed deliveries do not come back as one synchronized stampede the moment the receiver recovers.

Be precise about what counts as success:

• Success is a 2xx response. Nothing else. Not a 3xx, do not follow redirects (that is a request forgery vector).
• 429 and 503 mean back off harder. Honor Retry-After if present.
• 4xx other than 429 means the request itself is bad. Retrying a 400 or a 401 forever wastes everyone's time. Retry a few times (endpoints misdeploy), then stop faster than you would for a 5xx.
• Timeouts are failures. Set a hard client timeout, typically 10 to 30 seconds. A receiver that takes 60 seconds to answer is telling you they are doing heavy work inline, which is their bug, not your queue's problem.

And critically: retries must be per-endpoint and per-event, isolated from each other. One dead endpoint must never delay deliveries to healthy endpoints. That means a real queue with per-destination concurrency, not a single worker loop iterating over subscriptions.

The outbox pattern: never lose the event in the first place

The most common silent failure is not a delivery failure at all. It is the event never being enqueued. Your code updates the database, then publishes the webhook job, and the process crashes between the two. The order exists, the event does not, and no retry policy on earth can resend an event that was never recorded.

The fix is the transactional outbox. Write the event into an outbox table in the same database transaction as the state change. A separate dispatcher process polls that table (or tails the change stream) and hands events to the delivery queue, marking them as dispatched.

CREATE TABLE webhook_outbox (
  id            BIGSERIAL PRIMARY KEY,
  event_id      TEXT NOT NULL UNIQUE,
  event_type    TEXT NOT NULL,
  payload       JSONB NOT NULL,
  created_at    TIMESTAMPTZ NOT NULL DEFAULT now(),
  dispatched_at TIMESTAMPTZ
);

Now the event's existence is guaranteed by the same ACID transaction that committed the business change. The dispatcher can crash, restart, and re-dispatch; the consumer's deduplication absorbs the duplicates. This one pattern eliminates the worst class of webhook bug: the event that vanished without a trace.

Sign everything, and sign the timestamp too

An unsigned webhook endpoint is an unauthenticated write API. Anyone who discovers the URL can forge events. "Mark this invoice paid" is a fun payload to forge.

The standard mechanism is an HMAC-SHA256 signature over the raw request body, keyed with a per-endpoint secret, delivered in a header. Two details separate a correct implementation from a vulnerable one:

• Include a timestamp in the signed content and reject requests older than a tolerance window (five minutes is typical). Without it, a captured request can be replayed forever.
• Compare signatures in constant time. A naive string comparison leaks timing information.

const crypto = require("crypto");

function verify(rawBody, header, secret, toleranceSec = 300) {
  const { t, v1 } = parseHeader(header); // "t=1720083262,v1=5257a8..."
  if (Math.abs(Date.now() / 1000 - t) > toleranceSec) return false;
  const expected = crypto
    .createHmac("sha256", secret)
    .update(`${t}.${rawBody}`)
    .digest("hex");
  return crypto.timingSafeEqual(Buffer.from(v1), Buffer.from(expected));
}

Two operational notes. First, verification must run against the raw bytes of the body, before any JSON parsing or middleware re-serialization touches it. Half of all "signature mismatch" support tickets trace back to a framework quietly re-encoding the body. Second, support secret rotation without downtime: allow two active secrets per endpoint, send signatures for both during the overlap window, and let integrators roll at their own pace. The broader threat model here overlaps heavily with API security best practices, including SSRF: if users can register arbitrary webhook URLs, your delivery workers are making requests on their behalf, so block private IP ranges and your own metadata endpoints at the egress layer.

Ordering is a lie, so stop promising it

Retries destroy ordering. If event A fails and event B succeeds, B arrives first, then A shows up an hour later. Guaranteeing ordered delivery across retries means serializing all deliveries per endpoint behind the oldest failure, which turns one flaky event into a total blockage. No serious webhook provider promises order, and you should not either.

Instead, make out-of-order delivery harmless:

• Put a created timestamp and, for events about the same object, a monotonically increasing sequence or updated_at in the payload. Consumers ignore events older than the state they already hold.
• For state-critical flows, document the "thin event, fetch truth" pattern: treat the webhook as a doorbell, then call your API for the current state of the object. The webhook says something changed; the API says what is true now.

That second pattern also caps the blast radius of stale payloads and keeps sensitive data out of a channel that transits third-party infrastructure.

Dead letters, auto-disable, and the recovery path

After the retry schedule is exhausted, the event needs somewhere to go. Deleting it is how you end up in an angry enterprise customer's postmortem. Route exhausted deliveries to a dead letter store, keyed by endpoint, retained for at least the length of your event history window.

Around that store, build three behaviors:

• Alert the endpoint owner early. Email or dashboard notification after a sustained failure streak, not after three days of silence.
• Auto-disable endpoints that fail for days. An endpoint returning 404 for a week is abandoned. Keep hammering it and you waste worker capacity and look like an attacker to their WAF. Disable it, notify loudly, and make re-enabling one click.
• Offer replay. A dashboard where an integrator can see delivery attempts, response codes, response bodies, and press "redeliver" is the single highest-leverage feature in the entire system. It converts support tickets into self-service.

Back all of this with an events API: GET /v1/events?since=... returning the same payloads that were pushed. Serious consumers will run a nightly reconciliation job against it, and after any incident on either side, that endpoint is how everyone gets back to consistent state. Webhooks push, the events API is the source of truth. Instrument every hop of this pipeline; the observability practices we recommend apply directly, with delivery success rate, end-to-end latency, and per-endpoint failure streaks as the metrics that matter.

The consumer contract you should publish

Your docs shape how integrators behave, so tell them exactly what a good handler does:

1. Verify the signature against the raw body. Reject on failure with a 401.
2. Enqueue the event to an internal queue or job system, keyed by event ID for deduplication.
3. Return 200 immediately, typically in under a second.
4. Process asynchronously, with your own retries around your own logic.
5. Reconcile nightly against the events API.

Handlers that do heavy work inline before responding are the top cause of timeout-driven redelivery storms. Say so in bold in the docs.

Build or buy: a quick decision framework

Everything above is a real engineering project: queues, schedulers, signing, a delivery dashboard, egress controls. Whether to build it depends on where webhooks sit in your product.

• Buy or adopt (Svix, Hookdeck, Convoy, or an open source dispatcher) when webhooks are a supporting feature, your team is small, and you need the dashboard and retry machinery on day one. Typical integration effort is days, not months.
• Build on your own queue infrastructure (SQS, RabbitMQ, or a Postgres-backed job runner) when webhooks are core to your product's value, you have unusual compliance or data residency constraints, or delivery volume makes per-message pricing painful. Budget for the dashboard and replay tooling too, because that is half the value.

Either way, the payload design, signing scheme, idempotency contract, and events API remain your job. Those are product decisions, and they are exactly the kind of API ergonomics that decide whether integrators enjoy your platform, a theme we cover in designing APIs developers love.

The reliability checklist

Before you call your webhook system production-grade, verify each of these:

1. Events are written via a transactional outbox, never published directly from request handlers.
2. Every event has a unique, stable ID and a versioned, fact-based payload.
3. Retries use exponential backoff with jitter over at least 24 hours, isolated per endpoint.
4. Only 2xx counts as success; 4xx and 5xx follow different retry rules; redirects are never followed.
5. Payloads are HMAC-signed with a timestamp, verified against raw bytes, with dual-secret rotation.
6. Egress blocks private IP ranges and cloud metadata endpoints.
7. Exhausted deliveries land in a dead letter store with alerting, auto-disable, and one-click replay.
8. An events API exists for reconciliation, with documented retention.
9. Delivery metrics (success rate, latency, failure streaks per endpoint) feed real alerts.
10. Docs specify the consumer contract: verify, enqueue, ack fast, process async, reconcile.

Most systems we audit pass three or four of these. The gap between "sends HTTP requests" and "developers trust it with money-moving events" lives in the other six.

How Innovation T can help

Innovation T designs and builds this kind of infrastructure for a living: webhook pipelines, event-driven backends, and the API platforms around them, from the outbox table to the replay dashboard. Our software and cloud engineering services cover architecture reviews of existing webhook systems as well as greenfield builds on your stack.

If your integrations page is generating support tickets instead of trust, talk to us. We will tell you which of the ten checklist items you are missing and what it takes to close the gap.