Why architecture matters here
The defining constraint is that the consumer is outside your control plane. Internal messaging lets you fix slow consumers, bound their downtime, and coordinate deploys. Webhook endpoints are written by thousands of customers at every skill level; at any moment a few percent are broken. Without architecture that isolates per-endpoint failure, the platform degrades to its worst consumer: workers pile up on a hanging endpoint, queues back up across tenants, and a single customer's misconfigured firewall delays everyone's deliveries. The fan-out unit must therefore be the (event, endpoint) pair, each with independent state, retries, and circuit breaking.
Durability ordering matters more than in most systems because webhooks are frequently the only signal a customer gets: a missed payment.succeeded event is not an observability gap, it is an unshipped order. The platform must persist the event before acking the producer, must never lose a delivery record while an endpoint is down for a weekend, and must make redelivery a self-service customer action. At-least-once is the only honest contract over the public internet — which pushes idempotency onto consumers, so the platform must supply the tools: stable event IDs, idempotency keys in headers, and documentation that treats duplicate delivery as normal.
Finally, webhooks are a security product. The platform makes outbound requests to customer-supplied URLs from inside your network — the textbook SSRF vector — so egress must be locked down as deliberately as any ingress. And consumers must be able to verify payloads came from you and not an attacker who found their endpoint URL, which makes signing, timestamping, and secret rotation part of the core architecture rather than documentation footnotes.
The architecture: every piece explained
Top row: from event to deliveries. Producers — your domain services — publish events to the ingest service, which validates the schema, assigns a globally unique event ID, deduplicates on the producer's idempotency key (producers retry too), and persists the event to durable storage before returning 202. That write is the platform's source of truth; everything downstream is derivable from it. The subscription matcher loads the tenant's registered endpoints with their event-type filters (often with payload-level filter expressions) and emits one delivery record per match — event X to endpoint A and endpoint B are separate rows with separate lifecycles from this point on.
Delivery records enqueue onto the delivery queue keyed for two properties: per-endpoint FIFO ordering where the tenant requested it (an ordering key of endpoint ID, or endpoint+resource for finer grains), and per-tenant fairness so a tenant emitting a million events cannot starve a tenant emitting ten — typically weighted round-robin across per-tenant subqueues. Delivery workers lease a record, construct the request — canonical JSON body, event ID and timestamp headers, an HMAC signature over timestamp plus body using the endpoint's current secret (and, during rotation, the previous secret in a second header) — and POST with strict budgets: connect timeout ~3s, total timeout ~10–30s, no following redirects to new hosts, TLS verification always on. Only a 2xx counts as delivered; a 200 with a broken body is the consumer's bug, but a 3xx, 4xx, 5xx, or timeout is a failure with distinct retry semantics (410 Gone can auto-disable the endpoint; 429 respects Retry-After).
Middle row: failure machinery. Endpoint health tracks a rolling success rate per endpoint and opens a circuit breaker when a target is clearly down — deliveries then skip straight to scheduled retry without burning worker time on 30-second timeouts, and a probe cadence tests recovery. The retry scheduler owns failed deliveries: exponential backoff (e.g., 30s, 2m, 10m, 1h, 4h, up to a day) with full jitter, because a popular consumer coming back from an outage must not receive every tenant's backlog in one synchronized wave. After the schedule exhausts — commonly 8–72 hours of attempts — the delivery lands in the dead-letter store with its full attempt history.
Bottom row: the product surface. The delivery log records every attempt — timestamp, response code, latency, response-body snippet — queryable per endpoint, because 'did you send it' is the first question every integration debug session asks, and showing the customer their own 500s converts support tickets into self-service. The portal exposes replay (single delivery or time-range bulk), secret rotation with dual-secret overlap windows, endpoint pause/resume, and payload inspection. Egress infrastructure — published static IP ranges, SSRF-guarded resolution that refuses private address space, per-tenant outbound rate limits — completes the picture.
End-to-end flow
Step 1 — publish. The billing service emits invoice.paid for tenant Acme with idempotency key inv_991:paid. Ingest validates, dedupes (a producer retry of the same key gets the same 202 and no second event), persists event evt_7f3a, and acks. Step 2 — fan-out. The matcher finds two Acme subscriptions matching invoice.*: their ERP integration and a Slack-notifier endpoint. Two delivery records are created and enqueued under ordering key = endpoint ID; the ERP endpoint has FIFO ordering enabled, the Slack one does not care.
Step 3 — first attempts. A worker leases the Slack delivery, signs (timestamp + body, HMAC-SHA256 with the endpoint secret), POSTs, gets 200 in 180ms — delivered, logged, done. Another worker leases the ERP delivery and gets a connect timeout: Acme's ERP gateway is mid-deploy. The attempt is logged, the failure feeds endpoint health (3 consecutive failures now), and the retry scheduler books attempt #2 for ~30s later with jitter. Step 4 — circuit opens. Attempts 2 and 3 also fail; the breaker opens. Sixty more Acme events fan out over the next hour, and their ERP deliveries skip straight to the schedule without consuming worker timeouts — queued behind the breaker, ordered, waiting. The delivery log shows Acme a red endpoint with 'circuit open since 14:02' instead of silence.
Step 5 — recovery and drain. At 16:40 the half-open probe gets a 200. The breaker closes and the scheduler drains the endpoint's backlog in order, rate-limited to the endpoint's configured ceiling (say 50 req/s) so the recovering ERP is not flattened by its own backlog — this drain throttle is the difference between recovery and a relapse. Step 6 — the stragglers. Two deliveries from the outage window exhausted their schedule during a longer secondary failure and sit in the dead-letter store. Acme's integration engineer gets the configured DLQ notification, opens the portal, inspects the two payloads, fixes the ERP-side bug, and clicks bulk replay for the affected window. Replayed deliveries carry the original event ID and an incremented delivery attempt header — the ERP's idempotency layer (keyed on event ID, as your docs insisted) makes the replay safe. End state: zero lost events, one support ticket that never got filed.