A notification system looks like a thin pipe from "something happened" to "a phone buzzed". In practice it is a decision engine bolted to four unreliable delivery networks. It has to decide whether a person should hear about an event at all, on which channel, now or later, alone or in a digest. Then it has to get the message through providers that throttle, drop, delay and revoke, and absorb their feedback so tomorrow's decisions are better. Get the decision wrong and you send a 3 a.m. push or a duplicate payment alert. Get the delivery wrong and a password reset never arrives.
This article designs the system from the contract down: the guarantees, the components, the data model, the per-recipient decision pipeline in code, the realities of APNs, FCM, email and SMS, a worked celebrity fan-out, and the failure modes that show up in production. It assumes you know queues and idempotency; if not, start with message queues and idempotency.
The contract before the boxes
Write the promises down before drawing boxes, because they decide most of the architecture:
- No lost events. Once the producing transaction commits, the notification decision will eventually run. Internally the system is at-least-once.
- No visible duplicates. Each (event, recipient, channel) is delivered at most once as far as we control it. At-least-once plus idempotency keys gives effectively-once up to the provider boundary; past it, collapse identifiers limit damage.
- Priority classes with different SLOs. Security and transactional messages (login codes, payment receipts) target seconds and bypass quiet hours and caps. Social messages target under a minute. Marketing is batch and fully subject to consent and caps.
- Consent is enforced centrally. No producer can send to a user who opted out, and every decision is auditable.
Priority classes are the most important design input. They get separate queues and worker pools, so a marketing blast of ten million emails can never delay a one-time passcode.
The contract before the boxes
Write the promises down before drawing boxes, because they decide most of the architecture:
- No lost events. Once the producing transaction commits, the notification decision will eventually run. Internally the system is at-least-once.
- No visible duplicates. Each (event, recipient, channel) is delivered at most once as far as we control it. At-least-once plus idempotency keys gives effectively-once up to the provider boundary; past it, collapse identifiers limit damage.
- Priority classes with different SLOs. Security and transactional messages (login codes, payment receipts) target seconds and bypass quiet hours and caps. Social messages target under a minute. Marketing is batch and fully subject to consent and caps.
- Consent is enforced centrally. No producer can send to a user who opted out, and every decision is auditable.
Priority classes are the most important design input. They get separate queues and worker pools, so a marketing blast of ten million emails can never delay a one-time passcode.
The architecture, component by component
Follow an event through the diagram above. A product service writes its own row (a comment, an order) and, in the same database transaction, an outbox row describing the event. A relay publishes outbox rows to the event log, so an event exists if and only if the business change committed; see the outbox pattern. Producers never name channels or templates; they publish facts like comment.created with the IDs needed to render.
Fan-out workers resolve the audience. For a direct event (reply to your comment) that is one user. For a broadcast (a creator posts) it is a follower list, expanded in pages of a few hundred recipient IDs, each page becoming one task so retries are bounded and progress is checkpointed.
The orchestrator runs once per recipient: build the dedupe key, load preferences, check consent, apply quiet hours and frequency caps, decide immediate versus digest, render the template in the user's locale, and write a notification row plus one attempt row per chosen channel. Only then does it enqueue channel jobs. Partitioning the log by recipient keeps one user's decisions ordered, which makes caps and digests race-free.
Channel workers own one provider each. They hold connections, sign requests, map provider errors to retry, drop or suppress, and update the attempt row. Jobs that exhaust retries go to a dead letter queue with enough context to replay.
The feedback path ingests delivery receipts, bounces, spam complaints and dead device tokens, and turns them into state: suppressed addresses, deleted tokens, engagement history.
The data model
Four tables carry the system. The unique constraint on the dedupe key is what makes retries safe.
CREATE TABLE notification (
id BIGINT PRIMARY KEY,
dedupe_key TEXT NOT NULL UNIQUE, -- hash(event_id, recipient_id, kind)
recipient_id BIGINT NOT NULL,
kind TEXT NOT NULL, -- e.g. comment.created
priority TEXT NOT NULL, -- security | transactional | social | marketing
payload JSONB NOT NULL, -- rendered title/body per channel
state TEXT NOT NULL, -- decided | digested | suppressed
created_at TIMESTAMPTZ NOT NULL
);
CREATE TABLE delivery_attempt (
notification_id BIGINT REFERENCES notification(id),
channel TEXT NOT NULL, -- push | email | sms | inapp
target TEXT NOT NULL, -- device token id, address, phone
status TEXT NOT NULL, -- queued | sent | delivered | failed | dropped
provider_id TEXT, -- provider message id for receipts
tries INT NOT NULL DEFAULT 0,
PRIMARY KEY (notification_id, channel, target)
);
CREATE TABLE preference (
user_id BIGINT, category TEXT, channel TEXT, enabled BOOLEAN,
version BIGINT, PRIMARY KEY (user_id, category, channel)
);
CREATE TABLE device_token (
token_id BIGINT PRIMARY KEY, user_id BIGINT, platform TEXT,
token TEXT UNIQUE, last_seen TIMESTAMPTZ, revoked_at TIMESTAMPTZ
);The in-app inbox is simply the notification table read by recipient, which makes it the source of truth: the bell count comes from the database, and WebSocket delivery is a best-effort hint that tells an open client to refetch.
The decision pipeline in code
The orchestrator is a pure function of the event, the user's state and the clock, wrapped in idempotent writes. Keeping it pure makes it testable with recorded inputs.
from datetime import datetime, time
from zoneinfo import ZoneInfo
BYPASS = {"security", "transactional"}
def decide(event, user, prefs, caps, now_utc):
key = f"{event.id}:{user.id}:{event.kind}"
if not prefs.consented(event.category):
return Decision(key, "suppressed", reason="no consent")
channels = [ch for ch in ("push", "email", "sms", "inapp")
if prefs.enabled(event.category, ch)]
if event.priority not in BYPASS:
local = now_utc.astimezone(ZoneInfo(user.tz)).time()
if in_quiet_hours(local, user.quiet_start, user.quiet_end):
channels = [ch for ch in channels if ch == "inapp"] # silent only
if caps.exceeded(user.id, "push", per_day=user.push_cap):
channels = [ch for ch in channels if ch != "push"]
if event.digestible:
return Decision(key, "digested", channels=[ch for ch in channels if ch != "push"], window="1h")
return Decision(key, "decided", channels=channels)
def in_quiet_hours(t, start: time, end: time) -> bool:
return start <= t < end if start < end else t >= start or t < end # wraps midnight
def handle(event, user, store, queues, clock):
d = decide(event, user, store.prefs(user.id), store.caps, clock.now())
created = store.insert_notification_if_absent(d) # unique dedupe_key
if not created:
return # a retry: already decided
for ch in d.channels:
queues[ch, event.priority].put(store.attempt(d, ch))Two subtleties. Quiet hours need the user's IANA time zone, not an offset, or daylight saving moves the window twice a year. And frequency caps are counters that must survive restarts; implement them with the same machinery as API limits, described in the rate limiter deep dive. A deferred notification is not dropped: it is scheduled for the end of the quiet window or folded into the next digest.
Channel adapters and provider realities
Each provider has its own failure language, and the channel worker's main job is translating it.
Apple (APNs). An HTTP/2 API, usually authenticated with a JWT signed by a .p8 key; Apple asks providers to refresh that token no more often than every 20 minutes and at least every 60. Requests carry apns-push-type, apns-priority (10 for immediate, 5 for power-considerate), apns-expiration and optionally apns-collapse-id, which makes the device show only the latest of a series. HTTP 410 means the token is no longer active for your app: mark it revoked and stop sending. 400 BadDeviceToken often means a sandbox and production mix-up.
Google (FCM). Use the HTTP v1 API (POST /v1/projects/{id}/messages:send with an OAuth 2.0 access token); the legacy APIs have been shut down. UNREGISTERED (404) means delete the token, INVALID_ARGUMENT (400) is a bug to fix rather than retry, QUOTA_EXCEEDED (429) and UNAVAILABLE (503) mean back off exponentially and honour Retry-After.
Email. Deliverability is a reputation you earn. Since February 2024 Gmail requires senders of more than 5,000 messages a day to its users to authenticate with SPF, DKIM and DMARC, offer one-click unsubscribe on marketing mail, honour it within two days, and keep reported spam below 0.3%; Yahoo set similar rules. Process bounces and complaints from your provider immediately: a hard bounce or complaint suppresses the address for that category.
List-Unsubscribe: <https://example.com/unsub/8f2c1e>, <mailto:unsub@example.com?subject=8f2c1e>
List-Unsubscribe-Post: List-Unsubscribe=One-ClickSMS. The expensive channel. A single segment holds 160 GSM-7 characters or 70 UCS-2 characters, and one emoji switches the whole message to UCS-2, so a 100-character message with an emoji costs two segments. Carrier registration (for example US A2P 10DLC) gates throughput. Reserve SMS for security and transactional messages and fallbacks.
In-app. Write the row, then publish a small "you have news" event to the WebSocket gateway. If the user is offline nothing is lost; the client reads the inbox on reconnect.
Worked example: a two-million-follower post
A creator with 2,000,000 followers posts. Product wants pushes delivered within five minutes.
- Fan-out pages the follower list in chunks of 500: 4,000 tasks on the social queue.
- Preferences filter: say 40% have post pushes enabled and are outside quiet hours, so 800,000 pushes; the rest get in-app rows only.
- Required send rate: 800,000 / 300 s, about 2,700 requests per second.
- Little's law sizes concurrency: with 100 ms provider latency, in-flight requests = 2,700 x 0.1, about 270. HTTP/2 multiplexes many streams per connection, so a handful of connections per worker carries it.
- Dead tokens: if 5% of tokens are stale, 40,000 410 or UNREGISTERED responses arrive within minutes. The token cleaner must absorb them asynchronously, not in the send path.
Two more checks. Confirm the provider quota for your project before launch rather than assuming it. And make sure the 4,000 fan-out tasks are spread across partitions, otherwise one hot partition serialises the whole post. If the five-minute target is not a hard requirement, spreading sends over a longer window also flattens the traffic spike the push opens create on your own API.
Failure modes
- Duplicate sends after a crash. A worker sends, then dies before marking the attempt sent. Use the attempt row as the lock, accept that the provider boundary is at-least-once, and set collapse identifiers so a duplicate replaces rather than stacks.
- Retry storms. A provider outage plus immediate retries multiplies load at recovery. Use capped exponential backoff with jitter and a circuit breaker per provider.
- Head-of-line blocking. Marketing and passcodes share a queue and passcodes wait. Separate queues and pools per priority.
- Stale preferences. A cached opt-in is used after the user opted out. Version preferences and invalidate on write; consent checks must read a version no older than seconds.
- Time-zone bugs. Quiet hours computed in server time, or with fixed offsets.
- Token rot. Never pruning dead tokens inflates cost and can harm sender standing.
- Template failures. A missing locale string renders blank. Validate templates at deploy and fall back to a default locale.
Trade-offs
| Choice | Gain | Cost |
|---|---|---|
| Decide per recipient at send time | Fresh preferences and caps | More work per broadcast |
| Pre-materialise audiences | Fast fan-out | Stale follower lists |
| Digest by default for social | Less fatigue | Lower immediacy |
| Strict per-user ordering | Race-free caps and digests | Hot users limit throughput |
| One provider per channel | Simple | No failover during outages |
What to do next
- Write the four promises and priority classes down and get product to agree to the SLOs.
- Add a unique dedupe key to your notification table and make every writer insert-if-absent.
- Split queues and worker pools by priority; prove a marketing blast cannot delay a passcode.
- Map every APNs and FCM error to retry, drop or revoke, and build the token cleaner.
- Verify SPF, DKIM, DMARC and one-click unsubscribe; watch complaint rates weekly.
- Load-test a broadcast fan-out with Little's law numbers and stale tokens included.