Chapter 10 · System Design Fundamentals

Design a Notification System

One service that reliably reaches every user across push, SMS, email, in-app, and web — at scale, on time, and only when wanted. The functional surface looks trivial ("send a message"); the design lives in the queue between intent and delivery, and in the preferences that decide whether to send at all.

▶ Open the companion slides

Reading time ~12 min Prerequisites Ch 4 · Rate Limiter, Ch 2 · Estimation Audio 🔊 Hinglish read-aloud Next Design a News Feed System

A notification system is not a "sender" — it is an orchestrator over a handful of channel-specific delivery contracts. Producers post one logical event; the system decides the channels, renders the copy, respects the user's preferences, absorbs the burst in a queue, and fans out to third-party providers that each fail in their own special way. Everything hard — spikes, duplicates, provider outages, consent — is about controlling that pipeline, not about the send itself.

Primary source

Alex Xu, System Design Interview (Vol. 1), Chapter 10. Companion deck: slides — jump to any slide with the Slide N chips. Go deeper: the AWS SNS Developer Guide — a production pub/sub-to-many-channels service that mirrors this design.

Five channels, five contracts Slide 2

Start with the surfaces. Each channel carries its own provider, payload limit, identity model, and cost profile — and each has a different failure mode. Treating them uniformly inside the core and isolating the quirks at the edges is the central design challenge. Note the cost asymmetry: push and in-app are nearly free, SMS costs real money per message, email sits between.

Channel	Provider	Identity	Cost / limit
Mobile push	APNs (iOS), FCM (Android)	Device token per install	~free · ~4 KB payload
SMS	Twilio / MessageBird	E.164 phone number	cents each · 160 chars/segment
Email	SendGrid / SES	Email address	cheap · reputation-gated
In-app	Your own backend	User id + read state	free · no third party
Web push	Web Push / VAPID	Browser subscription URL	~free · browser-gated

A good system steers each notification to the cheapest channel that still satisfies the user's intent. It also keeps the channels independent so a slow email provider never starves push.

What the system must do Slide 3

Before drawing boxes, pin down the volume, the latency budget, and the rules the system must respect on the user's behalf. The functional list is short; the non-functional list is where the design pressure lives.

Functional

One API for many producers; deliver across all channels from a single logical request; per-channel and per-topic opt-outs; both transactional sends (one user, now) and scheduled campaigns (many users); server-rendered templates so non-engineers can edit copy.

Non-functional

Absorb bursty traffic (launches and incidents spike over a calm baseline); a few seconds end-to-end for transactional, minutes for campaigns; no duplicate deliveries for one logical event; survive a single provider's outage without dropping messages.

volume

10M / day

all channels

average

~115 / s

10M ÷ 86,400s

peak

~5K / s

campaign spike

P95 latency

≤ 2 s

transactional

The gap between a 115/s average and a 5,000/s peak is the whole story — the system must absorb roughly a 40× surge without losing messages or melting providers. That is a queueing problem, and the same peak-vs-average reflex from Chapter 2 sizes the fleet.

A single front door, many delivery lanes Slide 4

Read the pipeline first — it is the system. Producers post a logical notification once to the Notification API, which authenticates, applies preferences, and renders a template. The message drops into a queue (one topic per channel) that absorbs the burst. Per-channel workers pull at their provider's allowed rate and translate to provider calls. Every delivery event streams back to a tracking store.

Producers post once; the queue absorbs the burst; workers translate to provider calls at each provider's rate; delivery events stream back to tracking, closing the loop.

One API, server-rendered templates Slide 5

Callers describe what happened and who it concerns — a logical event, not a rendered message. The notification system decides how to reach them. That indirection is the point: copy changes and channel routing never require re-deploying every producer.

1 · accept an event envelope

a small struct: event, recipient, channels, a data bag, an idempotency_key, and a priority

2 · resolve the template

each event maps to versioned templates, one per channel and locale, rendered server-side — never on the device

3 · validate up front, then enqueue

reject unknown events, missing variables, unverified senders synchronously — cheap failure beats a broken downstream send

Template variables come from the request's data field plus a profile lookup (name, locale, time zone). Rendering inside the service means the wording can change without an app release, and a staging step catches a broken template before it reaches production.

A queue between intent and delivery Slide 6

Producers run on their own schedules; providers enforce their own rate limits. The queue is the shock absorber that keeps each side from breaking the other. A bursty input meets a steady, rate-limited output; the queue holds the difference so neither end has to flex unnaturally.

Backpressure is automatic: when a worker is slow the queue grows; when it catches up the queue drains. Partitioning by user_id preserves per-user ordering while still fanning out.

Producer-side win

The API returns success the moment the message lands in the queue. Producers never wait on the provider hop and never block when a provider is slow.

Consumer-side win

Workers pull at exactly their provider's allowed rate. One topic per channel means each worker fleet scales and tunes independently.

Providers — each is its own snowflake Slide 7

All the messy realities of the outside world live in the channel workers. Each worker integrates with one provider through a thin adapter and shields the rest of the system from its quirks — payload sizes, carrier filters, warm-up rules, rate-limit shapes. Above the adapter sits one uniform interface: send, cancel, status.

Channel	Provider	Rate-limit shape & gotcha
Push iOS	APNs (HTTP/2)	Connection-level concurrency, no hard QPS — but back off on 429.
Push Android	FCM	Project-level QPS; multicast batches many tokens into one call.
SMS	Twilio / MessageBird	Per-sender QPS and country throttles; carrier filtering is separate.
Email	SendGrid / SES	Daily quotas tied to reputation; new domains need a warm-up.
Web push	VAPID endpoints	Per-endpoint, set by the browser; `410` means gone for good.

Per-provider rate buckets

Each worker enforces its own token bucket — the same idea as Chapter 4. If the budget is 300 emails/s, the worker holds at 300 even when the queue holds millions; the queue absorbs the rest.

Failover providers

For SMS especially, a second provider per region keeps deliveries flowing during an outage. The worker tries the secondary after a few failed attempts on the primary — a contained refactor, not a rewrite.

Retries, backoff, and a dead-letter queue Slide 8

Providers fail; networks blip. The worker's job is to know which failures are worth retrying, how long to wait between tries, and what to do with messages that never go through. The rule: retry the transient, drop the permanent, and dead-letter the rest so nothing is ever silently lost.

Retries handle transient failure with exponential backoff; the DLQ catches the rest so they can be inspected and replayed — not dropped. Alert when the DLQ grows past a small baseline.

Classify before retrying

A 4xx usually means the message itself is bad — wrong token, unverified sender, malformed payload. Retrying just burns quota. Only 5xx, timeouts, and explicit 429s are retried.

Backoff with jitter

Each retry waits roughly twice as long, with random jitter so a fleet of workers does not all re-hit a wounded provider on the same second. Bound attempts to five or six, then dead-letter.

Deduplication & idempotency Slide 9

Distributed systems retry. Caller scripts retry. Queues redeliver. The system must guarantee the user sees each logical message at most once, no matter how many times its components try. The tool is a stable idempotency key chosen by the producer — for a shipped order, order-A-7710-shipped — checked at two layers.

1 · check at the API

a fast SET NX on Redis keyed by idempotency_key, TTL 24–48h — a second request with the same key returns the original response

2 · check again at the worker

just before the provider call, keyed by (user, channel, key) — this catches redeliveries from the queue itself

3 · key the logical event, not the request

two services sending "order shipped" for the same order must produce the same key, so the duplicate is suppressed

At-least-once is the default; at-most-once is the goal

Queues and retries give you at-least-once delivery for free. The idempotency key is what turns that into at-most-once per (user, channel, event). Without it, one flaky provider response can fan out into three copies of the same push landing on a user's phone.

User preferences are a first-class gate Slide 10

Even a perfectly delivered notification is wrong if the user did not want it. Preferences are enforced centrally, in the request path — not as an afterthought filter — so every channel and every event respects them. Four gates run in order; any failure drops or defers the message before it reaches the queue.

1 · channel opt-out?

drop if the user disabled this whole channel — "no SMS, ever" at the account level

2 · topic muted?

each event is tagged with a topic; keep transactional while muting marketing

3 · quiet hours?

defer sends during the user's local night until morning, except allowed high-priority events

4 · frequency cap?

a counter per (user, channel) skips the send once the daily / weekly limit is hit

On top sits a regulatory floor: GDPR, CAN-SPAM, and TCPA demand explicit consent for marketing channels, a working unsubscribe link in every email, and deletion of preference history on request. The preference service is the audit trail — the right notification at the wrong time is still a wrong notification.

Tracking — did it land, did it work? Slide 11

Every send emits a stream of events — delivered, opened, clicked — that flow into one pipeline. Without this loop the system runs blind and channel routing stays stuck on guesswork. The same event stream feeds dashboards, the warehouse, the frequency-cap counters, and the token-cleanup job.

Every funnel step emits an event. Bounces trigger token cleanup; the warehouse feeds the routing decisions that pick the best channel next time. Email-open pixels are blocked often, so that metric is a lower bound; push taps are reliable.

Principles Slide 12

Decouple with a queue

The queue is the heart of the design. Producers stay fast, providers stay within limits, and the system survives spikes neither end could absorb alone.

Push quirks to the edges

Only the channel workers know APNs payload sizes, Twilio carrier filters, or SES warm-up. Everything inside speaks one clean shape.

Retry the transient, dead-letter the rest

Backoff with jitter for 5xx and timeouts; a DLQ with tooling and alerts for the rest. Never silently drop a message.

Idempotency ends at-least-once

A stable key per logical event, checked at the API and again at the worker. The user sees the message at most once even after five retries.

Preferences are a gate

Opt-outs, topic mutes, quiet hours, and frequency caps live in the request path. The right notification at the wrong time is still wrong.

Close the loop with tracking

Every delivery emits an event that powers dashboards, frequency caps, token cleanup, and the routing that picks the best channel next time.

Active recall

Cover the answers. Say each one out loud before you tap to check.

Why does the API accept a logical event, not a rendered message?

Who owns copy and routing?

So the notification system — not each producer — decides channels, wording, and locale. Copy and routing change without re-deploying every producer. Templates are rendered server-side, so a message can change without an app release.

What job does the central queue do?

Bursty in, steady out.

It is a shock absorber: it decouples bursty producers from rate-limited providers. The API returns as soon as the message is enqueued; workers pull at each provider's allowed rate; backpressure is automatic.

Which failures do you retry, and which do you drop?

Look at the status class.

Retry 5xx, timeouts, and 429 with exponential backoff + jitter. Drop 4xx (bad token, unverified sender, malformed payload) — retrying just burns quota. After N attempts, send to the dead-letter queue.

How do you stop a user seeing the same notification twice?

One key, two checks.

A producer-chosen idempotency key for the logical event, checked at the API (Redis SET NX, 24–48h TTL) and again at the worker (keyed by user, channel, key) to catch queue redeliveries.

Name the four preference gates, in order.

Channel, topic, time, count.

Channel opt-out → topic muted → quiet hours → frequency cap. They run in the request path; any failure drops or defers the message before it reaches the queue. A regulatory floor (consent, unsubscribe, deletion) sits on top.

Why is delivery tracking part of the core system, not an add-on?

The loop feeds decisions.

Delivered / opened / clicked events feed dashboards, the warehouse, frequency-cap counters, token cleanup, and channel-routing decisions. Without the loop the system is blind and routing is guesswork.

Check yourself

Q1 A campaign spikes traffic from ~115/s to ~5,000/s for two minutes. What absorbs it?

Why: The queue buffers the burst so producers stay fast and workers drain at each provider's allowed rate. Providers can't simply be told to accept 40× traffic.

Q2 A worker gets a 4xx "invalid device token" from APNs. What should it do?

Why: A 4xx means the message itself is bad; retrying wastes quota. The right move is to drop it and remove the dead token so it isn't used again.

Q3 Why is the idempotency key checked at the worker as well as the API?

Why: Queues give at-least-once delivery, so the same message can reach the worker twice. A second check keyed by (user, channel, key) makes the provider call exactly once.

Q4 Where do the provider-specific quirks (APNs sizes, SES warm-up) belong?

Why: Pushing quirks to the edges keeps the core uniform. Only the channel worker's adapter speaks the provider's protocol; everything inside uses send / cancel / status.

Q5 A user muted "marketing" but kept "order updates". Where is that enforced?

Why: Preferences are a first-class gate evaluated centrally before enqueue. The topic-muted check drops marketing sends while transactional ones pass through.