Chapter 12 · System Design Fundamentals
Design a Chat System
One persistent socket per device, and everything else — ordering, durability, presence, offline reach — hangs off it. A chat system feels simple to use and is deceptively deep to build: the hard part is not sending a message, it is guaranteeing it survives, arrives once, and looks the same on every screen.
▶ Open the companion slidesOrdinary web traffic is client-pull: the browser asks, the server answers, the connection closes. Chat inverts that — the server has to wake the client the instant someone else types. That single requirement, "the server must push," drives almost every decision that follows: the transport, the way users map to servers, how messages are stored and ordered, and what happens when a phone is asleep in someone's pocket. Read the whole chapter as consequences of that one inversion.
Alex Xu, System Design Interview (Vol. 1), Chapter 12. Companion deck: slides — jump to any slide with the Slide N chips. Go deeper: Discord Engineering — How Discord Stores Billions of Messages.
What the system has to do Slide 2
The chat domain is small in surface area but rich in failure modes. Pin the user-visible features first — each one quietly imposes a different storage or networking constraint, and naming that constraint up front is half the design. Six features carry the chapter.
| Feature | What users see | The hidden constraint |
|---|---|---|
| 1:1 chat | Two people message in near real time (~200 ms). | Server must push; messages survive an offline recipient. |
| Group chat | Up to a few hundred members see the same order. | Fan-out happens server-side, once per member. |
| Presence | Online / away / last-seen dots. | Highest-QPS surface in the whole system. |
| History | Durable, paginated, sorted by time, years deep. | Write-heavy, append-only, recency-biased reads. |
| Push | Lock-screen alert when the app is closed. | Only the OS gateway (APNs / FCM) can reach a dead socket. |
| Multi-device | Phone, laptop, web all show the same state. | Per-device cursors that must converge without user action. |
Notice how little of this is about "sending text." The functional surface is trivial; the non-functional requirements — durability, ordering, presence at scale, offline reach — are where the whole design lives.
Choosing the connection model Slide 3
A chat system stands or falls on its push channel. HTTP request-response can't wake a client on its own, so the question is: how do we keep a live path from server to device open? Four options exist; only one is truly bidirectional and cheap enough to hold open for a billion connections.
| Approach | How it works | Latency | Verdict |
|---|---|---|---|
| Short polling | Client asks "anything new?" every few seconds; most calls return empty. | Poor | Reject — pure waste |
| Long polling | Server holds the request open until a message arrives or a timeout fires. | OK | Fallback |
| SSE | One-way server→client stream over plain HTTP; client still POSTs to send. | Good | One-way only |
| WebSocket | Full-duplex channel over one upgraded HTTP connection; bytes flow both ways. | Excellent | Pick this |
A chat client is bidirectional by definition — it both sends and receives on the same device. WebSocket gives one persistent connection per device, framed messages, tiny per-packet overhead, and predictable wake-up on mobile radios. Everything else — presence, typing, receipts — rides that same socket. Long polling becomes the graceful fallback on networks that block the upgrade handshake.
The four moving parts Slide 4
Split the system so each concern scales on its own axis. The chat service is the only stateful tier — it terminates the live sockets; the rest stay stateless or sit behind storage. A load balancer fronts the sockets, a registry remembers which server holds whom, and presence, push, and storage hang off the chat tier as independent services.
Chat Service (stateful)
Each instance terminates thousands of live WebSockets, routes inbound messages to other chat servers, writes to storage, and hands off to push when a recipient has no socket.
Presence, Push, Storage
Presence is a Redis-with-TTL heartbeat tracker. Push is a thin gateway over APNs / FCM / Web Push. Storage is an append-only KV store keyed by chat_id, plus an RDBMS for users and rosters.
Mapping users to chat servers Slide 5
A user's socket lives on exactly one chat server at a time. To route Alice→Bob you must know
which box holds Bob's connection — so a shared registry (Redis or ZooKeeper) maps user_id → server.
The entry carries a TTL refreshed by heartbeats, so a crashed server's mappings simply
expire and the next reconnect rewrites them on a healthy node.
user→self to the registry. On send the origin
server looks up the recipient: local means push directly; remote means RPC to the holding server.1:1 chat — sender to recipient Slide 6
Two rules govern the happy path: a message must be durable before it is delivered, and ordered before it is acknowledged. The chat server persists first, then acks the sender, then routes to the recipient. If delivery fails, the message is still recoverable — because it was written before anyone promised anything.
One ID, one truth
A central ID generator — Snowflake, ULID, or a per-chat sequence — stamps every message with a monotonic, sortable identifier shared by all parties. Ordering by ID equals ordering by time.
The offline path
No socket for Bob? The message still sits in his inbox. Push fires APNs / FCM. On reconnect, Bob's device pulls everything newer than its cursor and catches up.
Storing messages at scale Slide 7
Reads and writes are skewed by conversation, not by user — so the conversation is the
unit of locality. Partition on chat_id and all of one chat's messages cluster onto one node,
where they can be read as a single contiguous log. The workload is write-heavy, append-only, and
recency-biased, which points away from a relational store and toward a wide-column KV store like Cassandra,
DynamoDB, or HBase.
Within a partition, IDs must strictly increase. A centralised Snowflake generator gives global monotonicity with no coordination on the hot path: its high bits encode a timestamp, so sorting by ID is sorting by time. That single property is what lets every device — receiving frames out of order over lossy networks — reconstruct the one true sequence locally.
message_id = timestamp · worker · sequence → sortable, no coordination
Because messages are never updated, only inserted, there are no cross-row transactions to worry about. Cold conversations can migrate to cheaper storage tiers — the append-only shape makes tiering trivial. Users, group rosters, and profiles, which do mutate, live separately in a relational store.
From one sender to many readers Slide 8
In a group the sender publishes once. The chat service writes the message a single time to
the conversation log — which assigns the canonical message_id — then enqueues a reference into
one per-recipient inbox per member. Each member fetches and acks at their own pace: Bob
reads instantly, Liu reads tomorrow, and "unread" counts stay trivial because each inbox tracks its own cursor.
Bounded group size
Per-recipient fan-out works up to a few hundred members. Above that — broadcast channels, huge rooms — flip the model: members pull from the shared log instead of each getting a copy.
Single source of order
The conversation log assigns one canonical message_id. Every inbox copy references it, so even devices that receive out of order can sort and dedupe by ID.
Online presence Slide 9
Presence looks trivial — "is this user online?" — and is the single hardest thing to scale. Each device
heartbeats every few seconds, and every contact wants the answer in real time. The mechanism is a
heartbeat plus a TTL: the server writes presence:user with a TTL of two to
three times the ping interval, and each heartbeat refreshes it. Miss enough heartbeats and the key expires
on its own — the user is implicitly offline, with no explicit "goodbye" needed.
Read fan-out is the real cost
A user with 500 contacts is 500 reads per second just to render online dots. Cache aggressively, and only compute presence for contacts the user can currently see on screen.
Push transitions, not state
Never stream presence continuously. Publish only on a change — online→offline — over pub/sub, scoped to friends and recent chats. Subscribers keep a cached view and apply diffs.
Read receipts, typing, and delivery status Slide 10
These ephemeral signals make a chat feel alive. They are cheap one at a time and brutal in aggregate — so design them as best-effort, and never store them as first-class messages. Delivery status is the one exception: it is three durable states a message walks through.
Read state is a cursor
Store one "highest read message_id" per (user, chat) — far cheaper than a row per (user, message). Group chats display the count of members whose cursor is above a given message.
Typing is pure transient
Client sends typing-start; the server fans it to connected members and auto-clears after ~5 s. Never persisted, first to drop under load.
Network retries are the rule, not the exception. Every ack carries the message_id and the
server treats duplicates as no-ops; every outbound send carries a client-generated client_msg_id
so the same message isn't stored twice across a reconnect. Rule of thumb: if losing a signal would be
merely annoying (typing, presence flicker) make it best-effort; if losing it would be
wrong (delivery status, ordering) make it durable.
Push notifications for offline users Slide 11
When no socket is alive, the app may be killed by the OS — the only way to reach it is the platform push gateway. The push service is deliberately decoupled from the chat path: an async queue, a worker pool, and a careful authority over what actually gets sent. The chat service stays fast because it never blocks on a third-party gateway. This is the same machinery covered in Chapter 10 · Notification System, reused here for the offline case.
Decoupled pipeline
Chat service → durable queue (Kafka / SQS) → push workers → APNs / FCM / Web Push. Workers build the payload, look up the device token, batch, and retry independently.
Tokens expire
Each device registers a push token. Workers must handle 410 / unregistered responses and purge dead tokens, or they'll waste calls forever on phones that are gone.
Cancel on read
If the phone reconnects and reads the message before the gateway delivers, the worker checks delivery state and silently drops the notification. No "ghost" alert for a message already seen.
Quiet hours & E2E
Workers consult per-user preferences and may collapse a burst into "5 new messages." Under end-to-end encryption, push carries only a wake-up signal — the ciphertext is fetched and decrypted on-device.
Keeping a user's devices in step Slide 12
An account isn't tied to one device. Phone, laptop, and web each connect independently and must converge on the same view of message order and read state — without the user lifting a finger. The trick is that the server keeps almost no per-session memory: each device persists its own cursor ("last message_id I've seen" per chat) and, on reconnect, simply asks for everything newer.
One cursor per device
On reconnect a device sends GET msgs > cursor and replays the gap. The server is stateless about sessions — the cursor lives on the client and in durable per-device state.
Read state is global
Read on your phone and the badge clears on your laptop. A read event on any device pushes the new read cursor to every other live socket for that account.
Deliver to all, ack from any
An incoming message is pushed to every live socket the user holds. "Delivered" flips as soon as one device acks; "Read" still needs an explicit read event.
Convergence, not coordination
Because ordering is fixed by the global message_id, devices never negotiate. Each catches up to the same log independently and lands on an identical view.
Principles Slide 13
WebSocket is the spine
One persistent, full-duplex connection per device carries everything — messages, presence, typing, receipts. Long polling is only a fallback.
Persist before you deliver
Write durably first, ack the sender, then route. Storage is the source of truth; sockets are just a fast path to the same data.
Partition by conversation
chat_id is the partition key, Snowflake message_id the sort key. Every conversation is a self-contained, time-ordered log.
Fan-out where it fits
Small groups copy into per-recipient inboxes; huge rooms flip to pull. The right model depends on the ratio of members to active readers.
Presence is the loud surface
Heartbeats + TTL are simple in concept, brutal in QPS. Cache hard, send only transitions, and never compute presence no one can see.
Plan for offline from day one
Push gateways, per-device cursors, idempotent acks, dedupe on client_msg_id. Networks fail constantly; the experience must not.
Active recall
Cover the answers. Say each one out loud before you tap to check.
Check yourself
chat_id clusters a whole chat on one node and lets it be read as one contiguous log.