Chapter 16 · System Design Fundamentals
Patterns & Lessons
Fifteen chapters of newsfeeds, chat systems, crawlers, and storage layers — and the same handful of shapes keep returning. This is the step back: naming the recurring building blocks, the patterns that bind them, and the tradeoffs you cannot design your way out of.
▶ Open the companion slidesMost of system design is not invention — it is selection. A small vocabulary of components and patterns covers the overwhelming majority of what you will ever be asked to sketch. The skill is knowing which shape matches the problem in front of you, picking it honestly, and naming the bill it comes with. This chapter distills the recurring moves from every earlier chapter into one place, so it doubles as a map back into the series.
Alex Xu, System Design Interview (Vol. 1) — the synthesis of Chapters 1–15.
Companion deck: slides — jump to any slide with the
Slide N chips.
Go deeper: the
System Design Primer — an open
catalogue of these same building blocks with worked prompts and further reading.
Six pieces show up in almost every design Slide 2
Every design you have drawn reached for some subset of the same six components. Knowing what each one is actually for — and when it is overkill — matters far more than memorising any particular product. Learn the role, not the brand.
| Piece | What it is for | Reach for it when |
|---|---|---|
| Load balancer | Spreads requests, hides dead nodes behind one stable endpoint, terminates TLS. | The moment one server is not enough. |
| Cache | Stores a computed answer close to where it is read. | The same answer is recomputed over and over; reads dominate writes. |
| Queue | Buffers work between a producer and a consumer. | Work can wait, or a slow consumer must not crash the producer. |
| Database | The source of truth — relational for joins, key-value/document for scale. | Always. Pick the engine by access pattern, not vibes. |
| CDN | Serves bytes from an edge close to the user. | Users are global and payloads are big or repetitive. |
| Blob store | Cheap, durable storage for large write-once objects. | Photos, video, backups, checkpoints — put a CDN in front. |
Notice the pairing habit: a blob store almost always wants a CDN, a database almost always wants a cache, a queue almost always wants a dead-letter queue. The components are simple; the art is in how they connect.
Scale outward, and keep the servers stateless Slide 3
Vertical scaling — a bigger box — is easy until it is not: you hit a ceiling, a single failure domain, and downtime on every deploy. Horizontal scaling buys elasticity, blast-radius limits, and rolling deploys — but only if no request carries hidden affinity to one machine. The enabling trick is to push every scrap of session state out of the app tier, so any server can serve any request. This is the same lever Chapter 1 introduced; here it becomes a design reflex.
Cache everywhere — the hard part is knowing when to throw it away Slide 4
Browsers cache, CDNs cache, app servers cache, databases cache. Each layer knocks a zero off the latency and shields the layer beneath it from a slice of traffic. But a cache is only a bet that a value is still true, and the one question that decides whether caching helps or hurts is: when does the cached value stop being true? The read side of every system in Chapter 6 lived or died on that answer.
| Layer | Typical latency | What it holds |
|---|---|---|
| Browser cache | ~0 ms | The last response for this one user. |
| CDN edge | ~10 ms | Static assets and popular, cacheable responses. |
| In-process cache | ~0.1 ms | Per-instance hot objects, right in RAM. |
| Distributed cache | ~1 ms | Shared hot keys and sessions (Redis / Memcached). |
| Database | ~5–50 ms | The source of truth, with its own buffer pool. |
| Cold storage | 100 ms+ | Everything that fell out of every cache above. |
Invalidation tools, sharpest last
TTLs are the lazy default — cheap, but always a little stale. Explicit deletes on write are sharper but easy to miss. Versioned keys (user:42:v17) sidestep invalidation entirely: a new version is a new key.
The pathologies to expect
Thundering herd when a hot key expires; one node melting under a single celebrity key; a cold cache stampeding the DB on restart. Cure with jittered TTLs, request coalescing, and deliberate warm-ups.
The old joke — the two hardest problems in computing are cache invalidation and naming things — is a system-design truth. Adding a cache is a five-minute change; being sure it never serves a wrong answer at a moment that matters is the actual work.
When work can wait, put it in a queue Slide 5
A synchronous call chains you to your slowest dependency: its latency becomes your latency, and its failure becomes your failure. A queue breaks the chain into independent stages that each scale, retry, and fail on their own. The messaging systems in Chapter 10 are this pattern taken to production scale. The rule of thumb is simple: keep synchronous only what you cannot answer without; queue the rest.
What you gain
Producers and consumers scale on their own timelines; a traffic spike fills the queue instead of toppling a service; retries become safe when consumers are idempotent and failures fall into a dead-letter queue.
What it costs
New things to watch — queue depth, age of the oldest message, consumer lag — and eventual, not immediate, results. You have traded instant feedback for resilience, so your dashboards must change too.
Replicate for availability. Shard for capacity. Slide 6
These are different levers for different problems, and most of the pain in distributed data comes from confusing them. Replication copies the same dataset onto several nodes: you survive a node loss and you scale reads, but you do not grow capacity — every replica holds everything. Sharding splits the dataset across nodes by a key: you grow write throughput and storage beyond one machine, but you inherit cross-shard joins and painful rebalancing. Choosing a shard key is the same consistent-hashing problem from Chapter 5.
Replication knobs
Synchronous replicas are safe but slow; asynchronous ones are fast but can lose the last few writes. The dial is how many nodes you can lose before quorum breaks — and how much replication lag your reads tolerate.
Sharding knobs
The shard key is the decision you cannot easily reverse: user, tenant, geo, or hash. Pick one that spreads load evenly and keeps related rows together, and plan the rebalance script before you need it.
Denormalize for reads. Accept that writers catch up later. Slide 7
A normalized schema is elegant and slow: every read reassembles the answer with joins. Once reads outnumber writes by orders of magnitude — feeds, search, profile pages — you copy the data into shapes that match each query, so a read becomes a single lookup. The price is duplication and the discipline of keeping copies eventually consistent. The fan-out designs of Chapter 11 are exactly this bargain.
Materialise the view
Precompute the feed, the leaderboard, the search index. The read collapses to one key lookup instead of a query planner's best effort.
Fan-out on write
When someone posts, push the item into every follower's inbox. The write is expensive; every read afterwards is trivial. Great — until a creator has millions of followers.
Fan-out on read
For celebrities, flip it: assemble the feed at read time and cache it briefly. Most real systems use a hybrid, switching strategy by follower count.
Keep a rebuild path
Copies drift — when, not if. You need a job that recomputes any derived view from the source of truth, so a bug becomes a rerun, not a data-loss incident.
Accepting eventual consistency is not sloppiness — it is a product decision. A like count that lags by a few seconds costs nothing; a bank balance that lags costs everything. The craft is naming which reads may be stale, and treating the rest differently.
CAP, in practice: when the network splits, pick a side Slide 8
Partitions are rare, brief, and rarely total — so the textbook theorem is starker than daily life. The useful framing is per-operation: when a network split happens, each call must either pause until it can be consistent (CP) or answer with possibly stale data and reconcile later (AP). Different products want different answers, and often the same database makes both choices for different operations.
CP — pause when partitioned
Banking ledgers, ticket inventory, distributed locks. Better unavailable for ten seconds than double-spending or overselling. Consistency wins; availability yields.
AP — answer with what you have
Social feeds, product catalogues, DNS. A slightly stale "last seen" is fine; an error page is not. Availability wins; the copies reconcile afterwards.
Latency vs throughput. Cost vs reliability. Slide 9
No system is good at all four at once, and every knob drags a neighbour along. A designer's real job is to name where each dial sits today, what moving it costs, and which other dial it pulls. Two pairs come up in almost every review.
| The tension | Push one way… | …and you pay |
|---|---|---|
| Latency ↔ throughput | Batch, buffer, and pipeline to raise aggregate throughput. | Any single request waits longer while work piles up to a batch. |
| Cost ↔ reliability | Add regions, replicas, and quorums for more nines. | Each extra nine costs roughly an order of magnitude more. |
| Consistency ↔ availability | Insist every read is fresh. | Some requests block or fail during a partition (see CAP). |
| Simplicity ↔ scale | Add caches, shards, and queues to grow. | More moving parts to operate, monitor, and debug at 3 a.m. |
When you are unsure which move to make, resist adding machinery on instinct. Follow the cheapest ladder that the numbers justify, and re-measure after each rung — the interview framework of Chapter 3 is the same disciplined loop.
still slow? → precompute a view · writes hot? queue + workers · then measure again
How to actually keep getting better at this Slide 10
Reading chapters is the easy part; the skill compounds through habit. Three practices carry most of the growth — and none of them happen by accident.
1 · Build the smallest version
Stand up a multi-node cache and kill a node mid-write. Wire a queue between two services and slow the consumer to watch lag pile up. Shard a database and write the rebalance script before you need it. Running a thing and breaking it teaches what diagrams cannot.
2 · Read other people's source
Pick one open-source system you depend on — Postgres, Kafka, Envoy, etcd — and read its design docs end to end. Trace one request through the code, noting every queue, lock, and retry. Follow engineering blogs that publish real incident write-ups, not marketing.
3 · Design on paper, weekly
Pick a product you use and sketch the part you understand least, as if a stranger will implement it — numbers, not adjectives. Re-read it a month later and note what was wrong. Keep a running list of "things I do not yet understand"; that list is your real curriculum.
The meta-skill
Every one of these forces the vague to become concrete. The engineers who improve fastest are not the ones who read the most — they are the ones who keep making falsifiable predictions and checking them against reality.
Five principles to leave with Slide 11
Everything above collapses into five sentences. If you remember nothing else from these sixteen chapters, remember these.
Active recall
Cover the answers. Say each one out loud before you tap to check.