Nobody designs for millions of users on day one. You start with one server, then fix one
bottleneck at a time. Every layer you add is there to solve a specific problem the
previous setup ran into. This chapter climbs that ladder, one rung at a time.
Every box in the diagram below — load balancer, cache, queue, shards — earns its place by
fixing a problem you can actually measure. Not one you imagined might happen.
The diagram shows where this chapter ends up: the shape a system grows into after it has
survived real traffic. Don't try to memorise it. We'll build it one piece at a time, and
each piece will make sense as the answer to one question: what just broke?
Where we're headed. Static assets come from the CDN; everything
else enters through the load balancer, which spreads requests across an
interchangeable web tier. Reads hit the cache first, writes go to the
primary DB (replicas serve more reads), and slow work is handed to a
queue for workers to drain.
Primary source
Alex Xu, System Design Interview — An Insider's Guide (Vol. 1),
Chapter 1. The author's site, bytebytego.com,
is the highest-trust companion. This lesson mirrors the
slide deck; jump straight to any slide with the
Slide N chips below. Go deeper: High Scalability —
A
Beginner's Guide to Scaling to 11 Million+ Users on AWS walks this same ladder
with real AWS building blocks.
Twelve moves take you from a hobby project to a system that serves the whole internet.
Each rung below names the move and the failure it fixes. Just
skim it for now. The rest of the chapter walks through every rung in detail, with the
matching slide one tap away.
1Single server Slide 3Web app + database on one box. Fails when one process starves the other, or the box dies.
2Pick the right database Slide 4SQL by default; NoSQL for flexible schema or huge writes. The wrong choice locks in pain later.
3Scale the web tier out Slide 5Many identical servers, not one giant one. A single big box is still a single point of failure.
4Load balancer Slide 6One public IP fronts many private servers. Removes the web tier as a single point of failure.
5Database replication Slide 7One primary for writes, replicas for reads. Now the database isn't the lone weak point either.
6Caching tier Slide 8Hot data in memory, in front of the DB. Most reads should never touch disk.
7CDN Slide 9Static assets from an edge near the user. You can't beat the speed of light — so move closer to it.
8Stateless web tier Slide 10Push session state to shared storage. Sticky sessions quietly defeat horizontal scaling.
9Multiple data centers Slide 11GeoDNS routes users to the nearest healthy region. A whole region can go offline.
10Message queue Slide 12Decouple producers from consumers; absorb spikes. Not every task needs to finish inside the request.
11Logging, metrics, automation Slide 13Observability + CI/CD + IaC. At scale you can't SSH in and eyeball it.
12Database sharding Slide 14Split data across many DBs by a shard key. The last resort, when the primary can't grow further.
One VM runs everything. Fastest to ship, cheapest to run, easiest to debug
— and the right place for almost every system to start.
The user's browser looks up your domain name, gets this server's IP address back from
DNS, and talks to it directly over HTTP. The first thing that goes wrong is that nothing
is isolated from anything else. One runaway process eats all the CPU or memory, and the
whole site goes down with it. The entire system is one dead machine away from an outage.
Key idea
Start with the simplest thing that could possibly work. What tells you to add a layer
is real production data — how slow requests are, how many are failing, how much work is
piling up — not a diagram you drew before launch.
This is the first decision that is genuinely hard to undo later. Base it on the
shape of your data and how you'll read it — not on which option sounds more
modern. Both are the right answer for different jobs.
Transactions, joins, strong consistency. The default for CRUD.Flexible schema, huge writes, horizontal scale out of the box.
The honest answer
For your first million users, a managed Postgres almost always wins. NoSQL solves
problems that most applications never actually run into. Pick it because something you
really need forces you to, not because it sounds like it scales better.
Scaling up needs no code changes, but you eventually hit the biggest
machine money can buy — and it is still one machine that can die. Scaling out has
no such ceiling. The price is that your servers must stop keeping user data in their own
memory (stage 8).
The rule
Start by scaling up, because it costs you no work. Switch to scaling out as soon as
staying online matters more than staying simple. Only scaling out survives one machine
dying.
The public sees one IP address. Behind it, many servers share the work. The
load balancer keeps checking whether each server is healthy and quietly stops sending
traffic to any that isn't. The user never notices.
You get a security benefit for free. Only the load balancer has a public IP address; the
web servers sit on a private network where the internet can't reach them at all. How the
load balancer picks a server is a dial you can turn: send requests to each server in turn
(round-robin), or to whichever one is least busy (least-connections), or always send the
same user to the same server (IP-hash) when that matters.
One primary database takes every write. Replicas keep a copy of
it and answer reads. Most apps read far more than they write, so this buys a lot of
breathing room cheaply. If the primary dies, one replica is promoted to take its
place.
Watch out · replication lag
Replicas are always a few seconds behind the primary. So when a user writes something
and immediately reads it back — "I just posted, show me my post" — the read can land on a
replica that hasn't caught up yet, and the post seems to have vanished. Fix it by sending
those particular reads to the primary, or by designing the screen so a short delay doesn't
look broken.
Memory is roughly 100,000× faster than disk. If the cache has the
answer (a hit), the app replies straight away. If it doesn't (a miss), the
app asks the database, saves that answer into the cache, and then replies.
Two settings decide whether a cache actually helps. The TTL (time to
live) is how long an entry is allowed to stay before it's thrown away. Set it too long and
you serve users out-of-date data; set it too short and the cache is empty so often it
barely pays for itself. Data that is read constantly but changes rarely is the sweet spot.
The eviction policy decides what to delete when memory fills up — the
usual choice is LRU, which drops whatever was used least recently.
Watch out · cache stampede
When a very popular entry expires, a thousand requests can miss at the same moment and
all rush to rebuild it at once, hammering the database. Two fixes: let only one request do
the rebuild while the others wait, or add a little randomness to each TTL so entries don't
all expire at the same instant.
This speed-up comes from physics, not clever code. A round trip from Mumbai to
Frankfurt takes about 150 msat best; Mumbai to a server in Mumbai
takes about 10 ms. You cannot make light travel faster, so you move the data
closer to the user instead.
Getting rid of old files matters
Put a version in the filename, like app.v42.js. Asking the CDN to delete
old copies by hand is slow and easy to get wrong. If the version is part of the URL, every
deploy is simply a brand-new file that no edge has cached yet — nothing to clear.
Move anything about a specific user out of the web server's own memory: login
sessions into Redis, uploaded files into object storage. Now any request can be handled
by any server, and servers become interchangeable — you replace one without a
second thought.
The trap here is the sticky session. If user A's login session is stored
in server 1's memory, the load balancer has no choice but to keep sending user A back to
server 1. When server 1 dies, that user is logged out — which defeats the whole point of
running many servers. Once sessions live in shared storage instead, servers can be added
and removed automatically, and you can deploy new code by restarting servers a few at a
time with nobody noticing.
Key idea
"Stateless" does not mean the system remembers nothing. It means the remembering
happens in services built for it — databases, caches, file storage — instead of in the
memory of a web server that could disappear at any moment.
An entire region can go dark. GeoDNS looks at where each user is and sends
them to the closest data center that is still healthy, moving traffic away from one that
stops responding.
The hard part is the data. Each data center needs its own copy, and copying between
regions happens in the background rather than instantly — you accept that the two
copies are briefly out of step in exchange for surviving a regional outage. Also test your
failover: a plan nobody has ever rehearsed will fail on the day you need it, so practise
it deliberately. And be honest about the cost — you are roughly doubling your
infrastructure bill. Do it because you promised customers a level of uptime, not because
the diagram looks impressive.
The web server drops a job on the queue and replies to the user straight away.
Workers pick jobs off the queue whenever they are free. A sudden flood of signups piles
up harmlessly in the queue instead of crashing the email service.
Splitting the work this way lets each side grow on its own. Too much
background processing? Add workers. Too much incoming traffic? Add web servers. Neither
change forces the other. For plain job queues, SQS or RabbitMQ do the job. Reach for Kafka
when you need to keep the events around and replay them later.
Once you have more than a handful of servers, you can't log into one and puzzle out the
problem by reading it. Being able to see what your system is doing, and having it fix
routine things by itself, is not a "later" luxury. It is how the system stays alive, and
how you know which bottleneck to attack next.
Logging — collect logs from every server into one searchable place
(ELK, Loki, CloudWatch). Give each request an ID that follows it everywhere, so you can
pull up the whole story of one user's request.
Metrics — watch the four that matter most: how slow requests are,
how many are coming in, how many are failing, and how full your machines are.
Alerting — wake someone up for things users can feel, like requests
getting slow or failing. Don't wake them for "CPU is at 80%", which may be perfectly
fine. Alert on causes instead of symptoms and your team learns to ignore the pager.
Automation — automatic testing and deploys, servers defined in code
rather than clicked together by hand, automatic scaling and failover. If you've done
something manually twice, script it.
The shard key is the piece of data that decides which database a row
lives on. Choose it badly and you get a hot shard: one database doing most of the
work while the others sit idle.
Last resort, not first
Sharding makes your system permanently more complicated. Queries that need data from
two shards become slow or impossible, and changing the number of shards later is a huge
migration. Use up your simpler options first — read replicas, caching, a bigger machine.
This is the one rung on the ladder you can't easily step back down from.
Everything above is just a toolbox. What actually gets you through an interview — and
through a real night on call — is knowing when to reach for each tool:
Fix the bottleneck in front of you, not the one you imagine. Building
too much too early goes wrong faster than building too little. Add complexity when
something real tells you to.
Keeping servers free of user data pays off everywhere. It is what
makes it possible to add servers, deploy without downtime, and lose a machine
calmly.
Cache generously, expire carefully. Most reads can be answered from
memory. The hard part is not the caching — it's knowing when the cached copy is too old
to use.
Doing work in the background contains failures. A queue stops one
slow component from dragging everything else down with it. You accept a bit of delay and
get resilience in return.
You can't fix what you can't see. Good logs and metrics are how you
know which bottleneck is next.
Things breaking is normal. Disks die, networks split in two, whole
regions go offline. Assume every part can fail at any moment, and design so that it
doesn't take the rest with it.
Active recall
Cover the answers. Say each one out loud before you tap to check.
Why start with a single server even for a "serious" product?
Think shipping speed and where the signal to scale comes from.
It's the fastest to ship, cheapest to run, and easiest to debug.
You add layers in response to real production metrics, not predictions — so the
single box is both the right start and the source of the data that tells you what
to do next.
What must be true of the web tier before horizontal scaling works?
It's about where session state lives.
It must be stateless: no information about a
particular user sitting in one server's memory. Sessions move to a shared store like
Redis, uploaded files move to object storage. Once that's true, any server can handle
any request, and no user is tied to a machine that might die.
Primary/replica replication scales reads — what bug does it introduce?
Replicas aren't instantaneous.
Replication lag. Replicas trail the primary by
seconds, so a read-after-write flow ("see the post I just made") can show stale
data. Route those reads to the primary or design the UI to tolerate the lag.
Why does a CDN reduce latency — and what limit does it work around?
Physics, not software.
The speed of light is fixed, so a far-away round trip has an
irreducible floor (~150 ms Mumbai↔Frankfurt). A CDN serves assets from an edge
near the user (~10 ms), moving the data closer rather than trying to move light
faster.
What is cache stampede, and how do you prevent it?
A hot key expires…
When a hot key expires, many requests miss at once and all rebuild
it together, hammering the DB. Prevent it with a rebuild lock (one request
repopulates) or jittered TTLs so keys don't expire in lockstep.
Why is sharding the last resort on the ladder?
Think reversibility.
It makes the system permanently more complicated. Queries that need
data from two shards become slow or impossible, and changing the number of shards later
is a huge migration. Replicas, caching, and a bigger machine are all easy to undo, so
you use those up first.
Check yourself
Q1 Your read-heavy app's database is the bottleneck and a single point of failure. What's the most natural next move?
Why: Primary/replica replication directly removes the
DB as a single point of failure and scales reads — exactly the read-heavy
case. Sharding is a later, heavier, harder-to-reverse step.
Q2 A user reports that immediately after posting, their post sometimes doesn't appear. Most likely cause?
Why: Read-after-write against a replica that trails the
primary by seconds is the classic symptom. Fix by routing that read to the primary.
Q3 Why do sticky sessions undermine horizontal scaling?
Why: If state lives in one server's RAM, the LB must
always route that user there — and a dead server means a dropped session. Moving
state to shared storage makes servers interchangeable.
Q4 Which is the best reason to put a message queue between your web tier and an email-sending worker?
Why: The queue lets the web tier return immediately and
absorbs bursts; workers drain the backlog at a sustainable rate, and the two sides
scale independently.
Q5 You're starting a new CRUD product expecting its first users. Which database choice does the chapter actually recommend?
Why: For most CRUD apps up to a million users, managed
Postgres wins on transactions, joins, and simplicity. NoSQL solves problems most
apps never have — pick it for a real access pattern, not the scalability halo.