Replication

Why replicate at all

Three reasons, and it matters that you can name them separately, because a design that serves one may actively harm another:

1. Availability — a machine dies and the data is still there.
2. Read throughput — spread reads across copies.
3. Locality — put a copy near the user so reads don't cross an ocean.

The tension: availability wants copies to be independent, consistency wants them identical, and you can only keep them identical by making writes wait. Every replication scheme below is a different answer to how long the write waits, and for whom.

Single-leader replication

One node accepts writes. Everyone else follows.

The leader appends each write to a log and ships it to followers. What's in that log is a real design decision:

Synchronous, asynchronous, and the useful middle

The knob that matters is when the leader tells the client "done".

• Asynchronous — leader acks immediately. Fast. If the leader dies before the log ships, those acknowledged writes are gone. Not delayed — gone.
• Synchronous — leader waits for follower confirmation. Durable, but now your write latency includes a network round trip, and if that follower is slow, writes stall.
• Semi-synchronous — wait for one follower, let the rest lag. This is the common production choice: you survive a single-node loss without paying for the slowest replica.

Failover is the hard part

Leader dies. Now what?

1. Detect it — usually a timeout. Too short: you fail over on a GC pause. Too long: you're down for that whole window. There is no correct value, only a chosen tradeoff.
2. Elect a new leader — needs consensus, or you get two.
3. Redirect clients, and make the old leader stand down when it returns.

Two failure modes worth naming:

• Split brain — old leader comes back thinking it's still leader; two nodes accept conflicting writes. Fencing tokens (a monotonically increasing epoch number that storage checks and rejects when stale) are the standard defense.
• Lost writes — the new leader never received the tail of the old leader's log. GitHub had a well-known incident of exactly this shape, where MySQL autoincrement IDs already handed out by the dead leader were reused by the new one, and Redis keys then pointed at the wrong records.

Replication lag, and the three guarantees you actually want

With async replication, followers trail the leader. Usually milliseconds. Under load, seconds. During a backlog, minutes. Three distinct anomalies follow — and the fix for each is different, which is why they have separate names:

Read-your-own-writes. You write, then read from a lagging follower, and your own change is missing. That's the hook at the top. Fixes: route reads that could have been affected by your own recent writes to the leader; or track the client's last-write timestamp/LSN and only serve from a replica caught up past it.

Monotonic reads. You read from replica A (fresh), refresh, hit replica B (stale), and time appears to run backwards. Fix: pin each user to one replica, usually by hashing the user ID — not randomly per request.

Consistent prefix reads. You see an answer before the question. Fix: ensure causally related writes go through the same partition, or track causal dependencies explicitly.

Multi-leader replication

Accept writes in more than one place. You'd do this for:

• Multi-datacenter — writes are local and fast; cross-DC replication is async.
• Offline-capable clients — a phone with a local database is a leader.
• Real-time collaborative editing — every cursor is a leader.

The cost is a genuinely hard problem: write conflicts. Two leaders accept conflicting writes to the same record, and there is no global order to appeal to.

Leaderless replication

Dynamo-style. The client (or a coordinator) writes to several nodes at once and reads from several at once. No failover, because there's nothing to fail over.

With N replicas, W write acks required, and R read responses required:

If R + W > N, the read set and write set must overlap by at least one node, so a read is guaranteed to touch a node holding the newest write.

With N=3, W=2, R=2 you tolerate one node down for both reads and writes. With W=3, R=1 you get fast reads and no write availability under any failure. Staleness is repaired in the background by read repair (fix the stale replica during a read that noticed it) and anti-entropy (a background process comparing replicas, usually via Merkle trees).

Worked numbers: what synchronous replication costs in Europe

Approximate round-trip times:

Take a service doing a 1 ms commit, targeting 200 ms p99 for a request that makes 4 sequential writes.

• Async, single region: 4 × 1 ms = 4 ms. Fine. Risk: acked writes lost on leader death.
• Sync to a same-AZ replica: 4 × 1.5 ms = 6 ms. Basically free. Survives a machine. Does not survive an AZ.
• Sync Zurich → Frankfurt: 4 × 9 ms = 36 ms. Survives losing a whole datacenter, and still fits the budget — this is the sweet spot for EU-regulated workloads.
• Sync Zurich → us-east-1: 4 × 91 ms = 364 ms. Blows the budget by itself.

The design move that falls out: synchronous within a region, asynchronous across regions, and accept that a region loss may lose the last few seconds of writes. If that loss is unacceptable, you need consensus across regions and you pay the 90 ms — there is no third option, and saying so plainly is the mark of a senior answer.

What to take away

• Replication buys availability, read throughput, and locality — and every scheme trades one against consistency.
• Single-leader is the default. Its hard parts are failover and lag, not the happy path.
• Read-your-writes, monotonic reads, and consistent prefix are three different anomalies with three different fixes. Name them precisely.
• Multi-leader and leaderless both hand you conflict resolution. Last-write-wins is a data loss policy wearing a costume.
• R + W > N bounds staleness. It does not give you ordering.