Three reasons, three different systems
"Let's go multi-region" sounds like one decision. It is three, and the wrong one buys an expensive system that solves a problem you did not have.
Latency and proximity. Singapore users are 160 ms from Frankfurt. A read-path problem, solved with edge caching and read replicas, leaving the write path alone.
Disaster recovery. A region can genuinely disappear — fibre cuts, control-plane outages, a provider's bad day. A continuity problem, measured in RTO and RPO.
Data residency. EU personal data must stay in the EU — GDPR Article 44, Swiss FADP, sector rules for health and finance. A constraint, not an optimisation: it makes nothing faster and cannot be traded away.
In Europe the third is usually the real driver, and the one candidates handle worst: if a regulator says German records may not leave the EU, the topology is chosen for you.
The topologies
| Topology | Buys you | Costs you | RTO / RPO |
|---|---|---|---|
| Single region + backups | Cheapest | Region loss is an outage | Hours; RPO = backup interval |
| Active-passive standby | DR, no conflicts | Idle capacity; standby rots | 5–60 min; RPO = replication lag |
| Read-local, write-global | Fast local reads | Writes cross to the owner | Write-region failover; RPO ≈ 0 |
| Partitioned active-active | Local reads and writes | Routing complexity | Per partition; RPO = its lag |
| Full active-active | Writes anywhere | Merge semantics leak into product | ≈ 0 both; convergence ≠ correctness |
Active-passive is where most teams are, and its failure mode is not technical: the standby is never exercised, so the first real failover is the first test. Full active-active is what people reach for and can least defend: two regions writing one record means you own a merge function, and there is none for "decrement the balance".
The physics
| Path | RTT |
|---|---|
| Zurich ↔ Frankfurt | ~8 ms |
| Zurich ↔ Dublin | ~28 ms |
| Zurich ↔ us-east-1 | ~90 ms |
What matters is RTT times the number of sequential writes per request. Replication does the arithmetic: four sequential 1 ms commits cost 4 ms locally, 36 ms sync to Frankfurt, 364 ms sync to us-east-1 — blowing a 200 ms p99 budget alone. Hence: synchronous within a region, asynchronous across regions.
Then price it. Async replication loses every write not yet shipped when a region dies — normally a couple of hundred milliseconds, but under the load spike that often precedes failure, seconds. At 3,000 writes/second with 5 seconds of lag, roughly 15,000 acknowledged writes vanish. Get that number agreed in advance.
RTO and RPO
- RTO — how long the service may be unavailable before it is back.
- RPO — how much data may be lost, as time: RPO of 30 seconds means the last 30 seconds of acknowledged writes may vanish.
They are independent: RTO 10 s with RPO 5 min is a fast failover to a lagging replica; RTO an hour with RPO zero is a careful restore. Conflating them is a tell.
Partition by home region
Give every tenant exactly one home region, authoritative for its data. Zurich owns the Swiss tenants, Dublin the Irish ones; other regions hold read-only replicas and writes route home.
Nothing needs reconciling. Conflict resolution is not hard here, it is absent — one writer per record means conflicting concurrent writes cannot exist. You get local reads everywhere and per-tenant failover instead of all-or-nothing.
The costs are bounded: routing must know each tenant's home, cross-region tenant-spanning work becomes distributed transactions you should refuse to build, and migration needs a write freeze. This is partitioning applied geographically, the trade PACELC names — consistency locally, latency globally.
Cell-based architecture
Home regions partition the data. Cells partition everything.
A cell is a complete stack — load balancers, app servers, database, caches, queues — serving a subset of customers. Ten cells, each with 10% of tenants, sharing no state, each sized to be deployed, load-tested and lost as a unit.
The point is blast radius. Redundancy assumes independent failures, and availability math shows the flaw: the correlated-failure term does not shrink as you add replicas. A bad deploy or one tenant's pathological query hits every replica sharing the code path or data store. Cells cut that term: no shared component exists for correlation to travel through. Redundancy makes each instance less likely to fail; cells make a failure affect fewer people.
- Bounded impact. A poison-pill request that crashes the handler takes one cell: a 10% outage, not a 100% one.
- Deploy by cell. Ship to cell 1, watch real traffic for an hour, then cell 2. A bad release is caught at 1/N of traffic by real users, the most valuable property in practice.
- Residency boundary. EU cells hold EU data, so compliance is a list of cells.
Cell routing is the one shared component, so it can take everything down at once. It must be boring: a thin customer-to-cell mapping in a tiny replicated store, no business logic, changing on a timescale of weeks. Every feature added there — rate limiting, auth, rewriting — rebuilds the failure domain cells exist to remove.
Failover is the hard part
Automatic failover risks split brain: the standby cannot distinguish "the primary is dead" from "I cannot reach the primary", so both may accept writes. It also flaps: a 30-second blip promotes the standby, the old primary returns, and you have two divergent histories. That is the partial-failure ambiguity in failure models; the defence is a fencing token from a consensus system, per coordination.
Manual failover pays for a human: paging, diagnosis, a decision under pressure. Realistic RTO is 15 to 60 minutes, worse at 03:00. The usual compromise: automatic detection, a scripted runbook, human approval on one button.
Untested failover does not work. Not "might not" — does not. DNS TTLs nobody checked, pools that never re-resolve, a standby stuck read-only, a runbook naming a dead host. The only evidence is having done it recently, in production.
Residency versus caches
A global CDN happily replicates a response containing EU personal data to a PoP in Ohio — that is what a CDN is for. The work sits in the layers people forget: cache keys and PoP regions, log pipelines shipping to one store, backups replicated cross-region, analytics warehouses aggregating by design. Residency-aware regional caching works, and costs hit rate, because the cache is split along a boundary unrelated to access patterns.
What to take away
- Latency, disaster recovery and residency are three problems with three designs.
- Synchronous within a region, asynchronous across regions; state the loss window in writes.
- RTO is time to recover, RPO is data lost; RPO zero across regions costs latency per write.
- Home-region partitioning removes conflicts: exactly one authoritative writer per record.
- Cells bound blast radius and catch a bad release at 1/N of traffic; keep the router boring.
- Failover you have not rehearsed in production does not work.
Check yourself
-
A European fintech must keep customer data inside the EU. How does that change the design?
Residency removes the choice rather than tuning it, and the database is the easy part: the leaky paths are CDN PoPs, central log pipelines, backups and analytics. Encryption does not help, since the rule constrains location.
-
A service replicates asynchronously from Zurich to Dublin with 4 seconds of lag at 2,000 writes/second. Zurich is lost. What is the realistic RPO?
Async replication acknowledges before the remote region holds the data, so everything in flight is lost: lag times write rate. The 28 ms figure is what synchronous replication would have cost; promotion time is RTO.
-
What do cells give you that adding more replicas within a region does not?
Replicas share a code version, a deploy pipeline and often a data store, so the correlated term does not shrink as you add them. Cells share nothing, containing a bad release to 1/N of customers, though they cost more to operate.
-
Which measure most directly addresses the split-brain risk of automatic promotion?
A standby cannot tell a dead primary from an unreachable one. A lease makes write authority exclusive and revocable, so the old primary is rejected on return. Faster health checks make flapping likelier and longer TTLs slow recovery.