Multi-Tenancy

The isolation spectrum

One question, asked repeatedly: how much infrastructure do two customers share?

Left to right, isolation is bought with money and effort. A shared Postgres serving 5,000 small tenants might cost two dollars per tenant per month; a dedicated cluster has a floor of a few hundred whether that tenant sends ten requests a day or ten million. And the ops cliff is steep: a migration taking 4 seconds on one database takes over five hours serially across 5,000, failing partway on some. That is not a migration, it is a resumable job runner with per-tenant state and a fleet running two schema versions for hours.

Hybrid needs a routing layer — a tenant record saying where that tenant's data lives, resolved per request. Build it on day one.

The bug that ends companies

The most dangerous defect here is not downtime. It is one missing WHERE tenant_id = ? — silent, usually found by the customer, and unrecoverable in the sense that matters: you cannot un-show data. Defences, strongest first.

1. Row-level security in the database. A policy filters every query by the session's tenant context, so a forgotten clause returns zero rows instead of everyone's — the only defence that fails closed. Costs: the session variable must be set on every connection checkout (easy to break with a pooler), plans can degrade, and table owners bypass policies.

2. A data layer that cannot express an unscoped query. No raw SQL in handlers; the repository takes a tenant context as its first argument and offers no constructor without one. Unscoped access lives in one audited file for migrations and admin jobs. Weaker than RLS — convention, not enforcement — but where most teams start.

3. Mandatory tenant context in the request path. Resolve the tenant once, at the edge, from the authenticated principal — never from a body field, query parameter or client header. Taking tenant_id from user input turns authorisation into a guess.

4. Two-tenant tests. Seed two tenants with identically shaped data, run every endpoint as A then as B, and diff: no identifier from one may appear in the other. Generate them from your route table.

The noisy neighbour

Shared infrastructure means one tenant's behaviour becomes everyone's latency.

Shared CPU and IO. One tenant opens a report that aggregates 40 million rows unindexed. The buffer cache is evicted, disk queue depth spikes, and p99 for every other tenant goes from 80ms to 4 seconds. Nobody did anything wrong; the query was merely permitted.

Shared connection pool. Your service holds 100 connections. One tenant's spike, or a retry storm from their integration, takes 95. Everyone else queues, times out, retries — and the retries make it worse. A queueing collapse, not a capacity problem.

Hot shards. One tenant 100x the median.

The fixes, each with a cost. Per-tenant rate limits and quotas — a token bucket keyed by tenant on requests, rows scanned and storage; cheap, but they cap request count not request cost, so pair them with statement timeouts. Per-tenant pool caps — bulkheads, in the vocabulary of resilience: no tenant holds more than 20 of the 100 connections, so it can only degrade itself, paid for in idle headroom. Queue fairness — per-tenant queues with weighted round-robin, so a tenant enqueuing 2 million jobs does not delay one with three. Moving the outlier out — quotas alone make a huge tenant unhappy.

Skew is the default

Tenant sizes are heavy-tailed: the largest is routinely 100x to 1000x the median. That wrecks the obvious partitioning choice. Sharding by tenant_id gives beautiful locality — a tenant's tables land together, joins stay local, per-tenant operations are trivial — and one hopeless shard holding your biggest customer. Better hashing cannot help: the unit being hashed is indivisible, so a tenant either fits on a shard or it does not.

Two fixes. Make the key composite, (tenant_id, entity_id), so a giant spans shards at the cost of its single-shard query. Or lift the outlier onto dedicated infrastructure — usually cheaper.

The operations nobody plans for

Point-in-time restore for one tenant. A customer bulk-deletes 40,000 records at 14:30 and calls at 16:00. With database-per-tenant you restore theirs to 14:29. On a shared schema you restore the entire database to a scratch instance, extract that tenant's rows across 80-odd tables in foreign-key order, and merge them back while live tables take writes — days of work, improvised mid-incident. Raise it in an interview; it separates people who have run a multi-tenant system from those who have drawn one.

Export on offboarding. Contracts increasingly demand a machine-readable dump in a fixed window. Build it early; it doubles as your migration tool.

Complete deletion. GDPR erasure covers replicas, caches, search indexes, the warehouse, logs and backups. Backups are the real problem — you cannot delete rows from an immutable snapshot. The answer is crypto-shredding: a per-tenant key, destroyed.

Cost, rollout, and the control plane

You cannot price what you cannot measure. Tag metrics with the tenant — requests, CPU-seconds, stored bytes, egress, queue wait — so observability can answer what each tenant costs. Without it a flat per-seat price silently subsidises the heaviest 1%. Watch cardinality: 5,000 tenants is fine, per-user tags are not.

Schema changes on shared infrastructure must be backward compatible, because two code versions run against one schema: expand, backfill, switch reads, contract — never a rename in one step. Per-tenant flags then stage the rollout: internal tenant, five friendly tenants, 5%, everyone.

Finally, split control plane (provisioning, routing, metadata, keys) from data plane (serving requests). The control plane must never sit in the request path — if every request calls it to resolve a shard, its availability multiplies into yours. Cache routing with a TTL: its outage should stop onboarding, not serving.

What to take away

• Isolation is a spectrum; mature SaaS ends up hybrid, so build the routing layer early.
• A missing tenant filter is the defining failure mode — defend with RLS and a scoping data layer.
• Code review is not a control; controls fail closed on their own.
• Contain noisy neighbours with per-tenant quotas, bulkheaded pools and queue fairness.
• Tenant sizes are heavy-tailed, so sharding by tenant creates hot shards; move outliers out.
• Single-tenant restore, export and erasure are hard on a shared schema and always urgent.