Resilience Patterns

Containment, not prevention

Failure models covers timeouts from measured p99, deadline propagation, backoff with jitter, and retry budgets. Assume all of it. This lesson contains the failure that gets through anyway. Every pattern below is the same trade: give up work, capacity or fidelity, and the failure stays local.

Circuit breakers

A breaker wraps one dependency and holds three states.

Closed — normal. Requests pass; outcomes go into a rolling window.

Open — the dependency is presumed broken, so calls fail in microseconds: no socket, no thread parked. It helps twice, and people forget the second half. The caller stops burning resources on doomed calls, and the dependency stops receiving load it cannot serve, which is often the only way it recovers.

Half-open — after a cooldown of 5–30 seconds, a few probes are admitted. All succeed, it closes. Any fails, it re-opens with a longer cooldown, so a service that just came back does not take full traffic on a cold cache.

What trips it

An error rate over a rolling window — never a raw count. Twenty failures means nothing until you know whether it was out of 25 requests or 25,000. But rates carry their own trap: at low volume a rate is mostly noise.

Take a 50% threshold over a 10-second window, on an instance seeing 4 requests per window, against a dependency whose true error rate is a healthy 5%. Two unlucky errors read as 50%. That happens in about 1.4% of windows — roughly 120 false trips a day, per instance. Widen the gate to 20 requests and seeing 10+ failures at a 5% true rate is under one in a million.

So the rate needs a volume threshold beside it: do not evaluate until the window holds enough requests to mean anything. Hystrix's defaults — 20 requests minimum, 50% errors, a 10-second window — are a sane start because that gate is in them.

Bulkheads

Named for ship compartments: a separate pool per dependency — threads, connections, or a concurrency semaphore.

With one shared 200-thread pool, as in the hook, the slowest dependency eventually owns all 200 and unrelated endpoints die. Give recommendations its own pool of 20: when those are parked, call 21 is rejected instantly, 180 threads remain, /login never notices. Blast radius shrinks from the service to one feature.

The cost is utilisation: pools sized for each dependency's peak sum to more than one shared pool sized for the aggregate peak, because a partition cannot lend idle capacity. A pool per tenant applies the idea to noisy neighbours (multi-tenancy).

Load shedding

Above capacity you queue the excess or reject it. Queueing feels kinder and is wrong: a queue adds no capacity, it converts overload into unbounded latency.

A service serving 800 rps receives 1,000. The backlog grows 200 per second; after a minute it holds 12,000 requests and the newest arrival waits 12,000 / 800 = 15 seconds. Every caller timed out at 1 second. You are at 100% utilisation and near-zero goodput. A rejection costs microseconds; a timeout costs the full deadline on both sides.

Shed by priority. Classify at the edge and drop in order: analytics beacons and prefetch, then recommendations, then browse, and only last checkout, payment and auth. Never shed health checks, or the load balancer pulls the instance and hands its traffic to a fleet already struggling.

LIFO under overload is counter-intuitive and real. FIFO hands you the oldest request — the one whose caller has most likely given up — so the work is wasted and the next one is just as stale. LIFO serves the request with the most deadline budget left. The cost is honest unfairness: the tail of the queue starves, so pair it with a deadline check.

Graceful degradation

The arithmetic from availability math returns here. Serial dependencies multiply: five services at 99.9% give 0.999⁵ = 99.5%, about 43 hours down a year. Degradation is not a nicer error page — it removes a term from the product. Make recommendations, avatars and personalisation optional and checkout depends on two services, not five: 0.999² = 99.8%, a four-fold cut in downtime with no new hardware.

Recommendations time out → serve an hourly popular-items list; conversion dips, the page renders. Fraud scoring times out → auto-approve below a value threshold and queue for asynchronous review, decline above it: a decision about accepted loss that engineering does not own. Ranking down → return unranked lexical results.

Chaos engineering

Untested failover is a hypothesis. The breaker whose fallback throws on its first real call. The bulkhead pool sized at 200 on a 200-thread server, so it is not a bulkhead. Both pass review and are discovered at 03:00.

The method: state a steady-state hypothesis in business terms — orders per minute, not CPU — then inject one failure at the smallest survivable blast radius, with an automatic abort.

The value is not the experiments that pass. It is finding the failure modes you did not design for — the fallback that itself calls the failing dependency, or the dashboard that goes blank exactly when needed, a verdict on your observability as much as your architecture.

The patterns side by side

What to take away

• Breakers trip on an error rate over a rolling window, gated by a minimum request volume.
• Half-open probes decide whether to close; over-sensitive thresholds flap and amplify incidents.
• Bulkheads trade utilisation for a blast radius of one dependency, not the service.
• Shedding beats queueing: a queue turns overload into unbounded latency and zero goodput.
• Shed by priority, and use LIFO under overload — the newest request still has a caller.
• Degradation removes a term from the serial availability product; chaos testing proves it.