Failure, Timeouts & Partial Failure

Partial failure is the whole problem

In a single process, failure is total: the process crashes and everything stops. Reasoning is easy because there are two states.

In a distributed system, some things work and others don't, and you can't reliably tell which. A node is up but unreachable from you and reachable from someone else. A disk is returning corrupt data while answering health checks cheerfully. A network link works in one direction only. These are all normal Tuesday occurrences at scale.

The eight fallacies

Peter Deutsch's list, still perfectly accurate, and each one is a real outage:

1. The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4. The network is secure.
5. Topology doesn't change.
6. There is one administrator.
7. Transport cost is zero.
8. The network is homogeneous.

The useful exercise isn't memorising them, it's noticing which one a given design assumes. A synchronous call with no timeout assumes #1 and #2. A chatty API assumes #2 and #7. A service that caches DNS forever assumes #5.

You cannot distinguish slow from dead

This deserves its own heading because everything else follows from it.

You send a request and get no reply. Possible causes: the node crashed; the node is alive but paused (GC, VM migration, a disk stall); the request was lost; the reply was lost; the network is partitioned; the node is fine but overloaded.

From your side, every one of these looks identical. There is no message you can send and no observation you can make that distinguishes them. The only tool is a timeout, and a timeout is a guess.

The two generals problem makes this formal: two parties communicating over a lossy channel can never be certain they agree, because the last message's delivery is always unconfirmed. There's no protocol that fixes it — you can only make the uncertainty window small and design so that being wrong is survivable.

Choosing a timeout

Too short: you declare healthy nodes dead. They're still processing your request, you retry, now there are two. Under load, response times rise, more requests time out, more retries arrive, and you've built a positive feedback loop into an outage.

Too long: you hold a thread and a connection for 30 seconds while the user stares at a spinner, and your own caller times out anyway.

Practical guidance:

• Set timeouts from measured p99 latency, not from a round number. A timeout of 3× p99 is a reasonable starting point.
• Deadline propagation beats per-hop timeouts. Pass a deadline down the call chain — if the caller allowed 300 ms and 250 ms has elapsed, the downstream call gets 50 ms, not its own default 5 seconds. gRPC does this natively.
• Timeouts must shrink as you go deeper. If a handler with a 1 s budget calls a service with a 2 s timeout, the inner timeout never fires and does nothing.
• Prefer phi-accrual failure detectors over fixed thresholds where available. Instead of a binary alive/dead at a hard cutoff, they output a suspicion level from the observed distribution of heartbeat arrivals, adapting to a network that's simply slower today.

Retries, and how they cause the outage

Retrying is the obvious response to failure, and it's how a small problem becomes a large one.

Retry amplification. A request passes through 4 services, each retrying 3 times. A failure at the bottom produces 3⁴ = 81 requests. Your struggling service gets hit with 81× load at exactly the moment it can least handle it. This is why an overloaded system often can't recover even after the original trigger is gone.

Defenses, in order of importance:

1. Retry only idempotent operations — or make them idempotent with a key. See the idempotency lesson.
2. Exponential backoff with jitter. Backoff alone still synchronises clients into waves; the jitter is what decorrelates them. Full jitter (sleep = random(0, base × 2^attempt)) is the standard.
3. Retry budgets — cap retries as a fraction of total requests (say 10%), not per request. This bounds amplification globally rather than locally.
4. Retry at one layer only. Retries at every layer multiply. Pick the layer that knows enough to retry meaningfully and disable it elsewhere.
5. Circuit breakers — stop calling a failing dependency and fail fast, giving it room to recover.

A taxonomy worth knowing

Almost all infrastructure assumes crash-recovery with omission and timing faults, and explicitly not Byzantine faults — because Byzantine tolerance needs 3f+1 nodes instead of 2f+1 and far more messages. If an interviewer asks about Byzantine fault tolerance, the useful answer usually includes "and this is why we don't do that inside one trust domain."

Designing for it

• Assume every remote call can fail, hang, or partially succeed. There is no such thing as a call that "just works".
• Make operations idempotent so retrying is safe. This single property removes most of the ambiguity at the top of this lesson.
• Bulkhead — separate thread pools and connection pools per dependency, so one slow downstream cannot exhaust everything.
• Shed load rather than queueing indefinitely. A fast 503 is better than a 30-second wait, for both parties.
• Degrade gracefully — as covered in availability math, turning a hard dependency into an optional one changes the arithmetic entirely.

What to take away

• Partial failure is the defining property: some things work, and you can't tell which.
• You can never distinguish a slow node from a dead one. Timeouts are guesses with consequences.
• The two generals problem means agreement over a lossy channel is never certain.
• Set timeouts from p99, propagate deadlines, and shrink them going down the stack.
• Retries multiply. Use backoff with jitter, retry budgets, and retry at one layer.
• A slow node damages you more than a dead one. Shed load and fail fast.