Latency Numbers & the Estimation Reflex

The numbers worth memorising

You don't need precision. You need to never be wrong by a factor of a thousand.

If the table is too much, keep the three ratios:

• Memory is ~100× faster than SSD.
• SSD is ~100× faster than a spinning disk seek.
• A cross-continent round trip is ~1,000,000× a memory reference.

Data sizes, similarly rounded: 2^10 ≈ thousand, 2^20 ≈ million, 2^30 ≈ billion. A UUID is 16 bytes, a timestamp 8, a typical JSON API response 1–10 KB, a photo 2 MB, a minute of 1080p video ~50 MB.

The estimation recipe

Five steps, always the same order. Say each number out loud as you go — the interviewer is grading the derivation, not the answer.

1. Users → requests. DAU × actions per user per day ÷ 86,400 = average QPS.
2. Average → peak. Multiply by 2–3 for daily peak. Higher if there's a scheduled event (a match kickoff, a flash sale, a 09:00 commute spike).
3. Requests → resources. Per request: how many bytes over the wire, how many DB reads, how much CPU.
4. Storage. Items × size × replication factor × retention. Then add index overhead.
5. Bandwidth. QPS × payload size, in both directions. Don't forget egress cost.

Useful constant: 86,400 seconds in a day ≈ 100,000. So 1 M events/day ≈ 12/sec, and 1 B/day ≈ 12,000/sec. That single approximation does most of the work.

Worked example: driver location pings

A ride-hailing service across Switzerland and Austria.

Given: 200,000 drivers online at peak, each pinging location every 4 seconds. A ping is driver_id, lat, lon, heading, timestamp — call it 64 bytes of payload, 200 bytes on the wire with headers.

Write throughput

200,000 drivers ÷ 4 s   = 50,000 writes/sec at peak
Average, off-peak       ≈ 15,000 writes/sec

50k writes/sec is the number that decides the architecture. It rules out "just write each ping to Postgres" — that's a sustained 50k row-inserts/sec, achievable but expensive and wasteful for data that's obsolete in 4 seconds.

Bandwidth

50,000 × 200 bytes = 10 MB/s  = 80 Mbit/s inbound

Trivial. Bandwidth is not the constraint here — which is itself worth saying, because it tells the interviewer you checked rather than assumed.

Storage, if we kept everything

50,000/s × 86,400 s   = 4.3 B pings/day
4.3 B × 64 bytes      = 276 GB/day raw
× 3 replicas          = 830 GB/day
× 90 days retention   = 75 TB

75 TB to answer "where was this driver last Tuesday". That forces the real design question: do we need the history at all? Usually the answer is: keep current position in memory (200k drivers × 64 bytes = 13 MB, which fits in a single Redis instance with room to spare), and downsample the trail — one point per 30 seconds, kept 30 days, is 1/8th the volume.

Why you cannot run servers at 95%

Here's the number people get wrong most often. A server at 90% utilisation does not have 10% headroom — its latency has already multiplied.

Queueing theory says that with random arrivals, average latency scales as 1/(1-ρ) where ρ is utilisation. At 50% you wait about as long as you're served. At 90%, ten times. At 99%, a hundred times. Drag the slider:

Two things to take from it:

The knee is real and it's around 70–80%. Below it, latency is roughly flat and traffic growth is cheap. Above it, small traffic increases produce large latency increases. This is why capacity plans target 60–70% peak utilisation — the remaining 30% isn't waste, it's the buffer that keeps p99 sane.

Pooling beats splitting. Push the server slider up. Sixteen servers behind one balancer at 90% utilisation sit at ~1.4× service time; a single server at 90% sits at 10×. Same utilisation, radically different latency, purely because a shared queue absorbs bursts that would stall an individual queue. This is the mathematical argument for load balancing, for connection pooling, and against pinning work to specific nodes.

And note what the tail does. The p99 stat is roughly ln(100) ≈ 4.6× the queueing component, so at 80% utilisation on one server your mean is 5× but your p99 is around 19×. Tail latency degrades far faster than the mean, which is why an alert on average response time will not fire until users have been suffering for a while.

Sanity numbers for servers

Rough, defensible, and enough to reason with:

State your assumption when you use one: "I'll assume 1,000 QPS per app server, so 50k peak needs 50 servers, call it 65 with headroom." An interviewer who disagrees with 1,000 will say so, and now you're having a design conversation instead of guessing silently.

What to take away

• Three buckets: ns for CPU/memory, µs for SSD/local network, ms for disks and distance.
• 86,400 ≈ 100,000. Peak ≈ 2–3× average.
• The five-step recipe: users → QPS → peak → per-request cost → storage and bandwidth.
• An estimate is a design tool. If it doesn't change anything, you asked the wrong question.
• Latency scales as 1/(1-ρ). Plan for 60–70% peak utilisation, and remember tails degrade faster than means.
• Pooling requests across servers beats partitioning them into separate queues.