Observability & SLOs

Monitoring answers questions you already asked

Monitoring is the questions you thought of in advance: you imagined a failure mode, built a panel and an alert, and now you find out when it happens. It works exactly as well as your imagination did. Observability is answering the questions you did not anticipate, without shipping code.

The failures you predicted got fixed; what remains is combinations nobody drew on a whiteboard, including the gray failures of failure-models — a slow-but-alive node passes every health check and surfaces only if you can slice latency by instance.

The three signals

Metrics cost the same at 10 requests or 10 million, but carry no per-request detail, and Prometheus holds 1–3 KB per active series — a million series is gigabytes of RAM.

Logs carry the detail metrics threw away, and scale with traffic: 1 KB per line at 50k req/s is 4 TB a day, billed on ingest. Sample routine paths; keep every error.

Traces are the only signal that answers which of the 14 services made this request slow. Metrics say the p99 rose; traces say which hop made it rise.

Structured logging

BAD:  ERROR: payment failed for user 8123 after 3 retries
GOOD: {"event":"payment_failed","user_id":8123,"gateway":"stripe","trace_id":"4bf9…"}

The prose line is greppable only if you guess the phrasing, and cannot be aggregated. The structured one answers event=payment_failed AND gateway=stripe grouped by hour — a query you type during the incident, not a regex you debug during it. A log line you cannot filter on is a log line you cannot use. Use the same keys everywhere, always including trace_id.

Distributed tracing

A trace is a tree. Each unit of work is a span carrying trace_id (constant for the whole request), its own span_id, and parent_span_id; those three rebuild the tree, the critical path, and per-service latency. The ids ride in headers — W3C traceparent:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             version  trace-id                     span-id          flags

Every service must forward it on every outbound call, HTTP, gRPC and queue messages alike. One that drops it severs the trace: everything downstream starts a fresh trace and is lost to you, and the gap reads as that hop being fast. Queues are the usual offender — the context must be written into the message and read back out.

Sampling. Head-based decides at ingress, before the request runs: cheap, stateless, and it throws away the rare error you wanted. Tail-based buffers a trace's spans at a collector and decides once the request finishes, so you keep every error and everything over 1 second — paid for in buffer memory. Tail-based is what you want; head-based is what you can afford at the top of scale.

RED and USE

RED — for things that serve requests (APIs, RPC handlers, queue consumers): Rate, Errors, Duration. User-visible, so this is what you alert on.

USE — for things that get consumed (CPU, disk, thread pools, connection pools, queues): Utilisation, Saturation (work queued waiting on the resource), Errors.

RED says users are affected; USE says what to do about it. A thread pool at 100% utilisation with a growing wait queue is the saturation signal that shedding and breakers in resilience react to.

Percentiles

Averages hide everything: 99% of requests at 50 ms and 1% at 5 s gives a mean of 99.5 ms — a number describing no request that ever happened, while one user in a hundred waits five seconds.

p99 matters more than its 1% suggests, because of fan-out. If a request touches N services, each independently p99-slow with probability 0.01, the chance of hitting at least one slow call is 1 − 0.99^N: 13% at N=14, 63% at N=100. Fan out far enough and p99 is the median experience — see latency-estimation.

Alert on symptoms, not causes

High CPU is not a symptom — a healthy service under load looks identical to a sick one. Page on what users feel: checkout success rate, error ratio, latency SLO.

Then alert on the burn rate of the error budget, not an instantaneous threshold. Budgets come from the SLO — see availability-math. A 99.9% monthly SLO permits 0.1% failures; burn rate 1 spends exactly that over 30 days, burn rate 14.4 spends 2% of it in an hour.

One threshold cannot win: tight enough to catch a total outage in minutes and it fires on every blip; loose enough to stay quiet and a steady 3% error rate burns the month silently. The standard answer is multi-window, multi-burn-rate.

Both windows must breach to fire: the long one suppresses blips, the short one makes the alert stop firing when the incident ends. Fast burn pages a human; slow burn opens a ticket, because a three-day leak does not need anyone awake. Compute all of this per region as well as globally — a green global SLO hides a dead region (multi-region).

Every page must be actionable. If the responder can only acknowledge it and go back to sleep, the alert is training the team to ignore pages — and it will be ignored on the night it is real. Track what fraction of pages led to an action; delete the rest.

Cardinality is the bill

Series count is the product of label cardinalities. Attach user_id in a service with two million users and you have created two million series, gigabytes of memory, and a finance ticket.

Keep labels bounded — templated routes (/orders/{id}, never /orders/88213), status class, region — and push unbounded dimensions into logs and traces. Exemplars link a histogram bucket to a trace id, so you jump from a p99 spike to a slow request.

What to take away

• Monitoring answers questions you asked in advance; observability answers the ones you did not.
• Metrics are flat-cost and cardinality-fragile; only traces attribute latency across services.
• Structured logs carrying trace_id are queryable; prose log lines are not.
• Every service must forward trace context, queues included, or it is severed downstream.
• RED for request-serving things, USE for consumable resources: RED alerts, USE diagnoses.
• Never average percentiles, and alert on burn rate — every page must be actionable.