Load Balancing

L4 vs L7

The layer decides what the balancer can see, and therefore what it can do.

L4 forwards packets, so it's fast and protocol-agnostic — good for raw TCP, databases, and gigantic throughput. L7 understands HTTP, so it can route /api/* to one pool and /static/* to another, rewrite headers, retry idempotent requests, and enforce rate limits.

Most production stacks use both: L4 at the edge for volume and DDoS absorption, L7 behind it for routing intelligence.

The algorithms

Round robin — next server in rotation. Simple, and fine when requests are uniform and servers identical. Neither is usually true.

Weighted round robin — round robin with capacity weights. The standard fix for a fleet with mixed instance sizes.

Least connections — send to whoever has the fewest in-flight requests. Naturally adapts to uneven request costs, because an expensive request keeps a connection occupied. A good default for HTTP.

Least response time — least connections, weighted by observed latency. Better in theory; can oscillate, since everyone piles onto whichever server just looked fast.

Hash-based — hash the client IP or a URL component to pick a server. Gives you stickiness without cookies, and is how cache tiers get key affinity. Use consistent hashing so adding a server doesn't reshuffle everything.

Power of two choices — pick two servers at random, send to whichever has fewer connections. This one deserves more attention than it gets: it needs no global state, and it gets you most of the benefit of least-connections while avoiding the herd behaviour where every balancer simultaneously decides the same server is idle.

Health checks, and how they cause outages

Active — the balancer probes /health on a schedule. Passive — it watches real traffic and ejects a backend after N consecutive failures. Use both: active catches a server that's idle-but-broken, passive catches failure modes your probe doesn't cover.

The important design decision is shallow vs deep:

• A shallow check returns 200 if the process is alive.
• A deep check verifies the database connection, the cache, downstream dependencies.

Deep checks sound more responsible, and they are how you turn a partial outage into a total one:

Related trap: health-check-driven cascading failure. One server is ejected, its traffic redistributes to the rest, they get slower, they start failing checks, they get ejected. The cluster eats itself one node at a time. Rate-limit ejections and never let the pool shrink below a floor.

Sessions, and why you should not need stickiness

Sticky sessions (a cookie pinning a client to one server) are easy and they cost you:

• The server can't be drained without dropping sessions.
• Load becomes uneven and stays uneven.
• Autoscaling doesn't help the overloaded server.

The alternative is to make servers stateless: put session state in a shared store (Redis) or in a signed token (JWT) the client carries. Then any server can serve any request, and almost every operational problem above disappears.

Stickiness is still legitimate for long-lived connections — WebSockets pin to a server by nature — and for cache affinity, where you want the same key on the same node.

Graceful shutdown

The most common self-inflicted 502. Deploying means removing servers, and the sequence matters:

1. Mark the instance unhealthy (or deregister it) so it stops receiving new connections.
2. Wait for the balancer to notice — it's polling, so this takes a health-check interval or two.
3. Finish in-flight requests (connection draining), with a timeout.
4. Then exit.

Skipping step 2 is the classic bug: the process exits the instant it deregisters, while the balancer is still sending it traffic. Those requests become 502s during every deploy.

Where balancers sit

• DNS / anycast — the first tier, spreading users across regions. Coarse, and DNS caching makes failover slow.
• Global load balancer — anycast IP, routes to the healthiest nearest region.
• Regional L4 — absorbs volume, spreads across availability zones.
• Regional L7 — path and host routing, retries, rate limits.
• Service mesh / client-side — inside the cluster, each caller balances across instances itself, with no extra hop. This is what Envoy sidecars do.

The balancer must not be a single point of failure itself. In practice: an active-passive pair sharing a virtual IP, or an anycast address advertised from several places.

What to take away

• L4 is fast and blind; L7 understands HTTP and can route, retry, and rate-limit.
• Least connections beats round robin whenever request costs vary — which is nearly always.
• Power of two choices gets most of the benefit with no global state and no herding.
• Deep health checks can eject an entire fleet at once. Keep them shallow and fail open.
• Stateless servers remove the need for stickiness and everything it breaks.
• Graceful shutdown means deregister, wait, drain, exit — in that order.