Caching

Where caches live

Every layer between user and data can hold one, and each has a different invalidation story:

In-process caches are the fastest possible (no network hop) and the hardest to invalidate, because every instance has its own copy and they drift. They're excellent for data that is small, hot, and tolerant of staleness — feature flags, config, reference data — with a short TTL to bound the drift.

The patterns

Cache-aside (lazy loading). The application checks the cache, and on a miss reads the database and populates it. The default, and the one you should assume unless told otherwise. Only requested data is cached; a cache failure degrades to slow rather than broken. The downside is that every miss pays both lookups, and the first request after a write is stale unless you invalidate on write.

Read-through. Same shape, but the cache library does the loading. Cleaner application code, and it means the cache is on the critical path — if it's down, everything is down.

Write-through. Write to cache and database together. The cache is never stale, but every write pays both costs, and you cache data that may never be read.

Write-behind (write-back). Write to cache, flush to the database asynchronously. Fastest writes by a distance, and you will lose data if the cache dies before flushing. Use only where loss is acceptable — view counters, analytics — or with a durable buffer in front.

Refresh-ahead. Proactively refresh entries nearing expiry, so users never hit a cold one. Effective for a small, predictable hot set; wasteful otherwise.

Hit rate is logarithmic in cache size

Real traffic is skewed — a small fraction of keys serves most requests. That's why caches work at all, and it has a consequence people find surprising: most of the benefit arrives in the first few percent of capacity.

Things to notice:

The curve is steep then flat. At moderate skew, caching just 5% of the keyspace already serves 40–50% of requests depending on policy. Going from there to 40% — eight times the memory — only takes you to about 80%. Each additional nine of hit rate costs exponentially more RAM. This is the argument for sizing a cache from a measured hit-rate curve rather than from "how much can we afford".

Skew decides everything. Drag the Zipf slider. At high skew a tiny cache is spectacular; at low skew even a huge cache is mediocre. If your access pattern is close to uniform, caching is the wrong tool and no amount of memory will fix it.

LFU beats LRU here, but not always. LFU wins on a stable popularity distribution because it keeps the genuinely hot keys. It loses badly when popularity shifts, because an old-but-formerly-popular key has a high count and refuses to leave. Real systems use hybrids — segmented LRU, or W-TinyLFU (what Caffeine uses) — which approximate frequency while letting stale entries age out.

Invalidation

The hard part, and the reason for the joke.

TTL is the workhorse: bound staleness, let entries expire. Simple, self-healing, and requires no coordination. Choose the TTL from how stale the data may be, not from how often it changes.

Explicit invalidation on write gives freshness at the cost of coupling — every writer must know every cache key derived from that data, and one forgotten path means permanent staleness.

Versioned keys sidestep invalidation: include a version or content hash in the key (user:123:v7). Writers bump the version; old entries are never read again and expire on their own. This is what asset fingerprinting does, and it's the most reliable option when you can afford the key churn.

The four failure modes

Thundering herd (stampede). A hot key expires; a thousand concurrent requests all miss and all hit the database simultaneously. Fixes: single-flight — the first miss takes a lock and the rest wait for its result; probabilistic early expiry — each reader independently decides to refresh slightly before the TTL, so one lucky request refreshes while others still get the cached value; or stale-while-revalidate — serve the expired value and refresh in the background.

Cache avalanche. Many keys expire at the same moment — typically because they were all populated together at startup with an identical TTL. Fix: jitter the TTL (ttl ± 10%). One line of code, entire class of outage removed.

Cache penetration. Requests for keys that don't exist find nothing in the cache and go to the database every time — a trivially easy accidental (or deliberate) attack. Fixes: cache the negative result with a short TTL, or a Bloom filter over existing keys.

Cold start. The hook at the top. After a restart or a scale-out, hit rate is zero and the database sees full traffic. Fixes: warm the cache before taking traffic, restart nodes gradually rather than all at once, and know whether your database can survive 100% miss rate — if it can't, the cache is a single point of failure and should be treated as one.

What not to cache

• Data that changes faster than it's read — you pay invalidation for nothing.
• Data with a near-uniform access distribution — see the skew slider.
• Anything where staleness is a correctness bug and you can't bound it: balances, permission checks, inventory at the moment of sale. Cache the lookup, verify at the decision.
• Personalised responses at a shared layer, unless the cache key includes the user — the classic and severe bug where one user sees another's data.

What to take away

• Caches exist at every layer; the further from the origin, the harder to invalidate.
• Cache-aside is the default. Write-behind is fast and loses data.
• Hit rate is logarithmic in size and dominated by traffic skew — measure the curve.
• LFU suits stable popularity, LRU suits shifting popularity, hybrids suit reality.
• Four failure modes: stampede, avalanche, penetration, cold start. Each has a standard fix.
• A cache your origin cannot survive losing is a dependency, not an optimisation.