Fast software rarely does the work twice. It answers from a cache — and hopes the cache is right.
Between a user and the truth sits a ladder of caches — the browser, a CDN at the edge, an in-memory store, the database, cold storage at the bottom. Each rung you can answer from is faster and cheaper than the one below it, but it answers with a saved copy that might be out of date. That trade — freshness for speed and cost — is why software feels quick or slow, and runs cheap or expensive. And its hard problem has a name: knowing the exact moment a cached copy became wrong.
Stack five caches at 90% each. How many requests reach the bottom?
Before you drive the ladder, stake a number. Picture five cache layers stacked over the database — the browser, a CDN, an edge cache, an in-memory app cache, a read replica — and give each one a 90% hit-rate: it answers 90% of the requests that reach it and passes the rest down.
One request, five rungs, five very different answers.
A request falls down the ladder until some layer can answer it. A hit high up returns fast and cheap; a miss falls through to the slower, pricier rung below. Toggle layers on and off, set how often each one already has the answer (its hit-rate), then send a request and watch where it stops. The readouts below are the maths over a thousand requests.
🔒 Locked — make your prediction above to unlock the ladder, then send a request and watch where it stops.
Why "just ask the database" isn't the whole story.
The naive model is one hop: every request recomputes the answer from the origin. Simple, always fresh — and, under load, slow and expensive, because the scarcest resource does all the work. A caching ladder answers most requests from a cheap, fast copy long before they reach the origin. The difference between the rungs isn't a tweak; it's orders of magnitude.
"The server just does it"
One hop, one latency, one cost — every request runs the full query against the database. Easy to reason about and always fresh, but the origin does all the work, so it's slow under load and the most expensive way to run.
A stack of caches
Most requests are answered by a fast, cheap copy before they ever touch the origin, which only does the real work once per change. Far faster and far cheaper — at the price of sometimes serving a copy that has gone stale.
| Rung | Answers from | Typical latency | Cost / 1k | Staleness risk |
|---|---|---|---|---|
| Browser cache | the user's own device (RAM / disk)nothing leaves the machine | ~1 ms | $0 | High — hard to purge remotely |
| CDN edge | a point-of-presence near the userhundreds of cities | ~20 ms | ~$0.02 | Medium — bound by its TTL |
| Application cache | in-memory store, in-region (Redis)~1 ms lookup, ~85 ms round-trip | ~85 ms | ~$0.10 | Medium — bound by its TTL |
| Database | SSD + query engine — the source of truth~10 ms query, ~130 ms round-trip | ~130 ms | ~$1.20 | None — always fresh |
| Cold storage | object store / another region / archivecheap to keep, slow to fetch | ~200–500 ms | ~$4.00 | None — always fresh |
Cacheability is a judgement call, not a default.
Whether to cache a piece of data — and for how long — comes down to two questions: how often does it change, and how bad is it to serve a stale copy? For each item below, make the call: don't cache it, cache it briefly, or cache it for a long time. Then see the reasoning.
Where the ladder shows up — and where it bites.
The same trade-off explains a system's best day and its worst outage. Four scenarios any team building for real traffic will recognise.
A global news site on a CDN
A wire story breaks and traffic spikes to millions of readers in minutes. The same article is served from CDN edges in hundreds of cities; the origin sees only a trickle of cache-fills.
A personalised feed
Every user's home feed is different, so there's no shared copy to cache at the edge — a cache hit for one person would be the wrong content for the next.
The price that went stale
A price drop shipped, but a 24-hour cache kept serving the old, lower number. Checkout honoured the cached price and the company undercharged every order for a day before anyone noticed.
The cache stampede
A hot key expired at midnight. Thousands of requests missed in the same instant and hammered the database in unison to rebuild it — the "thundering herd" — and the origin fell over under the pile-on.
The ladder is easy to draw. Living with it is the engineering.
"Just add a cache" is a one-line suggestion that quietly signs you up for a staleness problem. Teams that stay fast without shipping wrong answers treat caching as a deliberate trade — layer by layer, with a plan for how each copy gets invalidated. That's the systems judgement we build with the leaders and teams we coach: turning a diagram on a screen into calls you can defend.
Talk to us about coachingCaching trades freshness for speed
Every cache answers faster and cheaper with a copy that might be wrong. The question is never "cache or not?" in the abstract — it's how much staleness this particular data can tolerate.
Invalidation is the hard part
Adding a cache is easy; knowing the exact moment every copy became wrong is the part that ships bugs. Decide how each cached thing gets purged before you cache it — a TTL is a guess, not a guarantee.
Know your latency numbers
Memory is nanoseconds, SSD microseconds, a same-datacenter hop half a millisecond, a cross-region round trip a hundred. Which rung you answer from can change the user's wait by 100x. The numbers are the design.