A popular post takes down the database
By Zeeshan Adil · 4 min read
A cached comment list expires while 20,000 people a second are reading it. The database does not just slow down for a moment: it goes down and stays down. This challenge reproduces that with a fleet simulator and real code, then fixes it in the order that matters, measuring every step.
THE PROBLEM
The problem
Your Comments API serves GET /posts/p123/comments?limit=20. A post goes viral: 20,000 requests a second, spread over 20 API processes. A load test says PostgreSQL can safely run 500 of these queries a second. That is fine, because almost every answer comes from a cache, a reusable copy of the database result.
Then the cached entry for p123 expires. Thousands of requests miss in the same instant, all of them query PostgreSQL, PostgreSQL slows down, and the slower it gets the more requests pile up. That is a cache stampede. Randomizing expiry times, the usual advice, helps when many different entries expire together. It does nothing for thousands of requests missing the same one.
The product allows public comment lists to be up to two minutes old during an incident. Creating comments must stay available. Keep both promises.
STEP 1 · PREDICT
p123's cache entry expires. 20,000 reads a second, a database that safely runs 500 queries a second, and each read may wait 500 ms. How long until readers get comments again?
- Under a second: the first query refills the cache
- A few seconds, while the backlog drains
- Never, as long as the traffic lasts
Write your answer down. The rest of the page will tell you.
STEP 2 · BUILD IT
Build the read path of a Comments API that keeps PostgreSQL alive when a hot entry expires, and prove it against 20,000 reads a second.
Your solution should:
- Store each response with
source_read_at,refresh_after(30 s) anddiscard_after(120 s). Fresh: serve it. Stale: serve it at once and refresh behind it. Past the hard limit or missing: wait briefly for a refresh, otherwise 503. - Copies between caches keep their timestamps, and the age starts when the query started, not when its result arrived.
- In each process, keep a map of
cache key -> refresh in progress, so concurrent requests share one refresh. Waiters give up at their own deadline; a slow key never blocks another key. - Give refreshes their own budget, a rate and a concurrency cap, and divide a fleet-wide budget by the number of processes. Out of budget means serve stale or reject, never query anyway.
- Keep a bounded local cache in each process, so losing ElastiCache does not lose every copy, and keep writes on their own pool. A successful
POSTreturns the new comment.
Stuck? Hints
- Work out how long one query takes once every connection is busy: 500 queries sharing 8 cores. Compare it with the deadline.
- Serving stale without a per-key refresh map moves the stampede to the background. Readers are happy and the database still drowns.
- Coalescing protects one key. Ten thousand different missing keys need a budget.
STEP 3 · CHECK IT
Run the tests that hold it to the rules
git clone https://github.com/ZeeshanAdilButt/distributed-systems-challenges cd distributed-systems-challenges uv sync --all-packages uv run stampede # opens http://127.0.0.1:8091
No Postgres, no Docker, no AWS account, and it runs on Windows too. The page replays every scenario second by second: the fleet, ElastiCache, PostgreSQL's CPU and p123's entry. A second card drives the real code: 100 threads at one cold post through a real SQLite file. Each part also has its own script. What the six parts show:
| part | what you'll see |
|---|---|
| 1 · The stampede | 0% of reads served from the expiry on; PostgreSQL at 100% CPU finishes no query in time, and never recovers |
| 2 · Jitter is not enough | 10,000 posts: the worst second drops from 8,899 queries to 908. One hot post: 263 queries either way |
| 3 · Serve stale, refresh behind | readers are fine for 90 s while the database still drowns; a re-stamped copy serves 200 s old data |
| 4 · One refresh per key | 100% served, 20 queries per refresh, about 4% CPU; real threads: 100 callers, 1 read |
| 5 · A budget for the database | a cold start of 10,000 posts: 0% served without a budget, 84% after 40 s with one, about 100 queries a second |
| 6 · Failure drills | 77% served through an ElastiCache outage with a local cache, 0.6% without; the hit rate reads 100% while data ages to 120 s |
STEP 4 · UNDERSTAND
Try it yourself first. This is how the solution in the repo works.
Never
"Never" was right. Once all 500 connections are busy, 500 queries share 8 cores, so a query that needs 12 ms of CPU takes 500 × 12 / 8 = 750 ms. Each read may wait 500 ms. Every query is cancelled before it finishes, so nothing ever refills the cache, so every new request misses too. The trigger lasted a moment; the outage lasts as long as the traffic does. That shape is called a metastable failure.
The boundary is sharp. With a 750 ms deadline the first queries land and the system recovers in two seconds. At 5,000 reads a second the same expiry is a blip; at 10,000 it is an outage. Nothing in the code changes between those runs, which is why this ships: it passes every test that does not reproduce the real traffic.
The fix, in the order that matters
| What it does | Measured, 20,000 reads a second | |
|---|---|---|
| Jitter | spreads entries written together | helps 10,000 posts, does nothing for p123 |
| Soft and hard TTL | serve a stale list for up to two minutes, refresh behind it | readers fine for 90 s, the database still at 100% |
| One refresh per key | concurrent requests share the refresh in progress, per process | 20 queries per refresh, 100% served |
| Refresh budget | 5 refresh starts a second and 2 at once per process, short deadlines | a cold start stays near 100 queries a second |
| Local cache | a bounded copy in each process | 77% served through an ElastiCache outage |
Each one fixes what the one before it leaves. Serving stale keeps readers happy but makes every stale read start its own refresh, so the stampede moves behind them. Sharing one refresh per key fixes that for p123, but ten thousand different missing posts are ten thousand refreshes. The budget fixes that, and it has to follow the fleet: double the processes and "5 a second each" quietly becomes 200.
Twenty processes still make twenty queries for one key. A distributed lock could make it one, but it adds a dependency that can fail on exactly the bad day, to save nineteen reads. That trade is accepted on purpose.
Two small rules that quietly break everything
Copies keep their timestamps. In one drill PostgreSQL is down and a deploy restarts every API process. Each process refills its local cache from ElastiCache. If a copy is stamped as read just now, lists that were already 80 seconds old get served for two more minutes. The cache believed nothing it served was older than 120 seconds. The real age reached 200.
The age starts at the read, not the arrival. A query that takes a second returns data that is already a second old, so record the time before you query.
Watch age, not hit rate
With the database down, the hit rate reads 100% for 95 seconds while every response gets older. Measure the oldest data served, the database refresh rate, refresh failures and rejected reads. A cache that serves old data perfectly is still a cache that is failing.
What this does not do
- It is a simulation at fleet scale. PostgreSQL is modelled as 8 cores shared fairly between running queries. Real databases thrash under that load, and real clients retry, so real stampedes are worse, not better.
- Network latency and ElastiCache failover time are not modelled. The real-code parts run on threads and a SQLite file, not on AWS.
- Staleness is a product decision. Never apply it to authorisation, or to anything that must disappear at once.
STEP 5 · YOUR TURN
Break the promise on purpose
In stampede, pick Database outage, switch every defence on, then untick Keep timestamps on copy. How old is the oldest list anyone gets, and what does the cache think it served? Then find a dashboard metric that would have caught it.
THE SOLUTION
Compare yours with mine
The stampede simulator, the real Comments API and the page, the cache pieces in jiffy-cache, Go versions of coalescing, and a README, script, results and test for each of the six parts.
Finished part 4?
Star the repo so you hear when the next challenge lands, and post your results. Other machines are half the fun.