Challenge series · 4 partsDistributed Systems Challenges
By Zeeshan Adil
Runnable models of work spread across processes that crash, stall and retry, with tests and measured results. Fence a zombie worker, send exactly one message through a flaky provider, decide where every row lives as the cluster changes, and survive a cache stampede.
Each challenge takes one real problem, asks you to solve it in Python and PostgreSQL, and then tries hard to break your solution: real worker processes, real kill -9, a real Postgres crash, and a fake third-party API that times out and lies about ordering. The tests check the outcome from the outside, not from the code's own idea of what happened.
Read a challenge here, build your answer, run the repo's tests against it, then open the repo to compare with mine.
Python 3.12 · uv · PostgreSQL for 01 and 02 · Go optional for 04 · MIT
Star it to hear when the next series ships.
Keep the tools: jiffy-lease · jiffy-flaky · jiffy-ring · jiffy-cache · jiffy-chaos
WHAT YOU'LL LEARN
By the end you can
- stop a stalled worker from finishing a job someone else now owns
- send one message per action through a provider that times out and reorders webhooks
- choose between modulo, buckets, a hash ring and ranges, and know what moves when a database joins
- stop a cache stampede with stale serving, one refresh per key and a refresh budget, and measure it
The parts
- Fencing a zombie workerA worker stalls, its lease runs out, someone else takes the job, and the first worker wakes up and finishes anyway. Make its late work count for nothing.pytest challenges/01-zombie-worker -v
- One message per action through a flaky providerSend exactly one SMS per order through a provider that times out and sends duplicate, out-of-order webhooks, while workers get killed. Then share the workers fairly across 100 stores../verify
- Where does each comment live?Split comments across databases four ways on real SQLite shards, move data without losing writes, then meet the viral post no hash can spread and the stale server writing to the wrong place.uv run shardbox
- A popular post takes down the databaseA cached comment list expires under 20,000 reads a second and the database goes down for good. Fix it with stale serving, one refresh per key, a refresh budget and a local cache, all measured.uv run stampede
Where this comes from
These came up for real rebuilding failure handling around an outbox, atomic job claiming and exactly-once consumers. Read about that work, then come back and break the small version.