Challenges
Each series takes problems I've met in production and turns them into a runnable model of what goes wrong, with tests that check the outcome from the outside and results you can measure yourself. Read a part, write down a prediction, then break a real system on your own machine and see if you were right.
Distributed Systems Challenges
Runnable models of work spread across processes that crash, stall and retry, with tests and measured results. Fence a zombie worker, send exactly one message through a flaky provider, decide where every row lives as the cluster changes, and survive a cache stampede.
Python 3.12 · uv · PostgreSQL for 01 and 02 · Go optional for 04 · MIT
- Fencing a zombie workerA worker stalls, its lease runs out, someone else takes the job, and the first worker wakes up and finishes anyway. Make its late work count for nothing.pytest challenges/01-zombie-worker -v
- One message per action through a flaky providerSend exactly one SMS per order through a provider that times out and sends duplicate, out-of-order webhooks, while workers get killed. Then share the workers fairly across 100 stores../verify
- Where does each comment live?Split comments across databases four ways on real SQLite shards, move data without losing writes, then meet the viral post no hash can spread and the stale server writing to the wrong place.uv run shardbox
- A popular post takes down the databaseA cached comment list expires under 20,000 reads a second and the database goes down for good. Fix it with stale serving, one refresh per key, a refresh budget and a local cache, all measured.uv run stampede
Ran one and got different numbers? Open an issue on the repository with your output. Different machines are half the point.