One message per action through a flaky provider
By Zeeshan Adil · 3 min read
A store's order ships and the customer should get one text message about it. Not zero, not two. This is the challenge where exactly-once stops being a slogan and becomes a design you can test.
THE PROBLEM
The problem
Between the order and the text sit worker processes that get killed, a database that restarts, and an SMS provider that misbehaves: it times out, delivers its status webhooks twice or out of order, and now and then reports a message as failed a while after it said it was sent.
Challenge 01 stopped a zombie worker with an epoch that the receiver checked. This provider checks nothing of ours. What it does offer is the usual pair: an Idempotency-Key on send, and a lookup by that key. The job is to get exactly one message out of that pair, with the epoch still guarding everything on our side of the wire.
And it has to be fair. When one store drops thousands of messages at once, the other 99 stores shouldn't wait behind it.
STEP 1 · PREDICT
A send to the provider times out. You don't know whether the message went out. What's the safe next step?
- Send it again with a new idempotency key
- Mark the action failed and alert someone
- Look it up by its key, and only send again on a definite "not found", with the same key
Write your answer down. The rest of the page will tell you.
STEP 2 · BUILD IT
Send exactly one message per action through a provider that lies, while your workers get killed.
Your solution should:
- Write the send intent, an outbox row with a stable idempotency key, in the same transaction as the compare-and-set that proves the worker still holds the action.
- Send through a fake provider that ignores epochs: 5% timeouts, 2% duplicate webhooks, 10% out-of-order webhooks, 1% late "failed" after "sent".
- Reconcile unknown outcomes through the provider's lookup. Never resend blindly.
- Keep a slow but healthy worker on its job with a heartbeat, fenced by epoch.
- Claim fairly across 100 stores: with one store sending 10x the traffic, the other 99 keep their p95 latency within 10% of a baseline where every store is equal.
- Survive chaos: 200 random SIGKILLs across real worker processes plus a PostgreSQL restart. Prove from the provider's own log that every action had exactly one effect.
Stuck? Hints
- A timeout, a 5xx or a refused connection means "unknown", never "no".
- Count the attempt before the request leaves. Zero attempts means nothing was ever sent, so the next owner can send without a lookup.
- Merge every report, from a send, a lookup or a webhook, through a forward-only order: sending, accepted, sent, then delivered or failed.
queuedaftersentis a step backwards and gets ignored. - Record each webhook's event id in the same transaction as the merge, so a duplicate is a no-op.
- For fairness, claim from the store served least recently, among those with work and fewer than a cap of actions in flight.
STEP 3 · CHECK IT
Run the tests that hold it to the rules
git clone https://github.com/ZeeshanAdilButt/distributed-systems-challenges cd distributed-systems-challenges ./verify # every gate, about ten minutes python -m courier.chaos # just the chaos run: 2000 actions, 200 kills python -m courier.bench # just the fairness benchmark
Needs Linux or macOS (or WSL) and the PostgreSQL server binaries on PATH: the chaos run starts and crashes its own cluster. This is what the chaos run checks, from the provider's own log:
| check | my result |
|---|---|
| actions with exactly one provider effect | 2000 of 2000 |
| actions with more than one effect | 0 |
| actions with no effect | 0 |
| sends after an earlier send with no lookup in between | 0 |
| outcomes that differ from the provider's final status | 0 |
STEP 4 · UNDERSTAND
Try it yourself first. This is how the solution in the repo works.
One action, start to finish
The third answer is right. A timeout tells you nothing about whether the message went out, so a new key risks a second text and "failed" risks none at all. The key is what makes a resend safe, and the lookup is how you avoid most resends.
enqueue actions: pending
claim round robin across stores, lease + epoch N
write intent one transaction: compare-and-set on (owner, epoch N)
+ INSERT outbox (stable idempotency key)
deliver loop attempts > 0 ? look the key up first
found -> record what the provider says
not found -> count the attempt, send with the same key
unknown -> back off, ask again
settle actions: dispatched, lease released
webhooks/check final status by webhook, or an overdue lookup finds it
If a worker dies anywhere in there, its lease runs out, another worker claims the action at epoch N+1, finds the intent already written, and carries on from the lookup. A worker frozen past its lease can't even count an attempt, because the new epoch makes all of its writes fail. A request it already had on the wire carries the same key, so the provider replays instead of sending twice.
Fair claiming
Plain FIFO hands every free worker the oldest action in the whole queue. When one store drops a few thousand messages at once, every other store waits behind them. Round robin with a per-store cap stops that: each claim goes to the store served least recently, and a store with a deep backlog never holds more than its cap of workers.
| scenario | claim | quiet stores' p95 | vs baseline |
|---|---|---|---|
| every store equal | fair | 68 ms | 1.00x |
| one store 10x, steady | fair | 70 ms | 0.98x |
| one store 10x, bulk sends | fair | 70 ms | 0.99x |
| one store 10x, bulk sends | FIFO | 159 ms | 2.33x |
Two things I learned getting the benchmark to measure the right thing. Headroom is part of fairness: the noisy store adds 9% to total traffic, and at 60% busy that extra load alone moved everyone's p95 by about 10%. And measure the scheduler, not the disk: with synchronous_commit on, an occasional slow WAL flush on a laptop stalls every commit at once and swamps the difference you're trying to see.
The limits
- The idempotency key has to outlive every retry. The fake provider remembers keys for an hour; real ones often keep them for a day. An action stuck longer than that could be sent twice, so alert on old, unfinished intents.
- The lookup is an optimisation; the key is the guarantee. A lookup that lags its writes can answer "not found" for a message it already has. The resend reuses the key, so it's still deduplicated.
- Fair means fair per message, not per cost. A store whose messages take longer to send uses more worker time.
STEP 5 · YOUR TURN
Watch FIFO fail the quiet stores
Run python -m courier.bench and find the FIFO row. Under bulk sends from one noisy store, how much does every other store's p95 move, and why does fair claiming keep it flat?
THE SOLUTION
Compare yours with mine
The courier service (claims.py, outbox.py, states.py, webhooks.py), built on jiffy-lease and tested against the misbehaving jiffy-flaky provider, with mutation testing on the claim, finish and outbox code.
Finished part 2?
Star the repo so you hear when the next challenge lands, and post your results. Other machines are half the fun.