Fencing a zombie worker
By Zeeshan Adil · 2 min read
A job claimed with a lease can outlive that lease. This is the smallest version of a bug that shows up in every job queue, and the fix is about 200 lines you can read in one sitting.
THE PROBLEM
The problem
A worker claims a job and takes a lease on it: it owns the job for the next few seconds, and if it dies, the lease runs out and someone else can take over. That part works.
Now the worker doesn't die. It stalls: a long garbage collection pause, a hung network call, a laptop lid closing. The lease expires, a healthy worker reclaims the job and starts on it, and then the first worker wakes up and carries on as if nothing happened. It calls the external system, and it commits its result over the new owner's.
Timeouts don't save you here, because the zombie has no idea time passed. The question is what makes its late work count for nothing.
STEP 1 · PREDICT
Worker A claims a job with a short lease, then freezes. Worker B reclaims the job and is still working when A wakes up. With leases alone, what happens to A's commit?
- Postgres rejects it, because the lease expired
- It succeeds, and A's result replaces B's
- It blocks until B finishes
Write your answer down. The rest of the page will tell you.
STEP 2 · BUILD IT
Make a stalled worker's late work count for nothing, in Python and PostgreSQL.
Your solution should:
- Claim a job with one
UPDATE ... WHERE id = (SELECT ... FOR UPDATE SKIP LOCKED) RETURNINGthat takes a pending job, or a running one whose lease has expired. - Judge leases on the database clock, and give every claim a number that only goes up: the epoch, your fencing token.
- Let
completeandfailsucceed only through a compare-and-set on owner and epoch. When no row matches, raise and drop the result. - Give the external receiver every call's job id and epoch. It keeps the newest epoch per job, refuses anything older, and accepts the same epoch again so a worker can retry its own call.
- Prove it with a test that runs the zombie as a real second OS process, in both orderings: the new owner reaching the receiver first, and the zombie reaching it first.
Stuck? Hints
- Leases on the worker's clock break as soon as two machines disagree about the time.
now()in the database doesn't. - Call the receiver first, then commit. If the worker dies in between, the next owner delivers again with a higher epoch, which the receiver accepts.
- A good test fails when you remove any one guard: the owner and epoch check, the receiver's epoch check, or the increment on claim.
STEP 3 · CHECK IT
Run the tests that hold it to the rules
git clone https://github.com/ZeeshanAdilButt/distributed-systems-challenges cd distributed-systems-challenges export DATABASE_URL=postgresql://postgres@localhost/postgres pytest challenges/01-zombie-worker -v
Any Postgres works. Everything is created in a separate fencing_demo schema that the tests drop and recreate. Both orderings must end the same way:
| case | zombie's receiver call | zombie's commit | final state |
|---|---|---|---|
| new owner reaches the receiver first | rejected | rejected | epoch 2 |
| zombie reaches the receiver first | accepted, then superseded | rejected | epoch 2 |
STEP 4 · UNDERSTAND
Try it yourself first. This is how the solution in the repo works.
Why the epoch works
If you picked "it succeeds", you were right. Nothing in a plain UPDATE ... SET status = 'done' knows the lease expired, so the zombie's commit lands on top of the new owner's work.
The epoch fixes that by making ownership something you can check at write time. Every claim increments it, so a newer claim always holds a larger number than every claim before it. The zombie still believes it holds epoch 1, but the row now says epoch 2, so its compare-and-set matches no row and its result is dropped.
The receiver needs the same rule, because the zombie may reach it before it tries to commit. It remembers the newest epoch it has accepted per job and refuses older ones. In the second ordering it does accept the zombie's call first, and is then superseded by epoch 2. The final state is right either way.
What fencing does not do
- It needs the receiver to cooperate. A third party that ignores the epoch gets no protection from it. For those, send a stable key they deduplicate on, and look up their state before retrying a call whose outcome is unknown. That is challenge 02.
- It guarantees the zombie never wins, not that it is never heard. For an effect that can't be superseded, like sending an email or moving money, the receiver has to check the epoch at call time, or the effect has to go through an outbox written in the same transaction as the compare-and-set.
- Leases are judged on the database clock, so clock skew between workers doesn't matter. The database clock itself is trusted.
STEP 5 · YOUR TURN
Break one guard at a time
Remove AND epoch = :epoch from the compare-and-set and run the test. Put it back, then remove the epoch check from the receiver. Which ordering fails each time, and why?
THE SOLUTION
Compare yours with mine
The claim, the compare-and-set, the receiver and the two-process test, in about 200 lines. Challenge 02 grows the same idea into the reusable jiffy-lease package.
Finished part 1?
Star the repo so you hear when the next challenge lands, and post your results. Other machines are half the fun.