fix(backend): survive a lost worker instead of stranding the session (#398) #448
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/398-survive-a-lost-worker"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #398 (CRITICAL).
The defect
Celery acknowledges a task the moment it is received, not when it finishes. A worker killed mid-job therefore took the task with it: nothing redelivered, nothing logged. The session row had already been flipped to
processingon intake, and both the GM's retry endpoint and the admin's requirefailedand 409 onprocessing— so the session sat at "Processing…" permanently, with a hand-writtenUPDATEas the only exit.The trigger is not exotic. An ordinary
docker compose up -d --buildduring a three-hour transcription does it.What ships
task_acks_late+task_reject_on_worker_lost— the broker redelivers what a dying worker was holding. Safe only because #397 landed first: a redeliveredprocess_audionow refuses to overwrite an existing transcript rather than clobbering GM edits. That ordering was a stated precondition of this issue, and it held.Time limits and
worker_prefetch_multiplier=1, in the same change. This is the part the issue didn't ask for and the fix is wrong without. I found no time limits configured at all — soacks_lateon its own converts a task that kills its worker into one that comes back and kills the next worker, indefinitely. A 6h soft limit turns that into an ordinary task failure the existingon_failurepath already records asfailed, which the retry buttons can act on. Six hours sits just abovemax_recording_hours(6), so it is a bound on pathology and never cuts off a legitimate full-length transcription.prefetch=1keeps a worker loss to the one job actually in flight instead of a reserved batch.An hourly watchdog (
unstick_lost_tasks) for what redelivery cannot reach — a broker restart, an enqueue that never landed. Four pipelines flip a row to in-progress on intake and all four are covered: audioprocessing, lorepending/extracting/matching,LoreEntryDraft.generating, andGenerationResultpending. It moves rows only tofailed, because that is the single state the retry paths accept; it touches no files and leavesaudio_file_pathintact, so the recording is still there to reprocess. Idempotent and state-driven — a second pass changes nothing and writes no duplicate audit entry.The 8-hour threshold is deliberately above the 6h task limit: anything still in progress past it cannot be a running task any more.
Bounded retries on transport errors only.
process_audioretries twice with backoff onhttpx.TransportError/TimeoutError/ConnectionError. Deliberately not a blanketException— retrying a malformed input just delays the honest error the admin panel is waiting to show.Verification
Mutation-checked in both halves:
task_acks_late=True→False: 1 test fails.processingby a deploy.The idempotency test was tightened after the first mutation run: it passed vacuously against the stub, since "the second pass changed nothing" is also true of a watchdog that never ran. It now asserts the first pass did the work.
1,417 backend tests pass (was 1,409). Lint and format clean at CI's pinned ruff 0.4.4.
🤖 Generated with Claude Code