[Backend] Add Celery acks_late and a stuck-task watchdog across every async pipeline #398
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Severity: CRITICAL
Found in the August 2026 session lifecycle review (#319).
A routine deploy or an OOM-killed worker permanently strands a session mid-pipeline with no self-service way out, because Celery acknowledges tasks the instant they're received rather than after they finish. If the worker process dies while transcribing a multi-hour recording, Celery never redelivers the task — it's just gone — but the session row was already flipped to
processingbefore the work started, and nothing ever flips it back. The GM's own retry button, and even the admin console's retry button, refuse to act unless the status isfailed, so the only way out is a manual database edit.Evidence
webapp/backend/app/tasks/celery_app.py:27-60—celery_app.conf.update(...)sets notask_acks_lateand notask_reject_on_worker_lost; a repo-wide grep confirms these settings appear nowhere, so Celery's default (ack-on-receipt) applies to every task, includingprocess_audio.webapp/backend/app/tasks/reminder_tasks.py:1984—process_audiois declared withmax_retries=0, so even a task that does fail with an exception (e.g. a transient Whisper timeout on a 3-hour job) gets no automatic retry.webapp/backend/app/routers/sessions.py:891-897andwebapp/backend/app/routers/admin.py:174-183— both the GM and admin retry endpoints requireaudio_processing_status == AudioProcessingStatus.failed; a session stuck atprocessing409s on both.webapp/backend/app/tasks/reminder_tasks.py:2705-2712—enforce_retention's pass over on-disk audio explicitlycontinues past any session whose status is neither successfully processed norfailed— it deliberately skips stuck-processingsessions rather than acting on them; no watchdog exists.webapp/backend/app/tasks/reminder_tasks.py:2043-2060— the WAV-discovery/speakers.jsonparsing code (which can raiseFileNotFoundErroror a JSON error) runs beforeasync with task_session() as db:opens, so an exception there never reaches the status-setting failure path either — same stuck-processingoutcome from a different cause.extracting/matchingafter a SIGKILL — the UI hides the retry button for every in-progress state) and for wiki drafts (generatingwith no failure path, only a destructivediscard_draftescape) and WorkbenchGenerationResult(stuckpending, frontend gives up polling after ~60s).Failure scenario
An operator runs the routine
docker compose up -d --builddeploy while a 3-hour transcription job is mid-flight on the worker. The container is sent SIGTERM/SIGKILL, the in-flightprocess_audiotask vanishes with it, and the session (already flipped toprocessingon intake) never moves again. The GM sees "Processing…" forever; clicking retry 409s; admin's retry 409s too. The only fix is a support engineer running a manualUPDATE sessions SET audio_processing_status = 'failed' WHERE id = ...so the retry button lights up.Proposed fix
Set
task_acks_late=Trueandtask_reject_on_worker_lost=Trueincelery_app.confforprocess_audioand the lore/draft/workbench tasks. This is safe only once the "[Recording] Guard against re-submitting audio..." issue lands, since acks_late means a task can now be redelivered and re-run — without that guard in place first, a redelivery would clobber edits exactly like an unguarded manual re-submission would. Add a Celery Beat watchdog task that scans forprocessing/pending/generating/extracting/matchingrows older than a configurable threshold (e.g. 3-6 hours) and flips them tofailedwith a "worker lost" error, which makes the existing retry buttons work for this class of failure without new UI. Separately, raiseprocess_audio'smax_retriesabove 0 for genuinely transient exceptions (network/timeout), distinct from permanent ones (bad path, malformed input). (This watchdog is generalised across all four pipelines in the "[Backend] Build a generic stuck-task watchdog..." issue; this issue is the audio-pipeline-specific defect that motivates it.)Acceptance criteria
task_acks_late=Trueandtask_reject_on_worker_lost=Trueare set forprocess_audio.processingsessions older than a configured threshold tofailedwith a descriptive error.failed, with no manual SQL required.process_audioretries at least once on transient exceptions instead ofmax_retries=0.Scope: every pipeline, not just
process_audioMerged from a companion finding. The same "worker died, no exit" shape appears in four independent pipelines, so the fix should be shared infrastructure rather than four bespoke patches:
process_audio, statusprocessingextracting/matching, with noon_failurecoverage for a SIGKILLgenerate_lore_entry_drafthas noon_failurebase at allGenerationResultrows have no server-side timeout; only the frontend gives up pollingApply
task_acks_lateandtask_reject_on_worker_lostconsistently across all of them, and add a single Celery Beat watchdog that scans for any row in an in-progress state (processing,pending,extracting,matching,generating) older than a configurable threshold and transitions it to a terminalfailedstate with a "worker lost" error.That one mechanism closes the majority of the eleven enumerated stuck states at once.
Ordering constraint: the audio re-submission guard must land first. Enabling
acks_latemeans tasks get redelivered after a worker loss, and until re-submission is guarded a redelivery will clobber GM-edited transcripts, summaries and curated highlights.Added acceptance criteria
task_acks_lateandtask_reject_on_worker_lostapplied to all four pipelines, not onlyprocess_audioacks_lateis enabledImplemented in PR #448 (
fix/398-survive-a-lost-worker), awaiting CI.Precondition confirmed first: #397's re-submission guard is in place (
reminder_tasks.py:2175), so a redeliveredprocess_audiorefuses to overwrite an existing transcript instead of clobbering GM edits.acks_latewould have been actively unsafe before that landed.One thing the issue didn't specify, shipped in the same change. There were no Celery time limits configured at all. With
acks_lateand no limit, a task that kills its worker is redelivered and kills the next one, forever — the fix would have traded a stuck session for an unavailable worker pool. Sotask_soft_time_limit=6h/task_time_limit=6h10mandworker_prefetch_multiplier=1are part of this, not a follow-up. Six hours sits just abovemax_recording_hours, so it bounds pathology without ever cutting a legitimate full-length transcription.Watchdog scope: hourly, 8h threshold (above the 6h task limit, so anything past it cannot still be running). Covers all four pipelines that flip a row to in-progress on intake — audio
processing, lorepending/extracting/matching,LoreEntryDraft.generating,GenerationResultpending. It only ever moves rows tofailed, since that is the one state the retry endpoints accept; no files are touched andaudio_file_pathis left intact, so the recording is still there to reprocess. Writes atasks.unstuckaudit entry with counts.process_audioalso retries twice with backoff on transport errors specifically — not a blanketException, which would only delay the honest error the admin panel needs to show.Verification: mutation-checked both halves — reverting
acks_latefails 1 test, stubbing the watchdog fails 3 including this issue's own scenario. The idempotency test initially passed against the stub (vacuously — "the second pass changed nothing" is also true of a watchdog that never ran) and was tightened to assert the first pass did the work. 1,417 backend tests pass, up from 1,409.Will close once CI is green and this is merged.
Merged in PR #448 (CI green). Closing.
One addition after the first mutation run: the terminal-row test walked all four watchdog queries, but only the audio and lore assignments ever executed — a wrong column name in the draft or workbench branch would have passed CI and failed the first time a worker actually died. Added two tests that seed a stranded draft and a stranded generation result, which also assert the accumulated work survives (draft body,
iteration_count, generation params) since preserving it is the reason the watchdog fails rows instead of discarding them. Stubbing the watchdog now fails five tests rather than three.Final: 1,419 backend tests pass.