[Backend] Add Celery acks_late and a stuck-task watchdog across every async pipeline #398
Closed
opened 2026-08-25 20:44:38 +00:00 by claude-bot
·
2 comments
No Branch/Tag specified
main
feat/1328-fill-it-in
feat/1328-fill-frontend
feat/1328-fill-backend
feat/9-foundry-push
fix/9-foundry-live
feat/9-foundry-push-frontend
renovate/celery-5.x
feat/5-foundry-connect
feat/9-foundry-push-backend
feat/9-foundry-push-module
feat/5-foundry-frontend
feat/5-foundry-backend
feat/5-foundry-module
release/v4.8.0-rc1
docs/d341-convert-draft
feat/958-update-to-version
feat/958-convert-sheet
feat/958-pathbuilder-import-frontend
fix/1360-heading-order
feat/958-pathbuilder-import-backend
feat/11-weekly-digest
feat/1341-auto-update
docs/d332-superseded
chore/1345-names-cleanup
feat/1342-mentioned-pages
feat/1328-recipes
fix/1346-pgdg-key
perf/1247-shared-recording-stream
chore/132-gitleaks-allowlist
release/v4.7.1
spike/foundry-module
release/v4.7.1-rc1
docs/decisions-d332-d337
fix/bot-certifi
chore/release-4.7.0
chore/release-4.7.0-rc4
chore/deps-4.7.0
chore/release-4.7.0-rc3
fix/1280-quote-must-say-it
fix/1257-names-row-squeeze
chore/release-4.7.0-rc2
fix/1245-pg-upgrade-count
chore/release-4.7.0-rc1
feat/1257-names-in-prep
feat/1257-be
feat/1257-fe
feat/1257-rating-fixes
design/1328-fill-it-in
feat/1257-name-splits
fix/1326-redis-503
feat/1280-fact-suggestions
feat/1280-be
feat/1280-fe
design/1280-fact-suggestions
feat/1279-infobox
feat/1279-be
feat/1279-fe
feat/1321-shelf-copies
design/1279-infobox
feat/1243-table-find
feat/1257-who-rated
feat/1243-prep-drawer
feat/1243-sheet-pickers
design/1243-browse-in-place
feat/1315-hotwords-default
feat/1242-system-filters
feat/1257-more-name-styles
feat/506-leg3-hotwords
design/1242-system-filters
fix/1302-packs-polish
feat/1241-rules-library
design/1241-rules-library
feat/1245-postgres-18
integ/4.6.2
fix/1263-map-ui-2
fix/1263-map-server
fix/1263-map-ui-1
fix/1261-editor-link-menu
fix/1229-axe-4.14-name-mismatch
renovate/axllent-mailpit-1.x
renovate/eslint-monorepo
renovate/authlib-1.x
chore/release-4.6.0
chore/release-4.6.0-rc2
fix/460-rc2-screens
fix/460-rc2-server
fix/release-prerelease-exit
chore/release-4.6.0-rc1
feat/125-campaign-templates
feat/126-calendar
feat/1181-pictures-and-files
feat/1190-event-people
integ/4.6.0
feat/1191-line-of-the-night
feat/1200-starter-packs
chore/bot-multidict-and-playwright-group
fix/1202-this-week-answered-count
fix/122-124-leftovers
fix/1178-get-routes-that-write
fix/1203-journeys-check-names
design/125-campaign-templates
feat/126-calendar-engine
design/126-in-world-calendar
fix/1189-cited-stamp-in-range
fix/1189-v4-looser-check
fix/1189-richer-beats
fix/1113-export-bind-lists
fix/1154-release-notes-by-file
chore/release-4.5.0
chore/release-4.5.0-rc12
fix/1135-wiki-markup-as-text
fix/demo-usage-cost-in-window
chore/release-4.5.0-rc11
fix/1121-sheet-consistency
fix/1121-sheet-5e-terms-and-card-hp
copy/1122-busiest-pages
fix/1121-relevant-choices
fix/1121-wiki-create-npc-save-sheet-links
fix/1029-menu-stays-on-screen
docs/prototype-transcript-bin-wording
fix/1023-editor-save-row-back-scroll
fix/1020-search-exact-first
fix/1020-table-prep
fix/1020-shell-wordmark-footer-bell
fix/1020-wiki-article-party
fix/872-admin-kept-until-phone-tables
feat/l13-milestone-level-waiting
fix/1000-1001-bell-older-setup-context
fix/999-pronouns-in-prompts
fix/l19-other-class-feats
fix/1010-transcript-window-ordinary-confirm
docs/d254-d269-decisions
fix/702-member-search
fix/680-transcript-bin
fix/993-unidentified-prompts
fix/841-redis-timeouts
fix/990-binned-not-exported
fix/830-export-prep-shelf
fix/deps-audit-2026-10-01
fix/877-title-reads
fix/798-audio-received-at
build/lane11-17-j7
build/lane11-16-levelup
build/lane11-14-seams
build/lane11-14-seed
build/lane11-15-start
build/lane11-13-old-block-out
fix/980-review-r3
build/lane11-12-block
build/lane11-8-packs
fix/demo-seed-canonical-names
build/lane11-11-hot-half
docs/l11-decisions
build/lane11-7-convert
build/lane11-10-editing
fix/npm-audit-brace-expansion
build/lane11-9-sheet-page
fix/bot-pyjwt
build/lane11-6-content
fix/lane11-rank-names
build/lane11-2b-language
build/lane11-4-js-evaluator
build/lane11-3-server
build/lane11-5a-pf2e
build/lane11-5b-5e
feat/design-admin
build/lane11-2-evaluator
build/lane11-1-sheet-pieces
build/lane11-0-drawing
fix/943-quest-objectives-fk
fix/prepsheet-shelf-flake
fix/945-leftovers
build/demo-gm-copy
fix/901-email-landings
build/lane10-8-journeys
build/demo-seed-lanes-8-10
fix/887-gm-own-confirmation
build/lane9-5-j1-admin
fix/918-audience-click
fix/useasync-overtaken
ci/939-critical-path
test/937-backend-suite-speed
chore/claude-orchestration
fix/924-xp-gm-only
fix/session-stale-poll
docs/decisions-d239-d242
fix/930-pypdf
build/lane8-journeys
build/925-quest-editor
fix/894-private-page-pictures
build/lane8-p4-drafts
build/lane8-p7-map
build/lane8-p6-handouts-rules
fix/wiki-fixture-names
build/lane10-d227-test-usage
fix/rsvp-horizon-test-tuesday
fix/media-volume-owner
build/lane8-p3-editor
fix/912-unidentified-rename
build/lane8-p5-party-quests
build/lane8-handout-takeback
build/lane10-pickers
build/lane8-p2-article
build/lane8-6-map
build/lane10-7b-email-pages
build/lane8-5-handouts
build/lane10-7a-test-the-bot-pages
build/lane8-p1-pages
build/lane8-4-quests
build/lane8-2b-feeds
build/lane8-3-parties
build/lane8-2a-links
build/lane10-5-commands
build/lane9-6-seed
build/lane10-4b-reactions
build/lane9-4d-pages
build/lane9-4c-pages
build/lane10-4a-posts
build/lane10-3-email
fix/891-web-fonts
build/lane9-4b-pages
build/lane10-6-test-the-bot
build/lane10-2-notify-out
build/lane8-1-wiki-text
build/lane9-4a-frame
design/lane10-drawings
fix/j6-budget
fix/j5-phone-flake
fix/869-test-leaks
fix/843-record-start-followups
build/lane7-all
build/lane6-all
build/lane9-b2-server
chore/changelog-fragments
docs/decisions-d218
build/lane9-a-owner-overview
fix/rule-card-level-label
fix/834-retention-semantics
fix/o3c-record-without-announcement
docs/decisions-d179
docs/decisions-d131-d170
fix/851-commit-before-response
build/lane6-j3-seed
build/lane9-b1-d67-identity
fix/841-blocking-loop
build/lane9-c-backups-audit
fix/vote-race-and-start-commit
fix/recording-announcement
fix/proposal-read-leak
fix/last-admin-guard
build/lane6-p4c-planning
build/lane6-p4b-margin
build/lane6-p4a-sheet
build/lane5-j4-seed
build/lane6-base
build/lane7-pages-talk
build/lane5-pages
build/lane7-pages-safety
build/lane7-p3-safety
build/lane7-p2-notify
build/lane6-p3-planning
build/lane6-p2-history
build/lane6-p1b-items
build/lane7-p1-threads
build/lane5-p6-faces
build/lane6-p1a-sheet
build/lane5-p5-find
build/lane5-p4a-board
build/lane5-p4b-scenes
build/lane5-p3-live
build/lane5-base
build/lane5-p2-cards
docs/lane5-decisions
build/review-r2-wide
fix/review-r1
build/lane5-p1-scenes
build/demo-seed
build/lane4-journeys-j5
build/lane4-page-review-b
build/lane4-page-review-a
build/lane4-page-read
build/lane4-journeys
build/lane4-page-sess2
build/lane4-page-sess1
build/lane4-notes
build/lane4-discord-on-approval
build/lane4-scheduling
build/lane4-versions
build/760-session-number
build/lane4-decisions
build/797-erase-no-regenerate
build/test-race-audit
build/lane3-completions
build/794-dark-only
build/796-titles-scroll
build/791-retention-anchor
build/a11y-sweep
build/shell-followups
build/retention-preview
build/753-backup-readback
docs/d91-d93
build/lane3-j1
build/ci-harness-paths-selftest
fix/useauth-test-race
build/ci-harness-paths
build/777-ai-settings-read
build/lane3-settings-c
build/lane3-settings-b
build/lane3-j6
build/lane3-sessionform
build/lane3-settings-a
build/lane3-members
build/lane3-thisweek
build/lane3-story
chore/87-align-python-3.12
v4.8.0-rc1
v4.7.1
v4.7.1-rc1
v4.7.0
v4.7.0-rc4
v4.7.0-rc3
v4.7.0-rc2
v4.7.0-rc1
v4.6.2
v4.6.2-rc4
v4.6.2-rc3
v4.6.2-rc2
v4.6.2-rc1
v4.6.1
v4.6.1-rc1
v4.6.0
v4.6.0-rc2
v4.6.0-rc1
v4.5.3
v4.5.3-rc1
v4.5.2
v4.5.2-rc1
v4.5.1
v4.5.1-rc2
v4.5.1-rc1
v4.5.0
v4.5.0-rc12
v4.5.0-rc11
v4.5.0-rc10
v4.5.0-rc9
v4.5.0-rc8
v4.5.0-rc7
v4.5.0-rc6
v4.5.0-rc5
v4.5.0-rc4
v4.5.0-rc3
v4.5.0-rc2
v4.5.0-rc1
v4.4.1
v4.4.0
v4.3.0
v4.2.3
v4.2.2
v4.2.1
v4.2.0
v4.1.2
v4.1.1
v4.1.0
v4.0.1
v4.0.0
v3.11.4
v3.11.3
v3.11.2
v3.11.1
v3.11.0
v3.10.0
v3.9.1
v3.9.0
v3.8.0
v3.7.0
v3.6.1
v3.6.0
v3.5.0
v3.4.1
v3.4.0
v3.3.0
v3.2.0
v3.1.0
v3.0.0
v2.1.0
v2.0.0
v1.1.0
v1.0.0
v0.12.0
v0.11.0
v0.10.0
v0.9.1
v0.3.3
v0.9.0
v0.8.0
v0.3.2
v0.3.1
v0.3.0
v0.7.0
v0.6.0
v0.5.0
v0.4.0
v0.2.1
v0.2.0
v0.1.1
v0.1.0
Labels
Clear labels
backend
Issue related to the backend
bot
Issue related to the bot
bug
Something is not working
duplicate
This issue or pull request already exists
enhancement
New feature
frontend
Issue related to the frontend
help wanted
Need some help
invalid
Something is wrong
question
More information is needed
recording
Issue related to the recording system
security
Security hardening / vulnerability work
wontfix
This won't be fixed
No labels
backend
bot
bug
duplicate
enhancement
frontend
help wanted
invalid
question
recording
security
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
rbrooks/Quest-Board#398
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Severity: CRITICAL
Found in the August 2026 session lifecycle review (#319).
A routine deploy or an OOM-killed worker permanently strands a session mid-pipeline with no self-service way out, because Celery acknowledges tasks the instant they're received rather than after they finish. If the worker process dies while transcribing a multi-hour recording, Celery never redelivers the task — it's just gone — but the session row was already flipped to
processingbefore the work started, and nothing ever flips it back. The GM's own retry button, and even the admin console's retry button, refuse to act unless the status isfailed, so the only way out is a manual database edit.Evidence
webapp/backend/app/tasks/celery_app.py:27-60—celery_app.conf.update(...)sets notask_acks_lateand notask_reject_on_worker_lost; a repo-wide grep confirms these settings appear nowhere, so Celery's default (ack-on-receipt) applies to every task, includingprocess_audio.webapp/backend/app/tasks/reminder_tasks.py:1984—process_audiois declared withmax_retries=0, so even a task that does fail with an exception (e.g. a transient Whisper timeout on a 3-hour job) gets no automatic retry.webapp/backend/app/routers/sessions.py:891-897andwebapp/backend/app/routers/admin.py:174-183— both the GM and admin retry endpoints requireaudio_processing_status == AudioProcessingStatus.failed; a session stuck atprocessing409s on both.webapp/backend/app/tasks/reminder_tasks.py:2705-2712—enforce_retention's pass over on-disk audio explicitlycontinues past any session whose status is neither successfully processed norfailed— it deliberately skips stuck-processingsessions rather than acting on them; no watchdog exists.webapp/backend/app/tasks/reminder_tasks.py:2043-2060— the WAV-discovery/speakers.jsonparsing code (which can raiseFileNotFoundErroror a JSON error) runs beforeasync with task_session() as db:opens, so an exception there never reaches the status-setting failure path either — same stuck-processingoutcome from a different cause.extracting/matchingafter a SIGKILL — the UI hides the retry button for every in-progress state) and for wiki drafts (generatingwith no failure path, only a destructivediscard_draftescape) and WorkbenchGenerationResult(stuckpending, frontend gives up polling after ~60s).Failure scenario
An operator runs the routine
docker compose up -d --builddeploy while a 3-hour transcription job is mid-flight on the worker. The container is sent SIGTERM/SIGKILL, the in-flightprocess_audiotask vanishes with it, and the session (already flipped toprocessingon intake) never moves again. The GM sees "Processing…" forever; clicking retry 409s; admin's retry 409s too. The only fix is a support engineer running a manualUPDATE sessions SET audio_processing_status = 'failed' WHERE id = ...so the retry button lights up.Proposed fix
Set
task_acks_late=Trueandtask_reject_on_worker_lost=Trueincelery_app.confforprocess_audioand the lore/draft/workbench tasks. This is safe only once the "[Recording] Guard against re-submitting audio..." issue lands, since acks_late means a task can now be redelivered and re-run — without that guard in place first, a redelivery would clobber edits exactly like an unguarded manual re-submission would. Add a Celery Beat watchdog task that scans forprocessing/pending/generating/extracting/matchingrows older than a configurable threshold (e.g. 3-6 hours) and flips them tofailedwith a "worker lost" error, which makes the existing retry buttons work for this class of failure without new UI. Separately, raiseprocess_audio'smax_retriesabove 0 for genuinely transient exceptions (network/timeout), distinct from permanent ones (bad path, malformed input). (This watchdog is generalised across all four pipelines in the "[Backend] Build a generic stuck-task watchdog..." issue; this issue is the audio-pipeline-specific defect that motivates it.)Acceptance criteria
task_acks_late=Trueandtask_reject_on_worker_lost=Trueare set forprocess_audio.processingsessions older than a configured threshold tofailedwith a descriptive error.failed, with no manual SQL required.process_audioretries at least once on transient exceptions instead ofmax_retries=0.Scope: every pipeline, not just
process_audioMerged from a companion finding. The same "worker died, no exit" shape appears in four independent pipelines, so the fix should be shared infrastructure rather than four bespoke patches:
process_audio, statusprocessingextracting/matching, with noon_failurecoverage for a SIGKILLgenerate_lore_entry_drafthas noon_failurebase at allGenerationResultrows have no server-side timeout; only the frontend gives up pollingApply
task_acks_lateandtask_reject_on_worker_lostconsistently across all of them, and add a single Celery Beat watchdog that scans for any row in an in-progress state (processing,pending,extracting,matching,generating) older than a configurable threshold and transitions it to a terminalfailedstate with a "worker lost" error.That one mechanism closes the majority of the eleven enumerated stuck states at once.
Ordering constraint: the audio re-submission guard must land first. Enabling
acks_latemeans tasks get redelivered after a worker loss, and until re-submission is guarded a redelivery will clobber GM-edited transcripts, summaries and curated highlights.Added acceptance criteria
task_acks_lateandtask_reject_on_worker_lostapplied to all four pipelines, not onlyprocess_audioacks_lateis enabledImplemented in PR #448 (
fix/398-survive-a-lost-worker), awaiting CI.Precondition confirmed first: #397's re-submission guard is in place (
reminder_tasks.py:2175), so a redeliveredprocess_audiorefuses to overwrite an existing transcript instead of clobbering GM edits.acks_latewould have been actively unsafe before that landed.One thing the issue didn't specify, shipped in the same change. There were no Celery time limits configured at all. With
acks_lateand no limit, a task that kills its worker is redelivered and kills the next one, forever — the fix would have traded a stuck session for an unavailable worker pool. Sotask_soft_time_limit=6h/task_time_limit=6h10mandworker_prefetch_multiplier=1are part of this, not a follow-up. Six hours sits just abovemax_recording_hours, so it bounds pathology without ever cutting a legitimate full-length transcription.Watchdog scope: hourly, 8h threshold (above the 6h task limit, so anything past it cannot still be running). Covers all four pipelines that flip a row to in-progress on intake — audio
processing, lorepending/extracting/matching,LoreEntryDraft.generating,GenerationResultpending. It only ever moves rows tofailed, since that is the one state the retry endpoints accept; no files are touched andaudio_file_pathis left intact, so the recording is still there to reprocess. Writes atasks.unstuckaudit entry with counts.process_audioalso retries twice with backoff on transport errors specifically — not a blanketException, which would only delay the honest error the admin panel needs to show.Verification: mutation-checked both halves — reverting
acks_latefails 1 test, stubbing the watchdog fails 3 including this issue's own scenario. The idempotency test initially passed against the stub (vacuously — "the second pass changed nothing" is also true of a watchdog that never ran) and was tightened to assert the first pass did the work. 1,417 backend tests pass, up from 1,409.Will close once CI is green and this is merged.
Merged in PR #448 (CI green). Closing.
One addition after the first mutation run: the terminal-row test walked all four watchdog queries, but only the audio and lore assignments ever executed — a wrong column name in the draft or workbench branch would have passed CI and failed the first time a worker actually died. Added two tests that seed a stranded draft and a stranded generation result, which also assert the accumulated work survives (draft body,
iteration_count, generation params) since preserving it is the reason the watchdog fails rows instead of discarding them. Stubbing the watchdog now fails five tests rather than three.Final: 1,419 backend tests pass.