[Ops] Close the eleven manual-intervention stuck states across session, lore, draft, and workbench pipelines #416
Closed
opened 2026-08-25 20:44:57 +00:00 by claude-bot
·
4 comments
No Branch/Tag specified
main
feat/1328-fill-it-in
feat/1328-fill-frontend
feat/1328-fill-backend
feat/9-foundry-push
fix/9-foundry-live
feat/9-foundry-push-frontend
renovate/celery-5.x
feat/5-foundry-connect
feat/9-foundry-push-backend
feat/9-foundry-push-module
feat/5-foundry-frontend
feat/5-foundry-backend
feat/5-foundry-module
release/v4.8.0-rc1
docs/d341-convert-draft
feat/958-update-to-version
feat/958-convert-sheet
feat/958-pathbuilder-import-frontend
fix/1360-heading-order
feat/958-pathbuilder-import-backend
feat/11-weekly-digest
feat/1341-auto-update
docs/d332-superseded
chore/1345-names-cleanup
feat/1342-mentioned-pages
feat/1328-recipes
fix/1346-pgdg-key
perf/1247-shared-recording-stream
chore/132-gitleaks-allowlist
release/v4.7.1
spike/foundry-module
release/v4.7.1-rc1
docs/decisions-d332-d337
fix/bot-certifi
chore/release-4.7.0
chore/release-4.7.0-rc4
chore/deps-4.7.0
chore/release-4.7.0-rc3
fix/1280-quote-must-say-it
fix/1257-names-row-squeeze
chore/release-4.7.0-rc2
fix/1245-pg-upgrade-count
chore/release-4.7.0-rc1
feat/1257-names-in-prep
feat/1257-be
feat/1257-fe
feat/1257-rating-fixes
design/1328-fill-it-in
feat/1257-name-splits
fix/1326-redis-503
feat/1280-fact-suggestions
feat/1280-be
feat/1280-fe
design/1280-fact-suggestions
feat/1279-infobox
feat/1279-be
feat/1279-fe
feat/1321-shelf-copies
design/1279-infobox
feat/1243-table-find
feat/1257-who-rated
feat/1243-prep-drawer
feat/1243-sheet-pickers
design/1243-browse-in-place
feat/1315-hotwords-default
feat/1242-system-filters
feat/1257-more-name-styles
feat/506-leg3-hotwords
design/1242-system-filters
fix/1302-packs-polish
feat/1241-rules-library
design/1241-rules-library
feat/1245-postgres-18
integ/4.6.2
fix/1263-map-ui-2
fix/1263-map-server
fix/1263-map-ui-1
fix/1261-editor-link-menu
fix/1229-axe-4.14-name-mismatch
renovate/axllent-mailpit-1.x
renovate/eslint-monorepo
renovate/authlib-1.x
chore/release-4.6.0
chore/release-4.6.0-rc2
fix/460-rc2-screens
fix/460-rc2-server
fix/release-prerelease-exit
chore/release-4.6.0-rc1
feat/125-campaign-templates
feat/126-calendar
feat/1181-pictures-and-files
feat/1190-event-people
integ/4.6.0
feat/1191-line-of-the-night
feat/1200-starter-packs
chore/bot-multidict-and-playwright-group
fix/1202-this-week-answered-count
fix/122-124-leftovers
fix/1178-get-routes-that-write
fix/1203-journeys-check-names
design/125-campaign-templates
feat/126-calendar-engine
design/126-in-world-calendar
fix/1189-cited-stamp-in-range
fix/1189-v4-looser-check
fix/1189-richer-beats
fix/1113-export-bind-lists
fix/1154-release-notes-by-file
chore/release-4.5.0
chore/release-4.5.0-rc12
fix/1135-wiki-markup-as-text
fix/demo-usage-cost-in-window
chore/release-4.5.0-rc11
fix/1121-sheet-consistency
fix/1121-sheet-5e-terms-and-card-hp
copy/1122-busiest-pages
fix/1121-relevant-choices
fix/1121-wiki-create-npc-save-sheet-links
fix/1029-menu-stays-on-screen
docs/prototype-transcript-bin-wording
fix/1023-editor-save-row-back-scroll
fix/1020-search-exact-first
fix/1020-table-prep
fix/1020-shell-wordmark-footer-bell
fix/1020-wiki-article-party
fix/872-admin-kept-until-phone-tables
feat/l13-milestone-level-waiting
fix/1000-1001-bell-older-setup-context
fix/999-pronouns-in-prompts
fix/l19-other-class-feats
fix/1010-transcript-window-ordinary-confirm
docs/d254-d269-decisions
fix/702-member-search
fix/680-transcript-bin
fix/993-unidentified-prompts
fix/841-redis-timeouts
fix/990-binned-not-exported
fix/830-export-prep-shelf
fix/deps-audit-2026-10-01
fix/877-title-reads
fix/798-audio-received-at
build/lane11-17-j7
build/lane11-16-levelup
build/lane11-14-seams
build/lane11-14-seed
build/lane11-15-start
build/lane11-13-old-block-out
fix/980-review-r3
build/lane11-12-block
build/lane11-8-packs
fix/demo-seed-canonical-names
build/lane11-11-hot-half
docs/l11-decisions
build/lane11-7-convert
build/lane11-10-editing
fix/npm-audit-brace-expansion
build/lane11-9-sheet-page
fix/bot-pyjwt
build/lane11-6-content
fix/lane11-rank-names
build/lane11-2b-language
build/lane11-4-js-evaluator
build/lane11-3-server
build/lane11-5a-pf2e
build/lane11-5b-5e
feat/design-admin
build/lane11-2-evaluator
build/lane11-1-sheet-pieces
build/lane11-0-drawing
fix/943-quest-objectives-fk
fix/prepsheet-shelf-flake
fix/945-leftovers
build/demo-gm-copy
fix/901-email-landings
build/lane10-8-journeys
build/demo-seed-lanes-8-10
fix/887-gm-own-confirmation
build/lane9-5-j1-admin
fix/918-audience-click
fix/useasync-overtaken
ci/939-critical-path
test/937-backend-suite-speed
chore/claude-orchestration
fix/924-xp-gm-only
fix/session-stale-poll
docs/decisions-d239-d242
fix/930-pypdf
build/lane8-journeys
build/925-quest-editor
fix/894-private-page-pictures
build/lane8-p4-drafts
build/lane8-p7-map
build/lane8-p6-handouts-rules
fix/wiki-fixture-names
build/lane10-d227-test-usage
fix/rsvp-horizon-test-tuesday
fix/media-volume-owner
build/lane8-p3-editor
fix/912-unidentified-rename
build/lane8-p5-party-quests
build/lane8-handout-takeback
build/lane10-pickers
build/lane8-p2-article
build/lane8-6-map
build/lane10-7b-email-pages
build/lane8-5-handouts
build/lane10-7a-test-the-bot-pages
build/lane8-p1-pages
build/lane8-4-quests
build/lane8-2b-feeds
build/lane8-3-parties
build/lane8-2a-links
build/lane10-5-commands
build/lane9-6-seed
build/lane10-4b-reactions
build/lane9-4d-pages
build/lane9-4c-pages
build/lane10-4a-posts
build/lane10-3-email
fix/891-web-fonts
build/lane9-4b-pages
build/lane10-6-test-the-bot
build/lane10-2-notify-out
build/lane8-1-wiki-text
build/lane9-4a-frame
design/lane10-drawings
fix/j6-budget
fix/j5-phone-flake
fix/869-test-leaks
fix/843-record-start-followups
build/lane7-all
build/lane6-all
build/lane9-b2-server
chore/changelog-fragments
docs/decisions-d218
build/lane9-a-owner-overview
fix/rule-card-level-label
fix/834-retention-semantics
fix/o3c-record-without-announcement
docs/decisions-d179
docs/decisions-d131-d170
fix/851-commit-before-response
build/lane6-j3-seed
build/lane9-b1-d67-identity
fix/841-blocking-loop
build/lane9-c-backups-audit
fix/vote-race-and-start-commit
fix/recording-announcement
fix/proposal-read-leak
fix/last-admin-guard
build/lane6-p4c-planning
build/lane6-p4b-margin
build/lane6-p4a-sheet
build/lane5-j4-seed
build/lane6-base
build/lane7-pages-talk
build/lane5-pages
build/lane7-pages-safety
build/lane7-p3-safety
build/lane7-p2-notify
build/lane6-p3-planning
build/lane6-p2-history
build/lane6-p1b-items
build/lane7-p1-threads
build/lane5-p6-faces
build/lane6-p1a-sheet
build/lane5-p5-find
build/lane5-p4a-board
build/lane5-p4b-scenes
build/lane5-p3-live
build/lane5-base
build/lane5-p2-cards
docs/lane5-decisions
build/review-r2-wide
fix/review-r1
build/lane5-p1-scenes
build/demo-seed
build/lane4-journeys-j5
build/lane4-page-review-b
build/lane4-page-review-a
build/lane4-page-read
build/lane4-journeys
build/lane4-page-sess2
build/lane4-page-sess1
build/lane4-notes
build/lane4-discord-on-approval
build/lane4-scheduling
build/lane4-versions
build/760-session-number
build/lane4-decisions
build/797-erase-no-regenerate
build/test-race-audit
build/lane3-completions
build/794-dark-only
build/796-titles-scroll
build/791-retention-anchor
build/a11y-sweep
build/shell-followups
build/retention-preview
build/753-backup-readback
docs/d91-d93
build/lane3-j1
build/ci-harness-paths-selftest
fix/useauth-test-race
build/ci-harness-paths
build/777-ai-settings-read
build/lane3-settings-c
build/lane3-settings-b
build/lane3-j6
build/lane3-sessionform
build/lane3-settings-a
build/lane3-members
build/lane3-thisweek
build/lane3-story
chore/87-align-python-3.12
v4.8.0-rc1
v4.7.1
v4.7.1-rc1
v4.7.0
v4.7.0-rc4
v4.7.0-rc3
v4.7.0-rc2
v4.7.0-rc1
v4.6.2
v4.6.2-rc4
v4.6.2-rc3
v4.6.2-rc2
v4.6.2-rc1
v4.6.1
v4.6.1-rc1
v4.6.0
v4.6.0-rc2
v4.6.0-rc1
v4.5.3
v4.5.3-rc1
v4.5.2
v4.5.2-rc1
v4.5.1
v4.5.1-rc2
v4.5.1-rc1
v4.5.0
v4.5.0-rc12
v4.5.0-rc11
v4.5.0-rc10
v4.5.0-rc9
v4.5.0-rc8
v4.5.0-rc7
v4.5.0-rc6
v4.5.0-rc5
v4.5.0-rc4
v4.5.0-rc3
v4.5.0-rc2
v4.5.0-rc1
v4.4.1
v4.4.0
v4.3.0
v4.2.3
v4.2.2
v4.2.1
v4.2.0
v4.1.2
v4.1.1
v4.1.0
v4.0.1
v4.0.0
v3.11.4
v3.11.3
v3.11.2
v3.11.1
v3.11.0
v3.10.0
v3.9.1
v3.9.0
v3.8.0
v3.7.0
v3.6.1
v3.6.0
v3.5.0
v3.4.1
v3.4.0
v3.3.0
v3.2.0
v3.1.0
v3.0.0
v2.1.0
v2.0.0
v1.1.0
v1.0.0
v0.12.0
v0.11.0
v0.10.0
v0.9.1
v0.3.3
v0.9.0
v0.8.0
v0.3.2
v0.3.1
v0.3.0
v0.7.0
v0.6.0
v0.5.0
v0.4.0
v0.2.1
v0.2.0
v0.1.1
v0.1.0
Labels
Clear labels
backend
Issue related to the backend
bot
Issue related to the bot
bug
Something is not working
duplicate
This issue or pull request already exists
enhancement
New feature
frontend
Issue related to the frontend
help wanted
Need some help
invalid
Something is wrong
question
More information is needed
recording
Issue related to the recording system
security
Security hardening / vulnerability work
wontfix
This won't be fixed
No labels
backend
bot
bug
duplicate
enhancement
frontend
help wanted
invalid
question
recording
security
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
rbrooks/Quest-Board#416
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Severity: HIGH
Found in the August 2026 session lifecycle review (#319).
Across the session-recording, lore-generation, wiki-draft, and Workbench pipelines, there are eleven distinct ways a piece of in-progress work can get permanently stuck in a state that only a database edit, filesystem edit, or console action can unstick — each one is a guaranteed support ticket in a hosted product whose stated bar is "zero operator intervention." Several already have dedicated defect issues covering their root cause; this issue tracks the ones that don't, and exists so the full list is visible and closed out together rather than piecemeal.
Evidence — the eleven enumerated stuck states, with cross-references to issues that already cover the root cause where one exists:
audio_processing_status='processing'forever (worker killed mid-task, or a pre-DB-context exception, or an enqueue failure after intake commits) — root cause tracked in "[Backend] Add Celery acks_late and a watchdog...".lore_generation_status='pending'from a lost/failed enqueue — root cause tracked in "[Backend] Enqueue before committing lore_generation_status=pending...".lore_generation_status='extracting'/'matching'after a worker SIGKILL — the API would accept a re-trigger, butwebapp/frontend/src/pages/SessionDetail.jsx:108(LORE_STATUS_IN_PROGRESS) hides the re-run button for every in-progress state, so it's stuck from the user's chair even though the backend isn't fully blocking it. Not yet covered elsewhere — needs a UI fix (show a re-run option after some elapsed time) independent of the backend watchdog.generatingafter a worker crash — only self-service exit isdiscard_draft, which throws away all iteration work. Not yet covered elsewhere — needs its own failure path/watchdog coverage, or the discard-only escape needs a less destructive alternative.GenerationResultstuckpendingforever — the frontend gives up polling at ~60s (webapp/frontend/src/components/GeneratorPanel.jsx:40-41,MAX_POLLS = 40×POLL_INTERVAL_MS); self-serviceable by re-running (at the cost of paying for tokens again), but reads as "the tool ate my generation." Not yet covered elsewhere.content_approved_at409s re-approval (webapp/backend/app/routers/sessions.py:635-639), and there is no un-approve action, so a post-approval summary edit never reaches Discord. Not yet covered elsewhere.webapp/frontend/src/pages/CampaignDetail.jsx:1075) fires the notes-harvest and Discord fan-out irreversibly. Data isn't lost, but the state transition can't be undone. Not yet covered elsewhere.Failure scenario
Any one of the "not yet covered elsewhere" items above happens in isolation and turns into a support ticket with no self-service resolution — for example, a wiki draft's generation worker is killed mid-run, the draft is stuck at
generating, and the GM's only available action destroys the iteration work they were part-way through rather than letting them retry.Proposed fix
For each stuck state without an existing owner, add either (a) a watchdog-driven terminal-state transition (piggybacking on the cross-pipeline watchdog tracked in "[Backend] Build a generic stuck-task watchdog..."), or (b) where the backend already isn't the blocker (the lore in-progress re-run button case), a frontend fix to stop hiding the retry action indefinitely. For the one-way state transitions (approved summary, completed/cancelled sessions), add an explicit, audited un-approve/un-complete/un-cancel action rather than leaving the only escape as impossible or destructive.
Acceptance criteria
generatinghas a non-destructive recovery path that doesn't require discarding iteration work.Picking this up. PR #449 covers four of the five states that had no existing owner; the issue stays open for the fifth plus the resolution log below.
Status of all eleven
Six were cross-referenced to root-cause issues; three of those have now shipped:
audio_processing_status='processing'foreveracks_late+ hourly watchdoglore_generation_status='pending'from a lost enqueueThe five with no owner:
extracting/matching— UI hides retrygeneratingGenerationResultstuckpendingNotes on two of them
Workbench pending — this reads worse in the issue than it is.
GenerationHistory.jsxfilters only bytool_id, never by status, sopendingandfailedrows both render with a status badge and a working "Re-open" button; the backend's history endpoint has no status filter either. #398's watchdog now gives a stranded row a terminalfailedstate within the hour. The generation is visible and recoverable throughout, so I'd rather record this as closed than invent work to satisfy a checkbox. Happy to reopen it if you disagree.Lore in-progress retry — deliberately held back from #449. Doing it properly needs a
lore_generation_started_atcolumn (the only timestamp today isSession.updated_at, which any unrelated write bumps, and it isn't even exposed inSessionResponse). A migration doesn't belong smuggled into a PR of endpoint additions.A defect found while surveying that one, not in this issue
generate_lore_proposals(routers/sessions.py:253) guards onsession.lore_generation_status == LoreGenerationStatus.pendingonly.extractingandmatchingfall straight through, so calling the endpoint directly during either mid-pipeline state commitspendingand queues a second concurrent extraction over the same transcript — duplicate proposals and a doubled token bill. It's masked today purely becauseLORE_STATUS_IN_PROGRESShides the button in the UI for all three states, which is the same line the issue asks us to relax. Relaxing it without fixing the guard would turn a latent bug into a reachable one, so the two ship together.PR #449 verification
Mutation-checked — 13 of 14 new backend tests fail against the unfixed code. The 14th passed vacuously (asserting "the audio is still trashed" holds equally when the endpoint 404s) and was tightened until it failed too. Frontend tests were run in a container, which caught a real bug in the new test helper that reading alone had missed. 1,423 backend + 446 frontend tests pass.
PR #449 merged (CI green on both commits). All five states that had no other owner are closed:
uncomplete/uncancel, auditedunapprove, so approving repostsgeneratingreset— keeps every edit and iterationextracting/matching— UI hides retrylore_generation_progress_at+ a 30-min staleness thresholdGenerationResultstuckpendingThe lore item also closed a defect this issue didn't list:
generate_lore_proposalsguarded onpendingalone, so a call duringextracting/matchingqueued a second extraction over the same transcript. It was unreachable only because the UI hid the button — the exact rule this issue asked us to relax — so both halves had to ship together.Leaving this open for the six states tracked elsewhere. Three have shipped (#398, #399, and the audio half); still open: #413 (lore enqueue wedge), #411 (co-GM journal 500), the orphaned-audio-directory reconciliation, and the bot's silent vote-reaction failure. Once those four land, every acceptance criterion here is met and this can close.
Acceptance criteria status:
generatinghas a non-destructive recovery pathAll six acceptance criteria are met. Closing.
Verified against the code on
mainrather than from memory of what was merged:SessionDetail.jsx—LORE_STALE_AFTER_MS, mirroringLORE_GENERATION_STALE_AFTERinrouters/sessions.py, keyed on the newsessions.lore_generation_progress_atgeneratinghas a non-destructive recovery pathunstick_lost_taskstransitionsLoreDraftStatus.generatingpast the cutoff, leaving the iteration work intact — the discard-only escape is no longer the only exitPOST /sessions/{id}/unapprove(routers/sessions.py), audited assession.content_unapprovedPOST /sessions/{id}/uncompleteand/uncancel, auditedThe eleven, and where each was actually resolved:
audio_processing_status='processing'forever — #398 (acks_late+unstick_lost_tasks)lore_generation_status='pending'from a lost enqueue — #413 (enqueue failures now mark the runfailedwith a re-runnable message)lore_generation_status='extracting'/'matching'after SIGKILL — #416's own UI fix, abovegenerating— #398's watchdogGenerationResultstuckpending— #398's watchdogOne correction to this issue's own framing. The body says the Workbench
GenerationResultcase is "self-serviceable by re-running (at the cost of paying for tokens again)". By the time the watchdog work landed that was no longer the interesting part: the row now reaches a terminal state on its own, so the GM is told the generation failed rather than being left to infer it from a spinner that stopped. The frontend'sMAX_POLLScap is unchanged and still correct — it bounds the polling, it no longer bounds the information.Worth recording for whoever reads this next: on five of these, the issue's stated evidence had decayed by the time the work started — the root cause had already been fixed by a sibling issue, or the file/line cited had moved. Reading the linked issues before trusting an enumeration like this one is not optional.
Noted while doing #395 (v4.3.0): on
WikiDraftReview, the generating banner, and therefore both the new "Check again" control and this issue's reset, live inside the non-convert_statslayout branch, so aconvert_statsdraft stuck ingeneratingstill cannot reach either control. The top status banner now reports the stall in every mode, but the actions do not. Moving that block above the layout branch is about five lines and belongs with this issue's surface.