[Ops] Close the eleven manual-intervention stuck states across session, lore, draft, and workbench pipelines #416
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Severity: HIGH
Found in the August 2026 session lifecycle review (#319).
Across the session-recording, lore-generation, wiki-draft, and Workbench pipelines, there are eleven distinct ways a piece of in-progress work can get permanently stuck in a state that only a database edit, filesystem edit, or console action can unstick — each one is a guaranteed support ticket in a hosted product whose stated bar is "zero operator intervention." Several already have dedicated defect issues covering their root cause; this issue tracks the ones that don't, and exists so the full list is visible and closed out together rather than piecemeal.
Evidence — the eleven enumerated stuck states, with cross-references to issues that already cover the root cause where one exists:
audio_processing_status='processing'forever (worker killed mid-task, or a pre-DB-context exception, or an enqueue failure after intake commits) — root cause tracked in "[Backend] Add Celery acks_late and a watchdog...".lore_generation_status='pending'from a lost/failed enqueue — root cause tracked in "[Backend] Enqueue before committing lore_generation_status=pending...".lore_generation_status='extracting'/'matching'after a worker SIGKILL — the API would accept a re-trigger, butwebapp/frontend/src/pages/SessionDetail.jsx:108(LORE_STATUS_IN_PROGRESS) hides the re-run button for every in-progress state, so it's stuck from the user's chair even though the backend isn't fully blocking it. Not yet covered elsewhere — needs a UI fix (show a re-run option after some elapsed time) independent of the backend watchdog.generatingafter a worker crash — only self-service exit isdiscard_draft, which throws away all iteration work. Not yet covered elsewhere — needs its own failure path/watchdog coverage, or the discard-only escape needs a less destructive alternative.GenerationResultstuckpendingforever — the frontend gives up polling at ~60s (webapp/frontend/src/components/GeneratorPanel.jsx:40-41,MAX_POLLS = 40×POLL_INTERVAL_MS); self-serviceable by re-running (at the cost of paying for tokens again), but reads as "the tool ate my generation." Not yet covered elsewhere.content_approved_at409s re-approval (webapp/backend/app/routers/sessions.py:635-639), and there is no un-approve action, so a post-approval summary edit never reaches Discord. Not yet covered elsewhere.webapp/frontend/src/pages/CampaignDetail.jsx:1075) fires the notes-harvest and Discord fan-out irreversibly. Data isn't lost, but the state transition can't be undone. Not yet covered elsewhere.Failure scenario
Any one of the "not yet covered elsewhere" items above happens in isolation and turns into a support ticket with no self-service resolution — for example, a wiki draft's generation worker is killed mid-run, the draft is stuck at
generating, and the GM's only available action destroys the iteration work they were part-way through rather than letting them retry.Proposed fix
For each stuck state without an existing owner, add either (a) a watchdog-driven terminal-state transition (piggybacking on the cross-pipeline watchdog tracked in "[Backend] Build a generic stuck-task watchdog..."), or (b) where the backend already isn't the blocker (the lore in-progress re-run button case), a frontend fix to stop hiding the retry action indefinitely. For the one-way state transitions (approved summary, completed/cancelled sessions), add an explicit, audited un-approve/un-complete/un-cancel action rather than leaving the only escape as impossible or destructive.
Acceptance criteria
generatinghas a non-destructive recovery path that doesn't require discarding iteration work.Picking this up. PR #449 covers four of the five states that had no existing owner; the issue stays open for the fifth plus the resolution log below.
Status of all eleven
Six were cross-referenced to root-cause issues; three of those have now shipped:
audio_processing_status='processing'foreveracks_late+ hourly watchdoglore_generation_status='pending'from a lost enqueueThe five with no owner:
extracting/matching— UI hides retrygeneratingGenerationResultstuckpendingNotes on two of them
Workbench pending — this reads worse in the issue than it is.
GenerationHistory.jsxfilters only bytool_id, never by status, sopendingandfailedrows both render with a status badge and a working "Re-open" button; the backend's history endpoint has no status filter either. #398's watchdog now gives a stranded row a terminalfailedstate within the hour. The generation is visible and recoverable throughout, so I'd rather record this as closed than invent work to satisfy a checkbox. Happy to reopen it if you disagree.Lore in-progress retry — deliberately held back from #449. Doing it properly needs a
lore_generation_started_atcolumn (the only timestamp today isSession.updated_at, which any unrelated write bumps, and it isn't even exposed inSessionResponse). A migration doesn't belong smuggled into a PR of endpoint additions.A defect found while surveying that one, not in this issue
generate_lore_proposals(routers/sessions.py:253) guards onsession.lore_generation_status == LoreGenerationStatus.pendingonly.extractingandmatchingfall straight through, so calling the endpoint directly during either mid-pipeline state commitspendingand queues a second concurrent extraction over the same transcript — duplicate proposals and a doubled token bill. It's masked today purely becauseLORE_STATUS_IN_PROGRESShides the button in the UI for all three states, which is the same line the issue asks us to relax. Relaxing it without fixing the guard would turn a latent bug into a reachable one, so the two ship together.PR #449 verification
Mutation-checked — 13 of 14 new backend tests fail against the unfixed code. The 14th passed vacuously (asserting "the audio is still trashed" holds equally when the endpoint 404s) and was tightened until it failed too. Frontend tests were run in a container, which caught a real bug in the new test helper that reading alone had missed. 1,423 backend + 446 frontend tests pass.
PR #449 merged (CI green on both commits). All five states that had no other owner are closed:
uncomplete/uncancel, auditedunapprove, so approving repostsgeneratingreset— keeps every edit and iterationextracting/matching— UI hides retrylore_generation_progress_at+ a 30-min staleness thresholdGenerationResultstuckpendingThe lore item also closed a defect this issue didn't list:
generate_lore_proposalsguarded onpendingalone, so a call duringextracting/matchingqueued a second extraction over the same transcript. It was unreachable only because the UI hid the button — the exact rule this issue asked us to relax — so both halves had to ship together.Leaving this open for the six states tracked elsewhere. Three have shipped (#398, #399, and the audio half); still open: #413 (lore enqueue wedge), #411 (co-GM journal 500), the orphaned-audio-directory reconciliation, and the bot's silent vote-reaction failure. Once those four land, every acceptance criterion here is met and this can close.
Acceptance criteria status:
generatinghas a non-destructive recovery pathAll six acceptance criteria are met. Closing.
Verified against the code on
mainrather than from memory of what was merged:SessionDetail.jsx—LORE_STALE_AFTER_MS, mirroringLORE_GENERATION_STALE_AFTERinrouters/sessions.py, keyed on the newsessions.lore_generation_progress_atgeneratinghas a non-destructive recovery pathunstick_lost_taskstransitionsLoreDraftStatus.generatingpast the cutoff, leaving the iteration work intact — the discard-only escape is no longer the only exitPOST /sessions/{id}/unapprove(routers/sessions.py), audited assession.content_unapprovedPOST /sessions/{id}/uncompleteand/uncancel, auditedThe eleven, and where each was actually resolved:
audio_processing_status='processing'forever — #398 (acks_late+unstick_lost_tasks)lore_generation_status='pending'from a lost enqueue — #413 (enqueue failures now mark the runfailedwith a re-runnable message)lore_generation_status='extracting'/'matching'after SIGKILL — #416's own UI fix, abovegenerating— #398's watchdogGenerationResultstuckpending— #398's watchdogOne correction to this issue's own framing. The body says the Workbench
GenerationResultcase is "self-serviceable by re-running (at the cost of paying for tokens again)". By the time the watchdog work landed that was no longer the interesting part: the row now reaches a terminal state on its own, so the GM is told the generation failed rather than being left to infer it from a spinner that stopped. The frontend'sMAX_POLLScap is unchanged and still correct — it bounds the polling, it no longer bounds the information.Worth recording for whoever reads this next: on five of these, the issue's stated evidence had decayed by the time the work started — the root cause had already been fixed by a sibling issue, or the file/line cited had moved. Reading the linked issues before trusting an enumeration like this one is not optional.
Noted while doing #395 (v4.3.0): on
WikiDraftReview, the generating banner, and therefore both the new "Check again" control and this issue's reset, live inside the non-convert_statslayout branch, so aconvert_statsdraft stuck ingeneratingstill cannot reach either control. The top status banner now reports the stall in every mode, but the actions do not. Moving that block above the layout branch is about five lines and belongs with this issue's surface.