fix(lore): report a lost enqueue instead of wedging at pending (#413) #459

Merged

claude-bot merged 1 commit from fix/413-lore-enqueue-wedge into main

2026-08-31 04:16:21 +00:00

claude-bot commented

2026-08-31 03:49:55 +00:00

Contributor

Closes #413 (LOW). Also closes the last of #416's cross-referenced stuck states that is purely backend.

The defect

lore_generation_status is committed as pending and then the Celery task is enqueued. The broker is deliberately configured to fail fast — 2s timeouts, retry_on_timeout: False — so an unreachable broker is a routine event, not an edge case. The commit lands, nothing is queued, and the session shows "Lore generation queued…" with no work behind it. The endpoint's own guard then treats that ghost as a live run and refuses to re-trigger.

Reachable from two independent call sites: the GM-facing endpoint, and the on_session_summary_available fan-out after process_audio (where _safe_delay swallowed the exception entirely).

Deliberately not the reorder the issue proposes

The issue asks that pending be set only after a successful enqueue. That introduces a race: generate_lore_proposals does not require pending on pickup — it loads the session and proceeds. Enqueueing before the commit lets a fast worker set extracting, which the later pending commit then overwrites. That trades a visible wedge for a silent one, in a milestone about not losing state.

Marking failed on the way out is the same repair without the race:

The endpoint sets failed with a plain explanation and returns 503, so the caller learns the click did nothing rather than getting a 202 for work that was never queued.
The fan-out: _safe_delay now returns whether the enqueue landed, and process_audio marks lore failed when it did not. Swallowing remains correct for the tasks whose absence changes no stored state; it was only ever wrong for the one whose status the caller had already committed as queued.

failed specifically, because that is the state the retry paths accept — so the re-run button is back immediately, rather than after #416's 30-minute staleness threshold or #398's 8-hour watchdog. Both remain as backstops; this is the fast path.

Worth noting: the issue's severity has already dropped

When #413 was filed, the stated consequence was "The only fix is a manual database update." That is no longer true — #416 added a staleness threshold that lets a GM supersede a stalled run after 30 minutes, and #398's watchdog flips it to failed after 8 hours. This PR turns "wait 30 minutes" into "told at once", which is the remaining gap rather than the original one.

Verification

Mutation-checked: with the fix reverted, 5 of 6 tests fail.

One test exists specifically to stop this change regressing in the other direction — _safe_delay swallows so that one broker hiccup cannot block the other fan-out enqueues, and that behaviour is now pinned. Making the fan-out raise would be an easy "cleanup" for someone to attempt later.

1,476 backend tests pass (was 1,470). Lint clean at pinned ruff 0.4.4.

🤖 Generated with Claude Code

Closes #413 (LOW). Also closes the last of #416's cross-referenced stuck states that is purely backend. ## The defect `lore_generation_status` is committed as `pending` and *then* the Celery task is enqueued. The broker is deliberately configured to fail fast — 2s timeouts, `retry_on_timeout: False` — so an unreachable broker is a **routine event, not an edge case**. The commit lands, nothing is queued, and the session shows "Lore generation queued…" with no work behind it. The endpoint's own guard then treats that ghost as a live run and refuses to re-trigger. Reachable from two independent call sites: the GM-facing endpoint, and the `on_session_summary_available` fan-out after `process_audio` (where `_safe_delay` swallowed the exception entirely). ## Deliberately not the reorder the issue proposes The issue asks that `pending` be set **only after** a successful enqueue. That introduces a race: `generate_lore_proposals` does **not** require `pending` on pickup — it loads the session and proceeds. Enqueueing before the commit lets a fast worker set `extracting`, which the later `pending` commit then overwrites. That trades a visible wedge for a silent one, in a milestone about not losing state. Marking `failed` on the way out is the same repair without the race: - **The endpoint** sets `failed` with a plain explanation and returns **503**, so the caller learns the click did nothing rather than getting a 202 for work that was never queued. - **The fan-out**: `_safe_delay` now returns whether the enqueue landed, and `process_audio` marks lore `failed` when it did not. Swallowing remains correct for the tasks whose absence changes no stored state; it was only ever wrong for the one whose status the caller had *already committed as queued*. `failed` specifically, because that is the state the retry paths accept — so the re-run button is back **immediately**, rather than after #416's 30-minute staleness threshold or #398's 8-hour watchdog. Both remain as backstops; this is the fast path. ## Worth noting: the issue's severity has already dropped When #413 was filed, the stated consequence was *"The only fix is a manual database update."* That is no longer true — #416 added a staleness threshold that lets a GM supersede a stalled run after 30 minutes, and #398's watchdog flips it to `failed` after 8 hours. This PR turns "wait 30 minutes" into "told at once", which is the remaining gap rather than the original one. ## Verification Mutation-checked: with the fix reverted, **5 of 6 tests fail**. One test exists specifically to stop this change regressing in the other direction — `_safe_delay` swallows so that one broker hiccup cannot block the *other* fan-out enqueues, and that behaviour is now pinned. Making the fan-out raise would be an easy "cleanup" for someone to attempt later. **1,476 backend tests pass** (was 1,470). Lint clean at pinned ruff 0.4.4. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit

2026-08-31 03:49:56 +00:00

fix(lore): report a lost enqueue instead of wedging at pending (#413 )

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 47s

Details

CI / Docker image build (pull_request) Successful in 10s

Details

CI / Backend lint (ruff) (pull_request) Successful in 24s

Details

CI / Bot tests and audit (pull_request) Successful in 1m13s

Details

CI / Bot/backend version sync (pull_request) Successful in 45s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 3m9s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 10m34s

Details

2b03edb877

lore_generation_status is committed as `pending` and then the task is
enqueued. The broker is deliberately configured to fail fast — 2s timeouts,
no retry — so an unreachable broker is routine, not an edge case. The
commit lands, nothing is queued, and the session shows "queued…" forever;
the endpoint's own guard then treats that ghost as a live run.

Both call sites now report it. The endpoint marks the session `failed` with
a plain explanation and returns 503, so the caller knows the click did
nothing. The fan-out's _safe_delay returns whether the enqueue landed, and
process_audio marks lore `failed` when it did not — swallowing is right for
tasks whose absence changes no stored state, and wrong for the one whose
status the caller had already committed as queued.

Deliberately NOT the reorder the issue proposes. generate_lore_proposals
does not require `pending` on pickup, so enqueueing before the commit lets
a fast worker set `extracting` and have the later commit overwrite it —
trading a visible wedge for a silent one. Marking failed on the way out is
the same repair without the race.

`failed` specifically because that is the state the retry paths accept, so
the button is back immediately rather than after #416's 30-minute staleness
threshold or #398's 8-hour watchdog. Both of those remain as backstops; this
is the fast path.

Mutation-checked: with the fix reverted, 5 of 6 tests fail. One of them
pins the behaviour that motivated _safe_delay in the first place — one
broker hiccup must still not block the other enqueues — so the change
cannot regress into raising.

1,476 backend tests pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

claude-bot merged commit 5d6a9c2f02 into main

2026-08-31 04:16:21 +00:00

claude-bot deleted branch fix/413-lore-enqueue-wedge

2026-08-31 04:16:22 +00:00

claude-bot referenced this pull request from a commit

2026-08-31 04:16:23 +00:00

Merge PR #459: report a lost enqueue instead of wedging at pending (#413)