[Backend] Enqueue before committing lore_generation_status=pending to avoid a stuck wedge #413

New issue

Closed

opened 2026-08-25 20:44:54 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-08-25 20:44:54 +00:00

Collaborator

Severity: LOW

Found in the August 2026 session lifecycle review (#319).

The lore-generation status is committed to the database as pending before the Celery task that would actually process it is enqueued — so if the broker enqueue fails (which the broker is deliberately configured to fail fast on), the session is left showing pending with nothing ever going to run, the retry API 409s treating pending as already-in-progress, and the UI hides the retry button for every in-progress state — a self-inflicted stuck state from an ordering bug rather than an infrastructure failure.

Evidence

webapp/backend/app/routers/sessions.py:250-256 — the session's lore_generation_status is committed to pending before .delay() is called; if the broker is unreachable, .delay() raises after the commit has already landed, leaving pending with no task actually queued.
webapp/backend/app/tasks/celery_app.py:38-48 — broker_transport_options sets a 2-second socket_connect_timeout/socket_timeout with retry_on_timeout: False, explicitly so a broker outage fails fast rather than hanging — which is correct for the caller, but means this exact wedge is a routine, expected occurrence during any broker blip, not a rare edge case.
The same class recurs via _safe_delay in webapp/backend/app/services/summary_events.py:21-34, which swallows an enqueue exception after process_audio has already committed lore_generation_status = pending (reminder_tasks.py:2190-2194) as part of the fan-out — so the wedge is reachable from two independent call sites.

Failure scenario
The Redis broker has a brief connectivity blip exactly when a GM triggers lore generation (or right after a transcription completes and the fan-out tries to enqueue it). The DB commit of pending succeeds; the .delay() call fails and is logged (or silently swallowed via _safe_delay). The session now shows "Lore generation queued…" forever — the API refuses to let anyone re-trigger it because it looks already-in-progress, and the UI's re-run button is hidden for every in-progress state. The only fix is a manual database update.

Proposed fix
Reorder to enqueue first, and only commit pending to the database once the enqueue has actually succeeded — if .delay() raises, surface the failure immediately (to the caller for the synchronous endpoint case; as a failed status directly, skipping pending entirely, for the fan-out case) instead of leaving a status implying work is in flight when it never started.

Acceptance criteria

lore_generation_status is only set to pending after the Celery task has been successfully enqueued, not before.
A failed enqueue (from either the synchronous endpoint or the on_session_summary_available fan-out) results in a failed status with a clear error, not a silent pending wedge.
Regression test: simulate a broker enqueue failure and assert the session does not end up stuck at pending.

**Severity: LOW** Found in the August 2026 session lifecycle review (#319). The lore-generation status is committed to the database as `pending` before the Celery task that would actually process it is enqueued — so if the broker enqueue fails (which the broker is deliberately configured to fail fast on), the session is left showing `pending` with nothing ever going to run, the retry API 409s treating `pending` as already-in-progress, and the UI hides the retry button for every in-progress state — a self-inflicted stuck state from an ordering bug rather than an infrastructure failure. **Evidence** - `webapp/backend/app/routers/sessions.py:250-256` — the session's `lore_generation_status` is committed to `pending` before `.delay()` is called; if the broker is unreachable, `.delay()` raises after the commit has already landed, leaving `pending` with no task actually queued. - `webapp/backend/app/tasks/celery_app.py:38-48` — `broker_transport_options` sets a 2-second `socket_connect_timeout`/`socket_timeout` with `retry_on_timeout: False`, explicitly so a broker outage fails fast rather than hanging — which is correct for the caller, but means this exact wedge is a routine, expected occurrence during any broker blip, not a rare edge case. - The same class recurs via `_safe_delay` in `webapp/backend/app/services/summary_events.py:21-34`, which swallows an enqueue exception after `process_audio` has already committed `lore_generation_status = pending` (`reminder_tasks.py:2190-2194`) as part of the fan-out — so the wedge is reachable from two independent call sites. **Failure scenario** The Redis broker has a brief connectivity blip exactly when a GM triggers lore generation (or right after a transcription completes and the fan-out tries to enqueue it). The DB commit of `pending` succeeds; the `.delay()` call fails and is logged (or silently swallowed via `_safe_delay`). The session now shows "Lore generation queued…" forever — the API refuses to let anyone re-trigger it because it looks already-in-progress, and the UI's re-run button is hidden for every in-progress state. The only fix is a manual database update. **Proposed fix** Reorder to enqueue first, and only commit `pending` to the database once the enqueue has actually succeeded — if `.delay()` raises, surface the failure immediately (to the caller for the synchronous endpoint case; as a `failed` status directly, skipping `pending` entirely, for the fan-out case) instead of leaving a status implying work is in flight when it never started. **Acceptance criteria** - [ ] `lore_generation_status` is only set to `pending` after the Celery task has been successfully enqueued, not before. - [ ] A failed enqueue (from either the synchronous endpoint or the `on_session_summary_available` fan-out) results in a `failed` status with a clear error, not a silent `pending` wedge. - [ ] Regression test: simulate a broker enqueue failure and assert the session does not end up stuck at `pending`.

claude-bot added this to the v4.1.0 - Data Durability & Recovery milestone

2026-08-25 20:44:54 +00:00

claude-bot added the

backend

enhancement

labels

2026-08-25 20:44:54 +00:00

claude-bot referenced this issue

2026-08-29 23:22:03 +00:00

[Ops] Close the eleven manual-intervention stuck states across session, lore, draft, and workbench pipelines #416

claude-bot referenced this issue

2026-08-29 23:43:17 +00:00