[Backend] Enqueue before committing lore_generation_status=pending to avoid a stuck wedge #413
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Severity: LOW
Found in the August 2026 session lifecycle review (#319).
The lore-generation status is committed to the database as
pendingbefore the Celery task that would actually process it is enqueued — so if the broker enqueue fails (which the broker is deliberately configured to fail fast on), the session is left showingpendingwith nothing ever going to run, the retry API 409s treatingpendingas already-in-progress, and the UI hides the retry button for every in-progress state — a self-inflicted stuck state from an ordering bug rather than an infrastructure failure.Evidence
webapp/backend/app/routers/sessions.py:250-256— the session'slore_generation_statusis committed topendingbefore.delay()is called; if the broker is unreachable,.delay()raises after the commit has already landed, leavingpendingwith no task actually queued.webapp/backend/app/tasks/celery_app.py:38-48—broker_transport_optionssets a 2-secondsocket_connect_timeout/socket_timeoutwithretry_on_timeout: False, explicitly so a broker outage fails fast rather than hanging — which is correct for the caller, but means this exact wedge is a routine, expected occurrence during any broker blip, not a rare edge case._safe_delayinwebapp/backend/app/services/summary_events.py:21-34, which swallows an enqueue exception afterprocess_audiohas already committedlore_generation_status = pending(reminder_tasks.py:2190-2194) as part of the fan-out — so the wedge is reachable from two independent call sites.Failure scenario
The Redis broker has a brief connectivity blip exactly when a GM triggers lore generation (or right after a transcription completes and the fan-out tries to enqueue it). The DB commit of
pendingsucceeds; the.delay()call fails and is logged (or silently swallowed via_safe_delay). The session now shows "Lore generation queued…" forever — the API refuses to let anyone re-trigger it because it looks already-in-progress, and the UI's re-run button is hidden for every in-progress state. The only fix is a manual database update.Proposed fix
Reorder to enqueue first, and only commit
pendingto the database once the enqueue has actually succeeded — if.delay()raises, surface the failure immediately (to the caller for the synchronous endpoint case; as afailedstatus directly, skippingpendingentirely, for the fan-out case) instead of leaving a status implying work is in flight when it never started.Acceptance criteria
lore_generation_statusis only set topendingafter the Celery task has been successfully enqueued, not before.on_session_summary_availablefan-out) results in afailedstatus with a clear error, not a silentpendingwedge.pending.