Pipeline lifecycle hardening: clear-path row locks, tracked dispatch tasks, shared SSE LISTEN connection #53

New issue

Closed

opened 2026-07-18 05:31:18 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-18 05:31:18 +00:00

Contributor

(1) expire_alerts_job (scheduler.py:55-76) and poll_alerts_job →
_clear_missing_alerts/send_pending_lifted_notifications both select
uncleared/unlifted rows in separate sessions with no locking; overlap can
double-send all-clears. Use with_for_update(skip_locked=True) on those
selections (or funnel clearing through one job).
(2) Dispatch tasks are fire-and-forget (alert_processor.py:656-659); the
event loop holds only weak refs, so tasks can be GC-cancelled mid-send and
are not awaited on shutdown. Track them in a module-level set with
done-callbacks; cancel/await in lifespan shutdown.
(3) Each SSE client opens a dedicated asyncpg LISTEN connection
(app/api/sse.py:46-53) with no cap — including the public endpoint. Move to
one shared LISTEN connection with in-process fan-out to bounded per-client
queues.

Acceptance criteria:

Overlapping clear/lifted runs cannot double-dispatch (locking test or reasoning documented)
Dispatch tasks tracked and drained on shutdown
SSE uses one PG connection regardless of client count

Filed from the 2026-07-17 codebase audit (docs/.internal/report-2026-07-17.md), finding F-14.

(1) `expire_alerts_job` (scheduler.py:55-76) and `poll_alerts_job` → `_clear_missing_alerts`/`send_pending_lifted_notifications` both select uncleared/unlifted rows in separate sessions with no locking; overlap can double-send all-clears. Use `with_for_update(skip_locked=True)` on those selections (or funnel clearing through one job). (2) Dispatch tasks are fire-and-forget (`alert_processor.py:656-659`); the event loop holds only weak refs, so tasks can be GC-cancelled mid-send and are not awaited on shutdown. Track them in a module-level set with done-callbacks; cancel/await in lifespan shutdown. (3) Each SSE client opens a dedicated asyncpg LISTEN connection (app/api/sse.py:46-53) with no cap — including the public endpoint. Move to one shared LISTEN connection with in-process fan-out to bounded per-client queues. **Acceptance criteria:** - [ ] Overlapping clear/lifted runs cannot double-dispatch (locking test or reasoning documented) - [ ] Dispatch tasks tracked and drained on shutdown - [ ] SSE uses one PG connection regardless of client count --- _Filed from the 2026-07-17 codebase audit (`docs/.internal/report-2026-07-17.md`), finding F-14._

claude-bot added this to the v1.2.0 - Notification formatting & delivery reliability milestone

2026-07-18 05:31:18 +00:00

claude-bot added the

area:notifications

type:maintenance

labels

2026-07-18 05:31:18 +00:00

claude-bot referenced this issue

2026-07-18 05:32:12 +00:00

Migrate richer channels onto the shared notification formatter #14

claude-bot referenced this issue

2026-07-18 19:58:47 +00:00

Twilio send blocks the event loop — offload to a thread #56

rbrooks referenced this issue from a commit

2026-07-18 20:57:52 +00:00

Add delivery outbox with bounded retries; harden pipeline lifecycle

claude-bot referenced this issue

2026-07-18 20:58:08 +00:00

v1.2.0 Phase B: delivery outbox + pipeline lifecycle #100

claude-bot commented

2026-07-18 21:05:51 +00:00

Author

Contributor

Done in #100 (merged). (1) Clear/lifted row selections take with_for_update(skip_locked=True) so overlapping runs can't double-send all-clears. (2) Dispatch is now inline via the outbox (the GC-able fire-and-forget task is gone); the remaining background tasks (e.g. the heartbeat ping) are tracked in a module set and drained on shutdown. (3) app/api/sse.py now uses a single shared LISTEN broker fanning out to per-client bounded queues (drop-oldest on overflow), started/stopped in the app lifespan — replacing the one-Postgres-connection-per-client model.

Done in #100 (merged). (1) Clear/lifted row selections take `with_for_update(skip_locked=True)` so overlapping runs can't double-send all-clears. (2) Dispatch is now inline via the outbox (the GC-able fire-and-forget task is gone); the remaining background tasks (e.g. the heartbeat ping) are tracked in a module set and drained on shutdown. (3) `app/api/sse.py` now uses a single shared LISTEN broker fanning out to per-client bounded queues (drop-oldest on overflow), started/stopped in the app lifespan — replacing the one-Postgres-connection-per-client model.

claude-bot closed this issue

2026-07-18 21:05:55 +00:00

rbrooks referenced this issue from a commit

2026-07-19 04:21:21 +00:00

Extract dashboard assembly into a service; drop the redundant live SPC fan-out (#58)

claude-bot referenced this issue

2026-07-19 04:21:35 +00:00

v1.4.0 Phase 3: extract dashboard assembly, drop live SPC fan-out, consolidate match-context (#58) #110

claude-bot referenced this issue

2026-07-19 04:28:59 +00:00

Extract dashboard assembly from main.py; stop the 13-request live SPC fan-out per page load #58