v1.2.0 Phase B: delivery outbox + pipeline lifecycle #100

Merged

claude-bot merged 1 commit from feat/v1.2.0-phaseB-outbox into main

2026-07-18 21:04:35 +00:00

claude-bot commented

2026-07-18 20:58:08 +00:00

Contributor

The delivery-reliability core of v1.2.0 — the milestone's keystone.

#50 — delivery outbox (bounded retries)

Previously every dispatch path committed the dedup SentAlert before sending, so a transient channel failure was logged and then permanently dropped (at-most-once delivery). Now:

A notification_deliveries table (migration 0020) records per-(alert, channel, kind) attempts with status/attempts/last_error/next_attempt_at and a serialized payload (so a retry in a later process reconstructs the full warning text).
Dispatch commits a durable pending row, attempts the send inline (prompt initial delivery preserved), then a process_delivery_outbox scheduler job retries failed rows with exponential backoff (FOR UPDATE SKIP LOCKED) up to DELIVERY_MAX_ATTEMPTS/DELIVERY_MAX_AGE_HOURS, then dead. A sent row is never re-sent (idempotent).
Notifiers now signal outcome (notifiers/errors.py): send_alert/send_alert_lifted raise TransientDeliveryError (5xx/timeout/connection → retry) or PermanentDeliveryError (disabled/misconfig/4xx → dead) instead of swallowing HTTP errors — so the outbox observes and retries the real transient failures, not just unexpected exceptions. SPC/forecast sends keep their existing swallow behavior.
Scope: NWS warning path fully covered (initial + all-clear ledger). SPC/MCD sends record to the ledger for observability; auto-retry for SPC is a documented follow-up (entangled with impact-page/AI side effects).

#53 — pipeline lifecycle hardening

Row locks (with_for_update(skip_locked=True)) on the clear/lifted selections so overlapping runs can't double-send all-clears.
Tracked tasks: dispatch is inline now (the GC-able fire-and-forget task is gone); remaining background tasks are tracked and drained on shutdown.
Shared SSE LISTEN: one broker connection fans out to per-client bounded queues (drop-oldest), started/stopped in the lifespan — replacing one Postgres connection per SSE client.

#56 tail

TimezoneFinder is now a module-level singleton (was constructed per location-resolution).

Testing

Full suite green on the dev server: 593 passed (incl. discord/matrix delivery-error paths); migration 0020 applies cleanly to head on real Postgres. The core acceptance is covered end-to-end: a transient send failure leaves a failed row that a later outbox run delivers successfully; a permanent failure goes dead; a sent row is never re-sent.

Notes / judgment calls

Twilio/voip.ms API-level errors default to transient (retry until aged out) — precise 401→permanent classification would need provider-specific inspection.
The SSE broker reconnects when the next client connects; a lone long-lived client after a dropped shared connection is an edge case (a reconnect watchdog is a possible follow-up).

Closes #50, #53

🤖 Generated with Claude Code

The delivery-reliability core of v1.2.0 — the milestone's keystone. ## #50 — delivery outbox (bounded retries) Previously every dispatch path committed the dedup `SentAlert` *before* sending, so a transient channel failure was logged and then permanently dropped (at-most-once delivery). Now: - A `notification_deliveries` table (migration **0020**) records per-`(alert, channel, kind)` attempts with status/attempts/last_error/next_attempt_at and a serialized payload (so a retry in a later process reconstructs the full warning text). - Dispatch commits a durable `pending` row, attempts the send inline (prompt initial delivery preserved), then a `process_delivery_outbox` scheduler job retries `failed` rows with exponential backoff (`FOR UPDATE SKIP LOCKED`) up to `DELIVERY_MAX_ATTEMPTS`/`DELIVERY_MAX_AGE_HOURS`, then `dead`. A `sent` row is never re-sent (idempotent). - **Notifiers now signal outcome** (`notifiers/errors.py`): `send_alert`/`send_alert_lifted` raise `TransientDeliveryError` (5xx/timeout/connection → retry) or `PermanentDeliveryError` (disabled/misconfig/4xx → dead) instead of swallowing HTTP errors — so the outbox observes and retries the *real* transient failures, not just unexpected exceptions. SPC/forecast sends keep their existing swallow behavior. - **Scope**: NWS warning path fully covered (initial + all-clear ledger). SPC/MCD sends record to the ledger for observability; auto-retry for SPC is a documented follow-up (entangled with impact-page/AI side effects). ## #53 — pipeline lifecycle hardening - **Row locks** (`with_for_update(skip_locked=True)`) on the clear/lifted selections so overlapping runs can't double-send all-clears. - **Tracked tasks**: dispatch is inline now (the GC-able fire-and-forget task is gone); remaining background tasks are tracked and drained on shutdown. - **Shared SSE LISTEN**: one broker connection fans out to per-client bounded queues (drop-oldest), started/stopped in the lifespan — replacing one Postgres connection per SSE client. ## #56 tail `TimezoneFinder` is now a module-level singleton (was constructed per location-resolution). ## Testing Full suite green on the dev server: **593 passed** (incl. discord/matrix delivery-error paths); migration `0020` applies cleanly to head on real Postgres. The core acceptance is covered end-to-end: a transient send failure leaves a `failed` row that a later outbox run delivers successfully; a permanent failure goes `dead`; a `sent` row is never re-sent. ## Notes / judgment calls - Twilio/voip.ms API-level errors default to transient (retry until aged out) — precise 401→permanent classification would need provider-specific inspection. - The SSE broker reconnects when the next client connects; a lone long-lived client after a dropped shared connection is an edge case (a reconnect watchdog is a possible follow-up). Closes #50, #53 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit

2026-07-18 20:58:09 +00:00

Add delivery outbox with bounded retries; harden pipeline lifecycle

CI / test (pull_request) Successful in 4m3s

Details

f9d50c9fd3

Delivery outbox (#50): a notification_deliveries table (migration 0020) records
per-(alert, channel, kind) delivery attempts. After the dedup SentAlert is
committed, dispatch enqueues a durable pending row and attempts the send inline;
a process_delivery_outbox scheduler job retries failed rows with exponential
backoff (with_for_update skip_locked) up to DELIVERY_MAX_ATTEMPTS / _MAX_AGE_HOURS,
then marks them dead. Idempotent: a sent row is never re-sent.

Notifiers now signal delivery outcome (app/services/notifiers/errors.py):
send_alert/send_alert_lifted raise TransientDeliveryError (5xx/timeout/connection
-> retry) or PermanentDeliveryError (disabled/misconfig/4xx -> dead) instead of
swallowing HTTP errors, so the outbox actually observes and retries real transient
failures. SPC/forecast sends keep the swallowing behavior (unchanged).

Pipeline lifecycle (#53): clear/lifted selections take row locks so overlapping
runs can't double-send all-clears; dispatch is now inline (the GC-able
fire-and-forget task is gone) with remaining background tasks tracked and drained
on shutdown; app/api/sse.py uses a single shared LISTEN broker fanning out to
per-client bounded queues (started/stopped in the lifespan) instead of one PG
connection per client.

TimezoneFinder is now a module-level singleton (#56 tail).

Full suite green on the dev server: 593 passed; migration 0020 applies cleanly.

Closes #50, #53

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>