rbrooks/WeatherBot

Watch

Fork 0

Alert clearing treats NWS fetch failure as "no active alerts" — mass false all-clears #40

New issue

Closed

opened 2026-07-18 05:31:08 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-18 05:31:08 +00:00

Contributor

fetch_alerts_for_states returns [] on any httpx.HTTPError
(app/services/nws.py:163-174). poll_alerts_job passes that straight to
process_alerts (app/services/scheduler.py:48-50), whose
_clear_missing_alerts (app/services/alert_processor.py:307-342) then clears
every uncleared SentAlert — dispatching all-clear notifications to all
channels — because nothing is in active_matches. The next good poll re-sends
everything as new. One transient NWS error = false all-clears for every active
alert, including live warnings, followed by duplicate re-alerts.

Fix (recommended):

Distinguish fetch failure from empty feed: raise or return None from
fetch_alerts_for_states on HTTP error; skip process_alerts for that cycle.
Add bounded retry with jittered backoff (2-3 attempts) around the GET —
no retry exists on any NWS call today (all of nws.py is single-attempt).
Guard _clear_missing_alerts's disappeared branch when the loaded
locations list is empty (same mass-clear reachable via that path,
alert_processor.py:189-199).
Edge case while in there: alerts with neither ends nor expires get
expires=NULL and are invisible to the dashboard (expires > now filter,
app/main.py:176) and to all clearing paths (expires != None filters).
Store a synthetic expiry (e.g. onset/sent + a default window) instead.

Acceptance criteria:

A poll cycle whose NWS fetch fails performs no clearing and no dispatch
NWS alert GETs retry with backoff before the cycle is abandoned
Empty resolved-locations list cannot trigger the disappeared clear path
Alerts without ends/expires receive a synthetic expiry and clear normally
Regression test: fetch failure between two good polls sends zero all-clears

Filed from the 2026-07-17 codebase audit (docs/.internal/report-2026-07-17.md), finding F-01.

`fetch_alerts_for_states` returns `[]` on any `httpx.HTTPError` (app/services/nws.py:163-174). `poll_alerts_job` passes that straight to `process_alerts` (app/services/scheduler.py:48-50), whose `_clear_missing_alerts` (app/services/alert_processor.py:307-342) then clears every uncleared `SentAlert` — dispatching all-clear notifications to all channels — because nothing is in `active_matches`. The next good poll re-sends everything as new. One transient NWS error = false all-clears for every active alert, including live warnings, followed by duplicate re-alerts. Fix (recommended): - Distinguish fetch failure from empty feed: raise or return `None` from `fetch_alerts_for_states` on HTTP error; skip `process_alerts` for that cycle. - Add bounded retry with jittered backoff (2-3 attempts) around the GET — no retry exists on any NWS call today (all of nws.py is single-attempt). - Guard `_clear_missing_alerts`'s `disappeared` branch when the loaded `locations` list is empty (same mass-clear reachable via that path, alert_processor.py:189-199). - Edge case while in there: alerts with neither `ends` nor `expires` get `expires=NULL` and are invisible to the dashboard (`expires > now` filter, app/main.py:176) and to all clearing paths (`expires != None` filters). Store a synthetic expiry (e.g. onset/sent + a default window) instead. **Acceptance criteria:** - [ ] A poll cycle whose NWS fetch fails performs no clearing and no dispatch - [ ] NWS alert GETs retry with backoff before the cycle is abandoned - [ ] Empty resolved-locations list cannot trigger the `disappeared` clear path - [ ] Alerts without `ends`/`expires` receive a synthetic expiry and clear normally - [ ] Regression test: fetch failure between two good polls sends zero all-clears --- _Filed from the 2026-07-17 codebase audit (`docs/.internal/report-2026-07-17.md`), finding F-01._

claude-bot added this to the v1.0.2 - Alert correctness & integrity milestone

2026-07-18 05:31:08 +00:00

claude-bot added the

bug

area:notifications

labels

2026-07-18 05:31:08 +00:00

rbrooks referenced this issue from a commit

2026-07-18 06:51:12 +00:00

Fix NWS fetch failures causing false all-clears and dropped outbreak alerts

claude-bot referenced this issue from a pull request that will close it,

2026-07-18 06:51:28 +00:00

v1.0.2 — Alert correctness & integrity #92

claude-bot closed this issue

2026-07-18 06:56:43 +00:00

claude-bot commented

2026-07-18 06:57:14 +00:00

Author

Contributor

Fixed in #92 (merged to main, deployed to dev via CD). NWS fetches now retry with jittered backoff and return None on failure; poll_alerts_job skips the cycle entirely on None (no clearing, no dispatch), so a transient outage no longer fires false all-clears. Mass-clear is also gated on a non-empty locations list, and alerts lacking both ends and expires receive a synthetic 6h expiry. Regression tests cover fetch-failure→no-clear and the synthetic expiry.

Fixed in #92 (merged to `main`, deployed to dev via CD). NWS fetches now retry with jittered backoff and return `None` on failure; `poll_alerts_job` skips the cycle entirely on `None` (no clearing, no dispatch), so a transient outage no longer fires false all-clears. Mass-clear is also gated on a non-empty locations list, and alerts lacking both `ends` and `expires` receive a synthetic 6h expiry. Regression tests cover fetch-failure→no-clear and the synthetic expiry.

claude-bot referenced this issue

2026-07-28 01:37:35 +00:00

Don't clear alerts on incomplete information (#146, #147) #150

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/WeatherBot#40

No description provided.

Rows
Columns