rbrooks/WeatherBot

Watch

Fork 0

False all-clear when NWS replaces an alert (watch -> advisory/warning) #145

New issue

Closed

opened 2026-07-28 00:31:45 +00:00 by claude-bot · 2 comments

claude-bot commented

2026-07-28 00:31:45 +00:00

Contributor

Confirmed against production data. An alert that NWS replaces with a different product is treated as "the hazard is over" and triggers an all-clear notification, days before the hazard actually ends.

Evidence

Reported symptom: heat warnings issued for 7/24–7/28, bot announced on 7/24 that they had expired/been cancelled.

sent_alerts (dashboard rows) for the affected locations:

lifecycle	event	sent_at	expires	cleared_at
`NWS_KLSX_XH_A_0002`	Extreme Heat Watch	2026-07-24 06:13	2026-07-29 00:00	2026-07-24 17:16
`NWS_KSGF_XH_A_0001`	Extreme Heat Watch	2026-07-24 05:32	2026-07-29 00:00	2026-07-24 16:28
`NWS_KSGF_HT_Y_0004`	Heat Advisory	2026-07-24 16:28:03.81	2026-07-29 00:00	2026-07-26 05:29

VTEC on the cleared watch: /O.NEW.KLSX.XH.A.0002.260726T1600Z-260729T0000Z/ — valid through 07-29. expires was parsed and stored correctly. lifted_notified_at is set, so an all-clear really was delivered.

The watch was cleared 4.5 days before its own expiry, so this is not the time-based path. It is _clear_missing_alerts path 2 (alert_processor.py:517-533): still valid by time, but absent from active_matches, therefore "disappeared" → cleared → all-clear.

Why it left the feed: NWS replaced it. On the Rolla location the Heat Advisory HT.Y.0004 was inserted at 16:28:03.81 and the watch cleared at 16:28:03.96 — 150 ms apart, same poll cycle. A watch→advisory transition, read as a cancellation.

Not a one-off — the same pattern in June: NWS_KLSX_XH_A_0001, VTEC valid to 260703T0500Z, cleared 2026-06-27 18:16, ~5 days early.

Aggravating factor: the replacement can be invisible

The four LSX locations have min_severity = watch. The replacing Heat Advisory is advisory-severity, so it was filtered out before any record was created. Those locations got an all-clear and nothing else, in the middle of an escalating heat event. Only Rolla (min_severity = advisory) shows the replacement row.

The general defect, and why it is worse than heat

_clear_records and send_pending_lifted_notifications send an all-clear per (lifecycle, location) with no check for any other active alert at that location. Nothing distinguishes:

the hazard genuinely ended, versus
NWS replaced this product with a different one for the same hazard (watch → advisory → warning), which changes the phenomenon/significance and therefore the lifecycle id.

For heat this is confusing. For a Tornado Watch cancelled while a Tornado Warning is active at the same location, the same code path sends "all clear" during an active warning. That is a life-safety failure mode, not a cosmetic one.

Note SentAlert.upgrade_from already exists (migration 0010) but is only used for SPC outlook risk transitions — there is no equivalent for NWS watch→warning.

Suggested fix

Suppress the all-clear when another alert of equal or greater severity is active for that location. Cheapest, safest, and fixes the dangerous tornado case as well as this one.
Recognise supersession explicitly. The replacing product carries VTEC action UPG/CAN; the superseded lifecycle can be linked via upgrade_from and the message reworded ("upgraded to X") instead of an all-clear.
Reconsider severity filtering for replacements. A replacement below min_severity currently produces silence where the user has just been told the hazard ended. At minimum the all-clear should not fire when a lower-severity successor exists.

Fix 1 is the one to land first — it is small, and it is the difference between a confusing message and a dangerous one.

**Confirmed against production data.** An alert that NWS *replaces* with a different product is treated as "the hazard is over" and triggers an all-clear notification, days before the hazard actually ends. ## Evidence Reported symptom: heat warnings issued for 7/24–7/28, bot announced on 7/24 that they had expired/been cancelled. `sent_alerts` (dashboard rows) for the affected locations: | lifecycle | event | sent_at | expires | cleared_at | |---|---|---|---|---| | `NWS_KLSX_XH_A_0002` | Extreme Heat Watch | 2026-07-24 06:13 | **2026-07-29 00:00** | **2026-07-24 17:16** | | `NWS_KSGF_XH_A_0001` | Extreme Heat Watch | 2026-07-24 05:32 | **2026-07-29 00:00** | **2026-07-24 16:28** | | `NWS_KSGF_HT_Y_0004` | Heat Advisory | 2026-07-24 16:28:03.81 | 2026-07-29 00:00 | 2026-07-26 05:29 | VTEC on the cleared watch: `/O.NEW.KLSX.XH.A.0002.260726T1600Z-260729T0000Z/` — valid **through 07-29**. `expires` was parsed and stored correctly. `lifted_notified_at` is set, so an all-clear really was delivered. The watch was cleared **4.5 days before its own expiry**, so this is not the time-based path. It is `_clear_missing_alerts` path 2 (`alert_processor.py:517-533`): still valid by time, but absent from `active_matches`, therefore "disappeared" → cleared → all-clear. **Why it left the feed:** NWS replaced it. On the Rolla location the Heat Advisory `HT.Y.0004` was inserted at `16:28:03.81` and the watch cleared at `16:28:03.96` — **150 ms apart, same poll cycle**. A watch→advisory transition, read as a cancellation. Not a one-off — the same pattern in June: `NWS_KLSX_XH_A_0001`, VTEC valid to `260703T0500Z`, cleared `2026-06-27 18:16`, ~5 days early. ## Aggravating factor: the replacement can be invisible The four LSX locations have `min_severity = watch`. The replacing Heat Advisory is advisory-severity, so it was filtered out before any record was created. Those locations got an **all-clear and nothing else**, in the middle of an escalating heat event. Only Rolla (`min_severity = advisory`) shows the replacement row. ## The general defect, and why it is worse than heat `_clear_records` and `send_pending_lifted_notifications` send an all-clear per `(lifecycle, location)` with **no check for any other active alert at that location**. Nothing distinguishes: - the hazard genuinely ended, versus - NWS replaced this product with a different one for the same hazard (watch → advisory → warning), which changes the phenomenon/significance and therefore the lifecycle id. For heat this is confusing. For a Tornado Watch cancelled while a Tornado Warning is active at the same location, the same code path sends "all clear" during an active warning. That is a life-safety failure mode, not a cosmetic one. Note `SentAlert.upgrade_from` already exists (migration 0010) but is only used for SPC outlook risk transitions — there is no equivalent for NWS watch→warning. ## Suggested fix 1. **Suppress the all-clear when another alert of equal or greater severity is active for that location.** Cheapest, safest, and fixes the dangerous tornado case as well as this one. 2. **Recognise supersession explicitly.** The replacing product carries VTEC action `UPG`/`CAN`; the superseded lifecycle can be linked via `upgrade_from` and the message reworded ("upgraded to X") instead of an all-clear. 3. **Reconsider severity filtering for replacements.** A replacement below `min_severity` currently produces silence where the user has just been told the hazard ended. At minimum the all-clear should not fire when a lower-severity successor exists. Fix 1 is the one to land first — it is small, and it is the difference between a confusing message and a dangerous one.

claude-bot added the

bug

area:notifications

labels

2026-07-28 00:31:45 +00:00

claude-bot referenced this issue

2026-07-28 00:37:05 +00:00

Alert fetch silently truncates at the pagination cap, causing mass false all-clears #146

claude-bot referenced this issue

2026-07-28 00:37:05 +00:00

expire_alerts_job clears on timestamp alone, without any feed cross-check #147

claude-bot commented

2026-07-28 00:38:19 +00:00

Author

Contributor

Picking this up on branch fix/false-all-clear-supersession.

Design note after re-reading the pipeline: the obvious fix does not actually fix the reported case. Checking "is another alert still active at this location" only works when the replacement was recorded — and for the four LSX locations it was not. _process_single_alert filters in this order (alert_processor.py:319-330):

_location_matches_alert — geographic
_meets_threshold — severity vs Location.min_severity
excluded event types

active_matches only collects entries surviving all three. The replacing Heat Advisory is advisory-severity, those locations are min_severity = watch, so it was dropped at step 2 and no sent_alerts row ever existed. From the DB alone the hazard looks gone.

So the guard has to consult the feed at the point of geographic match, before severity filtering.

Approach

Two complementary guards:

A. Hazard-family presence (fixes this bug). Track (location_id, hazard_family) for every alert that matches a location geographically, regardless of whether it clears the severity threshold. When clearing a record whose family is still present at that location, mark it cleared but suppress the all-clear. Family comes from the VTEC phenomenon already embedded in SentAlert.nws_alert_id (NWS_{office}_{phen}_{sig}_{etn} → XH), mapped through a small table — heat is EH/HT/XH, so a watch→advisory transition stays inside one family even though the phenomenon code changes.

B. Equal-or-greater severity still active (fixes the dangerous case). Before sending any all-clear, suppress if another uncleared alert of equal or greater severity is active at that location. Placed in send_pending_lifted_notifications so it covers every clearing path — the poll sweep, _process_cancel, and expire_alerts_job (#147) alike — rather than only the "disappeared" path. This is what stops "all clear" going out while a Tornado Warning is live.

Suppression means setting lifted_notified_at at clear time, not leaving it NULL — otherwise the row is re-selected every cycle forever, which is exactly the loop #148 is stuck in.

Picking this up on branch `fix/false-all-clear-supersession`. Design note after re-reading the pipeline: **the obvious fix does not actually fix the reported case.** Checking "is another alert still active at this location" only works when the replacement was recorded — and for the four LSX locations it was not. `_process_single_alert` filters in this order (`alert_processor.py:319-330`): 1. `_location_matches_alert` — geographic 2. `_meets_threshold` — severity vs `Location.min_severity` 3. excluded event types `active_matches` only collects entries surviving all three. The replacing Heat Advisory is advisory-severity, those locations are `min_severity = watch`, so it was dropped at step 2 and no `sent_alerts` row ever existed. From the DB alone the hazard looks gone. So the guard has to consult the feed at the point of **geographic** match, before severity filtering. ## Approach Two complementary guards: **A. Hazard-family presence (fixes this bug).** Track `(location_id, hazard_family)` for every alert that matches a location geographically, *regardless* of whether it clears the severity threshold. When clearing a record whose family is still present at that location, mark it cleared but suppress the all-clear. Family comes from the VTEC phenomenon already embedded in `SentAlert.nws_alert_id` (`NWS_{office}_{phen}_{sig}_{etn}` → `XH`), mapped through a small table — heat is `EH`/`HT`/`XH`, so a watch→advisory transition stays inside one family even though the phenomenon code changes. **B. Equal-or-greater severity still active (fixes the dangerous case).** Before sending any all-clear, suppress if another uncleared alert of equal or greater severity is active at that location. Placed in `send_pending_lifted_notifications` so it covers **every** clearing path — the poll sweep, `_process_cancel`, and `expire_alerts_job` (#147) alike — rather than only the "disappeared" path. This is what stops "all clear" going out while a Tornado Warning is live. Suppression means setting `lifted_notified_at` at clear time, not leaving it NULL — otherwise the row is re-selected every cycle forever, which is exactly the loop #148 is stuck in.

rbrooks referenced this issue from a commit

2026-07-28 00:48:06 +00:00

Suppress false all-clears when a hazard is superseded (#145)

claude-bot referenced this issue from a pull request that will close it,

2026-07-28 00:48:28 +00:00

Suppress false all-clears when a hazard is superseded (#145) #149

claude-bot closed this issue

2026-07-28 00:58:49 +00:00

claude-bot referenced this issue from a commit

2026-07-28 00:58:51 +00:00

Merge pull request 'Suppress false all-clears when a hazard is superseded (#145)' (#149) from fix/false-all-clear-supersession into main

claude-bot commented

2026-07-28 00:59:08 +00:00

Author

Contributor

Fixed in #149 (merged to main). CI green in 4m30s.

Shipped: two guards.

A — hazard-family presence. (location_id, hazard_family) is now tracked for every alert covering a location geographically, before the severity and exclusion filters. A record whose family is still present was superseded rather than lifted: still cleared, all-clear suppressed. Family is derived from the VTEC phenomenon already embedded in nws_alert_id, so a watch→advisory transition stays in one family despite the phenomenon code changing (XH → HT) — the exact reason phenomenon-matching alone would have missed this.

B — equal-or-greater severity still active. No all-clear goes out while something at least as severe is live at that location. Placed in send_pending_lifted_notifications so it covers every clearing path, including expire_alerts_job (#147), not just the poll sweep that caused this report.

Suppression stamps lifted_notified_at rather than leaving it NULL, so suppressed rows are not re-selected forever — the failure mode #148 is currently in.

Verified locally in Docker before push: ruff clean, compileall OK, 797 SQLite-tier tests, migrations clean on Postgres 16, 4 Postgres-tier tests. Two of the new tests are deliberately controls — hazard genuinely gone, and only a lower-severity alert lingering — so the guards cannot silently over-suppress real all-clears.

Worth knowing:

The family table is deliberately incomplete-safe. An unmapped VTEC phenomenon returns None and falls through to guard B, degrading to previous behaviour rather than swallowing a real all-clear. If a hazard family shows up that should be grouped and is not, it is a one-line addition to _VTEC_HAZARD_FAMILIES.
This does not retroactively fix the stuck rows. The four LSX heat watches already have cleared_at and lifted_notified_at set from 7/24 — nothing re-sends or un-sends. The fix applies from the next supersession onward.
Guard B partially mitigates #147 but is not a substitute for it. expire_alerts_job still clears on stale timestamps during an NWS outage; guard B only stops the notification when something more severe happens to be recorded.

Fixed in #149 (merged to `main`). CI green in 4m30s. **Shipped:** two guards. **A — hazard-family presence.** `(location_id, hazard_family)` is now tracked for every alert covering a location *geographically*, before the severity and exclusion filters. A record whose family is still present was superseded rather than lifted: still cleared, all-clear suppressed. Family is derived from the VTEC phenomenon already embedded in `nws_alert_id`, so a watch→advisory transition stays in one family despite the phenomenon code changing (`XH` → `HT`) — the exact reason phenomenon-matching alone would have missed this. **B — equal-or-greater severity still active.** No all-clear goes out while something at least as severe is live at that location. Placed in `send_pending_lifted_notifications` so it covers every clearing path, including `expire_alerts_job` (#147), not just the poll sweep that caused this report. Suppression stamps `lifted_notified_at` rather than leaving it NULL, so suppressed rows are not re-selected forever — the failure mode #148 is currently in. **Verified locally in Docker before push:** ruff clean, `compileall` OK, 797 SQLite-tier tests, migrations clean on Postgres 16, 4 Postgres-tier tests. Two of the new tests are deliberately **controls** — hazard genuinely gone, and only a lower-severity alert lingering — so the guards cannot silently over-suppress real all-clears. **Worth knowing:** 1. **The family table is deliberately incomplete-safe.** An unmapped VTEC phenomenon returns `None` and falls through to guard B, degrading to previous behaviour rather than swallowing a real all-clear. If a hazard family shows up that should be grouped and is not, it is a one-line addition to `_VTEC_HAZARD_FAMILIES`. 2. **This does not retroactively fix the stuck rows.** The four LSX heat watches already have `cleared_at` and `lifted_notified_at` set from 7/24 — nothing re-sends or un-sends. The fix applies from the next supersession onward. 3. **Guard B partially mitigates #147 but is not a substitute for it.** `expire_alerts_job` still clears on stale timestamps during an NWS outage; guard B only stops the *notification* when something more severe happens to be recorded.

claude-bot referenced this issue

2026-07-28 01:37:35 +00:00

Don't clear alerts on incomplete information (#146, #147) #150

claude-bot referenced this issue

2026-07-28 03:14:35 +00:00

Webex all-clear 400s in an endless retry loop; response body not logged #148

claude-bot referenced this issue

2026-07-28 03:40:47 +00:00

CD deploys only to the dev host; the live instance is 6 days behind and manually managed #154