Alert fetch silently truncates at the pagination cap, causing mass false all-clears #146

New issue

Closed

opened 2026-07-28 00:37:04 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-28 00:37:04 +00:00

Contributor

fetch_alerts_for_states follows pagination.next but stops at a hard cap and still returns a successful result (app/services/nws.py:220):

pages = 1
next_url = (data.get("pagination") or {}).get("next")
while next_url and pages < 10:

When the cap is hit, features is silently truncated and the function returns a normal list[NWSAlert] — indistinguishable from a complete fetch. Every alert on page 11+ is therefore missing from active_matches, and _clear_missing_alerts clears it as "disappeared from the feed", firing all-clears for alerts that are still active.

The comment three lines above the loop states this is exactly what pagination was added to prevent:

Follow pagination.next (an absolute URL) so large multi-state outbreaks aren't truncated to page 1 (and then mass-cleared as "disappeared"). Bounded to a sane page cap.

The cap reintroduces the same failure at page 10 instead of page 1. It is strictly a question of feed size — a big enough multi-state outbreak triggers mass false all-clears, and the larger the outbreak the more likely it is, which is the worst possible correlation for an alerting system.

Fix

Hitting the cap must not look like success. Either:

treat a hit cap as a failed fetch — return None, so the caller skips the cycle rather than mass-clearing (consistent with the existing #40 skip-on-outage contract), or
return the partial result but signal incompleteness so _clear_missing_alerts skips the "disappeared" sweep for that cycle.

Either way, log loudly at WARNING when the cap is reached — today it is invisible. Raising the cap alone is not a fix; it just moves the cliff.

Same family as #145 (false all-clears). #145 is a supersession-detection gap; this is a data-completeness gap. Both end in an unwarranted all-clear.

`fetch_alerts_for_states` follows `pagination.next` but stops at a hard cap and still returns a **successful** result (`app/services/nws.py:220`): ```python pages = 1 next_url = (data.get("pagination") or {}).get("next") while next_url and pages < 10: ``` When the cap is hit, `features` is silently truncated and the function returns a normal `list[NWSAlert]` — indistinguishable from a complete fetch. Every alert on page 11+ is therefore missing from `active_matches`, and `_clear_missing_alerts` clears it as "disappeared from the feed", firing all-clears for alerts that are still active. The comment three lines above the loop states this is exactly what pagination was added to prevent: > Follow pagination.next (an absolute URL) so large multi-state outbreaks aren't truncated to page 1 (and then mass-cleared as "disappeared"). Bounded to a sane page cap. The cap reintroduces the same failure at page 10 instead of page 1. It is strictly a question of feed size — a big enough multi-state outbreak triggers mass false all-clears, and the larger the outbreak the more likely it is, which is the worst possible correlation for an alerting system. ## Fix Hitting the cap must not look like success. Either: - treat a hit cap as a failed fetch — return `None`, so the caller skips the cycle rather than mass-clearing (consistent with the existing `#40` skip-on-outage contract), **or** - return the partial result but signal incompleteness so `_clear_missing_alerts` skips the "disappeared" sweep for that cycle. Either way, log loudly at WARNING when the cap is reached — today it is invisible. Raising the cap alone is not a fix; it just moves the cliff. ## Related Same family as #145 (false all-clears). #145 is a supersession-detection gap; this is a data-completeness gap. Both end in an unwarranted all-clear.

claude-bot added the

bug

area:notifications

labels

2026-07-28 00:37:05 +00:00

claude-bot referenced this issue

2026-07-28 00:48:28 +00:00

Suppress false all-clears when a hazard is superseded (#145) #149

rbrooks referenced this issue from a commit

2026-07-28 01:37:13 +00:00

Don't clear alerts on incomplete information (#146, #147)

claude-bot referenced this issue from a pull request that will close it,

2026-07-28 01:37:34 +00:00

Don't clear alerts on incomplete information (#146, #147) #150

claude-bot closed this issue

2026-07-28 01:47:46 +00:00

claude-bot referenced this issue from a commit

2026-07-28 01:47:46 +00:00

Merge pull request 'Don't clear alerts on incomplete information (#146, #147)' (#150) from fix/incomplete-feed-guards into main

claude-bot commented

2026-07-28 02:08:26 +00:00

Author

Contributor

Fixed in #150 (merged). CI green.

Fetches now return an AlertFeed — a list subclass, so every existing caller and test is untouched — carrying a complete flag set False when pages remained at the cap. process_alerts still dispatches what it received (those alerts are real, and truncation happens during outbreaks when they matter most) but skips the clearing pass entirely, because absence from a partial feed proves nothing.

The cap moved 10 → 50, far above any plausible legitimate feed for the states one install monitors, so it reads as the runaway guard it is rather than an operating limit. Hitting it now logs at ERROR instead of passing silently.

Verified locally in Docker: ruff, 805 SQLite-tier tests, migrations on Postgres 16, 4 Postgres-tier tests. Includes a control test (pagination ending naturally → complete is True, feed still clears) so the guard cannot silently stop legitimate all-clears.

Fixed in #150 (merged). CI green. Fetches now return an `AlertFeed` — a `list` subclass, so every existing caller and test is untouched — carrying a `complete` flag set False when pages remained at the cap. `process_alerts` still **dispatches** what it received (those alerts are real, and truncation happens during outbreaks when they matter most) but skips the clearing pass entirely, because absence from a partial feed proves nothing. The cap moved 10 → 50, far above any plausible legitimate feed for the states one install monitors, so it reads as the runaway guard it is rather than an operating limit. Hitting it now logs at ERROR instead of passing silently. Verified locally in Docker: ruff, 805 SQLite-tier tests, migrations on Postgres 16, 4 Postgres-tier tests. Includes a control test (pagination ending naturally → `complete is True`, feed still clears) so the guard cannot silently stop legitimate all-clears.

claude-bot referenced this issue

2026-07-28 03:14:35 +00:00

Webex all-clear 400s in an endless retry loop; response body not logged #148

claude-bot referenced this issue

2026-07-28 03:40:47 +00:00

CD deploys only to the dev host; the live instance is 6 days behind and manually managed #154