rbrooks/WeatherBot

Watch

Fork 0

Radar history: capture per location from the IEM archive, with guaranteed lead-in and no dropped frames #210

New issue

Open

opened 2026-10-13 21:32:52 +00:00 by claude-bot · 3 comments

claude-bot commented

2026-10-13 21:32:52 +00:00

Collaborator

Decided with @rbrooks on 2026-10-13. Ships in v2.0.0 (@rbrooks's decision). Supersedes the per-alert capture from #130 and the cap discussion on #208. Parent: #22.

Why

Radar frames are captured per alert, live, with a 24-frame cap. That has three weaknesses:

Frames can be dropped for good. NOAA's MapServer serves "now" only, so anything not fetched at the time (restart, outage, failed fetch, cap reached) is unrecoverable.
Lead-in depends on luck. A warning's loop only has "before" frames if a watch happened to be active first.
Overlapping alerts duplicate work. A watch and a warning on one location each fetch and store the same picture every cycle.

The enabler

Iowa State's archive (IEM) serves the national NEXRAD base-reflectivity mosaic at 5-minute steps for arbitrary past times, with the same bbox request shape used for NOAA:

https://mesonet.agron.iastate.edu/cgi-bin/wms/nexrad/n0q-t.cgi?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap
  &LAYERS=nexrad-n0q-wmst&STYLES=&SRS=EPSG:4326&BBOX=<lon-2.5>,<lat-2.5>,<lon+2.5>,<lat+2.5>
  &WIDTH=600&HEIGHT=600&FORMAT=image/png&TRANSPARENT=true&TIME=<ISO8601 UTC>

Measured 2026-10-13 for the Wichita bbox:

Past times work back at least a year (2025-10-13 returned a real frame). 55–190 KB per frame, under half a second each.
Lag is under three minutes: the 21:30Z slot was served at 21:32:15Z.
A slot that does not exist yet returns HTTP 200 with a fully transparent 1,476-byte PNG, not an error. A missing frame is indistinguishable from clear sky by status code alone. The design has to handle this.

Design

Frames belong to a location and a radar time. New table keyed (location_id, valid_at) on a 5-minute grid. Overlapping alerts share one image.

Capture is "fill the window", idempotent. For each location, the wanted slots are the union over its qualifying alerts of [start − lead-in, end + tail]. The job fetches whichever wanted slots are missing. No per-alert cap; the window bounds the work.

Lead-in is retroactive. 60 minutes before first fire, fetched from the archive. Tail is 30 minutes after the alert ends.

Tiers (by NWS event name):

Tier	Events	Capture
Full	Tornado, Severe Thunderstorm, Flash Flood, Snow Squall, Extreme Wind, Dust Storm and Special Marine Warnings; Tornado and Severe Thunderstorm Watches	Every 5 min for the whole alert, +60 min before, +30 min after
Reduced	Winter Storm, Blizzard, Ice Storm, Lake Effect Snow, Hurricane, Tropical Storm, Storm Surge, Flood and High Wind Warnings; Flood and Winter Storm Watches	Every 15 min, first 24 h only
None	Everything else (heat, freeze, fire weather, fog, air quality, advisories, statements)	Nothing

Special Marine, Lake Effect Snow and Storm Surge Warnings were added as siblings of the approved lists.

Nothing dropped:

The job reconsiders every alert active in the last 48 hours on each run, so downtime and failed fetches are repaired on the next run.
Only slots at least 10 minutes old are requested. A fully transparent frame for a slot younger than 30 minutes is treated as "not available yet" and retried; older than that it is accepted as clear sky.
Each event exposes expected versus present frame counts. The Explorer caption shows both, and an admin alert fires when a Full-tier alert has been over for 30 minutes with under 90% of its frames.

Scope

Migration 0038: location_radar_frames
IEM archive fetcher (sequential, identified User-Agent, fail-soft)
Slot planner + fill job, replacing the per-alert periodic job
Explorer loop and still strip read the new frames, falling back to legacy alert_radar_frames for events that have none
Coverage accounting, Explorer caption, admin alert
Retention for the new table and its files on the unified horizon
Settings for lead-in, tail, steps, caps, lag, per-run budget, and a kill switch

Unchanged

The alert-time snapshot (#85) used by notifications and public pages stays on NOAA.
Existing alert_radar_frames rows and files stay readable and keep their retention. Regenerating old events from the archive is a possible follow-up, not part of this.

Known trade-offs

IEM's colour table differs from NOAA's, so new loops will not match old ones.
One more dependency on a university-run service (already used for SPC outlook maps).
Frames are larger than NOAA's (roughly 55–190 KB against 20–25 KB).
A new migration means the 1.8.1 → 2.0.0 production rehearsal has to be repeated, now 0033 → 0038.

Decided with @rbrooks on 2026-10-13. Ships in **v2.0.0** (@rbrooks's decision). Supersedes the per-alert capture from #130 and the cap discussion on #208. Parent: #22. ## Why Radar frames are captured per alert, live, with a 24-frame cap. That has three weaknesses: - **Frames can be dropped for good.** NOAA's MapServer serves "now" only, so anything not fetched at the time (restart, outage, failed fetch, cap reached) is unrecoverable. - **Lead-in depends on luck.** A warning's loop only has "before" frames if a watch happened to be active first. - **Overlapping alerts duplicate work.** A watch and a warning on one location each fetch and store the same picture every cycle. ## The enabler Iowa State's archive (IEM) serves the national NEXRAD base-reflectivity mosaic at 5-minute steps for arbitrary past times, with the same bbox request shape used for NOAA: ``` https://mesonet.agron.iastate.edu/cgi-bin/wms/nexrad/n0q-t.cgi?SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap &LAYERS=nexrad-n0q-wmst&STYLES=&SRS=EPSG:4326&BBOX=<lon-2.5>,<lat-2.5>,<lon+2.5>,<lat+2.5> &WIDTH=600&HEIGHT=600&FORMAT=image/png&TRANSPARENT=true&TIME=<ISO8601 UTC> ``` Measured 2026-10-13 for the Wichita bbox: - Past times work back at least a year (2025-10-13 returned a real frame). 55–190 KB per frame, under half a second each. - **Lag is under three minutes**: the 21:30Z slot was served at 21:32:15Z. - **A slot that does not exist yet returns HTTP 200 with a fully transparent 1,476-byte PNG**, not an error. A missing frame is indistinguishable from clear sky by status code alone. The design has to handle this. ## Design **Frames belong to a location and a radar time.** New table keyed `(location_id, valid_at)` on a 5-minute grid. Overlapping alerts share one image. **Capture is "fill the window", idempotent.** For each location, the wanted slots are the union over its qualifying alerts of `[start − lead-in, end + tail]`. The job fetches whichever wanted slots are missing. No per-alert cap; the window bounds the work. **Lead-in is retroactive.** 60 minutes before first fire, fetched from the archive. Tail is 30 minutes after the alert ends. **Tiers** (by NWS event name): | Tier | Events | Capture | |---|---|---| | Full | Tornado, Severe Thunderstorm, Flash Flood, Snow Squall, Extreme Wind, Dust Storm and Special Marine Warnings; Tornado and Severe Thunderstorm Watches | Every 5 min for the whole alert, +60 min before, +30 min after | | Reduced | Winter Storm, Blizzard, Ice Storm, Lake Effect Snow, Hurricane, Tropical Storm, Storm Surge, Flood and High Wind Warnings; Flood and Winter Storm Watches | Every 15 min, first 24 h only | | None | Everything else (heat, freeze, fire weather, fog, air quality, advisories, statements) | Nothing | Special Marine, Lake Effect Snow and Storm Surge Warnings were added as siblings of the approved lists. **Nothing dropped:** - The job reconsiders every alert active in the last 48 hours on each run, so downtime and failed fetches are repaired on the next run. - Only slots at least 10 minutes old are requested. A fully transparent frame for a slot younger than 30 minutes is treated as "not available yet" and retried; older than that it is accepted as clear sky. - Each event exposes expected versus present frame counts. The Explorer caption shows both, and an admin alert fires when a Full-tier alert has been over for 30 minutes with under 90% of its frames. ## Scope - [x] Migration `0038`: `location_radar_frames` - [x] IEM archive fetcher (sequential, identified User-Agent, fail-soft) - [x] Slot planner + fill job, replacing the per-alert periodic job - [x] Explorer loop and still strip read the new frames, falling back to legacy `alert_radar_frames` for events that have none - [x] Coverage accounting, Explorer caption, admin alert - [x] Retention for the new table and its files on the unified horizon - [x] Settings for lead-in, tail, steps, caps, lag, per-run budget, and a kill switch ## Unchanged - The alert-time snapshot (#85) used by notifications and public pages stays on NOAA. - Existing `alert_radar_frames` rows and files stay readable and keep their retention. Regenerating old events from the archive is a possible follow-up, not part of this. ## Known trade-offs - IEM's colour table differs from NOAA's, so new loops will not match old ones. - One more dependency on a university-run service (already used for SPC outlook maps). - Frames are larger than NOAA's (roughly 55–190 KB against 20–25 KB). - A new migration means the 1.8.1 → 2.0.0 production rehearsal has to be repeated, now `0033 → 0038`.

rbrooks referenced this issue from a commit

2026-10-13 21:47:07 +00:00

Keep radar history per location from the IEM archive, with lead-in and self-healing gaps (#210)

claude-bot referenced this issue

2026-10-13 21:47:25 +00:00

Keep radar history per location from the IEM archive, with lead-in and self-healing gaps (#210) #211

claude-bot referenced this issue from a commit

2026-10-13 21:55:08 +00:00

Merge pull request 'Keep radar history per location from the IEM archive, with lead-in and self-healing gaps (#210)' (#211) from feat/location-radar-archive into main

claude-bot commented

2026-10-13 21:55:09 +00:00

Author

Collaborator

Merged via #211 (851b4fe). CI run 261 green in 6m22s: ruff, alembic upgrade head through 0038 on a fresh Postgres, both test tiers, pip-audit, and the image build.

Not released or deployed yet; main is ahead of v2.0.0-rc6 by this and the #208 cadence fix (which this supersedes: the code it fixed is deleted).

Changes made in review beyond the original spec:

Full-tier windows capped at 24 hours (RADAR_ARCHIVE_FULL_MAX_HOURS), and alerts whose capped window predates the lookback are skipped, so a dashboard row that never clears cannot drive an endless backfill.
The incomplete-capture admin alert is skipped while a backlog remains, so an upgrade does not page for every recently ended alert.
An event's expected frame count comes from its own alerts' schedule. An event whose alerts schedule nothing gets no archive loop.

Leaving open until verified on dev

The fill job runs and backfills the last 48 hours (first real run of the job anywhere)
An Explorer loop assembled from archive frames, with lead-in visible
IEM's colour table over the Esri basemap looks acceptable
Caption reads "N of M frames" sensibly
No incomplete-capture alert spam on the first runs

Follow-ups not in this change

Repeat the production migration rehearsal, now 0033 → 0038.
A real hole in IEM's own archive is stored as a blank frame once it is over 30 minutes old. No detection for that yet.
Regenerating pre-#210 events from the archive.
No test asserts the Explorer caption text on the archive path directly.

Merged via #211 (`851b4fe`). CI run 261 green in 6m22s: ruff, `alembic upgrade head` through `0038` on a fresh Postgres, both test tiers, `pip-audit`, and the image build. Not released or deployed yet; `main` is ahead of `v2.0.0-rc6` by this and the #208 cadence fix (which this supersedes: the code it fixed is deleted). Changes made in review beyond the original spec: - Full-tier windows capped at 24 hours (`RADAR_ARCHIVE_FULL_MAX_HOURS`), and alerts whose capped window predates the lookback are skipped, so a dashboard row that never clears cannot drive an endless backfill. - The incomplete-capture admin alert is skipped while a backlog remains, so an upgrade does not page for every recently ended alert. - An event's expected frame count comes from its own alerts' schedule. An event whose alerts schedule nothing gets no archive loop. ## Leaving open until verified on dev - [ ] The fill job runs and backfills the last 48 hours (first real run of the job anywhere) - [ ] An Explorer loop assembled from archive frames, with lead-in visible - [ ] IEM's colour table over the Esri basemap looks acceptable - [ ] Caption reads "N of M frames" sensibly - [ ] No incomplete-capture alert spam on the first runs ## Follow-ups not in this change - Repeat the production migration rehearsal, now `0033 → 0038`. - A real hole in IEM's own archive is stored as a blank frame once it is over 30 minutes old. No detection for that yet. - Regenerating pre-#210 events from the archive. - No test asserts the Explorer caption text on the archive path directly.

claude-bot referenced this issue

2026-10-14 00:11:53 +00:00

Add v2.0.0-rc7 changelog entry #212

claude-bot commented

2026-10-14 00:48:39 +00:00

Author

Collaborator

Verified on dev (`2.0.0-rc7`), 2026-10-14 00:36–00:50 UTC

Deploy: tag v2.0.0-rc7 on fc25cc6, image digest sha256:81645a9a…4fae, pin bump Rhoving/iac-repo#640 (merged as 1f430dc), play ok=23 changed=3 failed=0. A pre-upgrade dump was taken first: /home/claude/weatherbot-dev-pre-2.0.0rc7-20261014.sql.gz on docker-test (7.6 MB, database at 0037).

Migration 0038 applied on a real database: Running upgrade 0037 -> 0038. New table present; the 492 legacy alert_radar_frames rows untouched.
The fill job runs and backfills. First run: wanted=155 missing=155 fetched=120 blank_deferred=0 failed=0 (stopped at the 120-per-run budget, 58 seconds). Second run: wanted=157 missing=37 fetched=37. Backlog cleared in two runs, zero failures.
Frames match the plan. Salina 55 (Severe Thunderstorm Watch issued 21:03Z; frames from 20:05Z, so the 60-minute lead-in is there). Wichita 70 (Flood Watch from 16:30Z at 15-minute steps, merged with the severe watch's 5-minute steps: one row per slot). Enid 32, all on the quarter hour, as a Flood Watch alone should give.
Loops assemble from archive frames. Built through the real get_event_radar_loop path inside the container: Salina 55 frames, 3.58 MB, 9.8 s; Wichita 70 frames, 4.30 MB, 6.8 s. Coverage reads 55 of 55 and 70 of 70.
IEM over the Esri basemap looks right. Frames are aligned with state lines and city labels, reflectivity reads clearly on the dark canvas. Checked first and last frames of the Salina loop: 20:05Z shows cells west of Salina; 00:35Z shows a line through Wichita and Emporia.
No alert spam. No incomplete-capture admin alert fired during the backlog.

This is also the first real convective radar loop seen in the Explorer, open on #22 since rc3.

Still not verified

The caption and loop as rendered in a browser on the Explorer event page (the checks above called the service functions directly).
A Full-tier warning: nothing reached the soak locations overnight, so these loops come from watches.
Blank-frame deferral on real data (blank_deferred stayed 0).
The incomplete-capture alert firing for a genuine shortfall.

Observations

157 frames took 14 MB on disk, about 80–90 KB each during active weather.
Both loops sit inside events titled "SPC Outlook Day 1", because the correlation layer attaches a watch as context to the SPC outlook's event. The loop is right; the title is not what someone looking for "the storm" would search for. Pre-existing behaviour, worth a look separately.
Unrelated, also seen in the log: Failed to parse hazard parameters for alert … once a minute for one Oklahoma alert. Fail-soft parsing in nws.py that swallows the exception without logging it.

## Verified on dev (`2.0.0-rc7`), 2026-10-14 00:36–00:50 UTC Deploy: tag `v2.0.0-rc7` on `fc25cc6`, image digest `sha256:81645a9a…4fae`, pin bump Rhoving/iac-repo#640 (merged as `1f430dc`), play `ok=23 changed=3 failed=0`. A pre-upgrade dump was taken first: `/home/claude/weatherbot-dev-pre-2.0.0rc7-20261014.sql.gz` on `docker-test` (7.6 MB, database at `0037`). - [x] **Migration `0038` applied on a real database**: `Running upgrade 0037 -> 0038`. New table present; the 492 legacy `alert_radar_frames` rows untouched. - [x] **The fill job runs and backfills.** First run: `wanted=155 missing=155 fetched=120 blank_deferred=0 failed=0` (stopped at the 120-per-run budget, 58 seconds). Second run: `wanted=157 missing=37 fetched=37`. Backlog cleared in two runs, zero failures. - [x] **Frames match the plan.** Salina 55 (Severe Thunderstorm Watch issued 21:03Z; frames from 20:05Z, so the 60-minute lead-in is there). Wichita 70 (Flood Watch from 16:30Z at 15-minute steps, merged with the severe watch's 5-minute steps: one row per slot). Enid 32, all on the quarter hour, as a Flood Watch alone should give. - [x] **Loops assemble from archive frames.** Built through the real `get_event_radar_loop` path inside the container: Salina 55 frames, 3.58 MB, 9.8 s; Wichita 70 frames, 4.30 MB, 6.8 s. Coverage reads `55 of 55` and `70 of 70`. - [x] **IEM over the Esri basemap looks right.** Frames are aligned with state lines and city labels, reflectivity reads clearly on the dark canvas. Checked first and last frames of the Salina loop: 20:05Z shows cells west of Salina; 00:35Z shows a line through Wichita and Emporia. - [x] **No alert spam.** No incomplete-capture admin alert fired during the backlog. This is also the first **real convective radar loop** seen in the Explorer, open on #22 since rc3. ## Still not verified - The caption and loop as rendered in a browser on the Explorer event page (the checks above called the service functions directly). - A Full-tier **warning**: nothing reached the soak locations overnight, so these loops come from watches. - Blank-frame deferral on real data (`blank_deferred` stayed 0). - The incomplete-capture alert firing for a genuine shortfall. ## Observations - 157 frames took 14 MB on disk, about 80–90 KB each during active weather. - Both loops sit inside events titled "SPC Outlook Day 1", because the correlation layer attaches a watch as context to the SPC outlook's event. The loop is right; the title is not what someone looking for "the storm" would search for. Pre-existing behaviour, worth a look separately. - Unrelated, also seen in the log: `Failed to parse hazard parameters for alert …` once a minute for one Oklahoma alert. Fail-soft parsing in `nws.py` that swallows the exception without logging it.

claude-bot commented

2026-10-14 00:48:44 +00:00

Author

Collaborator

Correction to the disk figure above: I measured 9.4 MB for the first 120 frames (about 78 KB each) and did not re-measure after the second run. "157 frames took 14 MB" was an estimate written as a measurement; at the measured rate 157 frames is about 12 MB.

Correction to the disk figure above: I measured **9.4 MB for the first 120 frames** (about 78 KB each) and did not re-measure after the second run. "157 frames took 14 MB" was an estimate written as a measurement; at the measured rate 157 frames is about 12 MB.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/WeatherBot#210

No description provided.

Rows
Columns

Radar history: capture per location from the IEM archive, with guaranteed lead-in and no dropped frames #210

Why

The enabler

Design

Scope

Unchanged

Known trade-offs

Leaving open until verified on dev

Follow-ups not in this change

Verified on dev (2.0.0-rc7), 2026-10-14 00:36–00:50 UTC

Still not verified

Observations

Verified on dev (`2.0.0-rc7`), 2026-10-14 00:36–00:50 UTC