rbrooks/WeatherBot
Watch
1
0
Fork
You've already forked WeatherBot
 0
Code Issues 10 Pull requests 2 Projects Releases 16 Packages 1 Wiki Activity Actions

Historical Weather Event Explorer #22

New issue
Open
opened 2026-06-23 23:15:42 +00:00 by rbrooks · 2 comments
rbrooks commented 2026-06-23 23:15:42 +00:00
Owner
Copy link

Target release: v2.0.0 — epic. Implementation is tracked in the child issues indexed below.

Add a historical event reconstruction system (calendar + event review) so users can pick a date and see weather activity across all locations they can access, reconstructing related products into understandable events (SPC outlook → MCDs → watches → warnings → all-clears → radar loops → delivered summaries).

Design pass (2026-07-27)

Key finding: the correlation spine already exists

SentAlert.nws_alert_id stores the VTEC-derived lifecycle id, not the raw NWS feature URN — alert_processor.py:910 assigns nws_alert_id=lifecycle_id, built by alert_identity.alert_lifecycle_id as NWS_{office}_{phenomenon}_{significance}_{etn}. That same string is already nws_alert_snapshots.lifecycle_id, public_alert_pages.lifecycle_id, half the radar snapshot filename, and the AI summary source_id. SPC lineage is already recorded explicitly in spc_outlook_risk_transitions.

The Explorer therefore does not need an event-correlation layer built from nothing — it needs a grouping layer over keys that already line up. This removes most of the risk originally assumed in this issue.

The real problems are elsewhere

  1. Capture gaps. The richest historical record, nws_alert_snapshots (full text, geometry, hazards), is only written when PUBLIC_ALERT_PAGES_ENABLED is on and the location is public_enabled. Both default off. Radar snapshots are likewise off by default. Lightning persists nothing at all. For a private location, all that survives an event today is event name, severity, headline, and times — not enough to reconstruct anything. History not captured is gone forever, so the capture issues must land first.
  2. Retention incoherence. Four windows disagree (sent_alerts 90d, snapshots 180d, SPC pages ~30d past expiry, AI summaries 30d), so an older event renders with holes.
  3. A data-loss trap. Location.sent_alerts cascades all, delete-orphan — deleting a location destroys its entire history.

Decisions taken

Question Decision
History horizon 13 months, unified — enables year-over-year comparison; all families age out together
Snapshot capture gating Ungate capture, keep exposure gated — capture and publication become independent concerns
Object storage Abstraction now, B2/S3 later — interface with a disk backend; no object-storage implementation in v2.0.0
Lightning history Full cluster archive, deduped, with its own shorter retention — pending a Weather Pulse terms review recorded on #132

Child issues

Capture — land first, every day unshipped is history lost

  • #129 Decouple historical alert-snapshot capture from public-page gating
  • #130 Capture radar snapshots for every alert, with an independent retention window
  • #131 Media storage abstraction (disk backend now, object storage later)
  • #132 Lightning cluster archive

Data integrity and retention

  • #133 Unified 13-month history horizon across all history tables
  • #134 Deleting a location must not destroy its alert history

Correlation

  • #136 Decide correlation rules for ambiguous product groupings (blocks #135)
  • #135 Event correlation layer (derived, rebuildable weather_events)

Views

  • #137 Explorer: calendar page
  • #138 Explorer: day detail view
  • #139 Explorer: event detail timeline
  • #140 Explorer: derived radar loop windows for event replay
  • #142 Explorer: export event artifacts
  • #23 Historical display of stored AI summaries

Cross-cutting

  • #141 History query performance, indexing, and Postgres-tier tests

Open questions — status

  • Retention for source text, radar/media metadata, summaries → 13-month unified horizon (#133)
  • Retention + storage backend for cached radar/media artifacts → #130, #131
  • Object storage as first target or later enhancement → later; abstraction now (#131)
  • How to group products when the meteorological relationship is ambiguous → #136
  • User annotate / manually merge-split groupings → #136
  • Which artifacts should be exportable → #142

Originally sourced from the retired docs/ROADMAP.md → Under Investigation → Historical Weather Event Explorer.

**Target release:** v2.0.0 — **epic**. Implementation is tracked in the child issues indexed below. Add a historical event reconstruction system (calendar + event review) so users can pick a date and see weather activity across all locations they can access, reconstructing related products into understandable events (SPC outlook → MCDs → watches → warnings → all-clears → radar loops → delivered summaries). --- ## Design pass (2026-07-27) ### Key finding: the correlation spine already exists `SentAlert.nws_alert_id` stores the **VTEC-derived lifecycle id**, not the raw NWS feature URN — `alert_processor.py:910` assigns `nws_alert_id=lifecycle_id`, built by `alert_identity.alert_lifecycle_id` as `NWS_{office}_{phenomenon}_{significance}_{etn}`. That same string is already `nws_alert_snapshots.lifecycle_id`, `public_alert_pages.lifecycle_id`, half the radar snapshot filename, and the AI summary `source_id`. SPC lineage is already recorded explicitly in `spc_outlook_risk_transitions`. The Explorer therefore does **not** need an event-correlation layer built from nothing — it needs a grouping layer over keys that already line up. This removes most of the risk originally assumed in this issue. ### The real problems are elsewhere 1. **Capture gaps.** The richest historical record, `nws_alert_snapshots` (full text, geometry, hazards), is only written when `PUBLIC_ALERT_PAGES_ENABLED` is on **and** the location is `public_enabled`. Both default off. Radar snapshots are likewise off by default. Lightning persists nothing at all. For a private location, all that survives an event today is event name, severity, headline, and times — not enough to reconstruct anything. **History not captured is gone forever, so the capture issues must land first.** 2. **Retention incoherence.** Four windows disagree (sent_alerts 90d, snapshots 180d, SPC pages ~30d past expiry, AI summaries 30d), so an older event renders with holes. 3. **A data-loss trap.** `Location.sent_alerts` cascades `all, delete-orphan` — deleting a location destroys its entire history. ### Decisions taken | Question | Decision | |---|---| | History horizon | **13 months, unified** — enables year-over-year comparison; all families age out together | | Snapshot capture gating | **Ungate capture, keep exposure gated** — capture and publication become independent concerns | | Object storage | **Abstraction now, B2/S3 later** — interface with a disk backend; no object-storage implementation in v2.0.0 | | Lightning history | **Full cluster archive**, deduped, with its own shorter retention — pending a Weather Pulse terms review recorded on #132 | --- ## Child issues **Capture — land first, every day unshipped is history lost** - #129 Decouple historical alert-snapshot capture from public-page gating - #130 Capture radar snapshots for every alert, with an independent retention window - #131 Media storage abstraction (disk backend now, object storage later) - #132 Lightning cluster archive **Data integrity and retention** - #133 Unified 13-month history horizon across all history tables - #134 Deleting a location must not destroy its alert history **Correlation** - #136 Decide correlation rules for ambiguous product groupings *(blocks #135)* - #135 Event correlation layer (derived, rebuildable `weather_events`) **Views** - #137 Explorer: calendar page - #138 Explorer: day detail view - #139 Explorer: event detail timeline - #140 Explorer: derived radar loop windows for event replay - #142 Explorer: export event artifacts - #23 Historical display of stored AI summaries **Cross-cutting** - #141 History query performance, indexing, and Postgres-tier tests --- ## Open questions — status - [x] Retention for source text, radar/media metadata, summaries → 13-month unified horizon (#133) - [x] Retention + storage backend for cached radar/media artifacts → #130, #131 - [x] Object storage as first target or later enhancement → later; abstraction now (#131) - [ ] How to group products when the meteorological relationship is ambiguous → #136 - [ ] User annotate / manually merge-split groupings → #136 - [ ] Which artifacts should be exportable → #142 Originally sourced from the retired `docs/ROADMAP.md` → Under Investigation → Historical Weather Event Explorer.
claude-bot commented 2026-08-01 07:25:10 +00:00
Contributor
Copy link

Engineering complete (2026-08-01)

Every child issue is closed. The full arc, as merged on main:

Foundation — capture ungated (#129–#132, earlier), unified 13-month horizon (#133), location soft-delete so deletion can't destroy history (#134, PR #169).
Correlation — rules decided and ratified (#136: cores-vs-context, 90-min gap, per-location events; manual merge/split deferred to #167), derived rebuildable weather_events layer (#135, PR #168).
Views — calendar (#137, PR #170), day detail with SPC risk band (#138, PR #171), event timeline with full official text, delivery record incl. failures, inline AI summaries (#139, PR #172; #23 closed as delivered), radar loop replay (#140, PR #173), JSON export (#142, PR #175).
Performance — bounded /stats, five benchmark-justified indexes, partitioning rejected with measurements on record, Postgres-tier tests (#141, PR #176; also resolved the #174 test flake at root cause).

All open questions in this epic's list are now answered on their respective issues. Remaining before this closes: cut the v2.0.0 release (changelog, tag → registry image, iac pin bump, play) and verify the Explorer against real production history. #158 (capture-table storage measurement after a convective season) stays open independently by design.

## Engineering complete (2026-08-01) Every child issue is closed. The full arc, as merged on `main`: **Foundation** — capture ungated (#129–#132, earlier), unified 13-month horizon (#133), location soft-delete so deletion can't destroy history (#134, PR #169). **Correlation** — rules decided and ratified (#136: cores-vs-context, 90-min gap, per-location events; manual merge/split deferred to #167), derived rebuildable `weather_events` layer (#135, PR #168). **Views** — calendar (#137, PR #170), day detail with SPC risk band (#138, PR #171), event timeline with full official text, delivery record incl. failures, inline AI summaries (#139, PR #172; #23 closed as delivered), radar loop replay (#140, PR #173), JSON export (#142, PR #175). **Performance** — bounded /stats, five benchmark-justified indexes, partitioning rejected with measurements on record, Postgres-tier tests (#141, PR #176; also resolved the #174 test flake at root cause). All open questions in this epic's list are now answered on their respective issues. Remaining before this closes: cut the **v2.0.0 release** (changelog, tag → registry image, iac pin bump, play) and verify the Explorer against real production history. #158 (capture-table storage measurement after a convective season) stays open independently by design.
claude-bot commented 2026-08-21 03:50:32 +00:00
Contributor
Copy link

Production migration rehearsal: PASS (2026-08-21)

The 1.8.1 → 2.0.0 jump has now been exercised against real production data. Dev only ever migrated incrementally across rc1/rc2/rc3, so the combined 0033 → 0037 path had been exercised nowhere — and v1.4.0 rolled back on exactly this class of gap (an empty throwaway hiding what real rows do).

Method. Latest production dump (weatherbot-20260821-033003.sql.gz, 4.6 MB) restored into a throwaway Postgres and migrated with the real weatherbot:2.0.0-rc3 image. Run on docker-host itself, deliberately: production data includes channel credentials, so copying it to another host to test it would have created a worse problem than the one being tested. Everything created was namespaced wb-rehearsal-* and torn down on exit; weatherbot-db-1 and its volume were never touched, and live production remains at 0033.

Result — all four migrations applied cleanly, exit code 0:

Running upgrade 0033 -> 0034, Add weather_events / weather_event_members tables (#135)
Running upgrade 0034 -> 0035, Add locations.deleted_at for soft-delete (#134)
Running upgrade 0035 -> 0036, Add the history-query indexes the #141 benchmark justified
Running upgrade 0036 -> 0037, Split notification_channels.radar_animated into radar_enabled + radar_animated (#184)

No data loss. Post-migration counts match live production exactly:

rehearsal (post-migration) live production
sent_alerts 1721 1721
locations 5 5
notification_channels 4 4
users 2 2

Schema objects verified present: weather_events, weather_event_members, locations.deleted_at, and 22 indexes across the history tables.

0037 backfill on real rows: radar_enabled IS DISTINCT FROM radar_animated returns 0 rows. All 4 production channels land t/t.

What this does NOT prove

Two honest limits worth recording rather than glossing:

  1. It proves the schema migration, not the application running on migrated data. In particular, the first correlation pass backfills weather_events from all retained history within 30 minutes of startup — on production that means deriving events across 1,721 sent_alerts plus snapshots and lightning, on hardware also serving live alert polling. That job has only ever run against dev's much smaller history. It is the most likely source of a GA surprise now that the migrations are cleared.
  2. The backfill's meaningful branch is still untested on real data. Every channel on both dev and production has radar_animated=true, so no real row exercises the false → radar_enabled=false case that 0037 exists to protect. It passes only in the Postgres-tier test.

Remaining before v2.0.0 final

  • Migrations verified against real production data
  • rc3 observed rendering a real convective event on dev (both the #182 loop fix and #184's animation still have only synthetic verification)
  • Bump the group pin 1.8.1 → 2.0.0 and deploy production

Also worth noting for the record: production's history now has a 9.7-day hole (2026-08-11 to 08-20) from the outage in #186. The Explorer will render those days as "no activity", which is indistinguishable from genuinely quiet weather.

## Production migration rehearsal: PASS (2026-08-21) The 1.8.1 → 2.0.0 jump has now been exercised against **real production data**. Dev only ever migrated incrementally across rc1/rc2/rc3, so the combined `0033 → 0037` path had been exercised nowhere — and v1.4.0 rolled back on exactly this class of gap (an empty throwaway hiding what real rows do). **Method.** Latest production dump (`weatherbot-20260821-033003.sql.gz`, 4.6 MB) restored into a throwaway Postgres and migrated with the real `weatherbot:2.0.0-rc3` image. Run **on `docker-host` itself**, deliberately: production data includes channel credentials, so copying it to another host to test it would have created a worse problem than the one being tested. Everything created was namespaced `wb-rehearsal-*` and torn down on exit; `weatherbot-db-1` and its volume were never touched, and live production remains at `0033`. **Result — all four migrations applied cleanly, exit code 0:** ``` Running upgrade 0033 -> 0034, Add weather_events / weather_event_members tables (#135) Running upgrade 0034 -> 0035, Add locations.deleted_at for soft-delete (#134) Running upgrade 0035 -> 0036, Add the history-query indexes the #141 benchmark justified Running upgrade 0036 -> 0037, Split notification_channels.radar_animated into radar_enabled + radar_animated (#184) ``` **No data loss.** Post-migration counts match live production exactly: | | rehearsal (post-migration) | live production | |---|---:|---:| | `sent_alerts` | 1721 | 1721 | | `locations` | 5 | 5 | | `notification_channels` | 4 | 4 | | `users` | 2 | 2 | **Schema objects verified present:** `weather_events`, `weather_event_members`, `locations.deleted_at`, and 22 indexes across the history tables. **0037 backfill on real rows:** `radar_enabled IS DISTINCT FROM radar_animated` returns **0 rows**. All 4 production channels land `t/t`. ## What this does NOT prove Two honest limits worth recording rather than glossing: 1. **It proves the schema migration, not the application running on migrated data.** In particular, the first correlation pass backfills `weather_events` from all retained history within 30 minutes of startup — on production that means deriving events across 1,721 `sent_alerts` plus snapshots and lightning, on hardware also serving live alert polling. That job has only ever run against dev's much smaller history. It is the most likely source of a GA surprise now that the migrations are cleared. 2. **The backfill's meaningful branch is still untested on real data.** Every channel on both dev and production has `radar_animated=true`, so no real row exercises the `false → radar_enabled=false` case that 0037 exists to protect. It passes only in the Postgres-tier test. ## Remaining before v2.0.0 final - [x] Migrations verified against real production data - [ ] rc3 observed rendering a real convective event on dev (both the #182 loop fix and #184's animation still have only synthetic verification) - [ ] Bump the group pin `1.8.1` → `2.0.0` and deploy production Also worth noting for the record: production's history now has a **9.7-day hole** (2026-08-11 to 08-20) from the outage in #186. The Explorer will render those days as "no activity", which is indistinguishable from genuinely quiet weather.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
renovate/python-minorpatch
renovate/cryptography-50.x
feat/event-export
feat/event-radar-loops
feat/explorer-event
feat/explorer-day
feat/explorer-calendar
feat/media-storage-abstraction
fix/allclear-image-url-space
fix/webex-permanent-failure
fix/incomplete-feed-guards
fix/false-all-clear-supersession
feat/radar-frames
feat/history-capture-ungate
renovate/cryptography-49.x
renovate/pillow-12.x
renovate/timezonefinder-8.x
v2.0.0-rc4
v2.0.0-rc3
v2.0.0-rc2
v2.0.0-rc1
v1.8.1
v1.8.0
v1.7.0
v1.6.2
v1.6.1
v1.6.0
v1.5.0
v1.4.0
v1.3.0
v1.2.0
v1.1.0
v1.0.3
v1.0.2
v1.0.1
v1.0.0
Labels
Clear labels   
area:ai
AI summarization

  
area:ci-cd
Continuous integration / deployment

  
area:notifications
Notification delivery & formatting

  
area:observability
Freshness, diagnostics, metrics, admin ops

  
area:public-pages
Public alert / SPC / link pages

  
backlog
Post-2.0 / not yet scheduled

  
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
type:decision
Requires a product/design decision before/with implementation

  
type:feature
New functionality

  
type:infra
CI/CD, build, deployment infrastructure

  
type:maintenance
Tech debt, deprecations, cleanup

  
type:security
Security / abuse mitigation

  
v1.0.1
Target release: v1.0.1 (maintenance & deprecations)

  
v1.1.0
Target release: v1.1.0 (operations & observability)

  
v1.2.0
Target release: v1.2.0 (notification formatting hardening)

  
v1.3.0
Target release: v1.3.0 (AI summarization expansion)

  
v2.0.0
Target release: v2.0.0 (Historical Weather Event Explorer)

  
wontfix
This won't be fixed

No labels
area:ai
area:ci-cd
area:notifications
area:observability
area:public-pages
backlog
bug
duplicate
enhancement
help wanted
invalid
question
type:decision
type:feature
type:infra
type:maintenance
type:security
v1.0.1
v1.1.0
v1.2.0
v1.3.0
v2.0.0
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/WeatherBot#22
No description provided.