rbrooks/WeatherBot
Watch
1
0
Fork
You've already forked WeatherBot
 0
Code Issues 11 Pull requests 6 Projects Releases 16 Packages 1 Wiki Activity Actions

Historical Weather Event Explorer #22

New issue
Open
opened 2026-06-23 23:15:42 +00:00 by rbrooks · 9 comments
rbrooks commented 2026-06-23 23:15:42 +00:00
Owner
Copy link

Target release: v2.0.0 — epic. Implementation is tracked in the child issues indexed below.

Add a historical event reconstruction system (calendar + event review) so users can pick a date and see weather activity across all locations they can access, reconstructing related products into understandable events (SPC outlook → MCDs → watches → warnings → all-clears → radar loops → delivered summaries).

Design pass (2026-07-27)

Key finding: the correlation spine already exists

SentAlert.nws_alert_id stores the VTEC-derived lifecycle id, not the raw NWS feature URN — alert_processor.py:910 assigns nws_alert_id=lifecycle_id, built by alert_identity.alert_lifecycle_id as NWS_{office}_{phenomenon}_{significance}_{etn}. That same string is already nws_alert_snapshots.lifecycle_id, public_alert_pages.lifecycle_id, half the radar snapshot filename, and the AI summary source_id. SPC lineage is already recorded explicitly in spc_outlook_risk_transitions.

The Explorer therefore does not need an event-correlation layer built from nothing — it needs a grouping layer over keys that already line up. This removes most of the risk originally assumed in this issue.

The real problems are elsewhere

  1. Capture gaps. The richest historical record, nws_alert_snapshots (full text, geometry, hazards), is only written when PUBLIC_ALERT_PAGES_ENABLED is on and the location is public_enabled. Both default off. Radar snapshots are likewise off by default. Lightning persists nothing at all. For a private location, all that survives an event today is event name, severity, headline, and times — not enough to reconstruct anything. History not captured is gone forever, so the capture issues must land first.
  2. Retention incoherence. Four windows disagree (sent_alerts 90d, snapshots 180d, SPC pages ~30d past expiry, AI summaries 30d), so an older event renders with holes.
  3. A data-loss trap. Location.sent_alerts cascades all, delete-orphan — deleting a location destroys its entire history.

Decisions taken

Question Decision
History horizon 13 months, unified — enables year-over-year comparison; all families age out together
Snapshot capture gating Ungate capture, keep exposure gated — capture and publication become independent concerns
Object storage Abstraction now, B2/S3 later — interface with a disk backend; no object-storage implementation in v2.0.0
Lightning history Full cluster archive, deduped, with its own shorter retention — pending a Weather Pulse terms review recorded on #132

Child issues

Capture — land first, every day unshipped is history lost

  • #129 Decouple historical alert-snapshot capture from public-page gating
  • #130 Capture radar snapshots for every alert, with an independent retention window
  • #131 Media storage abstraction (disk backend now, object storage later)
  • #132 Lightning cluster archive

Data integrity and retention

  • #133 Unified 13-month history horizon across all history tables
  • #134 Deleting a location must not destroy its alert history

Correlation

  • #136 Decide correlation rules for ambiguous product groupings (blocks #135)
  • #135 Event correlation layer (derived, rebuildable weather_events)

Views

  • #137 Explorer: calendar page
  • #138 Explorer: day detail view
  • #139 Explorer: event detail timeline
  • #140 Explorer: derived radar loop windows for event replay
  • #142 Explorer: export event artifacts
  • #23 Historical display of stored AI summaries

Cross-cutting

  • #141 History query performance, indexing, and Postgres-tier tests

Open questions — status

  • Retention for source text, radar/media metadata, summaries → 13-month unified horizon (#133)
  • Retention + storage backend for cached radar/media artifacts → #130, #131
  • Object storage as first target or later enhancement → later; abstraction now (#131)
  • How to group products when the meteorological relationship is ambiguous → #136
  • User annotate / manually merge-split groupings → #136
  • Which artifacts should be exportable → #142

Originally sourced from the retired docs/ROADMAP.md → Under Investigation → Historical Weather Event Explorer.

**Target release:** v2.0.0 — **epic**. Implementation is tracked in the child issues indexed below. Add a historical event reconstruction system (calendar + event review) so users can pick a date and see weather activity across all locations they can access, reconstructing related products into understandable events (SPC outlook → MCDs → watches → warnings → all-clears → radar loops → delivered summaries). --- ## Design pass (2026-07-27) ### Key finding: the correlation spine already exists `SentAlert.nws_alert_id` stores the **VTEC-derived lifecycle id**, not the raw NWS feature URN — `alert_processor.py:910` assigns `nws_alert_id=lifecycle_id`, built by `alert_identity.alert_lifecycle_id` as `NWS_{office}_{phenomenon}_{significance}_{etn}`. That same string is already `nws_alert_snapshots.lifecycle_id`, `public_alert_pages.lifecycle_id`, half the radar snapshot filename, and the AI summary `source_id`. SPC lineage is already recorded explicitly in `spc_outlook_risk_transitions`. The Explorer therefore does **not** need an event-correlation layer built from nothing — it needs a grouping layer over keys that already line up. This removes most of the risk originally assumed in this issue. ### The real problems are elsewhere 1. **Capture gaps.** The richest historical record, `nws_alert_snapshots` (full text, geometry, hazards), is only written when `PUBLIC_ALERT_PAGES_ENABLED` is on **and** the location is `public_enabled`. Both default off. Radar snapshots are likewise off by default. Lightning persists nothing at all. For a private location, all that survives an event today is event name, severity, headline, and times — not enough to reconstruct anything. **History not captured is gone forever, so the capture issues must land first.** 2. **Retention incoherence.** Four windows disagree (sent_alerts 90d, snapshots 180d, SPC pages ~30d past expiry, AI summaries 30d), so an older event renders with holes. 3. **A data-loss trap.** `Location.sent_alerts` cascades `all, delete-orphan` — deleting a location destroys its entire history. ### Decisions taken | Question | Decision | |---|---| | History horizon | **13 months, unified** — enables year-over-year comparison; all families age out together | | Snapshot capture gating | **Ungate capture, keep exposure gated** — capture and publication become independent concerns | | Object storage | **Abstraction now, B2/S3 later** — interface with a disk backend; no object-storage implementation in v2.0.0 | | Lightning history | **Full cluster archive**, deduped, with its own shorter retention — pending a Weather Pulse terms review recorded on #132 | --- ## Child issues **Capture — land first, every day unshipped is history lost** - #129 Decouple historical alert-snapshot capture from public-page gating - #130 Capture radar snapshots for every alert, with an independent retention window - #131 Media storage abstraction (disk backend now, object storage later) - #132 Lightning cluster archive **Data integrity and retention** - #133 Unified 13-month history horizon across all history tables - #134 Deleting a location must not destroy its alert history **Correlation** - #136 Decide correlation rules for ambiguous product groupings *(blocks #135)* - #135 Event correlation layer (derived, rebuildable `weather_events`) **Views** - #137 Explorer: calendar page - #138 Explorer: day detail view - #139 Explorer: event detail timeline - #140 Explorer: derived radar loop windows for event replay - #142 Explorer: export event artifacts - #23 Historical display of stored AI summaries **Cross-cutting** - #141 History query performance, indexing, and Postgres-tier tests --- ## Open questions — status - [x] Retention for source text, radar/media metadata, summaries → 13-month unified horizon (#133) - [x] Retention + storage backend for cached radar/media artifacts → #130, #131 - [x] Object storage as first target or later enhancement → later; abstraction now (#131) - [ ] How to group products when the meteorological relationship is ambiguous → #136 - [ ] User annotate / manually merge-split groupings → #136 - [ ] Which artifacts should be exportable → #142 Originally sourced from the retired `docs/ROADMAP.md` → Under Investigation → Historical Weather Event Explorer.
claude-bot commented 2026-08-01 07:25:10 +00:00
Collaborator
Copy link

Engineering complete (2026-08-01)

Every child issue is closed. The full arc, as merged on main:

Foundation — capture ungated (#129–#132, earlier), unified 13-month horizon (#133), location soft-delete so deletion can't destroy history (#134, PR #169).
Correlation — rules decided and ratified (#136: cores-vs-context, 90-min gap, per-location events; manual merge/split deferred to #167), derived rebuildable weather_events layer (#135, PR #168).
Views — calendar (#137, PR #170), day detail with SPC risk band (#138, PR #171), event timeline with full official text, delivery record incl. failures, inline AI summaries (#139, PR #172; #23 closed as delivered), radar loop replay (#140, PR #173), JSON export (#142, PR #175).
Performance — bounded /stats, five benchmark-justified indexes, partitioning rejected with measurements on record, Postgres-tier tests (#141, PR #176; also resolved the #174 test flake at root cause).

All open questions in this epic's list are now answered on their respective issues. Remaining before this closes: cut the v2.0.0 release (changelog, tag → registry image, iac pin bump, play) and verify the Explorer against real production history. #158 (capture-table storage measurement after a convective season) stays open independently by design.

## Engineering complete (2026-08-01) Every child issue is closed. The full arc, as merged on `main`: **Foundation** — capture ungated (#129–#132, earlier), unified 13-month horizon (#133), location soft-delete so deletion can't destroy history (#134, PR #169). **Correlation** — rules decided and ratified (#136: cores-vs-context, 90-min gap, per-location events; manual merge/split deferred to #167), derived rebuildable `weather_events` layer (#135, PR #168). **Views** — calendar (#137, PR #170), day detail with SPC risk band (#138, PR #171), event timeline with full official text, delivery record incl. failures, inline AI summaries (#139, PR #172; #23 closed as delivered), radar loop replay (#140, PR #173), JSON export (#142, PR #175). **Performance** — bounded /stats, five benchmark-justified indexes, partitioning rejected with measurements on record, Postgres-tier tests (#141, PR #176; also resolved the #174 test flake at root cause). All open questions in this epic's list are now answered on their respective issues. Remaining before this closes: cut the **v2.0.0 release** (changelog, tag → registry image, iac pin bump, play) and verify the Explorer against real production history. #158 (capture-table storage measurement after a convective season) stays open independently by design.
claude-bot commented 2026-08-21 03:50:32 +00:00
Collaborator
Copy link

Production migration rehearsal: PASS (2026-08-21)

The 1.8.1 → 2.0.0 jump has now been exercised against real production data. Dev only ever migrated incrementally across rc1/rc2/rc3, so the combined 0033 → 0037 path had been exercised nowhere — and v1.4.0 rolled back on exactly this class of gap (an empty throwaway hiding what real rows do).

Method. Latest production dump (weatherbot-20260821-033003.sql.gz, 4.6 MB) restored into a throwaway Postgres and migrated with the real weatherbot:2.0.0-rc3 image. Run on docker-host itself, deliberately: production data includes channel credentials, so copying it to another host to test it would have created a worse problem than the one being tested. Everything created was namespaced wb-rehearsal-* and torn down on exit; weatherbot-db-1 and its volume were never touched, and live production remains at 0033.

Result — all four migrations applied cleanly, exit code 0:

Running upgrade 0033 -> 0034, Add weather_events / weather_event_members tables (#135)
Running upgrade 0034 -> 0035, Add locations.deleted_at for soft-delete (#134)
Running upgrade 0035 -> 0036, Add the history-query indexes the #141 benchmark justified
Running upgrade 0036 -> 0037, Split notification_channels.radar_animated into radar_enabled + radar_animated (#184)

No data loss. Post-migration counts match live production exactly:

rehearsal (post-migration) live production
sent_alerts 1721 1721
locations 5 5
notification_channels 4 4
users 2 2

Schema objects verified present: weather_events, weather_event_members, locations.deleted_at, and 22 indexes across the history tables.

0037 backfill on real rows: radar_enabled IS DISTINCT FROM radar_animated returns 0 rows. All 4 production channels land t/t.

What this does NOT prove

Two honest limits worth recording rather than glossing:

  1. It proves the schema migration, not the application running on migrated data. In particular, the first correlation pass backfills weather_events from all retained history within 30 minutes of startup — on production that means deriving events across 1,721 sent_alerts plus snapshots and lightning, on hardware also serving live alert polling. That job has only ever run against dev's much smaller history. It is the most likely source of a GA surprise now that the migrations are cleared.
  2. The backfill's meaningful branch is still untested on real data. Every channel on both dev and production has radar_animated=true, so no real row exercises the false → radar_enabled=false case that 0037 exists to protect. It passes only in the Postgres-tier test.

Remaining before v2.0.0 final

  • Migrations verified against real production data
  • rc3 observed rendering a real convective event on dev (both the #182 loop fix and #184's animation still have only synthetic verification)
  • Bump the group pin 1.8.1 → 2.0.0 and deploy production

Also worth noting for the record: production's history now has a 9.7-day hole (2026-08-11 to 08-20) from the outage in #186. The Explorer will render those days as "no activity", which is indistinguishable from genuinely quiet weather.

## Production migration rehearsal: PASS (2026-08-21) The 1.8.1 → 2.0.0 jump has now been exercised against **real production data**. Dev only ever migrated incrementally across rc1/rc2/rc3, so the combined `0033 → 0037` path had been exercised nowhere — and v1.4.0 rolled back on exactly this class of gap (an empty throwaway hiding what real rows do). **Method.** Latest production dump (`weatherbot-20260821-033003.sql.gz`, 4.6 MB) restored into a throwaway Postgres and migrated with the real `weatherbot:2.0.0-rc3` image. Run **on `docker-host` itself**, deliberately: production data includes channel credentials, so copying it to another host to test it would have created a worse problem than the one being tested. Everything created was namespaced `wb-rehearsal-*` and torn down on exit; `weatherbot-db-1` and its volume were never touched, and live production remains at `0033`. **Result — all four migrations applied cleanly, exit code 0:** ``` Running upgrade 0033 -> 0034, Add weather_events / weather_event_members tables (#135) Running upgrade 0034 -> 0035, Add locations.deleted_at for soft-delete (#134) Running upgrade 0035 -> 0036, Add the history-query indexes the #141 benchmark justified Running upgrade 0036 -> 0037, Split notification_channels.radar_animated into radar_enabled + radar_animated (#184) ``` **No data loss.** Post-migration counts match live production exactly: | | rehearsal (post-migration) | live production | |---|---:|---:| | `sent_alerts` | 1721 | 1721 | | `locations` | 5 | 5 | | `notification_channels` | 4 | 4 | | `users` | 2 | 2 | **Schema objects verified present:** `weather_events`, `weather_event_members`, `locations.deleted_at`, and 22 indexes across the history tables. **0037 backfill on real rows:** `radar_enabled IS DISTINCT FROM radar_animated` returns **0 rows**. All 4 production channels land `t/t`. ## What this does NOT prove Two honest limits worth recording rather than glossing: 1. **It proves the schema migration, not the application running on migrated data.** In particular, the first correlation pass backfills `weather_events` from all retained history within 30 minutes of startup — on production that means deriving events across 1,721 `sent_alerts` plus snapshots and lightning, on hardware also serving live alert polling. That job has only ever run against dev's much smaller history. It is the most likely source of a GA surprise now that the migrations are cleared. 2. **The backfill's meaningful branch is still untested on real data.** Every channel on both dev and production has `radar_animated=true`, so no real row exercises the `false → radar_enabled=false` case that 0037 exists to protect. It passes only in the Postgres-tier test. ## Remaining before v2.0.0 final - [x] Migrations verified against real production data - [ ] rc3 observed rendering a real convective event on dev (both the #182 loop fix and #184's animation still have only synthetic verification) - [ ] Bump the group pin `1.8.1` → `2.0.0` and deploy production Also worth noting for the record: production's history now has a **9.7-day hole** (2026-08-11 to 08-20) from the outage in #186. The Explorer will render those days as "no activity", which is indistinguishable from genuinely quiet weather.
claude-bot commented 2026-10-13 16:26:51 +00:00
Collaborator
Copy link

v2.0.0-rc5 tagged and published (2026-10-13) — not yet deployed anywhere

Tag v2.0.0-rc5 on 83bd6d1. release.yml run 247 built and pushed git.rhoving.com/rbrooks/weatherbot:2.0.0-rc5 (digest sha256:e08bdd82…f5b5a), confirmed from the job log.

Since rc4, no Explorer changes and no migrations:

  • Security scan backlog (#195, PR #200): Starlette 1.8.0, cryptography 50.0.2, PyJWT and urllib3 pinned, two-stage image without the compiler.
  • SPC outlook issue times (#201, PR #202): NOAA's compact timestamps now parse; fire outlooks take their issue time from the feed's file date.

Also closed today: #186 (production cleanup done; outage detection handed to Rhoving/iac-repo#636, which does not gate v2.0.0).

Deliberately held until after 2.0.0: Renovate #178 (SQLAlchemy 2.1 and the rest of the minor/patch group), #199 (Python 3.15), #193 (timezonefinder 9), #197 (signal-cli 0.101).

State of the hosts

  • Production: 1.8.1, database 0033, healthy.
  • Dev: 2.0.0-rc4, database 0037, healthy. Two temporary soak locations still present (Jackson MS (rc3 soak), Huntsville AL (rc4 soak)).

Remaining before v2.0.0 final

  • Migrations verified against real production data (2026-08-21)
  • rc5 deployed to dev (iac-repo pin bump + play against docker-test)
  • rc5 starts cleanly: first run of the two-stage image's entrypoint and virtualenv
  • Starlette 1.x checked by hand on dev: sign-in and session, live dashboard updates, static assets
  • issued_at populating on new spc_outlook_pages rows, and the parse warning gone
  • A real convective radar loop seen in the Explorer (carried from rc3/rc4)
  • Animated notifier radar exercised; needs a Discord, Signal, Matrix or Pushover channel on dev (carried from rc3/rc4)
  • Remove the two soak locations from dev
  • Bump the production pin 1.8.1 → 2.0.0, with a dump taken first; an image-only rollback will crash-loop once 0034–0037 have run
## v2.0.0-rc5 tagged and published (2026-10-13) — not yet deployed anywhere Tag `v2.0.0-rc5` on `83bd6d1`. `release.yml` run 247 built and pushed `git.rhoving.com/rbrooks/weatherbot:2.0.0-rc5` (digest `sha256:e08bdd82…f5b5a`), confirmed from the job log. Since rc4, no Explorer changes and no migrations: - **Security scan backlog (#195, PR #200)**: Starlette 1.8.0, cryptography 50.0.2, PyJWT and urllib3 pinned, two-stage image without the compiler. - **SPC outlook issue times (#201, PR #202)**: NOAA's compact timestamps now parse; fire outlooks take their issue time from the feed's file date. Also closed today: #186 (production cleanup done; outage detection handed to Rhoving/iac-repo#636, which does not gate v2.0.0). Deliberately held until after 2.0.0: Renovate #178 (SQLAlchemy 2.1 and the rest of the minor/patch group), #199 (Python 3.15), #193 (timezonefinder 9), #197 (signal-cli 0.101). ## State of the hosts - Production: `1.8.1`, database `0033`, healthy. - Dev: `2.0.0-rc4`, database `0037`, healthy. Two temporary soak locations still present (`Jackson MS (rc3 soak)`, `Huntsville AL (rc4 soak)`). ## Remaining before v2.0.0 final - [x] Migrations verified against real production data (2026-08-21) - [ ] rc5 deployed to dev (iac-repo pin bump + play against `docker-test`) - [ ] rc5 starts cleanly: first run of the two-stage image's entrypoint and virtualenv - [ ] Starlette 1.x checked by hand on dev: sign-in and session, live dashboard updates, static assets - [ ] `issued_at` populating on new `spc_outlook_pages` rows, and the parse warning gone - [ ] A real convective radar loop seen in the Explorer (carried from rc3/rc4) - [ ] Animated notifier radar exercised; needs a Discord, Signal, Matrix or Pushover channel on dev (carried from rc3/rc4) - [ ] Remove the two soak locations from dev - [ ] Bump the production pin `1.8.1` → `2.0.0`, with a dump taken first; an image-only rollback will crash-loop once 0034–0037 have run
claude-bot commented 2026-10-13 16:56:58 +00:00
Collaborator
Copy link

rc5 deployed to dev, 2026-10-13 16:48 UTC

Pin bump Rhoving/iac-repo#637 (e3c981d, merged as b2bfe76); play against docker-test: ok=23 changed=3 failed=0. The dry run beforehand showed the image line as the only change. Production untouched (group pin still 1.8.1).

Verified on dev

  • The two-stage image runs. weatherbot:2.0.0-rc5 healthy, 0 restarts, PID 1 is uvicorn as uid 10001 from /opt/venv, reports v2.0.0-rc5, no gcc present. Image is 723 MB against 1.02 GB for rc4.
  • Versions inside the container: Starlette 1.8.0, cryptography 50.0.2, PyJWT 2.15.1, urllib3 2.8.0.
  • Startup: Alembic no-op at 0037, scheduler started, no errors.
  • HTTP from the host: /health 200, /metrics 200, / and /explorer 307 to login, /auth/login 200, /static/css/app.css 200.
  • SPC issue times (#201) on real data. After the first SPC poll on rc5: D1_202610131629 has issued_at 16:29Z and D3_202610130708 has 07:08Z, matching their product ids. "missing parseable issue time" warnings since start: 0 (rc4 logged about 1,900 a day). Same-risk reissues were recognised as continuations and dispatch was suppressed, so the fix re-sent nothing. Active records expire where expected: Day 1 at 2026-10-14 08:00Z, Day 3 at 2026-10-16 08:00Z.

Not verified (needs a browser session)

  • Sign-in through Authentik and session persistence across a restart
  • Live dashboard updates over SSE
  • Explorer pages rendering while signed in

Seen on dev, pre-existing, not caused by rc5

  • One admin alert at startup: "Product stale: NWS active alerts for TX: last success 212840 min ago". That is about 148 days, so a freshness row left behind by a location that no longer exists.
  • Two SPC Mesoscale Discussion records from 2026-07-11 are still uncleared with no expiry.

Checklist

  • rc5 deployed to dev
  • rc5 starts cleanly on the two-stage image
  • issued_at populating, parse warning gone
  • Starlette 1.x checked by hand while signed in
  • A real convective radar loop seen in the Explorer
  • Animated notifier radar exercised on a radar-capable channel
  • Remove the two soak locations from dev
  • Production: dump, then pin 1.8.1 → 2.0.0
## rc5 deployed to dev, 2026-10-13 16:48 UTC Pin bump Rhoving/iac-repo#637 (`e3c981d`, merged as `b2bfe76`); play against `docker-test`: `ok=23 changed=3 failed=0`. The dry run beforehand showed the image line as the only change. Production untouched (group pin still `1.8.1`). ### Verified on dev - **The two-stage image runs.** `weatherbot:2.0.0-rc5` healthy, 0 restarts, PID 1 is uvicorn as uid 10001 from `/opt/venv`, reports `v2.0.0-rc5`, no `gcc` present. Image is 723 MB against 1.02 GB for rc4. - **Versions inside the container:** Starlette 1.8.0, cryptography 50.0.2, PyJWT 2.15.1, urllib3 2.8.0. - **Startup:** Alembic no-op at `0037`, scheduler started, no errors. - **HTTP from the host:** `/health` 200, `/metrics` 200, `/` and `/explorer` 307 to login, `/auth/login` 200, `/static/css/app.css` 200. - **SPC issue times (#201) on real data.** After the first SPC poll on rc5: `D1_202610131629` has `issued_at` 16:29Z and `D3_202610130708` has 07:08Z, matching their product ids. "missing parseable issue time" warnings since start: **0** (rc4 logged about 1,900 a day). Same-risk reissues were recognised as continuations and dispatch was suppressed, so the fix re-sent nothing. Active records expire where expected: Day 1 at 2026-10-14 08:00Z, Day 3 at 2026-10-16 08:00Z. ### Not verified (needs a browser session) - Sign-in through Authentik and session persistence across a restart - Live dashboard updates over SSE - Explorer pages rendering while signed in ### Seen on dev, pre-existing, not caused by rc5 - One admin alert at startup: "Product stale: NWS active alerts for TX: last success 212840 min ago". That is about 148 days, so a freshness row left behind by a location that no longer exists. - Two SPC Mesoscale Discussion records from 2026-07-11 are still uncleared with no expiry. ### Checklist - [x] rc5 deployed to dev - [x] rc5 starts cleanly on the two-stage image - [x] `issued_at` populating, parse warning gone - [ ] Starlette 1.x checked by hand while signed in - [ ] A real convective radar loop seen in the Explorer - [ ] Animated notifier radar exercised on a radar-capable channel - [ ] Remove the two soak locations from dev - [ ] Production: dump, then pin `1.8.1` → `2.0.0`
claude-bot commented 2026-10-13 17:51:35 +00:00
Collaborator
Copy link

rc6 deployed to dev, 2026-10-13 17:50 UTC

Tag v2.0.0-rc6 on 421b02d; release.yml run 255 pushed weatherbot:2.0.0-rc6 (digest sha256:448a009b…9571). Pin bump Rhoving/iac-repo#639 (merged as bd7ed86); play against docker-test: ok=23 changed=3 failed=0, image line the only diff. Production untouched on 1.8.1.

Contents since rc5, all found while checking rc5 on dev, no migrations:

  • Freshness warning held on by an unmonitored state, and printing the wrong feed's age (#204, PR #205)
  • Explorer calendar: whole day cell is the click target (#204, PR #205)
  • Add Location form kept the previous location's values (PR #207)

Verified on dev

  • Healthy, 0 restarts, reports v2.0.0-rc6, database still 0037.
  • Freshness cleanup worked on the real orphan. First alert poll logged Pruned 1 NWS alert freshness row(s) for states no longer monitored. The nws_alerts freshness rows are now exactly the five monitored states (AL, KS, MO, MS, OK), all seconds old. nws_alerts:TX is gone.
  • Zero warnings or errors since start. rc5 logged the TX stale-product admin alert at startup; rc6 does not.
  • Both template changes are in the running image.

Not verified (needs a browser)

  • The banner is visibly gone from the dashboard
  • Clicking anywhere in an Explorer day cell opens the day
  • A second Add Location in a row opens blank with Watch selected

Convective soak in progress

SPC Day 1 (16:29Z) has a Slight Risk from northwest Missouri and Kansas into the southern High Plains, with initiation expected over central Kansas mid-to-late afternoon. @rbrooks added three temporary dev locations at Advisory severity: Salina, KS (rc5 soak), Wichita, KS (rc5 soak), Enid, OK (rc5 soak). All three resolved (ready) and are being polled. No convective warnings were active nationwide at 17:23Z.

Dev now carries five temporary soak locations to remove before GA (these three plus the Jackson MS and Huntsville AL ones).

## rc6 deployed to dev, 2026-10-13 17:50 UTC Tag `v2.0.0-rc6` on `421b02d`; `release.yml` run 255 pushed `weatherbot:2.0.0-rc6` (digest `sha256:448a009b…9571`). Pin bump Rhoving/iac-repo#639 (merged as `bd7ed86`); play against `docker-test`: `ok=23 changed=3 failed=0`, image line the only diff. Production untouched on `1.8.1`. Contents since rc5, all found while checking rc5 on dev, no migrations: - Freshness warning held on by an unmonitored state, and printing the wrong feed's age (#204, PR #205) - Explorer calendar: whole day cell is the click target (#204, PR #205) - Add Location form kept the previous location's values (PR #207) ### Verified on dev - Healthy, 0 restarts, reports `v2.0.0-rc6`, database still `0037`. - **Freshness cleanup worked on the real orphan.** First alert poll logged `Pruned 1 NWS alert freshness row(s) for states no longer monitored`. The `nws_alerts` freshness rows are now exactly the five monitored states (AL, KS, MO, MS, OK), all seconds old. `nws_alerts:TX` is gone. - **Zero warnings or errors since start.** rc5 logged the TX stale-product admin alert at startup; rc6 does not. - Both template changes are in the running image. ### Not verified (needs a browser) - The banner is visibly gone from the dashboard - Clicking anywhere in an Explorer day cell opens the day - A second Add Location in a row opens blank with Watch selected ### Convective soak in progress SPC Day 1 (16:29Z) has a Slight Risk from northwest Missouri and Kansas into the southern High Plains, with initiation expected over central Kansas mid-to-late afternoon. @rbrooks added three temporary dev locations at Advisory severity: `Salina, KS (rc5 soak)`, `Wichita, KS (rc5 soak)`, `Enid, OK (rc5 soak)`. All three resolved (`ready`) and are being polled. No convective warnings were active nationwide at 17:23Z. Dev now carries five temporary soak locations to remove before GA (these three plus the Jackson MS and Huntsville AL ones).
claude-bot commented 2026-10-13 19:21:06 +00:00
Collaborator
Copy link

rc6 browser checks passed (confirmed by @rbrooks, 2026-10-13)

  • Freshness banner gone from the dashboard (#204 closed)
  • Explorer calendar: whole day cell clickable
  • Add Location dialog opens blank on a second add
  • Signed-in pages render on Starlette 1.x (dashboard, Explorer and Locations were all exercised while checking the above)

Radar capture is running on rc6

As of 19:20Z dev has captured 24 radar frames since 17:29Z, with every alert_radar_frame and NOAA MapServer fetch recorded as a success. The frames are riding on Flood Watches for the Wichita and Enid soak locations, not on a convective warning yet, so this proves capture on the two-stage image but not storm motion in a loop.

Still open before v2.0.0 final

  • A real convective radar loop seen in the Explorer (SPC expects initiation over central Kansas mid-to-late afternoon today)
  • Animated notifier radar exercised on a radar-capable channel
  • Remove the five temporary soak locations from dev
  • Production: dump, then pin 1.8.1 → 2.0.0
## rc6 browser checks passed (confirmed by @rbrooks, 2026-10-13) - [x] Freshness banner gone from the dashboard (#204 closed) - [x] Explorer calendar: whole day cell clickable - [x] Add Location dialog opens blank on a second add - [x] Signed-in pages render on Starlette 1.x (dashboard, Explorer and Locations were all exercised while checking the above) ## Radar capture is running on rc6 As of 19:20Z dev has captured **24 radar frames since 17:29Z**, with every `alert_radar_frame` and NOAA MapServer fetch recorded as a success. The frames are riding on **Flood Watches** for the Wichita and Enid soak locations, not on a convective warning yet, so this proves capture on the two-stage image but not storm motion in a loop. ## Still open before v2.0.0 final - [ ] A real convective radar loop seen in the Explorer (SPC expects initiation over central Kansas mid-to-late afternoon today) - [ ] Animated notifier radar exercised on a radar-capable channel - [ ] Remove the five temporary soak locations from dev - [ ] Production: dump, then pin `1.8.1` → `2.0.0`
claude-bot commented 2026-10-14 05:28:23 +00:00
Collaborator
Copy link

Production migration rehearsal repeated for 0033 → 0038: PASS (2026-10-14)

@rbrooks approved the v2.0.0 production release on 2026-10-14 after checking the Kansas loops in the Explorer. Animated notifier radar will be checked on production, as @rbrooks decided.

Method. Same as 2026-08-21, run on docker-host so production data never left the host: the newest nightly dump (weatherbot-20261014-033459.sql.gz, 19 MB) restored into a throwaway postgres:16.15-alpine named wb-rehearsal-*, migrated with the real weatherbot:2.0.0-rc7 image. Then the scheduler's own correlation job function was run against it in a one-off container. Nothing but the migration and that one function ran: production's data includes channel credentials, and a full app against a copy would send real notifications. Torn down afterwards and confirmed (0 containers, 0 networks, 0 temp files left); the rc7 image it pulled was removed from the host.

Migrations: all five applied, 9 seconds, exit 0.

Running upgrade 0033 -> 0034, Add weather_events / weather_event_members tables (#135)
Running upgrade 0034 -> 0035, Add locations.deleted_at for soft-delete (#134)
Running upgrade 0035 -> 0036, Add the history-query indexes the #141 benchmark justified
Running upgrade 0036 -> 0037, Split notification_channels.radar_animated into radar_enabled + radar_animated (#184)
Running upgrade 0037 -> 0038, Add location_radar_frames for the IEM radar archive (#210)
live production rehearsal, before rehearsal, after
alembic_version 0033 0033 0038
sent_alerts 1798 1798 1798
locations 5 5 5
notification_channels 4 4 4
users 2 2 2

Restore errors: 0. New tables present; radar_enabled matches radar_animated on all 4 channels.

First correlation pass (the one flagged in August as the likeliest go-live surprise): 19.9 seconds, 297 events (287 hazard, 10 watch-only), 909 members, from 1,798 alerts and 297,384 lightning records. Second (steady-state) pass: 1.0 second, event count unchanged.

One thing to watch. The first pass peaked at 597 MB RSS in a standalone process. The production app container has a 1 GiB memory limit and sits at about 216 MiB. In-process the pass should land well under the limit, but an OOM kill during it would restart the app into another first pass, so memory gets watched when the pass runs (30 minutes after startup).

## Production migration rehearsal repeated for `0033 → 0038`: PASS (2026-10-14) @rbrooks approved the v2.0.0 production release on 2026-10-14 after checking the Kansas loops in the Explorer. Animated notifier radar will be checked on production, as @rbrooks decided. **Method.** Same as 2026-08-21, run on `docker-host` so production data never left the host: the newest nightly dump (`weatherbot-20261014-033459.sql.gz`, 19 MB) restored into a throwaway `postgres:16.15-alpine` named `wb-rehearsal-*`, migrated with the real `weatherbot:2.0.0-rc7` image. Then the scheduler's own correlation job function was run against it in a one-off container. **Nothing but the migration and that one function ran**: production's data includes channel credentials, and a full app against a copy would send real notifications. Torn down afterwards and confirmed (0 containers, 0 networks, 0 temp files left); the rc7 image it pulled was removed from the host. **Migrations: all five applied, 9 seconds, exit 0.** ``` Running upgrade 0033 -> 0034, Add weather_events / weather_event_members tables (#135) Running upgrade 0034 -> 0035, Add locations.deleted_at for soft-delete (#134) Running upgrade 0035 -> 0036, Add the history-query indexes the #141 benchmark justified Running upgrade 0036 -> 0037, Split notification_channels.radar_animated into radar_enabled + radar_animated (#184) Running upgrade 0037 -> 0038, Add location_radar_frames for the IEM radar archive (#210) ``` | | live production | rehearsal, before | rehearsal, after | |---|---:|---:|---:| | `alembic_version` | 0033 | 0033 | **0038** | | `sent_alerts` | 1798 | 1798 | 1798 | | `locations` | 5 | 5 | 5 | | `notification_channels` | 4 | 4 | 4 | | `users` | 2 | 2 | 2 | Restore errors: 0. New tables present; `radar_enabled` matches `radar_animated` on all 4 channels. **First correlation pass (the one flagged in August as the likeliest go-live surprise): 19.9 seconds, 297 events (287 hazard, 10 watch-only), 909 members**, from 1,798 alerts and 297,384 lightning records. Second (steady-state) pass: 1.0 second, event count unchanged. **One thing to watch.** The first pass peaked at **597 MB RSS** in a standalone process. The production app container has a **1 GiB memory limit** and sits at about 216 MiB. In-process the pass should land well under the limit, but an OOM kill during it would restart the app into another first pass, so memory gets watched when the pass runs (30 minutes after startup).
claude-bot commented 2026-10-14 05:55:36 +00:00
Collaborator
Copy link

v2.0.0 released and deployed to production, 2026-10-14 05:50 UTC

  • Tag v2.0.0 on 859a88a (identical to rc7 apart from CHANGELOG.md). release.yml run 268 pushed weatherbot:2.0.0, digest sha256:97ec7f3c…0197.
  • Pin Rhoving/iac-repo#641 (merged as 7b6537c): group pin 1.8.1 → 2.0.0; dev's temporary RC override deleted, so both hosts follow the group pin again.
  • Dev first: play ok=23 changed=3 failed=0, v2.0.0 healthy, 0 restarts, no errors.
  • Pre-upgrade production dump through weatherbot-backup.service: weatherbot-20261014-054752.sql.gz (19 MB), gzip verified, alembic_version 0033 inside, matching live. A copy is pinned as /opt/weatherbot-svc/backups/weatherbot-pre-2.0.0-20261014.sql.gz. Note that the backup unit's own pruning deletes weatherbot-*.sql.gz files older than 14 days, the pinned copy included.
  • Production play 05:48–05:51Z: ok=24 changed=3 failed=0, all pre-flight assertions passed, image line the only diff.

Production after the upgrade

  • weatherbot:2.0.0 healthy, 0 restarts, weatherbot.service active, reports v2.0.0.
  • Migrations 0033 → 0038 applied, all five, as rehearsed.
  • No data change: 1798 alerts, 5 locations, 4 channels, 2 users, exactly as before. All four channels radar_enabled=t, radar_animated=t.
  • Alert polling working: 4 of 4 runs succeeded in the first five minutes (Fetched 4 active NWS alerts across 1 states).
  • /health 200, /auth/login 200. Zero ERROR or traceback lines; the only warnings are discord.py's voice-support notices, unchanged from 1.8.1.
  • Memory 132 MiB of 1 GiB.

Still to watch

  • First correlation pass at about 06:20Z, and its peak memory
  • First radar archive run (expected to fetch nothing: production's last 48 hours hold only SPC outlook alerts)
  • First real notification on 2.0.0, which is also the first real test of animated notifier radar (the plan @rbrooks chose)
  • Security rescan of weatherbot:2.0.0 to close #195
## v2.0.0 released and deployed to production, 2026-10-14 05:50 UTC - **Tag** `v2.0.0` on `859a88a` (identical to rc7 apart from `CHANGELOG.md`). `release.yml` run 268 pushed `weatherbot:2.0.0`, digest `sha256:97ec7f3c…0197`. - **Pin** Rhoving/iac-repo#641 (merged as `7b6537c`): group pin `1.8.1` → `2.0.0`; dev's temporary RC override deleted, so both hosts follow the group pin again. - **Dev first**: play `ok=23 changed=3 failed=0`, `v2.0.0` healthy, 0 restarts, no errors. - **Pre-upgrade production dump** through `weatherbot-backup.service`: `weatherbot-20261014-054752.sql.gz` (19 MB), gzip verified, `alembic_version` `0033` inside, matching live. A copy is pinned as `/opt/weatherbot-svc/backups/weatherbot-pre-2.0.0-20261014.sql.gz`. Note that the backup unit's own pruning deletes `weatherbot-*.sql.gz` files older than 14 days, the pinned copy included. - **Production play** 05:48–05:51Z: `ok=24 changed=3 failed=0`, all pre-flight assertions passed, image line the only diff. ### Production after the upgrade - `weatherbot:2.0.0` healthy, 0 restarts, `weatherbot.service` active, reports `v2.0.0`. - **Migrations `0033 → 0038` applied**, all five, as rehearsed. - **No data change**: 1798 alerts, 5 locations, 4 channels, 2 users, exactly as before. All four channels `radar_enabled=t, radar_animated=t`. - **Alert polling working**: 4 of 4 runs succeeded in the first five minutes (`Fetched 4 active NWS alerts across 1 states`). - `/health` 200, `/auth/login` 200. Zero `ERROR` or traceback lines; the only warnings are discord.py's voice-support notices, unchanged from 1.8.1. - Memory 132 MiB of 1 GiB. ### Still to watch - [ ] First correlation pass at about 06:20Z, and its peak memory - [ ] First radar archive run (expected to fetch nothing: production's last 48 hours hold only SPC outlook alerts) - [ ] First real notification on 2.0.0, which is also the first real test of animated notifier radar (the plan @rbrooks chose) - [ ] Security rescan of `weatherbot:2.0.0` to close #195
claude-bot commented 2026-10-14 07:04:36 +00:00
Collaborator
Copy link

Post-deploy watch complete (2026-10-14 07:05Z)

  • First correlation pass on production at 06:21Z: derived 297 event(s) across 5 location(s) (full-history backfill), matching the rehearsal exactly. Container memory peaked at 711 MiB of 1 GiB, no OOM, no restart.
  • First radar archive runs: six runs, nothing to fetch (production's last 48 hours held only SPC outlook alerts), no errors.
  • Regression found and fixed: SPC outlook processing failed on every poll after the upgrade (#214, an unescaped LIKE in the per-day record match, latent since before 2.0.0). Hotfixed as v2.0.1, deployed to dev then production at 06:57Z with @rbrooks's approval. SPC outlook processing was down 05:55–07:02Z; NWS warning alerts were unaffected.

Production is now on weatherbot:2.0.1, healthy, 0 errors.

Still open

  • First real notification on 2.0.x, which is also the first real test of animated notifier radar (as @rbrooks chose).
  • Security rescan of the deployed image to close #195.
  • Remove the five soak locations from dev.

I'd close this epic once the first real notification has gone out cleanly.

## Post-deploy watch complete (2026-10-14 07:05Z) - [x] **First correlation pass on production** at 06:21Z: `derived 297 event(s) across 5 location(s) (full-history backfill)`, matching the rehearsal exactly. Container memory peaked at **711 MiB of 1 GiB**, no OOM, no restart. - [x] **First radar archive runs**: six runs, nothing to fetch (production's last 48 hours held only SPC outlook alerts), no errors. - [x] **Regression found and fixed**: SPC outlook processing failed on every poll after the upgrade (#214, an unescaped `LIKE` in the per-day record match, latent since before 2.0.0). Hotfixed as **v2.0.1**, deployed to dev then production at 06:57Z with @rbrooks's approval. SPC outlook processing was down 05:55–07:02Z; NWS warning alerts were unaffected. Production is now on **`weatherbot:2.0.1`**, healthy, 0 errors. ### Still open - [ ] First real notification on 2.0.x, which is also the first real test of animated notifier radar (as @rbrooks chose). - [ ] Security rescan of the deployed image to close #195. - [ ] Remove the five soak locations from dev. I'd close this epic once the first real notification has gone out cleanly.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
renovate/python-minorpatch
renovate/test-deps-(automerge)
renovate/timezonefinder-9.x
renovate/python-3.x
renovate/pillow-13.x
chore/changelog-2.0.1
fix/spc-day-match
chore/changelog-2.0.0
chore/changelog-rc7
feat/location-radar-archive
fix/radar-frame-cadence
chore/changelog-rc6
fix/location-form-reset
fix/freshness-banner-and-explorer-day-target
chore/changelog-rc5
fix/spc-issue-time
fix/security-scan-195
renovate/bbernhard-signal-cli-rest-api-0.x
feat/event-export
feat/event-radar-loops
feat/explorer-event
feat/explorer-day
feat/explorer-calendar
feat/media-storage-abstraction
fix/allclear-image-url-space
fix/webex-permanent-failure
fix/incomplete-feed-guards
fix/false-all-clear-supersession
feat/radar-frames
feat/history-capture-ungate
renovate/cryptography-49.x
renovate/pillow-12.x
renovate/timezonefinder-8.x
v2.0.1
v2.0.0
v2.0.0-rc7
v2.0.0-rc6
v2.0.0-rc5
v2.0.0-rc4
v2.0.0-rc3
v2.0.0-rc2
v2.0.0-rc1
v1.8.1
v1.8.0
v1.7.0
v1.6.2
v1.6.1
v1.6.0
v1.5.0
v1.4.0
v1.3.0
v1.2.0
v1.1.0
v1.0.3
v1.0.2
v1.0.1
v1.0.0
Labels
Clear labels   
area:ai
AI summarization

  
area:ci-cd
Continuous integration / deployment

  
area:notifications
Notification delivery & formatting

  
area:observability
Freshness, diagnostics, metrics, admin ops

  
area:public-pages
Public alert / SPC / link pages

  
backlog
Post-2.0 / not yet scheduled

  
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
security
Security hardening / vulnerability work

  
type:decision
Requires a product/design decision before/with implementation

  
type:feature
New functionality

  
type:infra
CI/CD, build, deployment infrastructure

  
type:maintenance
Tech debt, deprecations, cleanup

  
type:security
Security / abuse mitigation

  
v1.0.1
Target release: v1.0.1 (maintenance & deprecations)

  
v1.1.0
Target release: v1.1.0 (operations & observability)

  
v1.2.0
Target release: v1.2.0 (notification formatting hardening)

  
v1.3.0
Target release: v1.3.0 (AI summarization expansion)

  
v2.0.0
Target release: v2.0.0 (Historical Weather Event Explorer)

  
wontfix
This won't be fixed

No labels
area:ai
area:ci-cd
area:notifications
area:observability
area:public-pages
backlog
bug
duplicate
enhancement
help wanted
invalid
question
security
type:decision
type:feature
type:infra
type:maintenance
type:security
v1.0.1
v1.1.0
v1.2.0
v1.3.0
v2.0.0
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/WeatherBot#22
No description provided.