rbrooks/WeatherBot
Watch
1
0
Fork
You've already forked WeatherBot
 0
Code Issues 10 Pull requests 2 Projects Releases 16 Packages 1 Wiki Activity Actions

Lightning cluster archive #132

New issue
Closed
opened 2026-07-27 20:07:17 +00:00 by claude-bot · 2 comments
claude-bot commented 2026-07-27 20:07:17 +00:00
Contributor
Copy link

Parent: #22.

Lightning persists nothing today. parse_placefile results live in _LIGHTNING_CACHE, an in-memory dict with a 60-second TTL (app/services/lightning.py:57,62), capped at _MAX_CLUSTERS = 500 (:54). Proximity alerts dispatch a synthetic NWSAlert (lightning_alerts.py:63-89) that is never written to sent_alerts; the only durable trace is Location.lightning_last_seen_at / lightning_last_alerted_at.

Decision taken: archive full cluster data, not just proximity events, so the Explorer can replay strike activity on an event timeline.

Sizing (estimated — confirm against real data before locking retention)

poll_lightning runs every 2 minutes (config.py:156) = 720 polls/day, up to 500 clusters each. Saturated that is ~54 MB/day, but saturation never holds. At a ~3% annual duty cycle and ~200 clusters when active: ~230 MB/year per fetch region before dedup. Regions are metro-sized, since _LIGHTNING_CACHE keys on rounded lat/lon and co-located locations share one.

Clusters repeat heavily across consecutive polls — the feed carries recent strikes with an age_min field, so the same strike appears in many successive fetches. Dedup should cut this 5–10x, landing around 25–50 MB/year/region, with a bad-season tail into the low GB if dedup underperforms.

For contrast, proximity-events-only would be ~30 KB/location/year. The full archive is roughly three orders of magnitude larger, which is affordable but not free.

Tasks

  • Review the Weather Pulse / AllisonHouse terms on durable storage of ENTLN feed data before building. v1.6.0 deliberately kept lightning authenticated-only, with public display behind LIGHTNING_ON_PUBLIC_PAGES (default off), because the data is licensed to the subscriber. A durable archive is a stronger posture than a live overlay. Record the finding on this issue. If the terms forbid retention, fall back to persisting only edge-triggered proximity events and note that here.
  • New append-only table with a dedup key over (rounded lat, rounded lon, strike time) so repeated polls do not multiply rows.
  • Its own retention window, shorter than the 13-month record horizon — lightning volume does not justify parity. Propose 90 days, confirm against measured growth.
  • Instrument real row counts and bytes across a full storm season; post the numbers here before locking retention.
  • Keep archival strictly off the alert-dispatch hot path — fire-and-forget and fail-soft, like radar snapshots.
  • Archive stays authenticated-only. No public-page exposure without a separate explicit decision.
Parent: #22. Lightning persists **nothing** today. `parse_placefile` results live in `_LIGHTNING_CACHE`, an in-memory dict with a 60-second TTL (`app/services/lightning.py:57,62`), capped at `_MAX_CLUSTERS = 500` (`:54`). Proximity alerts dispatch a synthetic `NWSAlert` (`lightning_alerts.py:63-89`) that is never written to `sent_alerts`; the only durable trace is `Location.lightning_last_seen_at` / `lightning_last_alerted_at`. **Decision taken:** archive full cluster data, not just proximity events, so the Explorer can replay strike activity on an event timeline. ## Sizing (estimated — confirm against real data before locking retention) `poll_lightning` runs every 2 minutes (`config.py:156`) = 720 polls/day, up to 500 clusters each. Saturated that is ~54 MB/day, but saturation never holds. At a ~3% annual duty cycle and ~200 clusters when active: **~230 MB/year per fetch region** before dedup. Regions are metro-sized, since `_LIGHTNING_CACHE` keys on rounded lat/lon and co-located locations share one. Clusters repeat heavily across consecutive polls — the feed carries recent strikes with an `age_min` field, so the same strike appears in many successive fetches. Dedup should cut this 5–10x, landing around **25–50 MB/year/region**, with a bad-season tail into the low GB if dedup underperforms. For contrast, proximity-events-only would be ~30 KB/location/year. The full archive is roughly three orders of magnitude larger, which is affordable but not free. ## Tasks - [ ] **Review the Weather Pulse / AllisonHouse terms on durable storage of ENTLN feed data before building.** v1.6.0 deliberately kept lightning authenticated-only, with public display behind `LIGHTNING_ON_PUBLIC_PAGES` (default off), because the data is licensed to the subscriber. A durable archive is a stronger posture than a live overlay. Record the finding on this issue. If the terms forbid retention, fall back to persisting only edge-triggered proximity events and note that here. - [ ] New append-only table with a dedup key over (rounded lat, rounded lon, strike time) so repeated polls do not multiply rows. - [ ] Its own retention window, shorter than the 13-month record horizon — lightning volume does not justify parity. Propose 90 days, confirm against measured growth. - [ ] Instrument real row counts and bytes across a full storm season; post the numbers here before locking retention. - [ ] Keep archival strictly off the alert-dispatch hot path — fire-and-forget and fail-soft, like radar snapshots. - [ ] Archive stays authenticated-only. No public-page exposure without a separate explicit decision.
claude-bot commented 2026-07-28 14:26:25 +00:00
Author
Contributor
Copy link

Done — PR #156 merged to main as 12d47dd (CI run #5086 green: ruff, alembic upgrade head on Postgres, both test tiers, pip-audit). Branch feat/lightning-archive deleted. This closes the capture phase of the v2.0.0 epic (#22): #129, #130, #131, #132 are all in.

What shipped

  • lightning_clusters table (migration 0032) with a UNIQUE dedup_key built from the cluster's own coordinates and time at ~11 m resolution — deliberately not rounded to the ~1.1 km granularity of the fetch cache, which would have discarded genuinely distinct clusters.
  • Archiving hangs off the existing fetch in _process_location, so it costs no extra upstream requests against the paid subscriber feed.
  • Archiving happens before the radius filter, so clusters are captured on the polls where storms are building nearby but not yet in alert range.
  • RETENTION_LIGHTNING_CLUSTERS_DAYS (default 90) and LIGHTNING_ARCHIVE_ENABLED (default true, no-op unless LIGHTNING_PLACEFILE_URL is set).
  • Fail-soft throughout — a bookkeeping failure can never cost someone a proximity warning. Authenticated-only; no public-page exposure.

Task list status

  • Weather Pulse / AllisonHouse terms reviewed — operator confirmed licensing permits durable storage, so this archives full cluster data rather than falling back to proximity-events-only.
  • Append-only table with a dedup key over rounded coordinates + strike time.
  • Own retention window, shorter than the 13-month record horizon — 90 days as a starting point.
  • Archival kept off the alert-dispatch hot path (fire-and-forget, fail-soft, like radar snapshots).
  • Archive stays authenticated-only.
  • Instrument real row counts and bytes; post the numbers here before locking retention. Still open — carried into #133.

Carried forward

The sizing on this issue (~25–50 MB/year/region) is arithmetic, not measurement — as is #130's ~165 MB/location/year. Production started capturing snapshots, radar frames and lightning at ~03:50 UTC today, so real row counts and table sizes now exist. #133 (unified 13-month horizon) will query prod for measured figures and set the horizon against those rather than the estimates. Numbers will be posted back here when measured.

Known coverage limit: only locations with lightning_alert_radius_mi set are polled, so archive coverage follows proximity-alert opt-in. Widening it means fetching for every location and multiplying requests against a paid feed — a deliberate deferral, same shape as the #129 gap but with a real cost attached.

Done — PR #156 merged to `main` as `12d47dd` (CI run #5086 green: ruff, `alembic upgrade head` on Postgres, both test tiers, `pip-audit`). Branch `feat/lightning-archive` deleted. This closes the capture phase of the v2.0.0 epic (#22): #129, #130, #131, #132 are all in. ## What shipped - **`lightning_clusters` table** (migration `0032`) with a UNIQUE `dedup_key` built from the cluster's own coordinates and time at ~11 m resolution — deliberately *not* rounded to the ~1.1 km granularity of the fetch cache, which would have discarded genuinely distinct clusters. - **Archiving hangs off the existing fetch** in `_process_location`, so it costs **no extra upstream requests** against the paid subscriber feed. - **Archiving happens before the radius filter**, so clusters are captured on the polls where storms are building nearby but not yet in alert range. - **`RETENTION_LIGHTNING_CLUSTERS_DAYS`** (default 90) and **`LIGHTNING_ARCHIVE_ENABLED`** (default true, no-op unless `LIGHTNING_PLACEFILE_URL` is set). - Fail-soft throughout — a bookkeeping failure can never cost someone a proximity warning. Authenticated-only; no public-page exposure. ## Task list status - [x] Weather Pulse / AllisonHouse terms reviewed — **operator confirmed licensing permits durable storage**, so this archives full cluster data rather than falling back to proximity-events-only. - [x] Append-only table with a dedup key over rounded coordinates + strike time. - [x] Own retention window, shorter than the 13-month record horizon — 90 days as a starting point. - [x] Archival kept off the alert-dispatch hot path (fire-and-forget, fail-soft, like radar snapshots). - [x] Archive stays authenticated-only. - [ ] **Instrument real row counts and bytes; post the numbers here before locking retention.** Still open — carried into #133. ## Carried forward **The sizing on this issue (~25–50 MB/year/region) is arithmetic, not measurement** — as is #130's ~165 MB/location/year. Production started capturing snapshots, radar frames and lightning at ~03:50 UTC today, so real row counts and table sizes now exist. #133 (unified 13-month horizon) will query prod for measured figures and set the horizon against those rather than the estimates. Numbers will be posted back here when measured. **Known coverage limit:** only locations with `lightning_alert_radius_mi` set are polled, so archive coverage follows proximity-alert opt-in. Widening it means fetching for every location and multiplying requests against a paid feed — a deliberate deferral, same shape as the #129 gap but with a real cost attached.
claude-bot commented 2026-07-28 14:35:09 +00:00
Author
Contributor
Copy link

Measurement attempt — not yet possible, and why

Following up on the open "instrument real row counts and bytes" task. Prod queried at 2026-07-28 14:30 UTC, ~3 minutes after the archive deployed (12d47dd, migration 0032 applied). Full measurement set for every table is on #133 (comment).

lightning_clusters: 0 rows. The estimate here (~25–50 MB/year/region) therefore remains unvalidated — I have no measured figure to replace it with.

That is expected rather than broken. Verified live:

  • poll_lightning_job is scheduled and executing (interval[0:02:00], ran successfully at 14:29:48 UTC).
  • The placefile fetch is succeeding — media:lightning_placefile freshness reports success=True failure_count=0 on each tick.
  • archive_clusters() is on the fetch path before the radius filter, at lightning_alerts.py:155.

So the archive is wired and running; the feed simply has no strikes near the opted-in location right now. archive_clusters returns 0 without logging on an empty batch, so silence here is the no-lightning case, not a failure.

Coverage is narrower than the estimate assumed. Only 1 of 5 locations has lightning_alert_radius_mi set, so the archive currently covers a single fetch region — the known coverage limit recorded when this shipped, now with a number on it. Any measured MB/year/region figure will come from that one region until more locations opt in.

Re-measure after a convective season with actual strikes in range. Until then the 25–50 MB/year/region figure should keep its "arithmetic, not measurement" caveat, and #133 is proceeding on structural work rather than on a sizing that does not exist yet — lightning_clusters keeps its own 90-day override, deliberately outside the unified horizon.

## Measurement attempt — not yet possible, and why Following up on the open "instrument real row counts and bytes" task. Prod queried at 2026-07-28 14:30 UTC, ~3 minutes after the archive deployed (`12d47dd`, migration `0032` applied). Full measurement set for every table is on #133 ([comment](https://git.rhoving.com/rbrooks/WeatherBot/issues/133#issuecomment-18937)). **`lightning_clusters`: 0 rows.** The estimate here (~25–50 MB/year/region) therefore remains **unvalidated** — I have no measured figure to replace it with. That is expected rather than broken. Verified live: - `poll_lightning_job` is scheduled and executing (`interval[0:02:00]`, ran successfully at 14:29:48 UTC). - The placefile fetch is succeeding — `media:lightning_placefile` freshness reports `success=True failure_count=0` on each tick. - `archive_clusters()` is on the fetch path before the radius filter, at [lightning_alerts.py:155](app/services/lightning_alerts.py#L155). So the archive is wired and running; the feed simply has no strikes near the opted-in location right now. `archive_clusters` returns 0 without logging on an empty batch, so silence here is the no-lightning case, not a failure. **Coverage is narrower than the estimate assumed.** Only **1 of 5 locations** has `lightning_alert_radius_mi` set, so the archive currently covers a single fetch region — the known coverage limit recorded when this shipped, now with a number on it. Any measured MB/year/region figure will come from that one region until more locations opt in. Re-measure after a convective season with actual strikes in range. Until then the 25–50 MB/year/region figure should keep its "arithmetic, not measurement" caveat, and #133 is proceeding on structural work rather than on a sizing that does not exist yet — `lightning_clusters` keeps its own 90-day override, deliberately outside the unified horizon.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
renovate/python-minorpatch
renovate/cryptography-50.x
feat/event-export
feat/event-radar-loops
feat/explorer-event
feat/explorer-day
feat/explorer-calendar
feat/media-storage-abstraction
fix/allclear-image-url-space
fix/webex-permanent-failure
fix/incomplete-feed-guards
fix/false-all-clear-supersession
feat/radar-frames
feat/history-capture-ungate
renovate/cryptography-49.x
renovate/pillow-12.x
renovate/timezonefinder-8.x
v2.0.0-rc4
v2.0.0-rc3
v2.0.0-rc2
v2.0.0-rc1
v1.8.1
v1.8.0
v1.7.0
v1.6.2
v1.6.1
v1.6.0
v1.5.0
v1.4.0
v1.3.0
v1.2.0
v1.1.0
v1.0.3
v1.0.2
v1.0.1
v1.0.0
Labels
Clear labels   
area:ai
AI summarization

  
area:ci-cd
Continuous integration / deployment

  
area:notifications
Notification delivery & formatting

  
area:observability
Freshness, diagnostics, metrics, admin ops

  
area:public-pages
Public alert / SPC / link pages

  
backlog
Post-2.0 / not yet scheduled

  
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
type:decision
Requires a product/design decision before/with implementation

  
type:feature
New functionality

  
type:infra
CI/CD, build, deployment infrastructure

  
type:maintenance
Tech debt, deprecations, cleanup

  
type:security
Security / abuse mitigation

  
v1.0.1
Target release: v1.0.1 (maintenance & deprecations)

  
v1.1.0
Target release: v1.1.0 (operations & observability)

  
v1.2.0
Target release: v1.2.0 (notification formatting hardening)

  
v1.3.0
Target release: v1.3.0 (AI summarization expansion)

  
v2.0.0
Target release: v2.0.0 (Historical Weather Event Explorer)

  
wontfix
This won't be fixed

No labels
area:ai
area:ci-cd
area:notifications
area:observability
area:public-pages
backlog
bug
duplicate
enhancement
help wanted
invalid
question
type:decision
type:feature
type:infra
type:maintenance
type:security
v1.0.1
v1.1.0
v1.2.0
v1.3.0
v2.0.0
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/WeatherBot#132
No description provided.