Archive ENTLN lightning clusters (#132) #156

Merged

claude-bot merged 1 commit from feat/lightning-archive into main

2026-07-28 14:25:58 +00:00

claude-bot commented

2026-07-28 06:12:58 +00:00

Contributor

Closes #132. Final capture-phase issue of the v2.0.0 Historical Explorer epic (#22).

Licensing confirmed by the operator to permit durable storage, so this archives the full cluster data rather than falling back to proximity-events-only.

Problem

Lightning persisted nothing. Parsed clusters lived in a 60-second in-memory cache (lightning._LIGHTNING_CACHE), and proximity alerts dispatched a synthetic NWSAlert that was never written to sent_alerts — the only durable trace was Location.lightning_last_seen_at / lightning_last_alerted_at. An event timeline could never show lightning activity, however severe the storm.

Change

New lightning_clusters table (migration 0032) with a UNIQUE dedup_key.
Archiving hangs off the existing fetch in _process_location, so it costs no extra upstream requests against a paid subscriber feed.
Own retention window — RETENTION_LIGHTNING_CLUSTERS_DAYS, default 90.
LIGHTNING_ARCHIVE_ENABLED, default true, and a no-op unless LIGHTNING_PLACEFILE_URL is set.

Three decisions worth reviewing

Dedup granularity. poll_lightning runs every couple of minutes and the feed keeps returning the same recent clusters, so without dedup the row count would track the poll rate rather than actual lightning. The key is built from the cluster's own coordinates and time at ~11 m resolution — lossless within a location's polls, and deliberately not rounded to the ~1.1 km granularity of the fetch cache, which would discard genuinely distinct clusters. It is imperfect across locations, because the placefile is generated per requested centre so two distant locations can receive slightly different groupings of the same strikes. That is inherent to the feed, not something the schema can fix.

Archiving happens before the radius filter. The Explorer wants activity across the event window, not only what came close enough to alert on — and _process_location's if not nearby: return would otherwise skip archiving on exactly the polls where storms are building nearby but not yet in range.

Retention is shorter than the record horizon, on purpose. Lightning is by far the highest-volume history the app stores — thousands of rows per storm against a handful of alerts — and its value decays faster, being event-replay texture rather than the alert record itself. 90 days is a starting point to revisit with real numbers in #133.

Known coverage limit

Only locations with lightning_alert_radius_mi set are polled, so archive coverage follows proximity-alert opt-in. If no location has a radius, nothing is archived. Widening that would mean fetching for every location and multiplying requests against a paid feed — a deliberate decision, not something to slip into this PR. Flagging it because it is the same shape as the #129 gap (capture gated on an unrelated opt-in), just with a real cost attached to fixing it.

Safety

Fail-soft throughout — the caller is the lightning alert path, and a bookkeeping failure must never cost someone a proximity warning. Batch-internal duplicates are collapsed before insert; a race with an overlapping poll is absorbed by the UNIQUE constraint rather than logging a traceback. The archive is authenticated-only and does not reach the public pages.

Verification

Step	Result
`ruff check .`	passed
`python -m compileall app`	OK
SQLite bulk tier	848 passed (up from 838)
`alembic upgrade head` on fresh Postgres 16	clean through `0032`
`0032` downgrade → re-upgrade round trip	clean
Postgres integration tier	4 passed

tests/test_lightning_archive.py covers dedup-key stability, that nearby-but-distinct clusters do not collide, timezone normalisation, and the core property: three identical polls store 2 rows, not 6. Plus batch-internal collapsing, new-alongside-known, the disabled flag, malformed input survivability, and retention pruning by occurred_at.

🤖 Generated with Claude Code

Closes #132. Final capture-phase issue of the v2.0.0 Historical Explorer epic (#22). **Licensing confirmed by the operator to permit durable storage**, so this archives the full cluster data rather than falling back to proximity-events-only. ## Problem Lightning persisted **nothing**. Parsed clusters lived in a 60-second in-memory cache (`lightning._LIGHTNING_CACHE`), and proximity alerts dispatched a synthetic `NWSAlert` that was never written to `sent_alerts` — the only durable trace was `Location.lightning_last_seen_at` / `lightning_last_alerted_at`. An event timeline could never show lightning activity, however severe the storm. ## Change - **New `lightning_clusters` table** (migration `0032`) with a UNIQUE `dedup_key`. - **Archiving hangs off the existing fetch** in `_process_location`, so it costs **no extra upstream requests** against a paid subscriber feed. - **Own retention window** — `RETENTION_LIGHTNING_CLUSTERS_DAYS`, default 90. - **`LIGHTNING_ARCHIVE_ENABLED`**, default true, and a no-op unless `LIGHTNING_PLACEFILE_URL` is set. ## Three decisions worth reviewing **Dedup granularity.** `poll_lightning` runs every couple of minutes and the feed keeps returning the same recent clusters, so without dedup the row count would track the *poll rate* rather than actual lightning. The key is built from the cluster's own coordinates and time at ~11 m resolution — lossless within a location's polls, and **deliberately not** rounded to the ~1.1 km granularity of the fetch cache, which would discard genuinely distinct clusters. It is imperfect *across* locations, because the placefile is generated per requested centre so two distant locations can receive slightly different groupings of the same strikes. That is inherent to the feed, not something the schema can fix. **Archiving happens before the radius filter.** The Explorer wants activity across the event window, not only what came close enough to alert on — and `_process_location`'s `if not nearby: return` would otherwise skip archiving on exactly the polls where storms are building nearby but not yet in range. **Retention is shorter than the record horizon, on purpose.** Lightning is by far the highest-volume history the app stores — thousands of rows per storm against a handful of alerts — and its value decays faster, being event-replay texture rather than the alert record itself. 90 days is a starting point to revisit with real numbers in #133. ## Known coverage limit Only locations with `lightning_alert_radius_mi` set are polled, so **archive coverage follows proximity-alert opt-in**. If no location has a radius, nothing is archived. Widening that would mean fetching for every location and multiplying requests against a paid feed — a deliberate decision, not something to slip into this PR. Flagging it because it is the same shape as the #129 gap (capture gated on an unrelated opt-in), just with a real cost attached to fixing it. ## Safety Fail-soft throughout — the caller is the lightning alert path, and a bookkeeping failure must never cost someone a proximity warning. Batch-internal duplicates are collapsed before insert; a race with an overlapping poll is absorbed by the UNIQUE constraint rather than logging a traceback. The archive is authenticated-only and does not reach the public pages. ## Verification | Step | Result | |---|---| | `ruff check .` | passed | | `python -m compileall app` | OK | | SQLite bulk tier | **848 passed** (up from 838) | | `alembic upgrade head` on fresh Postgres 16 | clean through `0032` | | `0032` downgrade → re-upgrade round trip | clean | | Postgres integration tier | 4 passed | `tests/test_lightning_archive.py` covers dedup-key stability, that nearby-but-distinct clusters do **not** collide, timezone normalisation, and the core property: **three identical polls store 2 rows, not 6**. Plus batch-internal collapsing, new-alongside-known, the disabled flag, malformed input survivability, and retention pruning by `occurred_at`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added this to the v2.0.0 - Historical Weather Event Explorer milestone

2026-07-28 06:12:58 +00:00

claude-bot added 1 commit

2026-07-28 06:12:58 +00:00

Archive ENTLN lightning clusters (#132 )

CI / test (pull_request) Successful in 3m57s

Details

3d46489cbe

Lightning persisted nothing: parsed clusters lived in a 60-second in-memory
cache and proximity alerts dispatched a synthetic NWSAlert never written to
sent_alerts, leaving only timestamps on locations. An event timeline could
therefore never show lightning activity, however severe.

Licensing is confirmed to permit durable storage, so this archives the FULL
cluster data rather than only the edge-triggered proximity events -- the
Explorer replays strike activity across an event window instead of showing a
single "lightning was nearby" marker.

- New lightning_clusters table (migration 0032) with a UNIQUE dedup_key.
  poll_lightning runs every couple of minutes and the feed keeps returning the
  same recent clusters, so without dedup the row count would track the poll
  rate rather than actual lightning. The key is built from the cluster's own
  coordinates and time at ~11 m resolution: lossless within a location's polls,
  and deliberately NOT rounded to the ~1.1 km granularity of the fetch cache,
  which would discard genuinely distinct clusters.
- Archiving hangs off the existing fetch in _process_location, so it costs NO
  extra upstream requests against a paid subscriber feed. It runs BEFORE the
  radius filter: the Explorer wants activity across the window, and the
  `not nearby` early-return would otherwise skip archiving on exactly the polls
  where storms are building nearby but not yet in range.
- Fail-soft throughout, so a bookkeeping failure can never cost a proximity
  warning. Batch-internal duplicates are collapsed before insert, and a race
  with an overlapping poll is absorbed by the UNIQUE constraint rather than
  logging a traceback.
- Own retention window (RETENTION_LIGHTNING_CLUSTERS_DAYS, 90) -- deliberately
  shorter than the 13-month record horizon, since lightning is the highest
  volume history stored and is event-replay texture rather than the alert
  record itself.

The archive is authenticated-only; ENTLN data is licensed to the subscriber and
must not reach the public pages without a separate decision.

Known coverage limit: only locations with lightning_alert_radius_mi set are
polled, so archive coverage follows proximity opt-in. Widening it would mean
fetching for every location and multiplying vendor requests, which is a
deliberate decision rather than something to slip in here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

claude-bot merged commit 12d47ddc2a into main

2026-07-28 14:25:58 +00:00

claude-bot deleted branch feat/lightning-archive

2026-07-28 14:25:58 +00:00

claude-bot referenced this pull request from a commit

2026-07-28 14:25:58 +00:00

Merge pull request 'Archive ENTLN lightning clusters (#132)' (#156) from feat/lightning-archive into main

claude-bot referenced this pull request

2026-07-28 14:26:25 +00:00

Lightning cluster archive #132