rbrooks/WeatherBot
Watch
1
0
Fork
You've already forked WeatherBot
 0
Code Issues 10 Pull requests 2 Projects Releases 16 Packages 1 Wiki Activity Actions

Decide correlation rules for ambiguous product groupings #136

New issue
Closed
opened 2026-07-27 20:07:45 +00:00 by claude-bot · 2 comments
claude-bot commented 2026-07-27 20:07:45 +00:00
Contributor
Copy link

Parent: #22. Blocks the event correlation layer — settle this before that pass is written.

#22 asks "how to group products when the meteorological relationship is ambiguous" and "user annotate / manually merge-split groupings?". These need answers up front because they determine the data model: a user-editable grouping needs override storage that a purely derived one does not.

Questions to settle

  • Does a Tornado Warning inside a Severe Thunderstorm Watch belong to one event or two? (Proposal: one event, with the watch as parent context.)
  • Do two warnings at the same location separated by a lull belong to one event? What lull duration splits them?
  • Does an SPC Day-1 outlook covering a location join every warning that fires that day, or stay a separate "outlook context" band on the timeline? (Proposal: context band, not membership — otherwise every warning on an ENH day collapses into one enormous event.)
  • Do simultaneous events at different locations from the same storm system merge into one multi-location event, or stay per-location? (Proposal: per-location — the app's entire model is per-location, and visible_location_filter is per-location, so a cross-location event would need its own visibility semantics.)
  • Manual merge/split: in v2.0.0 or a later refinement? If in, the derived tables need a user-override layer that survives rebuilds — which is a materially larger design.

Record the decisions here, then implement them in the correlation layer issue.

Parent: #22. Blocks the event correlation layer — settle this before that pass is written. #22 asks "how to group products when the meteorological relationship is ambiguous" and "user annotate / manually merge-split groupings?". These need answers up front because they determine the data model: a user-editable grouping needs override storage that a purely derived one does not. ## Questions to settle - [ ] Does a Tornado Warning inside a Severe Thunderstorm Watch belong to one event or two? *(Proposal: one event, with the watch as parent context.)* - [ ] Do two warnings at the same location separated by a lull belong to one event? What lull duration splits them? - [ ] Does an SPC Day-1 outlook covering a location join every warning that fires that day, or stay a separate "outlook context" band on the timeline? *(Proposal: context band, not membership — otherwise every warning on an ENH day collapses into one enormous event.)* - [ ] Do simultaneous events at different locations from the same storm system merge into one multi-location event, or stay per-location? *(Proposal: per-location — the app's entire model is per-location, and `visible_location_filter` is per-location, so a cross-location event would need its own visibility semantics.)* - [ ] Manual merge/split: in v2.0.0 or a later refinement? If in, the derived tables need a user-override layer that survives rebuilds — which is a materially larger design. Record the decisions here, then implement them in the correlation layer issue.
claude-bot commented 2026-08-01 03:00:30 +00:00
Author
Contributor
Copy link

Proposed correlation rules (awaiting Ryan's sign-off)

Grounded in what the data actually provides: the VTEC lifecycle id (NWS_{office}_{phenomenon}_{significance}_{etn}) exposes significance (W warning / A watch / Y advisory / S statement), SentAlert.upgrade_from records explicit upgrade lineage, every hazard has an active interval (coalesce(onset, sent_at) → coalesce(cleared_at, expires)), MCDs carry real SPC product numbers, and lightning correlates by location + time only.

The organizing idea: cores vs. context

An event is a per-location cluster of impact-significance hazards (warnings, advisories, statements — significance W/Y/S) linked by time proximity. Long-lived setup products (watches, MCDs, outlooks) attach to events as context; they never glue events together. This one distinction answers most of the ambiguity below coherently.

Q1 — Tornado Warning inside a Severe Thunderstorm Watch → one event; watch attached as context

The warning is a core member; the watch (significance A) attaches as a context member to every event whose core interval overlaps the watch interval. A watch with no warnings at the location still forms its own watch-only event, so "we were under a watch but nothing fired" remains visible on the calendar. What a watch never does is chain: two separate warning rounds inside one 8-hour watch stay two events, each referencing the watch — otherwise Q2's lull rule would be meaningless whenever a watch is up (which is precisely when it matters).

Q2 — Lull duration → 60-minute gap rule over core members, transitive

Two core hazards merge when their active intervals overlap or the gap between them is ≤ 60 minutes (transitive closure). upgrade_from edges always merge regardless of gap. Rationale for 60: distinct convective rounds at a point are almost always > 1 h apart; sub-hour gaps are usually the same complex re-warning. It's a named constant (EVENT_GAP_MINUTES) baked into derivation_version — since the layer is derived and rebuildable (#135), tuning it later is a rebuild, not a migration.

Q3 — SPC outlooks → day-band context, not membership (as proposed)

Outlooks render as a day-scale band on the calendar/day views, keyed by the existing spc_outlook_pages / spc_outlook_risk_transitions rows. Not event members — an ENH day would otherwise collapse every warning into one mega-event. MCDs sit in between and go the watch route: attached as context members to events overlapping their valid window (they have real product numbers and hours-scale windows), and additionally shown on the day view.

Q4 — Multi-location → per-location events (as proposed)

Matches the app's entire visibility model (visible_location_filter is per-location). Cross-location affinity is still cheap to display without a cross-location entity: the same lifecycle id appearing in several locations' events lets the day view badge "also affected N other locations" — purely derived, no new visibility semantics.

Q5 — Manual merge/split → defer past v2.0.0

A user-override layer that survives rebuilds contradicts the "derived and rebuildable, never source of truth" constraint from #22 and roughly doubles the design (override storage, conflict semantics on re-derivation, UI). Ship the automatic rules, see where they actually misgroup, and revisit with evidence. Cheap interim affordance: the event view can show why members grouped (gap distances, upgrade edges), which makes misgroupings diagnosable.

Lightning (not in the original list, needs a rule)

Archived clusters (#132) form lightning episodes per location via the same 60-minute gap rule. An episode overlapping an event's core interval attaches as a context member (it corroborates, it doesn't extend the event). An episode with no overlapping event forms its own lightning-only event so a dry-lightning day is still explorable.

Interval edge cases

  • Never-cleared hazards (app down at expiry): interval ends at expires; if both cleared_at and expires are NULL, cap at sent_at + 6 h so one malformed record can't chain a whole day together.
  • Watch-only and lightning-only events carry an event_kind (hazard / watch_only / lightning_only) so the calendar can render them at different visual weight.

If these land as-is, #135 implements: weather_events + weather_event_members(kind, member_id, role∈{core,context}), EVENT_GAP_MINUTES=60, derivation_version=1.

## Proposed correlation rules (awaiting Ryan's sign-off) Grounded in what the data actually provides: the VTEC lifecycle id (`NWS_{office}_{phenomenon}_{significance}_{etn}`) exposes **significance** (W warning / A watch / Y advisory / S statement), `SentAlert.upgrade_from` records explicit upgrade lineage, every hazard has an active interval (`coalesce(onset, sent_at)` → `coalesce(cleared_at, expires)`), MCDs carry real SPC product numbers, and lightning correlates by location + time only. ### The organizing idea: cores vs. context An **event** is a per-location cluster of *impact-significance* hazards (warnings, advisories, statements — significance W/Y/S) linked by time proximity. Long-lived *setup* products (watches, MCDs, outlooks) **attach to** events as context; they never glue events together. This one distinction answers most of the ambiguity below coherently. ### Q1 — Tornado Warning inside a Severe Thunderstorm Watch → **one event; watch attached as context** The warning is a core member; the watch (significance A) attaches as a `context` member to every event whose core interval overlaps the watch interval. A watch with **no** warnings at the location still forms its own watch-only event, so "we were under a watch but nothing fired" remains visible on the calendar. What a watch never does is *chain*: two separate warning rounds inside one 8-hour watch stay two events, each referencing the watch — otherwise Q2's lull rule would be meaningless whenever a watch is up (which is precisely when it matters). ### Q2 — Lull duration → **60-minute gap rule over core members, transitive** Two core hazards merge when their active intervals overlap or the gap between them is ≤ **60 minutes** (transitive closure). `upgrade_from` edges always merge regardless of gap. Rationale for 60: distinct convective rounds at a point are almost always > 1 h apart; sub-hour gaps are usually the same complex re-warning. It's a named constant (`EVENT_GAP_MINUTES`) baked into `derivation_version` — since the layer is derived and rebuildable (#135), tuning it later is a rebuild, not a migration. ### Q3 — SPC outlooks → **day-band context, not membership** (as proposed) Outlooks render as a day-scale band on the calendar/day views, keyed by the existing `spc_outlook_pages` / `spc_outlook_risk_transitions` rows. Not event members — an ENH day would otherwise collapse every warning into one mega-event. **MCDs sit in between and go the watch route**: attached as `context` members to events overlapping their valid window (they have real product numbers and hours-scale windows), and additionally shown on the day view. ### Q4 — Multi-location → **per-location events** (as proposed) Matches the app's entire visibility model (`visible_location_filter` is per-location). Cross-location affinity is still cheap to *display* without a cross-location entity: the same lifecycle id appearing in several locations' events lets the day view badge "also affected N other locations" — purely derived, no new visibility semantics. ### Q5 — Manual merge/split → **defer past v2.0.0** A user-override layer that survives rebuilds contradicts the "derived and rebuildable, never source of truth" constraint from #22 and roughly doubles the design (override storage, conflict semantics on re-derivation, UI). Ship the automatic rules, see where they actually misgroup, and revisit with evidence. Cheap interim affordance: the event view can show *why* members grouped (gap distances, upgrade edges), which makes misgroupings diagnosable. ### Lightning (not in the original list, needs a rule) Archived clusters (#132) form **lightning episodes** per location via the same 60-minute gap rule. An episode overlapping an event's core interval attaches as a `context` member (it corroborates, it doesn't extend the event). An episode with no overlapping event forms its own lightning-only event so a dry-lightning day is still explorable. ### Interval edge cases - Never-cleared hazards (app down at expiry): interval ends at `expires`; if both `cleared_at` and `expires` are NULL, cap at `sent_at + 6 h` so one malformed record can't chain a whole day together. - Watch-only and lightning-only events carry an `event_kind` (`hazard` / `watch_only` / `lightning_only`) so the calendar can render them at different visual weight. If these land as-is, #135 implements: `weather_events` + `weather_event_members(kind, member_id, role∈{core,context})`, `EVENT_GAP_MINUTES=60`, `derivation_version=1`.
claude-bot commented 2026-08-01 03:15:16 +00:00
Author
Contributor
Copy link

✅ Decisions ratified (Ryan, 2026-08-01)

The proposal above is adopted as-is with one change: EVENT_GAP_MINUTES = 90 (not 60). Final answers to the checklist:

  • TOR warning inside SVR watch → one event; watch attached as context member. Watches never chain events; a watch with no warnings forms a watch-only event.
  • Lull duration → 90-minute gap rule over core members (W/Y/S significance), transitive closure; upgrade_from edges always merge. Constant baked into derivation_version; retuning = rebuild.
  • SPC outlooks → day-band context, not membership. MCDs attach as context members to overlapping events.
  • Multi-location → per-location events; cross-location affinity displayed via shared lifecycle ids ("also affected N locations"), no cross-location entity.
  • Manual merge/split → deferred past v2.0.0, tracked in #167.

Plus the additions from the proposal: lightning episodes (same gap rule, context when overlapping an event, else lightning-only events), interval caps for never-cleared records (expires, else sent_at + 6h), and event_kind ∈ {hazard, watch_only, lightning_only}.

#135 is unblocked and implements: weather_events + weather_event_members(kind, member_id, role ∈ {core, context}), EVENT_GAP_MINUTES = 90, derivation_version = 1.

## ✅ Decisions ratified (Ryan, 2026-08-01) The proposal above is adopted **as-is with one change: `EVENT_GAP_MINUTES = 90`** (not 60). Final answers to the checklist: - [x] TOR warning inside SVR watch → **one event; watch attached as `context` member.** Watches never chain events; a watch with no warnings forms a watch-only event. - [x] Lull duration → **90-minute gap rule** over core members (W/Y/S significance), transitive closure; `upgrade_from` edges always merge. Constant baked into `derivation_version`; retuning = rebuild. - [x] SPC outlooks → **day-band context, not membership.** MCDs attach as `context` members to overlapping events. - [x] Multi-location → **per-location events**; cross-location affinity displayed via shared lifecycle ids ("also affected N locations"), no cross-location entity. - [x] Manual merge/split → **deferred past v2.0.0**, tracked in #167. Plus the additions from the proposal: lightning episodes (same gap rule, `context` when overlapping an event, else lightning-only events), interval caps for never-cleared records (`expires`, else `sent_at + 6h`), and `event_kind ∈ {hazard, watch_only, lightning_only}`. #135 is unblocked and implements: `weather_events` + `weather_event_members(kind, member_id, role ∈ {core, context})`, `EVENT_GAP_MINUTES = 90`, `derivation_version = 1`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
renovate/python-minorpatch
renovate/cryptography-50.x
feat/event-export
feat/event-radar-loops
feat/explorer-event
feat/explorer-day
feat/explorer-calendar
feat/media-storage-abstraction
fix/allclear-image-url-space
fix/webex-permanent-failure
fix/incomplete-feed-guards
fix/false-all-clear-supersession
feat/radar-frames
feat/history-capture-ungate
renovate/cryptography-49.x
renovate/pillow-12.x
renovate/timezonefinder-8.x
v2.0.0-rc4
v2.0.0-rc3
v2.0.0-rc2
v2.0.0-rc1
v1.8.1
v1.8.0
v1.7.0
v1.6.2
v1.6.1
v1.6.0
v1.5.0
v1.4.0
v1.3.0
v1.2.0
v1.1.0
v1.0.3
v1.0.2
v1.0.1
v1.0.0
Labels
Clear labels   
area:ai
AI summarization

  
area:ci-cd
Continuous integration / deployment

  
area:notifications
Notification delivery & formatting

  
area:observability
Freshness, diagnostics, metrics, admin ops

  
area:public-pages
Public alert / SPC / link pages

  
backlog
Post-2.0 / not yet scheduled

  
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
type:decision
Requires a product/design decision before/with implementation

  
type:feature
New functionality

  
type:infra
CI/CD, build, deployment infrastructure

  
type:maintenance
Tech debt, deprecations, cleanup

  
type:security
Security / abuse mitigation

  
v1.0.1
Target release: v1.0.1 (maintenance & deprecations)

  
v1.1.0
Target release: v1.1.0 (operations & observability)

  
v1.2.0
Target release: v1.2.0 (notification formatting hardening)

  
v1.3.0
Target release: v1.3.0 (AI summarization expansion)

  
v2.0.0
Target release: v2.0.0 (Historical Weather Event Explorer)

  
wontfix
This won't be fixed

No labels
area:ai
area:ci-cd
area:notifications
area:observability
area:public-pages
backlog
bug
duplicate
enhancement
help wanted
invalid
question
type:decision
type:feature
type:infra
type:maintenance
type:security
v1.0.1
v1.1.0
v1.2.0
v1.3.0
v2.0.0
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/WeatherBot#136
No description provided.