Radar velocity + derived rotation/hail products (BV/SRV, NROT, MESH, VIL) #128

New issue

Open

opened 2026-07-27 19:55:28 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-07-27 19:55:28 +00:00

Contributor

Parked, not forgotten. Originally filed to capture the v1.7.0 CHANGELOG.md "Notes" entry, which recorded this reasoning but had no tracking issue. Substantially revised 2026-07-27 after a design pass — see "What changed" at the bottom.

Not included in v1.7.0

Velocity — Base Velocity (BV) and Storm-Relative Velocity (SRV)
Derived rotation / hail products — NROT (normalized rotation), MESH (maximum estimated size of hail), VIL (vertically integrated liquid)

This is really two problems, not one

Derived gridded products (MESH, VIL, NROT) — tractable

These are not per-radar polar data. They come from MRMS (Multi-Radar Multi-Sensor), which already mosaics ~160 radars onto a regular CONUS lat/lon grid (~0.01°, ~2-minute cadence) and publishes GRIB2. There is no polar-to-cartesian problem: decode GRIB2 → reproject to Web Mercator → apply a color table → emit PNG. Standard cfgrib/pygrib + rasterio + PIL work.

Note a correction to the original framing: MESH and NROT are MRMS products, not NEXRAD Level III. Level III has its own older hail index products, but "MESH" as normally meant is MRMS. VIL exists in both. v1.7.0 already added MRMS precipitation accumulation as a layer, so MRMS is a known source in this codebase.

Velocity (BV/SRV) — the genuinely hard half

Doppler velocity is measured relative to the emitting radar. A CONUS velocity mosaic is meteorologically meaningless — you cannot average two radars viewing the same storm from different angles. That, not licensing, is the actual reason no velocity tile service exists anywhere. It is not an oversight to be routed around.

So velocity must be rendered per site. That happens to fit this app's model well: Location.nearest_radar already stores the site id, resolved at location creation.

If built from scratch, per site: fetch Level II from the public AWS open-data bucket (noaa-nexrad-level2, ~5–10 MB per volume scan, new scan every ~4–6 min in precip mode) → decode (MetPy, or Py-ART for more capability at the cost of a large scientific stack) → select the lowest elevation sweep → dealias (raw Doppler folds at the Nyquist limit, so a strong outbound wind wraps and displays as inbound — the step that makes naive velocity rendering actively misleading rather than merely ugly) → project polar to Web Mercator accounting for beam geometry and earth curvature → colorize to GRLevelX/NWS conventions → cache and serve.

Two shortcuts if ever built here: prefer Level III over Level II (the NWS RPG has already dealiased it and already computed storm-relative velocity using its own storm motion vector, so SRV comes free), and do not build a tile pyramid — radar.py already fetches one PNG per location and disk-caches it, so render one image per location per product and use a Leaflet ImageOverlay rather than a TileLayer.

The binding constraint if built in-process

FastAPI, APScheduler, the Discord bot and the Matrix client all share one asyncio event loop. Radial decode and rasterization are CPU-bound work measured in seconds. Running that in-process would stall the event loop and delay alert polling — the app's life-safety function — to make a picture prettier. That trade is not acceptable. It would need a ProcessPoolExecutor or, better, a separate service.

However: `rbrooks/Radome` already does all of this

Radome — a self-hosted NEXRAD Level II viewer — already implements every product this issue asks for. It should be treated as an integration target, not a thing to rebuild.

From specs/radome-spec-03-api.md, the tile API serves per-tilt products:

GET /api/tiles/{site_id}/{scan_id}/{product}/{tilt}.png — products REF, VEL, SRM, SW, ZDR, CC, PHI, KDP, HC, RR, NROT, AZDIV

and multi-tilt aggregates:

GET /api/tiles/{site_id}/{scan_id}/{product}.png — CREF, VIL, VILD, ET18, ET30, ET50

plus MRMS: GET /api/mrms/{product}/{timestamp}.png, with MRMS_MESH named explicitly in the spec's example response.

Mapping this issue's wish list onto what already exists:

Wanted here	Radome product
Base Velocity (BV)	`VEL`
Storm-Relative Velocity (SRV)	`SRM` — and Radome has storm-motion vector CRUD, incl. auto-populate from NWS Level 3 STI, which is exactly what SRV needs
NROT	`NROT` (per-tilt)
MESH	`MRMS_MESH`
VIL	`VIL` / `VILD`

The output contract is also already the right shape. Tiles are transparent RGBA PNGs served with an X-Radar-Bounds: {"north":…,"south":…,"east":…,"west":…} header and Cache-Control: public, max-age=3600, plus a metadata-only variant at …/{tilt}/meta returning bounds and URL. That is precisely a Leaflet ImageOverlay contract, and this dashboard is already Leaflet — so client-side integration is close to free wherever auth permits.

Real blockers for integration

Capacity — the actual reason this is still parked. Per specs/radome-spec-06-render-latency.md, on a 4-core host a single site's 12 sweep renders cost ~380s sequentially; with PARALLEL_SWEEPS=3 / KDT_WORKERS=2 that drops to ~160s, giving ~2 min to first partial tile and ~4 min to a complete scan. At a ~4–6 minute scan cadence, one site roughly saturates a 4-core box continuously. Radome is architected around PRIMARY_SITE + SECONDARY_SITES with an on-demand queue — not all ~160 sites. WeatherBot locations can span many radars. This is the same server-capacity constraint that has Radome paused.
Auth model mismatch. Radome implements no auth of its own — it sits behind Caddy + Authentik forward auth and only reads X-Authentik-Username for audit. WeatherBot also uses Authentik, but server-side WeatherBot→Radome calls would need to traverse forward auth (service token or bypass rule), and browser-side embedding needs the viewer's Authentik session to cover Radome's origin. Public pages could never show these products — they are unauthenticated by definition.
Coverage. Only sites Radome actually ingests. Matching is trivial (Location.nearest_radar is already the site id); the question is purely whether that site is being rendered.
Latency vs. the alert path. WeatherBot polls alerts every 60s; Radome needs ~2 min to a first partial tile and ~4 min to a complete scan. Fine for a dashboard overlay. Awkward for radar-at-alert snapshots or notification attachments, which would lag the alert they illustrate.

Caveat: the above is read from Radome's specs. Spec 06 cites production timing from 2026-05-18 logs against commit ca72619, so the core render path is demonstrably real, but several spec sections are marked pre-Phase-3 or future (on-demand worker for uncached sites, B2 archive replay). Verify implementation state against the code before planning against any of it.

Recommended path

Do not build a radar-rendering pipeline in WeatherBot. Radome already is one, and a second would be strictly worse.
Unblocking this is a Radome capacity question, not a WeatherBot feature question. It stays parked until Radome is unpaused and can render the sites WeatherBot's locations need.
When that happens, scope the integration as: a RADOME_BASE_URL setting, a product picker entry, and a Leaflet ImageOverlay fed by X-Radar-Bounds — small, given the contract already fits.
Decide the auth traversal (service token vs. forward-auth bypass for a server-side path) before any code.
Keep these products authenticated-only; they cannot appear on public pages.
Separately and independently: MRMS-derived gridded products (MESH/VIL/rotation) could be sourced without Radome if a tile/WMS service for them turns up — worth a check of IEM's services, since v1.7.0 already consumes both IEM and MRMS.

What changed

The original version of this issue concluded that these products would require building a dedicated radar-rendering pipeline, on the basis that no free web-tile source exists and premium Weather Pulse data is desktop GRLevelX format. Both those facts still hold. What the design pass added is that rbrooks/Radome already implements every one of these products, so the question is no longer "build a pipeline?" but "integrate, once Radome has the capacity" — a much smaller and differently-shaped problem.

**Parked, not forgotten.** Originally filed to capture the v1.7.0 `CHANGELOG.md` "Notes" entry, which recorded this reasoning but had no tracking issue. Substantially revised 2026-07-27 after a design pass — see "What changed" at the bottom. ## Not included in v1.7.0 - **Velocity** — Base Velocity (BV) and Storm-Relative Velocity (SRV) - **Derived rotation / hail products** — NROT (normalized rotation), MESH (maximum estimated size of hail), VIL (vertically integrated liquid) ## This is really two problems, not one ### Derived gridded products (MESH, VIL, NROT) — tractable These are not per-radar polar data. They come from **MRMS** (Multi-Radar Multi-Sensor), which already mosaics ~160 radars onto a regular CONUS lat/lon grid (~0.01°, ~2-minute cadence) and publishes GRIB2. There is no polar-to-cartesian problem: decode GRIB2 → reproject to Web Mercator → apply a color table → emit PNG. Standard `cfgrib`/`pygrib` + `rasterio` + PIL work. Note a correction to the original framing: **MESH and NROT are MRMS products, not NEXRAD Level III.** Level III has its own older hail index products, but "MESH" as normally meant is MRMS. VIL exists in both. v1.7.0 already added MRMS precipitation accumulation as a layer, so MRMS is a known source in this codebase. ### Velocity (BV/SRV) — the genuinely hard half Doppler velocity is measured **relative to the emitting radar**. A CONUS velocity mosaic is meteorologically meaningless — you cannot average two radars viewing the same storm from different angles. **That, not licensing, is the actual reason no velocity tile service exists anywhere.** It is not an oversight to be routed around. So velocity must be rendered per site. That happens to fit this app's model well: `Location.nearest_radar` already stores the site id, resolved at location creation. If built from scratch, per site: fetch Level II from the public AWS open-data bucket (`noaa-nexrad-level2`, ~5–10 MB per volume scan, new scan every ~4–6 min in precip mode) → decode (`MetPy`, or `Py-ART` for more capability at the cost of a large scientific stack) → select the lowest elevation sweep → **dealias** (raw Doppler folds at the Nyquist limit, so a strong outbound wind wraps and displays as inbound — the step that makes naive velocity rendering actively misleading rather than merely ugly) → project polar to Web Mercator accounting for beam geometry and earth curvature → colorize to GRLevelX/NWS conventions → cache and serve. Two shortcuts if ever built here: prefer **Level III over Level II** (the NWS RPG has already dealiased it and already computed storm-relative velocity using its own storm motion vector, so SRV comes free), and **do not build a tile pyramid** — `radar.py` already fetches one PNG per location and disk-caches it, so render one image per location per product and use a Leaflet `ImageOverlay` rather than a `TileLayer`. ## The binding constraint if built in-process FastAPI, APScheduler, the Discord bot and the Matrix client all share **one asyncio event loop**. Radial decode and rasterization are CPU-bound work measured in seconds. Running that in-process would stall the event loop and delay alert polling — the app's life-safety function — to make a picture prettier. That trade is not acceptable. It would need a `ProcessPoolExecutor` or, better, a separate service. --- ## However: `rbrooks/Radome` already does all of this **Radome — a self-hosted NEXRAD Level II viewer — already implements every product this issue asks for.** It should be treated as an integration target, not a thing to rebuild. From `specs/radome-spec-03-api.md`, the tile API serves per-tilt products: `GET /api/tiles/{site_id}/{scan_id}/{product}/{tilt}.png` — products `REF`, **`VEL`**, **`SRM`**, `SW`, `ZDR`, `CC`, `PHI`, `KDP`, `HC`, `RR`, **`NROT`**, `AZDIV` and multi-tilt aggregates: `GET /api/tiles/{site_id}/{scan_id}/{product}.png` — `CREF`, **`VIL`**, `VILD`, `ET18`, `ET30`, `ET50` plus MRMS: `GET /api/mrms/{product}/{timestamp}.png`, with `MRMS_MESH` named explicitly in the spec's example response. Mapping this issue's wish list onto what already exists: | Wanted here | Radome product | |---|---| | Base Velocity (BV) | `VEL` | | Storm-Relative Velocity (SRV) | `SRM` — and Radome has storm-motion vector CRUD, incl. auto-populate from NWS Level 3 STI, which is exactly what SRV needs | | NROT | `NROT` (per-tilt) | | MESH | `MRMS_MESH` | | VIL | `VIL` / `VILD` | **The output contract is also already the right shape.** Tiles are transparent RGBA PNGs served with an `X-Radar-Bounds: {"north":…,"south":…,"east":…,"west":…}` header and `Cache-Control: public, max-age=3600`, plus a metadata-only variant at `…/{tilt}/meta` returning bounds and URL. That is precisely a Leaflet `ImageOverlay` contract, and this dashboard is already Leaflet — so client-side integration is close to free wherever auth permits. ### Real blockers for integration 1. **Capacity — the actual reason this is still parked.** Per `specs/radome-spec-06-render-latency.md`, on a 4-core host a single site's 12 sweep renders cost ~380s sequentially; with `PARALLEL_SWEEPS=3` / `KDT_WORKERS=2` that drops to ~160s, giving ~2 min to first partial tile and ~4 min to a complete scan. At a ~4–6 minute scan cadence, **one site roughly saturates a 4-core box continuously.** Radome is architected around `PRIMARY_SITE` + `SECONDARY_SITES` with an on-demand queue — not all ~160 sites. WeatherBot locations can span many radars. This is the same server-capacity constraint that has Radome paused. 2. **Auth model mismatch.** Radome implements no auth of its own — it sits behind Caddy + Authentik forward auth and only reads `X-Authentik-Username` for audit. WeatherBot also uses Authentik, but server-side WeatherBot→Radome calls would need to traverse forward auth (service token or bypass rule), and browser-side embedding needs the viewer's Authentik session to cover Radome's origin. **Public pages could never show these products** — they are unauthenticated by definition. 3. **Coverage.** Only sites Radome actually ingests. Matching is trivial (`Location.nearest_radar` is already the site id); the question is purely whether that site is being rendered. 4. **Latency vs. the alert path.** WeatherBot polls alerts every 60s; Radome needs ~2 min to a first partial tile and ~4 min to a complete scan. Fine for a dashboard overlay. Awkward for radar-at-alert snapshots or notification attachments, which would lag the alert they illustrate. Caveat: the above is read from Radome's **specs**. Spec 06 cites production timing from 2026-05-18 logs against commit `ca72619`, so the core render path is demonstrably real, but several spec sections are marked pre-Phase-3 or future (on-demand worker for uncached sites, B2 archive replay). Verify implementation state against the code before planning against any of it. ## Recommended path - [ ] **Do not build a radar-rendering pipeline in WeatherBot.** Radome already is one, and a second would be strictly worse. - [ ] Unblocking this is a **Radome capacity question**, not a WeatherBot feature question. It stays parked until Radome is unpaused and can render the sites WeatherBot's locations need. - [ ] When that happens, scope the integration as: a `RADOME_BASE_URL` setting, a product picker entry, and a Leaflet `ImageOverlay` fed by `X-Radar-Bounds` — small, given the contract already fits. - [ ] Decide the auth traversal (service token vs. forward-auth bypass for a server-side path) before any code. - [ ] Keep these products **authenticated-only**; they cannot appear on public pages. - [ ] Separately and independently: MRMS-derived gridded products (MESH/VIL/rotation) could be sourced without Radome if a tile/WMS service for them turns up — worth a check of IEM's services, since v1.7.0 already consumes both IEM and MRMS. ## What changed The original version of this issue concluded that these products would require building a dedicated radar-rendering pipeline, on the basis that no free web-tile source exists and premium Weather Pulse data is desktop GRLevelX format. Both those facts still hold. What the design pass added is that **`rbrooks/Radome` already implements every one of these products**, so the question is no longer "build a pipeline?" but "integrate, once Radome has the capacity" — a much smaller and differently-shaped problem.

claude-bot added this to the Backlog (post-2.0, unscheduled) milestone

2026-07-27 19:55:33 +00:00

claude-bot added the

backlog

type:feature

labels

2026-07-27 19:55:39 +00:00

claude-bot referenced this issue

2026-07-28 04:54:17 +00:00

Media storage abstraction (disk backend now, object storage later) #131

claude-bot referenced this issue

2026-07-28 05:27:02 +00:00