Freshness banner permanently degraded: SPC day 4–8 text fetch targets nonexistent SWODY4–8 products #161

New issue

Closed

opened 2026-08-01 01:25:47 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-08-01 01:25:47 +00:00

Contributor

Symptom (reported on prod, 2026-08-01): the dashboard shows a data-freshness warning even though the last update was one minute ago.

Root cause

app/services/nws.py fetches SPC outlook narrative text per day for days 1–8 using _SPC_OUTLOOK_AWIPS_IDS = {…, 4: "SWODY4", …, 8: "SWODY8"} (nws.py:686). SPC does not issue per-day text products for days 4–8 — it issues one combined product, AWIPS SWOD48 (WMO ACUS48). Verified against the live NWS API tonight: recent SWO products carry only SWODY1, SWODY2, SWODY3, and SWOD48.

So spc:text_swody4–spc:text_swody8 have never succeeded. Prod log tonight:

Freshness update: category=spc_fetch key=spc:text_swody8 success=False failure_count=25986
Freshness update: category=spc_fetch key=spc:text_swody5 success=False failure_count=25991

~26k consecutive failures at the 5-minute cadence ≈ 90 days — this predates the registry migration by months; the migration just prompted a close look at the dashboard.

Why the banner contradicts itself

dashboard_freshness_summary() (observability.py:225) marks the whole spc_fetch category unhealthy when any row has failure_count > 0, but displays the category-wide max(last_success_at) — which is minutes old thanks to the healthy day-1/2/3 keys. Hence "degraded" next to "last update 1 min ago".

Suggested fix

Map days 4–8 to the combined product: fetch SWOD48 (AWIPS SWOD48, WMO ACUS48) once and reuse it for any day-4–8 context; track it under a single freshness key (e.g. spc:text_swod48).
Update ai_context.py:151 key derivation to match.
Clean up the five dead spc:text_swody4..8 ProductFreshness rows (or they trip the banner until deleted — no fetch will ever reset them once the keys are retired).
Optionally: make the banner's detail line name the failing key so "degraded" and "last update" can't read as contradictory.

Source: post-migration prod verification; investigation in Claude session 2026-08-01.

**Symptom (reported on prod, 2026-08-01):** the dashboard shows a data-freshness warning even though the last update was one minute ago. ## Root cause `app/services/nws.py` fetches SPC outlook narrative text per day for days 1–8 using `_SPC_OUTLOOK_AWIPS_IDS = {…, 4: "SWODY4", …, 8: "SWODY8"}` (nws.py:686). **SPC does not issue per-day text products for days 4–8** — it issues one combined product, AWIPS `SWOD48` (WMO `ACUS48`). Verified against the live NWS API tonight: recent `SWO` products carry only `SWODY1`, `SWODY2`, `SWODY3`, and `SWOD48`. So `spc:text_swody4`–`spc:text_swody8` have **never succeeded**. Prod log tonight: ``` Freshness update: category=spc_fetch key=spc:text_swody8 success=False failure_count=25986 Freshness update: category=spc_fetch key=spc:text_swody5 success=False failure_count=25991 ``` ~26k consecutive failures at the 5-minute cadence ≈ 90 days — this predates the registry migration by months; the migration just prompted a close look at the dashboard. ## Why the banner contradicts itself `dashboard_freshness_summary()` (observability.py:225) marks the whole `spc_fetch` category unhealthy when *any* row has `failure_count > 0`, but displays the category-wide `max(last_success_at)` — which is minutes old thanks to the healthy day-1/2/3 keys. Hence "degraded" next to "last update 1 min ago". ## Suggested fix 1. Map days 4–8 to the combined product: fetch `SWOD48` (AWIPS `SWOD48`, WMO `ACUS48`) once and reuse it for any day-4–8 context; track it under a single freshness key (e.g. `spc:text_swod48`). 2. Update `ai_context.py:151` key derivation to match. 3. Clean up the five dead `spc:text_swody4..8` `ProductFreshness` rows (or they trip the banner until deleted — no fetch will ever reset them once the keys are retired). 4. Optionally: make the banner's `detail` line name the failing key so "degraded" and "last update" can't read as contradictory. Source: post-migration prod verification; investigation in Claude session 2026-08-01.