v1.1.0 Phase 3: observability core #96

Merged

claude-bot merged 1 commit from feat/v1.1.0-phase3-observability into main

2026-07-18 17:27:39 +00:00

claude-bot commented

2026-07-18 17:21:31 +00:00

Contributor

The observability wave — the milestone's core value: making the app's health visible and its failures loud.

Backend

#73 — Prometheus /metrics: exports product freshness age, failure counts, up-state, and 24h event counts (NWS/SPC polls, notifier dispatch per channel, cache hit/miss) from existing observability data. Hand-rendered text format (no new dependency). Gated by METRICS_ENABLED; optional METRICS_TOKEN bearer.
#8 — Proactive stale/failing alerting: a scheduler job that alerts admins when a critical feed (nws_alerts, spc_fetch) stays stale/failing past STALE_PRODUCT_ALERT_MINUTES, debounced on state transitions.
#77 — Poller heartbeat / dead-man's-switch: a watchdog alerting when no successful NWS poll happened within HEARTBEAT_STALE_MINUTES, plus an optional external heartbeat ping (HEARTBEAT_PING_URL) after each good poll so an external monitor catches a fully-down app.
Shared admin_alerts.py: posts to ADMIN_ALERT_WEBHOOK when set (else logs), with in-process debounce so alerts don't spam.

UI

#74 — Degraded-data banner on the dashboard: "⚠ NWS data last updated N min ago" when a key feed is stale/failing, hidden when fresh — converts silent staleness into honest uncertainty for a safety app.
#9 — Freshness trend views on /diagnostics: 48h hourly-bucketed success-ratio sparklines per feed (dependency-free inline SVG) via a new admin-only /api/diagnostics/trends.

Config (added to `.env.example`)

METRICS_ENABLED, METRICS_TOKEN, ADMIN_ALERT_WEBHOOK, HEARTBEAT_STALE_MINUTES, HEARTBEAT_PING_URL, STALE_PRODUCT_ALERT_MINUTES — all optional with safe defaults (admin alerting/ping disabled until configured; metrics on, unauthenticated for homelab scraping).

Testing

Full suite green on the dev server: 458 passed (29 new tests) across metrics format/auth, heartbeat watchdog + debounce, stale-product alerting, admin-alert webhook/log fallback, degraded banner, and trend bucketing.

Notes

SSE client count isn't tracked anywhere, so it's intentionally omitted from /metrics rather than emitting a misleading constant.
The /metrics endpoint means external Alertmanager could also drive alerting; the in-app #8/#77 alerting is kept as the self-contained default.

Closes #73, #8, #77, #74, #9

🤖 Generated with Claude Code

The observability wave — the milestone's core value: making the app's health visible and its failures loud. ## Backend - **#73 — Prometheus `/metrics`**: exports product freshness age, failure counts, up-state, and 24h event counts (NWS/SPC polls, notifier dispatch per channel, cache hit/miss) from existing observability data. Hand-rendered text format (no new dependency). Gated by `METRICS_ENABLED`; optional `METRICS_TOKEN` bearer. - **#8 — Proactive stale/failing alerting**: a scheduler job that alerts admins when a critical feed (`nws_alerts`, `spc_fetch`) stays stale/failing past `STALE_PRODUCT_ALERT_MINUTES`, debounced on state transitions. - **#77 — Poller heartbeat / dead-man's-switch**: a watchdog alerting when no successful NWS poll happened within `HEARTBEAT_STALE_MINUTES`, plus an optional external heartbeat ping (`HEARTBEAT_PING_URL`) after each good poll so an external monitor catches a fully-down app. - Shared `admin_alerts.py`: posts to `ADMIN_ALERT_WEBHOOK` when set (else logs), with in-process debounce so alerts don't spam. ## UI - **#74 — Degraded-data banner** on the dashboard: "⚠ NWS data last updated N min ago" when a key feed is stale/failing, hidden when fresh — converts silent staleness into honest uncertainty for a safety app. - **#9 — Freshness trend views** on `/diagnostics`: 48h hourly-bucketed success-ratio **sparklines** per feed (dependency-free inline SVG) via a new admin-only `/api/diagnostics/trends`. ## Config (added to `.env.example`) `METRICS_ENABLED`, `METRICS_TOKEN`, `ADMIN_ALERT_WEBHOOK`, `HEARTBEAT_STALE_MINUTES`, `HEARTBEAT_PING_URL`, `STALE_PRODUCT_ALERT_MINUTES` — all optional with safe defaults (admin alerting/ping disabled until configured; metrics on, unauthenticated for homelab scraping). ## Testing Full suite green on the dev server: **458 passed** (29 new tests) across metrics format/auth, heartbeat watchdog + debounce, stale-product alerting, admin-alert webhook/log fallback, degraded banner, and trend bucketing. ## Notes - SSE client count isn't tracked anywhere, so it's intentionally omitted from `/metrics` rather than emitting a misleading constant. - The `/metrics` endpoint means external Alertmanager could also drive alerting; the in-app #8/#77 alerting is kept as the self-contained default. Closes #73, #8, #77, #74, #9 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit

2026-07-18 17:21:32 +00:00

Add observability core: /metrics, admin alerting, heartbeat, banner, trends

CI / test (pull_request) Successful in 3m23s

Details

5c7de27193

Backend:
- Prometheus /metrics endpoint (#73) exporting product freshness age, failure
  counts, up-state, and 24h event counts from the existing observability data;
  gated by METRICS_ENABLED and an optional METRICS_TOKEN bearer.
- Proactive stale/failing product alerting (#8): a scheduler job that notifies
  admins when a critical feed (nws_alerts, spc_fetch) stays stale/failing beyond
  STALE_PRODUCT_ALERT_MINUTES, debounced on state transitions.
- Poller heartbeat / dead-man's-switch (#77): a watchdog job that alerts when no
  successful NWS poll has happened within HEARTBEAT_STALE_MINUTES, plus an
  optional external heartbeat ping (HEARTBEAT_PING_URL) after each good poll.
- Shared admin-alert helper (app/services/admin_alerts.py): posts to
  ADMIN_ALERT_WEBHOOK when set, always logs, with in-process debounce.

UI:
- Degraded-data banner on the dashboard (#74): shows "NWS data last updated N
  min ago" when a key feed is stale/failing, hidden when fresh.
- Richer freshness trend views on /diagnostics (#9): 48h hourly-bucketed
  success-ratio sparklines per feed (dependency-free inline SVG) via a new
  admin-only /api/diagnostics/trends endpoint.

Full suite green on the dev server: 458 passed (29 new tests).

Closes #73, #8, #77, #74, #9

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>