Poller heartbeat / dead-man's-switch #77

New issue

Closed

opened 2026-07-18 05:31:39 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-18 05:31:39 +00:00

Contributor

The scariest failure for an alerting app isn't a wrong alert — it's silence nobody notices. If poll_alerts_job dies (or the whole app is down), no one is told; users assume no weather. Add a watchdog that alerts an admin channel when there's been no successful NWS poll in N minutes, plus an optional external ping (healthchecks.io-style) so an app that's fully down still trips an out-of-band alarm. Missed because "it just runs" — the failure is invisible until the day it matters. Effort S, value High. Pairs with I-03 (metrics) and I-04 (banner) — together they're the observability milestone's real point.

Filed from the 2026-07-17 codebase audit (docs/.internal/report-2026-07-17.md), finding I-07.

The scariest failure for an alerting app isn't a wrong alert — it's silence nobody notices. If `poll_alerts_job` dies (or the whole app is down), no one is told; users assume no weather. Add a watchdog that alerts an admin channel when there's been no successful NWS poll in N minutes, plus an optional external ping (healthchecks.io-style) so an app that's fully down still trips an out-of-band alarm. Missed because "it just runs" — the failure is invisible until the day it matters. Effort S, value High. Pairs with I-03 (metrics) and I-04 (banner) — together they're the observability milestone's real point. --- _Filed from the 2026-07-17 codebase audit (`docs/.internal/report-2026-07-17.md`), finding I-07._

claude-bot added this to the v1.1.0 - Operations, observability & CD milestone

2026-07-18 05:31:39 +00:00

claude-bot added the

area:observability

type:feature

labels

2026-07-18 05:31:39 +00:00

rbrooks referenced this issue from a commit

2026-07-18 17:21:18 +00:00

Add observability core: /metrics, admin alerting, heartbeat, banner, trends

claude-bot referenced this issue

2026-07-18 17:21:32 +00:00

v1.1.0 Phase 3: observability core #96

claude-bot commented

2026-07-18 17:27:57 +00:00

Author

Contributor

Done in #96 (merged). Two layers: (1) an in-app watchdog (heartbeat_watchdog_job) that alerts admins when there's been no successful NWS alert poll within HEARTBEAT_STALE_MINUTES (debounced); (2) an optional external heartbeat — after each successful poll it fire-and-forgets a GET to HEARTBEAT_PING_URL (healthchecks.io-style), so a fully-down app is caught out-of-band when the pings stop. Both disabled by default until configured.

Done in #96 (merged). Two layers: (1) an in-app watchdog (`heartbeat_watchdog_job`) that alerts admins when there's been no successful NWS alert poll within `HEARTBEAT_STALE_MINUTES` (debounced); (2) an optional external heartbeat — after each successful poll it fire-and-forgets a GET to `HEARTBEAT_PING_URL` (healthchecks.io-style), so a fully-down app is caught out-of-band when the pings stop. Both disabled by default until configured.

claude-bot closed this issue

2026-07-18 17:28:06 +00:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/WeatherBot#77

No description provided.

Rows
Columns