Expose Prometheus metrics for the things that fail quietly (#15) #124

Merged

claude-bot merged 1 commit from feat/15-metrics into main

2026-09-04 23:23:03 +00:00

claude-bot commented

2026-09-04 23:10:51 +00:00

Contributor

Closes #15.

GET /metrics, off unless METRICS_ENABLED. Counters for scheduler job outcomes and WLED push attempts, a histogram of job durations, and gauges for controller reachability, firmware support, AI usage against the cap, and the last successful run of each job. Scrape config, the full metric list and three alert rules are in docs/setup.md.

The metric that matters

iris_scheduler_last_success_timestamp_seconds{job}. Iris is set-and-forget, so the failure worth paging on is the nightly push that never happened — and that's an absence, not an error. Nothing is logged for a job that didn't run, so there's no line to alert on. A last-success timestamp turns it into something Prometheus can express:

- alert: IrisLightsDidNotComeOn
  expr: time() - iris_scheduler_last_success_timestamp_seconds{job="on_job"} > 129600

Related: a down controller reports iris_wled_up 0 rather than omitting the series. An absent series can't be alerted on with == 0 — it just stops matching, which is precisely the failure mode the metric exists to catch.

Instrumented at chokepoints, not call sites

Every scheduler job already funnels through log_run and every WLED write through _try_push, so the counters are incremented there — one place each. Sprinkling inc() calls next to each job would create a second accounting of the same events that can silently drift from the first.

Given this milestone has already produced three variants of "the test agreed with the code rather than reality" (leds.segs, the preset mock aliasing, the config round-trip), I wrote the drift check explicitly: it writes N schedule-log rows and asserts the counter moved by exactly N, exercising the real log_run rather than calling the recorder. Verified it fails when the instrumentation is removed — five tests go red.

State gauges are computed at scrape time from the controller status and schedule_log, the same sources the API reports from, so a gauge can't disagree with what the UI shows.

They're deliberately not derived counters: schedule_log is pruned after SCHEDULE_LOG_RETAIN_DAYS, and a counter that goes down breaks every rate() query built on it.

Auth

Unauthenticated when enabled, as every exporter is — Prometheus can't complete an OIDC flow, the same constraint the calendar feed hit in #16. The difference is that a feed exposes the user's calendar while this exposes counts, and nothing is labelled with a host, path or credential, so a token would add friction without protecting anything. Documented as "keep it on the container network."

Safety

Recording never raises. The instrumentation sits inside log_run and _try_push, and an exception there would take down a scheduled job for the sake of a counter. There's a test for that too.

prometheus-client uses a private registry rather than the global default, which would otherwise pad the output with process and GC collectors that say nothing about whether the lights came on.

Verification

572 backend tests pass (21 new), ruff, mypy clean
uvx pre-commit run --all-files passes
Drift check verified by mutation: removing record_job_run from log_run fails five tests

🤖 Generated with Claude Code

Closes #15. `GET /metrics`, off unless `METRICS_ENABLED`. Counters for scheduler job outcomes and WLED push attempts, a histogram of job durations, and gauges for controller reachability, firmware support, AI usage against the cap, and the last successful run of each job. Scrape config, the full metric list and three alert rules are in `docs/setup.md`. ### The metric that matters `iris_scheduler_last_success_timestamp_seconds{job}`. Iris is set-and-forget, so the failure worth paging on is the nightly push that **never happened** — and that's an *absence*, not an error. Nothing is logged for a job that didn't run, so there's no line to alert on. A last-success timestamp turns it into something Prometheus can express: ```yaml - alert: IrisLightsDidNotComeOn expr: time() - iris_scheduler_last_success_timestamp_seconds{job="on_job"} > 129600 ``` Related: a down controller reports `iris_wled_up 0` rather than omitting the series. An absent series can't be alerted on with `== 0` — it just stops matching, which is precisely the failure mode the metric exists to catch. ### Instrumented at chokepoints, not call sites Every scheduler job already funnels through `log_run` and every WLED write through `_try_push`, so the counters are incremented **there** — one place each. Sprinkling `inc()` calls next to each job would create a second accounting of the same events that can silently drift from the first. Given this milestone has already produced three variants of "the test agreed with the code rather than reality" (`leds.segs`, the preset mock aliasing, the config round-trip), I wrote the drift check explicitly: it writes N schedule-log rows and asserts the counter moved by exactly N, exercising the real `log_run` rather than calling the recorder. Verified it fails when the instrumentation is removed — five tests go red. State gauges are computed **at scrape time** from the controller status and `schedule_log`, the same sources the API reports from, so a gauge can't disagree with what the UI shows. They're deliberately *not* derived counters: `schedule_log` is pruned after `SCHEDULE_LOG_RETAIN_DAYS`, and a counter that goes down breaks every `rate()` query built on it. ### Auth Unauthenticated when enabled, as every exporter is — Prometheus can't complete an OIDC flow, the same constraint the calendar feed hit in #16. The difference is that a feed exposes the user's calendar while this exposes counts, and nothing is labelled with a host, path or credential, so a token would add friction without protecting anything. Documented as "keep it on the container network." ### Safety Recording never raises. The instrumentation sits *inside* `log_run` and `_try_push`, and an exception there would take down a scheduled job for the sake of a counter. There's a test for that too. `prometheus-client` uses a private registry rather than the global default, which would otherwise pad the output with process and GC collectors that say nothing about whether the lights came on. ### Verification - 572 backend tests pass (21 new), `ruff`, `mypy` clean - `uvx pre-commit run --all-files` passes - Drift check verified by mutation: removing `record_job_run` from `log_run` fails five tests 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit

2026-09-04 23:10:51 +00:00

Expose Prometheus metrics for the things that fail quietly (#15 )

CI / Dockerfile lint (pull_request) Successful in 9s

Details

CI / Python lint & type-check (pull_request) Successful in 1m2s

Details

CI / Pre-commit hooks (pull_request) Successful in 2m19s

Details

CI / Frontend lint, test & build (pull_request) Successful in 1m9s

Details

CI / Alembic migration check (pull_request) Successful in 1m58s

Details

CI / Python tests (pull_request) Successful in 2m41s

Details

CI / Docker build, health smoke & E2E (pull_request) Successful in 1m54s

Details

59786bc5db

GET /metrics, off unless METRICS_ENABLED. Counters for scheduler job
outcomes and WLED push attempts, a histogram of job durations, and gauges
for controller reachability, firmware support, AI usage against the cap,
and the last successful run of each job.

That last one is the point. Iris is set-and-forget, so the failure worth
paging on is the nightly push that never happened -- and that is an
absence, not an error: nothing is logged for a job that did not run, so
there is no line to alert on. A last-success timestamp turns it into
something Prometheus can express.

Instrumented at the two chokepoints every event already passes through --
log_run for scheduler jobs, _try_push for WLED writes -- rather than at
each call site. Sprinkling inc() calls next to each job would create a
second accounting of the same events that can silently drift from the
first, which is the failure this milestone has already hit three times in
other guises. A test writes N schedule-log rows and asserts the counter
moved by exactly N, and it fails when the instrumentation is removed.

State gauges are computed when Prometheus scrapes, reading the controller
status and schedule_log -- the same sources the API reports from -- rather
than being mirrored into variables that could disagree with the UI. They
are deliberately not derived counters: schedule_log is pruned after
SCHEDULE_LOG_RETAIN_DAYS, and a counter that goes down breaks every rate()
query built on it.

A down controller reports iris_wled_up 0 rather than omitting the series,
because an absent series cannot be alerted on with == 0; it simply stops
matching, which is the failure mode the metric exists to catch.

The endpoint is unauthenticated when enabled, as every exporter is:
Prometheus cannot complete an OIDC flow, the same constraint the calendar
feed hit in #16. Unlike a feed it exposes counts rather than data, and no
host, path or credential is labelled -- so a token would add friction
without protecting anything.

Recording never raises. Instrumentation sits inside log_run and _try_push,
and an exception there would take down a scheduled job for the sake of a
counter.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

claude-bot merged commit 5af94761dc into main

2026-09-04 23:23:03 +00:00

claude-bot deleted branch feat/15-metrics

2026-09-04 23:23:04 +00:00

claude-bot referenced this pull request

2026-09-04 23:23:16 +00:00

Prometheus /metrics endpoint & observability #15