rbrooks/Iris-WLED

Watch

Fork 0

Prometheus /metrics endpoint & observability #15

New issue

Closed

opened 2026-07-07 16:52:23 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-07-07 16:52:23 +00:00

Contributor

Goal

Expose operational metrics so Iris can be monitored (and alerted on) in a homelab Prometheus/Grafana stack.

Why it's valuable

Iris is a set-and-forget scheduler; when a nightly push silently fails, the user wants an alert, not to notice dark lights. v1 has structured JSON logging and a schedule_log, but no scrape target.

Sketch

Add prometheus-client, expose GET /metrics (env-gated).
Metrics: push success/failure counters, WLED connection up/down gauge, scheduler job run durations, AI request counts vs. cap, last-successful-push timestamp.
Optional: a shipped Grafana dashboard JSON.

Acceptance criteria

/metrics exposes counters/gauges for pushes, scheduler jobs, WLED connectivity
Endpoint env-gated, off by default
Documented scrape config

Proposed enhancement (not in the original spec).

## Goal Expose operational metrics so Iris can be monitored (and alerted on) in a homelab Prometheus/Grafana stack. ## Why it's valuable Iris is a set-and-forget scheduler; when a nightly push silently fails, the user wants an alert, not to notice dark lights. v1 has structured JSON logging and a `schedule_log`, but no scrape target. ## Sketch - Add `prometheus-client`, expose `GET /metrics` (env-gated). - Metrics: push success/failure counters, WLED connection up/down gauge, scheduler job run durations, AI request counts vs. cap, last-successful-push timestamp. - Optional: a shipped Grafana dashboard JSON. ## Acceptance criteria - [ ] `/metrics` exposes counters/gauges for pushes, scheduler jobs, WLED connectivity - [ ] Endpoint env-gated, off by default - [ ] Documented scrape config Proposed enhancement (not in the original spec).

claude-bot added the

labels

2026-07-07 16:53:40 +00:00

claude-bot added this to the v1.1.0 milestone

2026-07-15 15:32:59 +00:00

rbrooks referenced this issue from a commit

2026-09-04 23:10:37 +00:00

Expose Prometheus metrics for the things that fail quietly (#15)

claude-bot referenced this issue from a pull request that will close it,

2026-09-04 23:10:51 +00:00

Expose Prometheus metrics for the things that fail quietly (#15) #124

claude-bot closed this issue

2026-09-04 23:23:03 +00:00

claude-bot referenced this issue from a commit

2026-09-04 23:23:05 +00:00

Expose Prometheus metrics for the things that fail quietly (#15)

claude-bot commented

2026-09-04 23:23:16 +00:00

Author

Contributor

Done — #124 merged, CI green.

/metrics exposes counters/gauges for pushes, scheduler jobs, WLED connectivity
Endpoint env-gated, off by default (METRICS_ENABLED)
Documented scrape config — plus the metric table and three alert rules, in docs/setup.md

The metric that matters

iris_scheduler_last_success_timestamp_seconds{job}. Iris is set-and-forget, so the failure worth paging on is the nightly push that never happened — and that is an absence, not an error. Nothing is logged for a job that did not run, so there is no line to alert on:

- alert: IrisLightsDidNotComeOn
  expr: time() - iris_scheduler_last_success_timestamp_seconds{job="on_job"} > 129600

Same reasoning made a down controller report iris_wled_up 0 rather than omitting the series. An absent series cannot be alerted on with == 0; it just stops matching, which is precisely the failure the metric exists to catch.

Instrumented at chokepoints

Every scheduler job already funnels through log_run and every WLED write through _try_push, so the counters increment there — one place each, rather than an inc() beside every job that would form a second accounting able to drift from the first.

Given this milestone has produced three variants of "the test agreed with the code rather than reality", I wrote the drift check explicitly: it writes N schedule-log rows and asserts the counter moved by exactly N, exercising the real log_run. Removing the instrumentation turns five tests red — checked, not assumed.

State gauges are computed at scrape time from the controller status and schedule_log, the same sources the API reports from, so a gauge cannot disagree with the UI. They are deliberately not derived counters: schedule_log is pruned after SCHEDULE_LOG_RETAIN_DAYS, and a counter that goes down breaks every rate() query built on it.

Auth

Unauthenticated when enabled, as every exporter is — Prometheus cannot complete an OIDC flow, the same constraint #16 hit. The difference from a feed is that this exposes counts rather than data, and nothing is labelled with a host, path or credential, so a token would add friction without protecting anything. Documented as "keep it on the container network".

Safety

Recording never raises. The instrumentation sits inside log_run and _try_push, where an exception would take down a scheduled job for the sake of a counter. The registry is private rather than the global default, which would otherwise pad the output with process and GC collectors that say nothing about whether the lights came on.

Done — #124 merged, CI green. - [x] **`/metrics` exposes counters/gauges for pushes, scheduler jobs, WLED connectivity** - [x] **Endpoint env-gated, off by default** (`METRICS_ENABLED`) - [x] **Documented scrape config** — plus the metric table and three alert rules, in `docs/setup.md` ### The metric that matters `iris_scheduler_last_success_timestamp_seconds{job}`. Iris is set-and-forget, so the failure worth paging on is the nightly push that **never happened** — and that is an *absence*, not an error. Nothing is logged for a job that did not run, so there is no line to alert on: ```yaml - alert: IrisLightsDidNotComeOn expr: time() - iris_scheduler_last_success_timestamp_seconds{job="on_job"} > 129600 ``` Same reasoning made a down controller report `iris_wled_up 0` rather than omitting the series. An absent series cannot be alerted on with `== 0`; it just stops matching, which is precisely the failure the metric exists to catch. ### Instrumented at chokepoints Every scheduler job already funnels through `log_run` and every WLED write through `_try_push`, so the counters increment **there** — one place each, rather than an `inc()` beside every job that would form a second accounting able to drift from the first. Given this milestone has produced three variants of "the test agreed with the code rather than reality", I wrote the drift check explicitly: it writes N schedule-log rows and asserts the counter moved by exactly N, exercising the real `log_run`. Removing the instrumentation turns five tests red — checked, not assumed. State gauges are computed at scrape time from the controller status and `schedule_log`, the same sources the API reports from, so a gauge cannot disagree with the UI. They are deliberately not *derived counters*: `schedule_log` is pruned after `SCHEDULE_LOG_RETAIN_DAYS`, and a counter that goes down breaks every `rate()` query built on it. ### Auth Unauthenticated when enabled, as every exporter is — Prometheus cannot complete an OIDC flow, the same constraint #16 hit. The difference from a feed is that this exposes counts rather than data, and nothing is labelled with a host, path or credential, so a token would add friction without protecting anything. Documented as "keep it on the container network". ### Safety Recording never raises. The instrumentation sits *inside* `log_run` and `_try_push`, where an exception would take down a scheduled job for the sake of a counter. The registry is private rather than the global default, which would otherwise pad the output with process and GC collectors that say nothing about whether the lights came on.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/Iris-WLED#15

No description provided.

Rows
Columns