Controller watchdog: notice when the lights aren't showing what Iris pushed, and put them back

claude-bot commented

2026-09-26 05:01:05 +00:00

Collaborator

Problem

Iris pushes state at transitions (on, dim, off, ramp steps, the morning window) and then assumes the controller keeps it. Nothing checks. When a controller loses power or reboots, which is routine for a roofline controller on an outdoor circuit, a GFCI trip or a storm, it comes back in its own boot preset, not in what Iris last pushed. Iris doesn't find out until the next scheduled transition, which can be hours later. The same applies after a firmware update, or after someone pulls the plug to move an inflatable.

For a set-and-forget tool, this is the most likely way the lights will be wrong on a given night, and today nothing reports it.

What exists (verified against main):

No job compares live state with expected state. run_dim_job and run_off_ramp_job call get_state() only to skip when on is false (services/schedule_service.py).
The push-failure paths don't cover this. They only react when a push fails (services/wled_push.py push_with_fallback). A reboot after a successful push is invisible to them.
last_known_good pushes the wrong state. The last_known_good policy's reconnect watcher re-pushes the state that failed, not the module's stored _last_known_good, which is set on success but only exposed via get_last_known_good(). The name promises something the code doesn't do; worth fixing or renaming here.
Nothing reads info.uptime. It is the cheapest reboot signal WLED offers.

Design

A single "what should be showing now" resolver comes first. Today that knowledge is spread across _push_today_scheme, get_active_times, _ramp_ceiling, quick_push_manager.get_status(), resolve_no_event_state and the envelope builder, and it is expressed by pushing, not by returning a state. Refactor it into expected_state(now) -> ExpectedState | Indeterminate, then make both the scheduler's pushes and the watchdog call it.

This is the lesson from #93 and #74: two code paths answering one question drift apart.
The refactor is most of the work, and it is worth doing for its own sake.

The watchdog job runs every WATCHDOG_INTERVAL_MIN minutes (default 5). It is an interval job, registered like aurora_poll, and on by default. On each tick it:

Reads /json/info (uptime) and /json/state. If the controller is unreachable, it records that and leaves recovery to the existing reconnect path, so there is no double handling.
Detects a reboot: uptime is lower than the last observed value. That is unambiguous, and it is the only case the watchdog corrects without question.
Detects drift: it compares only the fields Iris itself sets (on, bri, and each segment's col, fx, sx, ix, pal), with a small tolerance on bri.
Skips while a ramp is in flight (ramp.py steps legitimately move bri), a preview is active (preview_manager), or expected_state is indeterminate.
- A quick-push or /remote override is the expected state until it reverts, so compare against that.

Don't fight a human. The hard part is someone using the WLED app on purpose. If the watchdog reverted every manual change within 5 minutes, it would be infuriating. So there is a policy setting (runtime-settable):

reboot_only (default): correct only after a detected reboot, or a reachable-again transition. Report drift, but don't correct it.
enforce: correct any drift. Right for installs where nobody touches the controller.
off.

Report every correction:

A state_corrected webhook event, {reason: "reboot"|"drift", expected, observed_summary}.
iris_watchdog_corrections_total{reason} and iris_watchdog_last_check_timestamp_seconds metrics.
A schedule_log row.

Drift that isn't corrected (reboot_only) is logged and sent as state_drift, once per episode, not every tick.

Edge cases

Reboot outside the on-envelope: the expected state is "off", and WLED's boot preset may be on. Correct it. Otherwise a 3 a.m. brownout leaves the lights burning until sunset the next day.
Uptime wraps or resets for other reasons, such as a WLED OTA update. A reboot is a reboot; the correction is still right.
Several controllers (#13): the per-controller last-uptime must be keyed by controller from the start.
Demo mode: DemoController never reboots. The watchdog must still run against it for coverage. Add a test hook that simulates an uptime reset.

Acceptance

An expected_state(now) resolver exists, and the scheduler's own pushes use it. There is a test proving the scheduler's push and expected_state agree across the envelope: on, dim, ramp, morning window, off, quick push and no-event.
A simulated reboot (uptime reset plus the boot-preset state) is corrected within one interval, in and out of the envelope.
Manual drift in reboot_only mode is reported once and not corrected. In enforce mode it is corrected.
No corrections during ramps or previews.
Webhook events, metrics, docs (docs/setup.md alert rule: increase(iris_watchdog_corrections_total[1d]) > 3 suggests a flaky circuit) and a CHANGELOG entry.
Verified once on the real controller by pulling its power mid-evening.

## Problem Iris pushes state at transitions (on, dim, off, ramp steps, the morning window) and then assumes the controller keeps it. Nothing checks. When a controller loses power or reboots, which is routine for a roofline controller on an outdoor circuit, a GFCI trip or a storm, it comes back in **its own boot preset**, not in what Iris last pushed. Iris doesn't find out until the next scheduled transition, which can be hours later. The same applies after a firmware update, or after someone pulls the plug to move an inflatable. For a set-and-forget tool, this is the most likely way the lights will be wrong on a given night, and today nothing reports it. **What exists** (verified against `main`): - **No job compares live state with expected state.** `run_dim_job` and `run_off_ramp_job` call `get_state()` only to skip when `on` is false (`services/schedule_service.py`). - **The push-failure paths don't cover this.** They only react when a push *fails* (`services/wled_push.py` `push_with_fallback`). A reboot after a successful push is invisible to them. - **`last_known_good` pushes the wrong state.** The `last_known_good` policy's reconnect watcher re-pushes the state that *failed*, not the module's stored `_last_known_good`, which is set on success but only exposed via `get_last_known_good()`. The name promises something the code doesn't do; worth fixing or renaming here. - **Nothing reads `info.uptime`.** It is the cheapest reboot signal WLED offers. ## Design **A single "what should be showing now" resolver comes first.** Today that knowledge is spread across `_push_today_scheme`, `get_active_times`, `_ramp_ceiling`, `quick_push_manager.get_status()`, `resolve_no_event_state` and the envelope builder, and it is expressed by *pushing*, not by returning a state. Refactor it into `expected_state(now) -> ExpectedState | Indeterminate`, then make both the scheduler's pushes and the watchdog call it. - This is the lesson from #93 and #74: two code paths answering one question drift apart. - The refactor is most of the work, and it is worth doing for its own sake. **The watchdog job** runs every `WATCHDOG_INTERVAL_MIN` minutes (default 5). It is an interval job, registered like `aurora_poll`, and on by default. On each tick it: 1. Reads `/json/info` (uptime) and `/json/state`. If the controller is unreachable, it records that and leaves recovery to the existing reconnect path, so there is no double handling. 2. **Detects a reboot:** `uptime` is lower than the last observed value. That is unambiguous, and it is the only case the watchdog corrects without question. 3. **Detects drift:** it compares only the fields Iris itself sets (`on`, `bri`, and each segment's `col`, `fx`, `sx`, `ix`, `pal`), with a small tolerance on `bri`. 4. **Skips** while a ramp is in flight (`ramp.py` steps legitimately move `bri`), a preview is active (`preview_manager`), or `expected_state` is indeterminate. - A quick-push or `/remote` override *is* the expected state until it reverts, so compare against that. **Don't fight a human.** The hard part is someone using the WLED app on purpose. If the watchdog reverted every manual change within 5 minutes, it would be infuriating. So there is a policy setting (runtime-settable): - `reboot_only` (**default**): correct only after a detected reboot, or a reachable-again transition. Report drift, but don't correct it. - `enforce`: correct any drift. Right for installs where nobody touches the controller. - `off`. **Report every correction:** - A `state_corrected` webhook event, `{reason: "reboot"|"drift", expected, observed_summary}`. - `iris_watchdog_corrections_total{reason}` and `iris_watchdog_last_check_timestamp_seconds` metrics. - A `schedule_log` row. Drift that isn't corrected (`reboot_only`) is logged and sent as `state_drift`, once per episode, not every tick. ## Edge cases - **Reboot outside the on-envelope:** the expected state is "off", and WLED's boot preset may be *on*. Correct it. Otherwise a 3 a.m. brownout leaves the lights burning until sunset the next day. - **Uptime wraps or resets for other reasons,** such as a WLED OTA update. A reboot is a reboot; the correction is still right. - **Several controllers (#13):** the per-controller last-uptime must be keyed by controller from the start. - **Demo mode:** `DemoController` never reboots. The watchdog must still run against it for coverage. Add a test hook that simulates an uptime reset. ## Acceptance - [ ] An `expected_state(now)` resolver exists, and the scheduler's own pushes use it. There is a test proving the scheduler's push and `expected_state` agree across the envelope: on, dim, ramp, morning window, off, quick push and no-event. - [ ] A simulated reboot (uptime reset plus the boot-preset state) is corrected within one interval, in and out of the envelope. - [ ] Manual drift in `reboot_only` mode is reported once and not corrected. In `enforce` mode it is corrected. - [ ] No corrections during ramps or previews. - [ ] Webhook events, metrics, docs (`docs/setup.md` alert rule: `increase(iris_watchdog_corrections_total[1d]) > 3` suggests a flaky circuit) and a CHANGELOG entry. - [ ] Verified once on the real controller by pulling its power mid-evening.

claude-bot added this to the v1.2.0 milestone

2026-09-26 05:02:35 +00:00

claude-bot added the

labels

2026-09-26 05:02:41 +00:00

claude-bot referenced this issue

2026-09-26 05:03:09 +00:00

Review nudges: say which upcoming nights need approval, before they go dark #149

claude-bot referenced this issue

2026-09-26 05:03:10 +00:00

Vacation / away mode: make the house look lived-in while nobody's home #159

claude-bot referenced this issue

2026-09-26 05:03:11 +00:00

Schedule music-synced light shows on Falcon Player (FPP) / xLights #158

claude-bot referenced this issue

2026-09-26 05:03:13 +00:00

Switch non-pixel decorations on and off: inflatables, spotlights and projectors on smart plugs #156

claude-bot referenced this issue

2026-09-26 05:04:08 +00:00

Energy and cost report: what the lights actually used this season #153

claude-bot referenced this issue

2026-09-26 05:04:09 +00:00

Installable app with push notifications (PWA) #154

claude-bot referenced this issue

2026-09-26 05:04:10 +00:00

Design: make a scheme describe intent, so it can drive devices other than WLED #155

claude-bot referenced this issue

2026-09-26 07:46:41 +00:00

Review nudges: say which upcoming nights need approval, before they go dark #149

rbrooks referenced this issue from a commit

2026-09-26 08:31:46 +00:00

Decide each night through one shared day plan (#149)

claude-bot referenced this issue

2026-09-26 08:32:02 +00:00

Warn before an unapproved night goes dark (#149) #162

claude-bot commented

2026-09-26 09:28:18 +00:00

Author

Collaborator

Picking this up now that #149 (day_plan) and #150 (the guard in the plan) are merged.

expected_state(now) is built on top of plan_for_date. The plan answers "what for this night"; the new layer adds "what at this moment": the envelope (on, dim, overnight-off, morning-on, off), ramps, quick-push and /remote overrides, and previews. The on, dim and off jobs will push what expected_state says at their fire time, with a parity test, so the watchdog and the scheduler can't disagree.

Default policy is reboot_only, as the issue body says. The last_known_good naming mismatch gets fixed here too.

Picking this up now that #149 (`day_plan`) and #150 (the guard in the plan) are merged. **`expected_state(now)` is built on top of `plan_for_date`.** The plan answers "what for this night"; the new layer adds "what at this moment": the envelope (on, dim, overnight-off, morning-on, off), ramps, quick-push and `/remote` overrides, and previews. The on, dim and off jobs will push what `expected_state` says at their fire time, with a parity test, so the watchdog and the scheduler can't disagree. **Default policy is `reboot_only`,** as the issue body says. The `last_known_good` naming mismatch gets fixed here too.

rbrooks referenced this issue from a commit

2026-09-26 10:42:45 +00:00

Work out what the lights should be showing at any moment (#148)

rbrooks referenced this issue from a commit

2026-09-26 10:42:45 +00:00

Make "last known good" retry what should be showing now (#148)

rbrooks referenced this issue from a commit

2026-09-26 10:42:45 +00:00

Put the lights back when the controller reboots (#148)

claude-bot referenced this issue from a pull request that will close it,

2026-09-26 10:43:02 +00:00

Put the lights back when the controller reboots (#148) #175

claude-bot referenced this issue

2026-09-26 10:43:17 +00:00

iris_scheduler_last_success_timestamp_seconds is off by the UTC offset when TZ isn't UTC #176

claude-bot closed this issue

2026-09-26 10:53:50 +00:00

claude-bot referenced this issue from a commit

2026-09-26 10:53:52 +00:00

Put the lights back when the controller reboots (#148) (#175)

claude-bot commented

2026-09-26 10:54:00 +00:00

Author

Collaborator

Done in #175 (squash-merged, CI green).

Delivered:

services/expected_state.py: expected_state(now), built on the day plan, and differences(), the comparator.
Held to the jobs by a parity test. It runs every job at its fire time against a recording DemoController across 8 configurations × approved, unapproved and no-event nights, and asserts that the controller then shows exactly expected_state.
last_known_good now means "retry when the controller is back", and on reconnect it pushes what should be showing now. The unused global is gone.
The watchdog: an interval job (5 minutes, on by default).
- Reboots are detected through info.uptime, persisted per host, so an Iris restart isn't mistaken for a reboot but a shared power cut is caught.
- Drift is checked on on, bri (±2) and each segment's col/fx/sx/ix/pal.
- Policies: reboot_only (the default), enforce and off.
- Reports through the state_corrected and state_drift webhooks and the iris_watchdog_* metrics.
Fixed along the way: the dim baseline and the on job used the server's date rather than the configured timezone's.

Still open, needs hardware: pull the roofline controller's power mid-evening and confirm it's corrected within one interval.

Noted, filed or already tracked:

#176: the metrics timestamp is off by the UTC offset.
#165: some settings don't reschedule.
#169: Push now bypasses the day plan.
Not filed: a successful push doesn't cancel an older pending retry chain. The watchdog stands aside while one is pending.

Done in #175 (squash-merged, CI green). **Delivered:** - **`services/expected_state.py`:** `expected_state(now)`, built on the day plan, and `differences()`, the comparator. - **Held to the jobs by a parity test.** It runs every job at its fire time against a recording `DemoController` across 8 configurations × approved, unapproved and no-event nights, and asserts that the controller then shows exactly `expected_state`. - **`last_known_good` now means "retry when the controller is back",** and on reconnect it pushes what should be showing *now*. The unused global is gone. - **The watchdog:** an interval job (5 minutes, on by default). - Reboots are detected through `info.uptime`, persisted per host, so an Iris restart isn't mistaken for a reboot but a shared power cut is caught. - Drift is checked on `on`, `bri` (±2) and each segment's `col`/`fx`/`sx`/`ix`/`pal`. - Policies: `reboot_only` (the default), `enforce` and `off`. - Reports through the `state_corrected` and `state_drift` webhooks and the `iris_watchdog_*` metrics. - **Fixed along the way:** the dim baseline and the on job used the server's date rather than the configured timezone's. **Still open, needs hardware:** pull the roofline controller's power mid-evening and confirm it's corrected within one interval. **Noted, filed or already tracked:** - #176: the metrics timestamp is off by the UTC offset. - #165: some settings don't reschedule. - #169: Push now bypasses the day plan. - Not filed: a successful push doesn't cancel an older pending retry chain. The watchdog stands aside while one is pending.

claude-bot referenced this issue

2026-09-26 11:07:25 +00:00

Energy and cost report: what the lights actually used this season #153

rbrooks referenced this issue from a commit

2026-09-26 11:18:38 +00:00

Get Iris's alerts on your phone as notifications (#154)

rbrooks referenced this issue from a commit

2026-09-26 11:18:38 +00:00

Notify on watchdog corrections, and add a Season shortcut (#154)

claude-bot referenced this issue

2026-09-26 11:18:57 +00:00

Install Iris on a phone and get its alerts there (#154) #178