rbrooks/Iris-WLED

Watch

Fork 0

Put the lights back when the controller reboots (#148) #175

Merged

claude-bot merged 3 commits from feat/148-watchdog into main

2026-09-26 10:53:50 +00:00

claude-bot commented

2026-09-26 10:43:02 +00:00

Collaborator

Closes #148.

Why

When a controller reboots (a GFCI trip, a storm, an OTA update), it comes back in its own boot preset. Iris didn't notice until the next transition, which could be hours later.

What (3 commits)

1. expected_state(now) (services/expected_state.py): what the lights should be showing at this moment.

It builds on #149's plan_for_date (which already carries #150's guard) and replays the night's transitions: on, ramps, late-night dim, overnight-off, morning-on and off.
A quick-push or /remote override is the expected state.
It returns Indeterminate during a ramp, a preview, within 120 s of a transition, with no location, or with no controller.
The jobs keep their behaviour, held by a parity test. It runs every job at its fire time against a recording DemoController across 8 configurations × approved, unapproved and no-event nights. After each job it asserts that the controller shows exactly expected_state, field for field. A separate test checks that the times it walks are the times schedule_today_jobs registers.
Two real fixes fell out of it (CHANGELOG, Fixed):
- The late-night dim's brightness baseline used the server's date.today(), so a 00:30 dim dimmed against the next night's event.
- The on job defaulted to the server's date rather than the configured timezone's.

2. last_known_good now means "retry when the controller is back". It always re-pushed the failed state anyway, and pushing the previous transition's state after a failed sunset push would put back last night's "off".

On reconnect it now pushes expected_state() for that moment, so a stale state can't overwrite a newer one.
The unused _last_known_good global is gone.
The settings label and docs are updated.

3. The watchdog (services/watchdog.py): an interval job, 5 minutes by default, on by default.

Reboot detection: info.uptime going backwards. The last uptime is stored per controller host in the settings table, so an Iris restart is never read as a controller reboot, but a shared power cut is detected.
Drift detection: on, bri (±2) and each managed segment's col/fx/sx/ix/pal.
Policy reboot_only (the default): corrects only after a reboot or a reconnect, and reports drift once per episode. It doesn't fight someone using the WLED app. enforce corrects all drift.
Stands aside while the retry or reconnect job is pending, or while the expected state is Indeterminate.
Reports through the state_corrected webhook (on by default) and state_drift (opt-in), the iris_watchdog_corrections_total{reason} counter, the iris_watchdog_last_check_timestamp_seconds and iris_watchdog_drift gauges, and a schedule_log row per correction.
Settings: a Watchdog card. The docs include the alert rule increase(iris_watchdog_corrections_total[1d]) > 3, which suggests a flaky circuit or PSU.

Verification

Backend: pytest 1145 passed; ruff and strict mypy are clean.
Frontend: lint, tsc, vitest (187 passed) and build.
E2E, run the CI way: 16/16, 3 of them new. With no controller configured, the watchdog was left to run at a 1-minute interval: no errors, no tracebacks, and no watchdog log lines.
Not verified on hardware: a real controller reboot (the #148 acceptance item "pull its power mid-evening"). That needs someone at the house.

Existing bugs noticed

iris_scheduler_last_success_timestamp_seconds is off by the UTC offset when TZ isn't UTC.
#165: some schedule settings don't reschedule when changed.
#169: Push now doesn't use the day plan. Under enforce, the watchdog undoes a Push now made outside lit hours; this is documented.
Without a ramp, an on time after late_night_time pushes full brightness; with a ramp it lands dimmed. The parity test holds both as they are.
A successful push doesn't cancel an older pending retry chain. The watchdog stands aside while one is pending.

🤖 Generated with Claude Code

Closes #148. ## Why When a controller reboots (a GFCI trip, a storm, an OTA update), it comes back in its own boot preset. Iris didn't notice until the next transition, which could be hours later. ## What (3 commits) **1. `expected_state(now)`** (`services/expected_state.py`): what the lights should be showing *at this moment*. - It builds on #149's `plan_for_date` (which already carries #150's guard) and replays the night's transitions: on, ramps, late-night dim, overnight-off, morning-on and off. - A quick-push or `/remote` override *is* the expected state. - It returns `Indeterminate` during a ramp, a preview, within 120 s of a transition, with no location, or with no controller. - **The jobs keep their behaviour, held by a parity test.** It runs every job at its fire time against a recording `DemoController` across 8 configurations × approved, unapproved and no-event nights. After each job it asserts that the controller shows exactly `expected_state`, field for field. A separate test checks that the times it walks are the times `schedule_today_jobs` registers. - **Two real fixes fell out of it** (CHANGELOG, Fixed): - The late-night dim's brightness baseline used the server's `date.today()`, so a 00:30 dim dimmed against the *next* night's event. - The on job defaulted to the server's date rather than the configured timezone's. **2. `last_known_good` now means "retry when the controller is back".** It always re-pushed the failed state anyway, and pushing the previous transition's state after a failed sunset push would put back last night's "off". - On reconnect it now pushes `expected_state()` for that moment, so a stale state can't overwrite a newer one. - The unused `_last_known_good` global is gone. - The settings label and docs are updated. **3. The watchdog** (`services/watchdog.py`): an interval job, 5 minutes by default, on by default. - **Reboot detection:** `info.uptime` going backwards. The last uptime is stored per controller host in the settings table, so an Iris restart is never read as a controller reboot, but a shared power cut is detected. - **Drift detection:** `on`, `bri` (±2) and each managed segment's `col`/`fx`/`sx`/`ix`/`pal`. - **Policy `reboot_only` (the default):** corrects only after a reboot or a reconnect, and reports drift once per episode. It doesn't fight someone using the WLED app. `enforce` corrects all drift. - **Stands aside** while the retry or reconnect job is pending, or while the expected state is Indeterminate. - **Reports** through the `state_corrected` webhook (on by default) and `state_drift` (opt-in), the `iris_watchdog_corrections_total{reason}` counter, the `iris_watchdog_last_check_timestamp_seconds` and `iris_watchdog_drift` gauges, and a `schedule_log` row per correction. - **Settings:** a Watchdog card. The docs include the alert rule `increase(iris_watchdog_corrections_total[1d]) > 3`, which suggests a flaky circuit or PSU. ## Verification - **Backend:** pytest 1145 passed; ruff and strict mypy are clean. - **Frontend:** lint, `tsc`, vitest (187 passed) and build. - **E2E, run the CI way:** 16/16, 3 of them new. With no controller configured, the watchdog was left to run at a 1-minute interval: no errors, no tracebacks, and no watchdog log lines. - **Not verified on hardware:** a real controller reboot (the #148 acceptance item "pull its power mid-evening"). That needs someone at the house. ## Existing bugs noticed - `iris_scheduler_last_success_timestamp_seconds` is off by the UTC offset when `TZ` isn't UTC. - #165: some schedule settings don't reschedule when changed. - #169: Push now doesn't use the day plan. Under `enforce`, the watchdog undoes a Push now made outside lit hours; this is documented. - Without a ramp, an on time after `late_night_time` pushes full brightness; with a ramp it lands dimmed. The parity test holds both as they are. - A successful push doesn't cancel an older pending retry chain. The watchdog stands aside while one is pending. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 3 commits

2026-09-26 10:43:02 +00:00

Work out what the lights should be showing at any moment (#148 ) 6bcbc4edaf

Add expected_state(now), built on the day plan from #149: it places the
night's plan on its envelope by replaying each transition the way its job
does (the dim skipping a dark controller, the on-ramp ceiling, the morning
window), and says when there is no single right answer (a ramp, a preview,
the two minutes either side of a transition, no controller or location).
A quick push is the expected state while it lasts. It also defines what
counts as matching it: only the fields Iris sets.

The jobs keep their own pushes. A parity test runs every job at its fire
time against a simulated controller and asserts the controller ends up
where expected_state says, across the morning window, both ramps, a dim
after midnight, a quick push, and approved, unapproved and no-event nights.

Two fixes fell out of it. The dim job read the showing scheme's brightness
for the calendar day, so a dim after midnight used the next night's event;
it now reads the night it belongs to, from the day plan. And the on job's
"today" is the configured timezone's rather than the server's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Make "last known good" retry what should be showing now (#148 ) 3f66ae3782

The last_known_good fallback promised the last state that was pushed
successfully, and a module variable tracked it, but nothing read it: the
reconnect watcher re-sent the push that had failed. For a scheduler that is
the right one to finish -- the last good push is the previous transition's,
so a failed sunset push would have been "recovered" with last night's off.

So the behaviour stays and its documented meaning becomes "retry when the
controller is back"; the unused variable is gone. On reconnect it now sends
expected_state for that moment, falling back to the failed push only when
that cannot be worked out, so a controller that returns after the dim gets
the dimmed scheme rather than the sunset one. The stored value keeps its
name, so existing settings are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Put the lights back when the controller reboots (#148 )

CI / Pre-commit hooks (pull_request) Successful in 50s

Details

CI / Dockerfile lint (pull_request) Successful in 8s

Details

CI / Alembic migration check (pull_request) Successful in 1m1s

Details

CI / Python lint & type-check (pull_request) Successful in 1m44s

Details

CI / Frontend lint, test & build (pull_request) Successful in 1m41s

Details

CI / Python tests (pull_request) Successful in 5m59s

Details

CI / Docker build, health smoke & E2E (pull_request) Successful in 2m5s

Details

b91cfae9dc

A controller that loses power -- a tripped GFCI, a storm, a firmware
update -- comes back in its own boot preset, and nothing noticed until the
next transition. A 3 a.m. brownout left the lights on all day.

A watchdog job now reads the controller back every watchdog_interval_min
minutes (default 5) and compares it with expected_state. A reboot is
info.uptime going down, remembered per controller host in memory and the
settings table, so an Iris restart is not read as one and a power cut that
takes Iris down too is still caught. A reconnect after an unreachable check
counts the same. Either is corrected through push_with_fallback. While the
controller is unreachable, or a retry or reconnect watcher is pending, the
watchdog leaves recovery to the push fallback.

watchdog_policy "reboot_only" (the default) reports any other difference
once per episode and leaves it -- a colour picked in the WLED app is not a
fault. "enforce" corrects it; "off" stops checking. New webhook events
state_corrected (on by default) and state_drift (opt-in), metrics
iris_watchdog_corrections_total{reason}, iris_watchdog_last_check_timestamp_seconds
and iris_watchdog_drift, a schedule_log row per correction, a Watchdog card
in Settings, and a demo-controller hook that simulates a reboot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude-bot merged commit cbf2b3b8ed into main

2026-09-26 10:53:50 +00:00

claude-bot deleted branch feat/148-watchdog

2026-09-26 10:53:50 +00:00

claude-bot referenced this pull request from a commit

2026-09-26 10:53:52 +00:00

Put the lights back when the controller reboots (#148) (#175)

claude-bot referenced this pull request

2026-09-26 10:54:00 +00:00

Controller watchdog: notice when the lights aren't showing what Iris pushed, and put them back #148