iris_scheduler_last_success_timestamp_seconds is off by the UTC offset when TZ isn't UTC #176

New issue

Open

opened 2026-09-26 10:43:17 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-09-26 10:43:17 +00:00

Collaborator

Bug

metrics._schedule_metrics calls .timestamp() on schedule_log.triggered_at, which is stored as naive UTC. Python interprets a naive datetime's .timestamp() as local time. Since #147, TZ actually reaches the container, so with TZ=America/Chicago the gauge reads 5 or 6 hours off.

This is the metric the recommended alert is built on (IrisLightsDidNotComeOn, docs/setup.md §8: time() - iris_scheduler_last_success_timestamp_seconds{job="on_job"} > 129600). An offset in the wrong direction makes that alert fire hours early or late.

test_metrics.py currently asserts the buggy behaviour. Found while building #148, whose new gauge uses UTC correctly.

Fix

Use triggered_at.replace(tzinfo=UTC).timestamp() (or store tz-aware values). Update test_metrics.py to run under a non-UTC TZ and assert the true epoch. Check the other .timestamp() calls on naive DB datetimes for the same mistake.

## Bug `metrics._schedule_metrics` calls `.timestamp()` on `schedule_log.triggered_at`, which is stored as **naive UTC**. Python interprets a naive datetime's `.timestamp()` as *local* time. Since #147, `TZ` actually reaches the container, so with `TZ=America/Chicago` the gauge reads 5 or 6 hours off. This is the metric the recommended alert is built on (`IrisLightsDidNotComeOn`, `docs/setup.md` §8: `time() - iris_scheduler_last_success_timestamp_seconds{job="on_job"} > 129600`). An offset in the wrong direction makes that alert fire hours early or late. `test_metrics.py` currently asserts the buggy behaviour. Found while building #148, whose new gauge uses UTC correctly. ## Fix Use `triggered_at.replace(tzinfo=UTC).timestamp()` (or store tz-aware values). Update `test_metrics.py` to run under a non-UTC `TZ` and assert the true epoch. Check the other `.timestamp()` calls on naive DB datetimes for the same mistake.

claude-bot added this to the v1.2.0 milestone

2026-09-26 10:43:29 +00:00

claude-bot added the

labels

2026-09-26 10:43:29 +00:00

claude-bot referenced this issue

2026-09-26 10:54:00 +00:00

Controller watchdog: notice when the lights aren't showing what Iris pushed, and put them back #148