The published image can't run jobs or push: prometheus-client, defusedxml and zeroconf aren't installed #173

Closed
opened 2026-09-26 10:22:20 +00:00 by claude-bot · 1 comment
Collaborator

Bug (critical)

Verified against the published git.rhoving.com/rbrooks/iris-wled:1.1.1 image:

prometheus_client MISSING ModuleNotFoundError
defusedxml MISSING ModuleNotFoundError
zeroconf MISSING ModuleNotFoundError
app.services.metrics FAILS: ModuleNotFoundError No module named 'prometheus_client'

schedule_service.log_run (so every scheduler job) and wled_push._try_push (every WLED push) import app.services.metrics without a guard. A container from this image starts and passes its health check, but can't run the on/off jobs or push to the lights. Off-site S3 backup (defusedxml) and controller discovery (zeroconf) are broken too. This affects 1.1.0 and 1.1.1; the production deploy (Rhoving/iac-repo#527) is on hold.

Found by the delegate building #154, and then confirmed against the registry image.

Cause

The Dockerfile installs from backend/requirements.txt, a hand-kept copy of pyproject.toml's dependencies. It was not updated when #15 (metrics), #18 (off-site backup) and #50 (discovery) added dependencies to pyproject.toml. It also installs pytest and respx into the production image. And audit.yml runs pip-audit -r requirements.txt, so the weekly dependency audit has been auditing the wrong set as well.

It's the same class of bug as #147: two lists that nothing keeps in step.

Why CI didn't catch it

Tests run from uv sync (so everything is installed), and the E2E container never runs a scheduler job or pushes to a controller. The image's health endpoint doesn't import metrics.

Fix

  • The image installs from uv.lock, via uv export --frozen --no-dev with hashes, so the image is byte-for-byte the locked set. requirements.txt is deleted, so there's no second list left to drift.
  • audit.yml audits the same exported set.
  • A CI smoke step imports every app.* module inside the built image, so any runtime import that the image can't satisfy fails the build.
  • Released as v1.1.2 from a release/1.1.x branch cut at v1.1.1, because main already has unreleased v1.2.0 features. The same fix goes to main.
## Bug (critical) Verified against the published `git.rhoving.com/rbrooks/iris-wled:1.1.1` image: ``` prometheus_client MISSING ModuleNotFoundError defusedxml MISSING ModuleNotFoundError zeroconf MISSING ModuleNotFoundError app.services.metrics FAILS: ModuleNotFoundError No module named 'prometheus_client' ``` `schedule_service.log_run` (so every scheduler job) and `wled_push._try_push` (every WLED push) import `app.services.metrics` without a guard. A container from this image starts and passes its health check, but **can't run the on/off jobs or push to the lights.** Off-site S3 backup (`defusedxml`) and controller discovery (`zeroconf`) are broken too. This affects 1.1.0 and 1.1.1; the production deploy (Rhoving/iac-repo#527) is on hold. Found by the delegate building #154, and then confirmed against the registry image. ## Cause The Dockerfile installs from `backend/requirements.txt`, a hand-kept copy of `pyproject.toml`'s dependencies. It was not updated when #15 (metrics), #18 (off-site backup) and #50 (discovery) added dependencies to `pyproject.toml`. It also installs `pytest` and `respx` into the production image. And `audit.yml` runs `pip-audit -r requirements.txt`, so the weekly dependency audit has been auditing the wrong set as well. It's the same class of bug as #147: two lists that nothing keeps in step. ## Why CI didn't catch it Tests run from `uv sync` (so everything is installed), and the E2E container never runs a scheduler job or pushes to a controller. The image's health endpoint doesn't import metrics. ## Fix - **The image installs from `uv.lock`,** via `uv export --frozen --no-dev` with hashes, so the image is byte-for-byte the locked set. `requirements.txt` is deleted, so there's no second list left to drift. - **`audit.yml`** audits the same exported set. - **A CI smoke step imports every `app.*` module inside the built image,** so any runtime import that the image can't satisfy fails the build. - **Released as v1.1.2** from a `release/1.1.x` branch cut at `v1.1.1`, because `main` already has unreleased v1.2.0 features. The same fix goes to `main`.
Author
Collaborator

Fixed and released.

  • v1.1.2 (tag on release/1.1.x, cut from v1.1.1) is published as sha256:6a5c1fb5…. :1.1.2, :1.1 and :latest all point to it.
    • Checked against the published image: every app.* module imports, and prometheus_client, defusedxml and zeroconf are present.
    • pytest and respx are gone.
    • schedule_service.log_run succeeds in a read-only, cap-dropped container. On 1.1.1 it raises ModuleNotFoundError.
  • main: #174 (squash-merged, CI green). The build installs the locked set exported from uv.lock with hashes, requirements.txt is deleted, audit.yml audits the same export, and the new CI step imports every app module inside the built image.
  • Production: Rhoving/iac-repo#527 now pins 1.1.2, and its hold is lifted.

Known limit: zeroconf is imported inside a function, so the import sweep can't detect its absence. It's installed from the lock now regardless.

Note: CI doesn't run on release/* branches, so the 1.1.2 image was verified by hand. A follow-up could extend ci.yml's triggers to release/**.

Fixed and released. - **v1.1.2** (tag on `release/1.1.x`, cut from `v1.1.1`) is published as `sha256:6a5c1fb5…`. `:1.1.2`, `:1.1` and `:latest` all point to it. - Checked against the **published** image: every `app.*` module imports, and `prometheus_client`, `defusedxml` and `zeroconf` are present. - `pytest` and `respx` are gone. - `schedule_service.log_run` succeeds in a read-only, cap-dropped container. On 1.1.1 it raises `ModuleNotFoundError`. - **`main`:** #174 (squash-merged, CI green). The build installs the locked set exported from `uv.lock` with hashes, `requirements.txt` is deleted, `audit.yml` audits the same export, and the new CI step imports every app module inside the built image. - **Production:** Rhoving/iac-repo#527 now pins 1.1.2, and its hold is lifted. **Known limit:** `zeroconf` is imported inside a function, so the import sweep can't detect its absence. It's installed from the lock now regardless. **Note:** CI doesn't run on `release/*` branches, so the 1.1.2 image was verified by hand. A follow-up could extend `ci.yml`'s triggers to `release/**`.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/Iris-WLED#173
No description provided.