CI: stop leaving our own images on the shared runner #86

Merged

claude-bot merged 1 commit from fix/reclaim-runner-disk into main 2026-08-31 16:55:37 +00:00

Conversation 0 Commits 1 Files changed 1 +17 -1

claude-bot commented 2026-08-31 05:45:41 +00:00

Contributor

Copy link

Follow-up to #78. Stops this repo leaking images onto the shared Forgejo runner.

What it does

docker-e2e built and loaded a ~350MB image every run and moved the iris:e2e tag, orphaning the previous layers. The reaper added in 43d3a32 removed containers and networks but never images, so nothing reclaimed them.

Cleanup now drops our own tags — iris:e2e plus the two registry tags applied on a push — so the leak never happens, rather than being swept up afterwards. On a push the registry already holds the image; these are only local tags.

The step also reports docker system df each run, so disk pressure shows as a visible trend instead of a mystery failure.

Scope narrowed after review

An earlier revision of this branch ran docker image prune -f --filter "until=2h". That would have removed dangling images belonging to every other repo on the shared daemon. Per the operator: general housekeeping is the infrastructure's job. This workflow now cleans up only what it creates.

What this does NOT fix

Two runs died on a full runner today:

• #7676 — mid-checkout: error: failed to write new configuration file ... .gitconfig.lock
• #7700 — inside buildkit: ResourceExhausted: ... no space left on device

Neither is fixed here. The exhaustion was in /var/lib/buildkit — build cache — which image removal does not touch, and which the infrastructure sweeps handle. I had earlier claimed build cache was "largely self-limiting because setup-buildx-action creates and removes a builder per run", and scoped my first attempt on that basis. That was wrong: the runner reported 20.19GB of build cache and ran out of space inside it.

This commit only stops Iris adding to the pile. The measured leak was real — the earlier revision's prune reclaimed 185.3MB of our orphaned layers in a single run — but it is roughly two orders of magnitude smaller than the build cache that actually filled the disk.

Evidence from the earlier revision's run

--- reclaiming dangling images older than 2h ---
Total reclaimed space: 185.3MB
Images:      13.65GB (10.48GB (76%) reclaimable)
Build Cache: 20.19GB (4.288GB reclaimable)

Follow-up to #78. Stops this repo leaking images onto the shared Forgejo runner. ## What it does `docker-e2e` built and loaded a ~350MB image every run and moved the `iris:e2e` tag, orphaning the previous layers. The reaper added in `43d3a32` removed containers and networks but **never images**, so nothing reclaimed them. Cleanup now drops our own tags — `iris:e2e` plus the two registry tags applied on a push — so **the leak never happens**, rather than being swept up afterwards. On a push the registry already holds the image; these are only local tags. The step also reports `docker system df` each run, so disk pressure shows as a visible trend instead of a mystery failure. ## Scope narrowed after review An earlier revision of this branch ran `docker image prune -f --filter "until=2h"`. That would have removed dangling images belonging to **every other repo** on the shared daemon. Per the operator: general housekeeping is the infrastructure's job. This workflow now cleans up only what it creates. ## What this does NOT fix Two runs died on a full runner today: - **#7676** — mid-checkout: `error: failed to write new configuration file ... .gitconfig.lock` - **#7700** — inside buildkit: `ResourceExhausted: ... no space left on device` **Neither is fixed here.** The exhaustion was in `/var/lib/buildkit` — build cache — which image removal does not touch, and which the infrastructure sweeps handle. I had earlier claimed build cache was "largely self-limiting because `setup-buildx-action` creates and removes a builder per run", and scoped my first attempt on that basis. That was wrong: the runner reported **20.19GB of build cache** and ran out of space inside it. This commit only stops Iris adding to the pile. The measured leak was real — the earlier revision's prune reclaimed `185.3MB` of our orphaned layers in a single run — but it is roughly two orders of magnitude smaller than the build cache that actually filled the disk. ## Evidence from the earlier revision's run ``` --- reclaiming dangling images older than 2h --- Total reclaimed space: 185.3MB Images: 13.65GB (10.48GB (76%) reclaimable) Build Cache: 20.19GB (4.288GB reclaimable) ```

claude-bot added 1 commit 2026-08-31 05:45:42 +00:00

CI: reclaim dangling images so the shared runner stops filling up ... 081a29832c

Run #7676 died mid-checkout with

  error: failed to write new configuration file .../.gitconfig.lock
  error: failed to write new configuration file .../.git/config.lock

Lock-file writes failing on both /tmp and the workspace is ENOSPC, not a code
failure -- the same content was green on the PR, and main was green again on the
next run once the condition cleared.

docker-e2e is a contributor. Every run builds and loads a ~350MB image with
load: true and moves the iris:e2e tag, untagging the previous layers. The reaper
added in 43d3a32 removes containers and networks but never images, so nothing
reclaimed them. Across ~15 runs today that is several GB, on a daemon shared
with every other repo on the instance.

Adds a dangling-image prune bounded by the same 2h cutoff as the container
sweep, so no in-flight build anywhere on the host can be affected. It runs
before the build rather than after, so the space is reclaimed when it is needed.
Non-fatal -- a failed prune must not break CI -- but it prints on failure rather
than swallowing it, and reports docker system df either way so the trend is
visible in the logs.

Verified: a stub docker whose prune and df both exit 1 still leaves the step at
exit 0; and `--filter until=` is accepted by the runner's Docker 29.7.2
(validated against a 10-year cutoff, which correctly reclaimed 0B).

Scoped to dangling images only. Build cache is not touched: setup-buildx-action
creates and removes a builder per run so it is largely self-limiting, and
pruning shared cache would slow every other repo's builds.

Refs #78.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

rbrooks force-pushed fix/reclaim-runner-disk from 081a29832c to 3326205f1d 2026-08-31 16:00:12 +00:00 Compare

rbrooks force-pushed fix/reclaim-runner-disk from 3326205f1d to 09b4c1ea1e 2026-08-31 16:05:13 +00:00 Compare

claude-bot changed title from CI: reclaim dangling images so the shared runner stops filling up to CI: stop leaving our own images on the shared runner 2026-08-31 16:05:27 +00:00

claude-bot merged commit 38931c33dd into main 2026-08-31 16:55:37 +00:00

claude-bot deleted branch fix/reclaim-runner-disk 2026-08-31 16:55:37 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

area/ai

AI suggestion pipeline

  

area/backend

FastAPI / Python backend

  

area/frontend

React / Vite frontend

  

area/infra

Docker, deployment, and infrastructure

  

area/scheduler

Scheduler and time envelope

  

area/wled

WLED controller integration

  

good-first-issue

Good for newcomers

  

priority/high

Blocks release or high impact

  

priority/low

Nice to have

  

priority/medium

Should be done soon

  

type/bug

Something isn't working

  

type/chore

Maintenance, release, or housekeeping

  

type/ci-cd

CI/CD pipeline, build, or release automation

  

type/docs

Documentation

  

type/feature

New capability or enhancement

  

type/qa

Testing, validation, or quality assurance

  

v1.0.0

Target release: v1.0.0

  

v1.1.0

Target release: v1.1.0

  

v1.2.0

Target release: v1.2.0

  

v2.0.0

Target release: v2.0.0

No labels

area/ai

area/backend

area/frontend

area/infra

area/scheduler

area/wled

good-first-issue

priority/high

priority/low

priority/medium

type/bug

type/chore

type/ci-cd

type/docs

type/feature

type/qa

v1.0.0

v1.1.0

v1.2.0

v2.0.0

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/Iris-WLED!86

No description provided.