rbrooks/Quest-Board

Watch

Fork 0

CI fills the runners' disks: each journeys run leaves its images behind #1081

New issue

Closed

opened 2026-10-04 15:42:58 +00:00 by claude-bot · 3 comments

claude-bot commented

2026-10-04 15:42:58 +00:00

Collaborator

CI failed three times on 2026-10-04 with "no space left on device" (runs 11967 and 11970 in full, and earlier on 2 October). None was caused by the code under test.

What was found (read-only look at the three runner VMs)

Jobs run in a Docker-in-Docker container on each runner.

Runner	Docker store	Images	Named `questboard-e2e-*`	Dangling volumes	Build cache
forgejo-runner (10.1.1.20)	96 GB disk, 46% used	798	679	270	14.4 GB
forgejo-runner-02 (10.1.1.22)	40 GB RAM tmpfs, 56% used when idle	123	109	74	5.4 GB
forgejo-runner-03 (10.1.1.26)	40 GB RAM tmpfs, 49% used when idle	42	27	29	5.5 GB

Cause. The journeys job names its compose project questboard-e2e-<run>-<project>, so every run builds its own set of images: nine per project, three projects per run, the Playwright one 2.7 to 3.8 GB. down -v removes the containers and volumes but not the images, and no later run reuses those names.
Why it bites now. On the two tmpfs runners the store is half full when idle, and three projects of one run each want about 4 GB for Playwright plus about 1.3 GB per service image. A burst of runs reaches the 40 GB limit before the runners' own clean-up timers (daily, hourly, and a pressure one at 80%) catch up. 4 October had about fifteen PR runs.
I could not tell which runner ran the failed jobs.

Fix in this repository

The journeys teardown removes the run's own images (--rmi local, with the tools profile so seed and playwright are included), when CI sets QB_E2E_RM_IMAGES=1. A local run keeps its images: its project name is fixed and the next run reuses them.

For the owner: the runners themselves

Nothing was changed on any runner. To free space now, inside forgejo_runner_dind on each, with no job running, most conservative first:

Run the role's own clean-up: sudo systemctl start forgejo-runner-pressure-gc.service, then forgejo-runner-gc.service.
docker volume prune -f: about 7.0 GB on .20, 3.3 GB on .22, 2.5 GB on .26.
docker image prune -a -f --filter until=24h: most of the 9.8 GB reclaimable on .20 and 8.7 GB on .26.
docker builder prune -a -f --filter until=24h: up to 14.4 GB on .20, about 4 GB on each of the others.

Longer term, in the runner's Ansible role: lower forgejo_runner_pressure_gc_high_pct below 80 for the tmpfs runners, or raise forgejo_runner_dind_ram_size above 40G.

Not checked

What the role's clean-up scripts actually remove, and whether the build cache is also growing without bound.

CI failed three times on 2026-10-04 with "no space left on device" (runs 11967 and 11970 in full, and earlier on 2 October). None was caused by the code under test. ## What was found (read-only look at the three runner VMs) Jobs run in a Docker-in-Docker container on each runner. | Runner | Docker store | Images | Named `questboard-e2e-*` | Dangling volumes | Build cache | |---|---|---|---|---|---| | forgejo-runner (10.1.1.20) | 96 GB disk, 46% used | 798 | 679 | 270 | 14.4 GB | | forgejo-runner-02 (10.1.1.22) | 40 GB RAM tmpfs, 56% used when idle | 123 | 109 | 74 | 5.4 GB | | forgejo-runner-03 (10.1.1.26) | 40 GB RAM tmpfs, 49% used when idle | 42 | 27 | 29 | 5.5 GB | - **Cause.** The journeys job names its compose project `questboard-e2e-<run>-<project>`, so every run builds its own set of images: nine per project, three projects per run, the Playwright one 2.7 to 3.8 GB. `down -v` removes the containers and volumes but not the images, and no later run reuses those names. - **Why it bites now.** On the two tmpfs runners the store is half full when idle, and three projects of one run each want about 4 GB for Playwright plus about 1.3 GB per service image. A burst of runs reaches the 40 GB limit before the runners' own clean-up timers (daily, hourly, and a pressure one at 80%) catch up. 4 October had about fifteen PR runs. - I could not tell which runner ran the failed jobs. ## Fix in this repository The journeys teardown removes the run's own images (`--rmi local`, with the `tools` profile so `seed` and `playwright` are included), when CI sets `QB_E2E_RM_IMAGES=1`. A local run keeps its images: its project name is fixed and the next run reuses them. ## For the owner: the runners themselves Nothing was changed on any runner. To free space now, inside `forgejo_runner_dind` on each, with no job running, most conservative first: 1. Run the role's own clean-up: `sudo systemctl start forgejo-runner-pressure-gc.service`, then `forgejo-runner-gc.service`. 2. `docker volume prune -f`: about 7.0 GB on .20, 3.3 GB on .22, 2.5 GB on .26. 3. `docker image prune -a -f --filter until=24h`: most of the 9.8 GB reclaimable on .20 and 8.7 GB on .26. 4. `docker builder prune -a -f --filter until=24h`: up to 14.4 GB on .20, about 4 GB on each of the others. Longer term, in the runner's Ansible role: lower `forgejo_runner_pressure_gc_high_pct` below 80 for the tmpfs runners, or raise `forgejo_runner_dind_ram_size` above 40G. ## Not checked What the role's clean-up scripts actually remove, and whether the build cache is also growing without bound.

claude-bot referenced this issue

2026-10-04 15:43:09 +00:00

A CI run of the journeys removes the images it built (#1081) #1082

claude-bot referenced this issue from a commit

2026-10-04 16:07:49 +00:00

Merge PR #1082: a CI run of the journeys removes the images it built (#1081)

claude-bot commented

2026-10-04 16:08:11 +00:00

Author

Collaborator

PR #1082 is merged. CI was green on its head (14 of 14 checks).

The change: a CI journeys run removes the nine images it built when its stack comes down (QB_E2E_RM_IMAGES=1, set by the journeys job and by its backstop step).
The owner's word (2026-10-04): the runners clear themselves automatically and had cleared by the time this ran, so no manual clean-up was needed.

Left open for one check: that a run really leaves no questboard-e2e-<run>-* images behind. That needs a look at a runner after a run or two. I will look read-only after the next few runs and close this with the count.

**PR #1082 is merged. CI was green on its head (14 of 14 checks).** - **The change:** a CI journeys run removes the nine images it built when its stack comes down (`QB_E2E_RM_IMAGES=1`, set by the journeys job and by its backstop step). - **The owner's word (2026-10-04):** the runners clear themselves automatically and had cleared by the time this ran, so no manual clean-up was needed. **Left open for one check:** that a run really leaves no `questboard-e2e-<run>-*` images behind. That needs a look at a runner after a run or two. I will look read-only after the next few runs and close this with the count.

claude-bot commented

2026-10-04 17:19:41 +00:00

Author

Collaborator

First read-only look at the runners after the fix (2026-10-04, about 17:25 UTC). Encouraging, not yet conclusive.

Runner	`questboard-e2e-*` images	Run ids they belong to	Store
.20	678	all 11969 or earlier (before the fix)	49% of 96 GB
.22	91	11967, 11968, 11970 (before the fix); 11984 (9 images)	69% of 40 GB
.26	46	11981 (27 images); 11984 (18 images)	49% of 40 GB

Runs 11971 to 11980 left no images on any runner. Those include the runs of #1082, #1078 and the rc4 release, which carry the fix.
11984 had containers running at the time, so its images were in use, not left over.
11981 has a full set of 27 images. That fits #1084's run: its branch was cut before the fix, so it would leave them. Not confirmed.
The old leftovers on .20 are for the runners' own clean-up to remove.

To close this: one more look when no journeys job is running, expecting no run id above 11984 to have images.

**First read-only look at the runners after the fix (2026-10-04, about 17:25 UTC). Encouraging, not yet conclusive.** | Runner | `questboard-e2e-*` images | Run ids they belong to | Store | |---|---|---|---| | .20 | 678 | all 11969 or earlier (before the fix) | 49% of 96 GB | | .22 | 91 | 11967, 11968, 11970 (before the fix); 11984 (9 images) | 69% of 40 GB | | .26 | 46 | 11981 (27 images); 11984 (18 images) | 49% of 40 GB | - **Runs 11971 to 11980 left no images on any runner.** Those include the runs of #1082, #1078 and the rc4 release, which carry the fix. - **11984** had containers running at the time, so its images were in use, not left over. - **11981** has a full set of 27 images. That fits #1084's run: its branch was cut before the fix, so it would leave them. Not confirmed. - The old leftovers on .20 are for the runners' own clean-up to remove. **To close this:** one more look when no journeys job is running, expecting no run id above 11984 to have images.

claude-bot commented

2026-10-05 01:50:48 +00:00

Author

Collaborator

Second read-only look at the runners (2026-10-05, about 01:50 UTC). The fix holds; closing.

Runner	Run ids with `questboard-e2e-<run>-*` images	Store
.20	11861 to 11969 (before the fix); 12020, 12028, 12029	46% of 96 GB
.22	12020, 12027, 12028	68% of 40 GB
.26	12027, 12029	53% of 40 GB

No run from 11971 to 12019 has an image left on any runner. That is about fifty runs since the fix.
12020 and 12027 to 12029 had jobs running when I looked (29 to 45 journeys containers on each runner), so their images were in use.
11981, the run from a branch cut before the fix, has been cleaned up by the runners' own timers.
The two RAM-backed runners now hold only the runs in flight: 69 and 43 images, where .22 held 123 before.

Not shown: that 12020 and 12027 to 12029 were removed when they ended. The fifty runs before them are the evidence for that.

Left for the runners' own clean-up: the leftovers on .20 from before the fix.

Not done, and the owner's if wanted: lowering the pressure threshold or raising the RAM store on the two tmpfs runners. With per-run images gone it may not be needed; .22 was at 68% with three runs in flight.

**Second read-only look at the runners (2026-10-05, about 01:50 UTC). The fix holds; closing.** | Runner | Run ids with `questboard-e2e-<run>-*` images | Store | |---|---|---| | .20 | 11861 to 11969 (before the fix); 12020, 12028, 12029 | 46% of 96 GB | | .22 | 12020, 12027, 12028 | 68% of 40 GB | | .26 | 12027, 12029 | 53% of 40 GB | - **No run from 11971 to 12019 has an image left on any runner.** That is about fifty runs since the fix. - **12020 and 12027 to 12029** had jobs running when I looked (29 to 45 journeys containers on each runner), so their images were in use. - **11981**, the run from a branch cut before the fix, has been cleaned up by the runners' own timers. - The two RAM-backed runners now hold only the runs in flight: 69 and 43 images, where .22 held 123 before. **Not shown:** that 12020 and 12027 to 12029 were removed when they ended. The fifty runs before them are the evidence for that. **Left for the runners' own clean-up:** the leftovers on .20 from before the fix. **Not done, and the owner's if wanted:** lowering the pressure threshold or raising the RAM store on the two tmpfs runners. With per-run images gone it may not be needed; .22 was at 68% with three runs in flight.

claude-bot closed this issue

2026-10-05 01:50:51 +00:00