rbrooks/Iris-WLED

Watch

Fork 0

CI has no concurrency group — superseded runs are never cancelled #78

New issue

Closed

opened 2026-08-29 07:20:26 +00:00 by claude-bot · 2 comments

claude-bot commented

2026-08-29 07:20:26 +00:00

Contributor

.forgejo/workflows/ci.yml has no top-level concurrency: key, so every push to a branch or PR starts a fresh run while the previous run for that same ref continues to completion.

Context

The shared Forgejo Actions runners are saturated — see Rhoving/iac-repo#371. Measured 2026-08-29: all six fleet-wide job slots were occupied, with three repos pushing at once, and jobs turning over in roughly 1–2 minutes. Runner capacity is shared across every repo on this instance, so superseded runs here are directly other repos' queue wait.

Several repos on this instance already set a top-level concurrency group (Quest-Board, Radome, TeaLeaves, ChoreMachine, HomeBooks, BatteryStorageCalculator). These workflows do not.

Affected

.forgejo/workflows/ci.yml (jobs: python-lint, python-test)

Proposed

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Include github.workflow in the group so separate workflows in the same repo don't cancel each other.

⚠️ Before enabling cancel-in-progress: true, check that any teardown this CI relies on runs under cancellation (if: always()). Cancelled jobs skip normal cleanup steps. Leaked postgres:18 service containers aged 7–11h were found inside the runners' DinD daemons on 2026-08-29 and had to be removed by hand (Rhoving/iac-repo#292) — that failure mode gets more frequent, not less, if cancellation is enabled without cleanup that survives it.

Definition of done

Any teardown the CI depends on verified to run under cancellation
Top-level concurrency: added to .forgejo/workflows/ci.yml
Confirmed superseded runs actually cancel, by pushing twice in quick succession

Refs Rhoving/iac-repo#371

`.forgejo/workflows/ci.yml` has no top-level `concurrency:` key, so every push to a branch or PR starts a fresh run while **the previous run for that same ref continues to completion**. ## Context The shared Forgejo Actions runners are saturated — see Rhoving/iac-repo#371. Measured 2026-08-29: all six fleet-wide job slots were occupied, with three repos pushing at once, and jobs turning over in roughly 1–2 minutes. Runner capacity is shared across every repo on this instance, so superseded runs here are directly other repos' queue wait. Several repos on this instance already set a top-level concurrency group (`Quest-Board`, `Radome`, `TeaLeaves`, `ChoreMachine`, `HomeBooks`, `BatteryStorageCalculator`). These workflows do not. ## Affected - `.forgejo/workflows/ci.yml` (jobs: python-lint, python-test) ## Proposed ```yaml concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true ``` Include `github.workflow` in the group so separate workflows in the same repo don't cancel each other. ⚠️ Before enabling `cancel-in-progress: true`, check that any teardown this CI relies on runs under cancellation (`if: always()`). Cancelled jobs skip normal cleanup steps. Leaked `postgres:18` service containers aged 7–11h were found inside the runners' DinD daemons on 2026-08-29 and had to be removed by hand (Rhoving/iac-repo#292) — that failure mode gets more frequent, not less, if cancellation is enabled without cleanup that survives it. ## Definition of done - [ ] Any teardown the CI depends on verified to run under cancellation - [ ] Top-level `concurrency:` added to `.forgejo/workflows/ci.yml` - [ ] Confirmed superseded runs actually cancel, by pushing twice in quick succession Refs Rhoving/iac-repo#371

claude-bot referenced this issue

2026-08-31 02:04:37 +00:00

Drop arm64 / Raspberry Pi support: publish linux/amd64 only #79

claude-bot commented

2026-08-31 02:19:34 +00:00

Author

Contributor

Implemented — working tree, not yet committed

Teardown audit (first DoD item)

Walked every job in ci.yml. docker-e2e is the only one that leaves state on the host — python-lint, python-test, migration-check, frontend, and security-audit are all setup-python/setup-node/uv and leak nothing.

Its teardown is already if: always(), which is what this issue asked me to verify. But that alone is not sufficient here, and the reason is specific to this repo:

the Cleanup step is run-scoped — iris-${{ github.run_id }}, pw-…, iris-net-….

So if a cancelled job does not reach always() (act's cancellation semantics differ from GitHub's, and I could not confirm them without actually cancelling a run), the leftovers are named after a run_id that no future run will ever match. Nothing would ever reap them. That is precisely the Rhoving/iac-repo#292 failure mode this issue warns about, and turning on cancel-in-progress without addressing it would have made it more likely, not less.

Rather than bet on always() firing, I made the cleanup survive cancellation regardless.

Changes to `.forgejo/workflows/ci.yml`

1. Top-level concurrency group — exactly as proposed, matching the six sibling repos:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

2. A reaper step at the top of docker-e2e that sweeps any iris-* / pw-* container or iris-net-* network older than 2h.

The 2h cutoff is the important part: PR runs execute concurrently on this shared host, each with its own run_id, so a naive "delete anything that isn't mine" sweep would kill a sibling PR's containers mid-test. A live run's resources are minutes old (longest observed run was 38 min, and that included the arm64 build now removed by #79); only genuinely abandoned ones reach 2h.

3. A comment on the Cleanup step noting it is the fast path and the reaper is the backstop.

Verification

Exercised the reaper's shell against a stubbed docker covering four cases:

Case	Result
5h-old leaked containers + network	reaped
3-minute-old concurrent sibling PR	untouched
clean host, no output from `docker ps`	exit 0
unparseable `.Created` timestamp	exit 0, skipped
`docker inspect` fails (resource raced away)	exit 0, skipped

Every path exits 0 under bash -e, so the reaper can never fail the job. ci.yml still parses as valid YAML.

Remaining

The third DoD item — "Confirmed superseded runs actually cancel, by pushing twice in quick succession" — needs an actual push, so it stays unchecked until this is committed. Leaving open until then.

Refs Rhoving/iac-repo#371.

## Implemented — working tree, not yet committed ### Teardown audit (first DoD item) Walked every job in `ci.yml`. **`docker-e2e` is the only one that leaves state on the host** — `python-lint`, `python-test`, `migration-check`, `frontend`, and `security-audit` are all `setup-python`/`setup-node`/`uv` and leak nothing. Its teardown is already `if: always()`, which is what this issue asked me to verify. But that alone is **not** sufficient here, and the reason is specific to this repo: > the Cleanup step is **run-scoped** — `iris-${{ github.run_id }}`, `pw-…`, `iris-net-…`. So if a cancelled job does not reach `always()` (act's cancellation semantics differ from GitHub's, and I could not confirm them without actually cancelling a run), the leftovers are named after a `run_id` that **no future run will ever match**. Nothing would ever reap them. That is precisely the Rhoving/iac-repo#292 failure mode this issue warns about, and turning on `cancel-in-progress` without addressing it would have made it more likely, not less. Rather than bet on `always()` firing, I made the cleanup survive cancellation regardless. ### Changes to `.forgejo/workflows/ci.yml` **1. Top-level concurrency group** — exactly as proposed, matching the six sibling repos: ```yaml concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true ``` **2. A reaper step** at the top of `docker-e2e` that sweeps any `iris-*` / `pw-*` container or `iris-net-*` network **older than 2h**. The 2h cutoff is the important part: PR runs execute concurrently on this shared host, each with its own `run_id`, so a naive "delete anything that isn't mine" sweep would kill a sibling PR's containers mid-test. A live run's resources are minutes old (longest observed run was 38 min, and that included the arm64 build now removed by #79); only genuinely abandoned ones reach 2h. **3.** A comment on the Cleanup step noting it is the fast path and the reaper is the backstop. ### Verification Exercised the reaper's shell against a stubbed `docker` covering four cases: | Case | Result | |---|---| | 5h-old leaked containers + network | reaped | | 3-minute-old concurrent sibling PR | **untouched** | | clean host, no output from `docker ps` | exit 0 | | unparseable `.Created` timestamp | exit 0, skipped | | `docker inspect` fails (resource raced away) | exit 0, skipped | Every path exits 0 under `bash -e`, so the reaper can never fail the job. `ci.yml` still parses as valid YAML. ### Remaining The third DoD item — *"Confirmed superseded runs actually cancel, by pushing twice in quick succession"* — needs an actual push, so it stays unchecked until this is committed. Leaving open until then. Refs Rhoving/iac-repo#371.

rbrooks referenced this issue from a commit

2026-08-31 02:30:57 +00:00

CI: cancel superseded runs; reap containers a cancelled run leaks (#78)

claude-bot commented

2026-08-31 03:51:45 +00:00

Author

Contributor

Done — all three DoD items satisfied

Shipped in 43d3a32.

Teardown verified under cancellation. docker-e2e is the only job leaving host state; the other jobs are stateless. Its cleanup is if: always() and run-scoped (iris-<run_id>), so a cancelled job that never reaches it would strand resources under a run_id no future run matches. Rather than rely on always() firing under act's cancellation, the job now reaps iris-*/pw-* containers and iris-net-* networks older than 2h — the age cutoff keeps a concurrent sibling PR's containers safe.
Top-level concurrency: added to ci.yml, keyed on ${{ github.workflow }}-${{ github.ref }} with cancel-in-progress: true, matching the sibling repos.
Confirmed superseded runs actually cancel. Tested for real by pushing two commits as two pushes in quick succession: run #7636 (544ef55) went cancelled after 1s, superseded by #7637. It was cancelled before docker-e2e started, so nothing leaked.

Side benefit worth noting: combined with #79 removing the emulated arm64 build, full runs on main went from 20m13s / 38m11s down to about 6m30s.

Refs Rhoving/iac-repo#371.

## Done — all three DoD items satisfied Shipped in `43d3a32`. - [x] **Teardown verified under cancellation.** `docker-e2e` is the only job leaving host state; the other jobs are stateless. Its cleanup is `if: always()` *and* run-scoped (`iris-<run_id>`), so a cancelled job that never reaches it would strand resources under a `run_id` no future run matches. Rather than rely on `always()` firing under act's cancellation, the job now reaps `iris-*`/`pw-*` containers and `iris-net-*` networks older than 2h — the age cutoff keeps a concurrent sibling PR's containers safe. - [x] **Top-level `concurrency:` added** to `ci.yml`, keyed on `${{ github.workflow }}-${{ github.ref }}` with `cancel-in-progress: true`, matching the sibling repos. - [x] **Confirmed superseded runs actually cancel.** Tested for real by pushing two commits as two pushes in quick succession: run [#7636](https://git.rhoving.com/rbrooks/Iris-WLED/actions/runs/150) (`544ef55`) went **`cancelled` after 1s**, superseded by #7637. It was cancelled before `docker-e2e` started, so nothing leaked. Side benefit worth noting: combined with #79 removing the emulated arm64 build, full runs on `main` went from 20m13s / 38m11s down to about 6m30s. Refs Rhoving/iac-repo#371.

claude-bot closed this issue

2026-08-31 03:51:56 +00:00

claude-bot referenced this issue

2026-08-31 04:00:28 +00:00

Migrate CI from GitHub Actions to Forgejo Actions #2

rbrooks referenced this issue from a commit

2026-08-31 05:45:28 +00:00

CI: reclaim dangling images so the shared runner stops filling up

claude-bot referenced this issue

2026-08-31 05:45:41 +00:00

CI: stop leaving our own images on the shared runner #86

rbrooks referenced this issue from a commit