rbrooks/Quest-Board

Watch

Fork 0

CI: limit how often the end-to-end journeys run while the runners are being fixed #1173

New issue

Closed

opened 2026-10-13 15:52:49 +00:00 by claude-bot · 2 comments

claude-bot commented

2026-10-13 15:52:49 +00:00

Collaborator

Raised from the runner side (Rhoving/iac-repo#635) at Ryan's request, so journeys runs can be limited here in the meantime. Nothing in this repo has been changed.

Why

The shared runners cap each job's own container (2 CPUs, 3 to 5 GB), but not what a job starts through Docker. A journeys job starts an 11-container Compose stack and builds about ten images, all outside that cap. Three of them start together on every run, and the runners take six jobs each. That load is the most likely cause of the runners' out-of-memory kills and out-of-space bursts, and it is what other repos queue behind.

This repo's load, last 7 days (to 2026-10-13)

96 ci.yml runs: 67 on pull requests, 29 on pushes to main.
49 succeeded, 16 failed, 31 were cancelled as superseded.
Triggered by: claude-bot 49, rbrooks 30, renovate-bot 17.
A successful run took a median of 35 minutes wall time (90th percentile 49, longest 64).
Every run that reaches journeys starts three stacks (desktop, phone, keyboard), and each builds its own image set under questboard-e2e-<run>-<project>-*: seven images of about 915 MB, a 2.7 GB Playwright image and the frontend. They share layers, so the disk cost is less than the sum, but each is built separately.
A cancelled run does not reach the Tear the stack down step: the runner skips if: always() steps on cancel (iac-repo#560). Its stack stays up until the runner's 6-hour sweep. On 2026-10-13 a stack from run 12525 was still running six and a half hours after it started.

Options, smallest change first

Each is independent. The audio_paths job already shows the pattern for 1 to 3, and its comment records that branch protection accepts a skipped CI / * check.

On pull requests, run desktop only; run all three projects on push to main.
67 of the 96 runs were pull requests, so this removes 134 of 288 journeys jobs a week (47%). Cost: a phone-only or keyboard-only regression is caught on main instead of on the PR.
Skip the journeys on a pull request that cannot affect them, with a changed-paths check like audio_paths: docs, the bot alone, the feedback site, evals. Fail open (run them) if the diff cannot be read. Saving depends on the mix of PRs; not measured.
Give Renovate pull requests a lighter gate (desktop only, or journeys only when webapp/ dependencies or the Playwright image change). 17 runs this week were Renovate's. Cost: auto-merge then rests on less; the docker job still catches a broken image.
Do not run the full suite twice for one change. A merged PR is tested on the PR and again on the push to main. With option 1 the push run becomes the full one and the PR run the light one, which removes most of the duplication without a separate rule.
Build the e2e images once per run and share them between the three journeys jobs, instead of three compose --profile tools build runs of the same Dockerfile. scripts/journeys.sh already has NO_BUILD=1. This needs somewhere to put the images: a registry the jobs can push to (I can add one on the LAN from iac-repo) or an artifact. Largest saving in build time and runner disk, and it removes three builds committing the same layers at once, which is the pattern seen just before the runner's build cache was corrupted on 2026-10-13 (iac-repo#633). Larger change than 1 to 4.
Make teardown survive a cancel. The runner does run an action's post: step after a cancel (read from the runner source, not yet tested here). Moving Tear the stack down into a small local action with a post: step would stop cancelled runs leaving stacks behind. A runner-side fix is also being investigated, so this is optional.

Suggested for now

Option 1, then option 2. They are a few lines each, need nothing from the runners, and can be reverted in one commit once the runners are bounded properly.

Not established

How much memory and store one journeys job uses. Being measured on the runners now (iac-repo#635).
The saving from options 2 and 3.
That a post: step runs on cancel on runner v12.13.2 (option 6).

Raised from the runner side (Rhoving/iac-repo#635) at Ryan's request, so journeys runs can be limited here in the meantime. Nothing in this repo has been changed. ## Why The shared runners cap each job's own container (2 CPUs, 3 to 5 GB), but not what a job starts through Docker. A journeys job starts an 11-container Compose stack and builds about ten images, all outside that cap. Three of them start together on every run, and the runners take six jobs each. That load is the most likely cause of the runners' out-of-memory kills and out-of-space bursts, and it is what other repos queue behind. ## This repo's load, last 7 days (to 2026-10-13) - 96 `ci.yml` runs: 67 on pull requests, 29 on pushes to `main`. - 49 succeeded, 16 failed, 31 were cancelled as superseded. - Triggered by: claude-bot 49, rbrooks 30, renovate-bot 17. - A successful run took a median of 35 minutes wall time (90th percentile 49, longest 64). - Every run that reaches `journeys` starts three stacks (`desktop`, `phone`, `keyboard`), and each builds its own image set under `questboard-e2e-<run>-<project>-*`: seven images of about 915 MB, a 2.7 GB Playwright image and the frontend. They share layers, so the disk cost is less than the sum, but each is built separately. - A cancelled run does not reach the `Tear the stack down` step: the runner skips `if: always()` steps on cancel (iac-repo#560). Its stack stays up until the runner's 6-hour sweep. On 2026-10-13 a stack from run 12525 was still running six and a half hours after it started. ## Options, smallest change first Each is independent. The `audio_paths` job already shows the pattern for 1 to 3, and its comment records that branch protection accepts a skipped `CI / *` check. 1. **On pull requests, run `desktop` only; run all three projects on push to `main`.** 67 of the 96 runs were pull requests, so this removes 134 of 288 journeys jobs a week (47%). Cost: a phone-only or keyboard-only regression is caught on `main` instead of on the PR. 2. **Skip the journeys on a pull request that cannot affect them**, with a changed-paths check like `audio_paths`: docs, the bot alone, the feedback site, evals. Fail open (run them) if the diff cannot be read. Saving depends on the mix of PRs; not measured. 3. **Give Renovate pull requests a lighter gate** (`desktop` only, or journeys only when `webapp/` dependencies or the Playwright image change). 17 runs this week were Renovate's. Cost: auto-merge then rests on less; the `docker` job still catches a broken image. 4. **Do not run the full suite twice for one change.** A merged PR is tested on the PR and again on the push to `main`. With option 1 the push run becomes the full one and the PR run the light one, which removes most of the duplication without a separate rule. 5. **Build the e2e images once per run and share them** between the three journeys jobs, instead of three `compose --profile tools build` runs of the same Dockerfile. `scripts/journeys.sh` already has `NO_BUILD=1`. This needs somewhere to put the images: a registry the jobs can push to (I can add one on the LAN from iac-repo) or an artifact. Largest saving in build time and runner disk, and it removes three builds committing the same layers at once, which is the pattern seen just before the runner's build cache was corrupted on 2026-10-13 (iac-repo#633). Larger change than 1 to 4. 6. **Make teardown survive a cancel.** The runner does run an action's `post:` step after a cancel (read from the runner source, not yet tested here). Moving `Tear the stack down` into a small local action with a `post:` step would stop cancelled runs leaving stacks behind. A runner-side fix is also being investigated, so this is optional. ## Suggested for now Option 1, then option 2. They are a few lines each, need nothing from the runners, and can be reverted in one commit once the runners are bounded properly. ## Not established - How much memory and store one journeys job uses. Being measured on the runners now (iac-repo#635). - The saving from options 2 and 3. - That a `post:` step runs on cancel on runner v12.13.2 (option 6).

rbrooks referenced this issue from a commit

2026-10-13 16:00:19 +00:00

ci: a pull request runs only the journeys it can break (#1173)

claude-bot referenced this issue

2026-10-13 16:00:29 +00:00

ci: a pull request runs only the journeys it can break (#1173) #1174

claude-bot commented

2026-10-13 16:00:38 +00:00

Author

Collaborator

Picked up. PR #1174 does options 1, 2 and 4, and most of 3 (a Renovate PR is a pull request like any other, so it gets desktop only, or nothing when it touches only the bot).

One change to option 1: a PR that touches the journeys themselves (webapp/frontend/e2e/), their stack (Dockerfile, compose files, scripts/journeys.sh, nginx.conf, package(-lock).json, ci.yml) or comes from a chore/release-* branch still runs all three. Three new specs on 12 to 13 Oct failed in one project only, and a release should not be tagged on desktop alone.

Not done here:

Option 5 (build the e2e images once per run): the larger saving, and it removes the three simultaneous builds seen before the cache fault on forgejo-runner-03. Needs the LAN registry from iac-repo; worth doing once that exists.
Option 6 (teardown that survives a cancel): waiting on whether the runner-side fix lands.

Not yet shown on a real run: the desktop-only and skipped paths, and that branch protection's CI / * is satisfied when the phone and keyboard checks are absent rather than skipped. The first ordinary PR after #1174 merges will show both.

Picked up. PR #1174 does options 1, 2 and 4, and most of 3 (a Renovate PR is a pull request like any other, so it gets `desktop` only, or nothing when it touches only the bot). One change to option 1: a PR that touches the journeys themselves (`webapp/frontend/e2e/`), their stack (`Dockerfile`, compose files, `scripts/journeys.sh`, `nginx.conf`, `package(-lock).json`, `ci.yml`) or comes from a `chore/release-*` branch still runs all three. Three new specs on 12 to 13 Oct failed in one project only, and a release should not be tagged on `desktop` alone. Not done here: - **Option 5** (build the e2e images once per run): the larger saving, and it removes the three simultaneous builds seen before the cache fault on forgejo-runner-03. Needs the LAN registry from iac-repo; worth doing once that exists. - **Option 6** (teardown that survives a cancel): waiting on whether the runner-side fix lands. Not yet shown on a real run: the `desktop`-only and skipped paths, and that branch protection's `CI / *` is satisfied when the phone and keyboard checks are absent rather than skipped. The first ordinary PR after #1174 merges will show both.

rbrooks referenced this issue from a commit

2026-10-13 16:02:52 +00:00

ci: the journeys plan sees renames and the whole backend tree (#1173)

claude-bot referenced this issue from a commit

2026-10-13 16:35:38 +00:00

Merge PR #1174: a pull request runs only the journeys it can break (#1173)

claude-bot referenced this issue

2026-10-13 20:39:24 +00:00

chore(release): v4.5.1-rc2 #1182

claude-bot referenced this issue

2026-10-17 04:21:18 +00:00

CI: the journeys check names read "(desktop) (desktop)" #1203

claude-bot referenced this issue

2026-10-27 17:38:39 +00:00

CI: build the end-to-end images once per run, and a teardown that survives a cancel #1248

claude-bot commented

2026-10-27 17:38:52 +00:00

Author

Collaborator

Closing. PR #1174 did options 1 to 4. It is now shown on real runs: on 27 Oct, #1230 (backend-only) skipped the journeys and still merged under CI / *, so absent checks satisfy branch protection. Options 5 and 6 are split into #1248.

Closing. PR #1174 did options 1 to 4. It is now shown on real runs: on 27 Oct, #1230 (backend-only) skipped the journeys and still merged under `CI / *`, so absent checks satisfy branch protection. Options 5 and 6 are split into #1248.

claude-bot closed this issue

2026-10-27 17:38:53 +00:00