CI: frontend vitest runs on one worker, and the journeys run serially — shorten the critical path #939

New issue

Closed

opened 2026-09-29 17:58:29 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-09-29 17:58:29 +00:00

Collaborator

Measured on run 11421 (push to main, 31 min):

Job	Wall time	Where it goes
Frontend	30 min	vitest 1,603 s, against ~160 s locally
Backend	29 min	pytest 1,588 s (#937 / PR #938 roughly halves this)
Journeys	25 min	Playwright 23.1 min for 118 tests. It also waits ~6 min on the `docker` job first.

Run 11416 was about 10 min slower in every job with identical cache hits, which points to contention on runner-01.

1. vitest. Its phase breakdown (environment 479 s, tests 796 s, import 133 s, setup 79 s, transform 38 s) sums to about the wall time, which means effectively one worker. CI jobs are capped at 2 CPUs and 3 GB (iac-repo #291), and vitest defaults to CPUs − 1 workers. Plan: reproduce under --cpus 2 --memory 3g locally, and compare the defaults with --maxWorkers=2. If that isn't enough, shard vitest across jobs.

2. Journeys. The suite is deliberately one worker, because the login rate limit is per IP and every journey shares one seeded database. Plan: a matrix with one job per Playwright project (desktop, phone, keyboard). Each gets its own stack, so each has its own database and IP, and nothing changes in the specs. Also check whether needs: docker is required, or only makes the job wait.

Done when: one CI run shows the new times, and the journeys still cover all three projects.

**Measured on run 11421** (push to main, 31 min): | Job | Wall time | Where it goes | |---|---|---| | Frontend | 30 min | vitest 1,603 s, against ~160 s locally | | Backend | 29 min | pytest 1,588 s (#937 / PR #938 roughly halves this) | | Journeys | 25 min | Playwright 23.1 min for 118 tests. It also waits ~6 min on the `docker` job first. | Run 11416 was about 10 min slower in every job with identical cache hits, which points to contention on runner-01. **1. vitest.** Its phase breakdown (environment 479 s, tests 796 s, import 133 s, setup 79 s, transform 38 s) sums to about the wall time, which means effectively one worker. CI jobs are capped at 2 CPUs and 3 GB (iac-repo #291), and vitest defaults to CPUs − 1 workers. **Plan:** reproduce under `--cpus 2 --memory 3g` locally, and compare the defaults with `--maxWorkers=2`. If that isn't enough, shard vitest across jobs. **2. Journeys.** The suite is deliberately one worker, because the login rate limit is per IP and every journey shares one seeded database. **Plan:** a matrix with one job per Playwright project (desktop, phone, keyboard). Each gets its own stack, so each has its own database and IP, and nothing changes in the specs. Also check whether `needs: docker` is required, or only makes the job wait. **Done when:** one CI run shows the new times, and the journeys still cover all three projects.

rbrooks referenced this issue from a commit

2026-09-29 18:15:23 +00:00

ci: shard vitest and split the journeys by project (#939)

claude-bot referenced this issue from a pull request that will close it,

2026-09-29 18:15:33 +00:00

ci: shard vitest and split the journeys by project (#939) #940

claude-bot closed this issue

2026-09-29 18:37:55 +00:00

claude-bot referenced this issue from a commit

2026-09-29 18:37:57 +00:00

Merge PR #940: shard vitest and split the journeys by project (#939)

claude-bot commented

2026-09-29 18:52:44 +00:00

Author

Collaborator

Measured on PR #940's own run (11427): 22 min 16 s, against 31–43 min before. Every job is green, with no OOM or exit-137 on either runner.

Job	Before (11421)	After (11427)
Frontend	30.1 min	shard 1/2: 11.7 min (includes lint, build and audit); shard 2/2: 5.8 min
Backend	29.3 min	18.2 min (#938's stub and guard; the base includes it)
Journeys	25.1 min, after docker	desktop 8.8 min, keyboard 7.8 min, phone 11.6 min
Docker build	~6 min	0.8 min (warm cache)

The last job to finish was journeys (phone), and it was queued, not slow. The docker job finished at 18:17:20, but phone only got a runner at 18:26:10. That's ~9 min waiting for one of the 8 slots, in a run of 14 jobs (the synthetic harness ran because ci.yml changed).

What's left, in order:

Runner slots. This is now the main lever: more capacity on runner-01 or a third runner. That's an iac-repo change.
Backend at 18 min is now the longest single job. On 2 CPUs, pytest-xdist would buy little. The remaining time is real work (the permission crawls, silent-track audio).
Frontend shard 1 carries lint, build and audit, which is why it's twice as long as shard 2. That's still well under the backend, so not worth rebalancing yet.

**Measured on PR #940's own run (11427): 22 min 16 s, against 31–43 min before.** Every job is green, with no OOM or exit-137 on either runner. | Job | Before (11421) | After (11427) | |---|---|---| | Frontend | 30.1 min | shard 1/2: 11.7 min (includes lint, build and audit); shard 2/2: 5.8 min | | Backend | 29.3 min | 18.2 min (#938's stub and guard; the base includes it) | | Journeys | 25.1 min, after docker | desktop 8.8 min, keyboard 7.8 min, phone 11.6 min | | Docker build | ~6 min | 0.8 min (warm cache) | The last job to finish was journeys (phone), and it was **queued, not slow**. The docker job finished at 18:17:20, but phone only got a runner at 18:26:10. That's ~9 min waiting for one of the 8 slots, in a run of 14 jobs (the synthetic harness ran because `ci.yml` changed). **What's left, in order:** 1. **Runner slots.** This is now the main lever: more capacity on runner-01 or a third runner. That's an iac-repo change. 2. **Backend at 18 min** is now the longest single job. On 2 CPUs, pytest-xdist would buy little. The remaining time is real work (the permission crawls, silent-track audio). 3. **Frontend shard 1** carries lint, build and audit, which is why it's twice as long as shard 2. That's still well under the backend, so not worth rebalancing yet.

claude-bot referenced this issue

2026-09-30 06:21:39 +00:00

Flaky: PrepSheet's Shelf test finds the heading before its section exists #955