CI: run the synthetic-session harness on a PR only when it touches the recording path #784

Merged

claude-bot merged 2 commits from build/ci-harness-paths into main 2026-09-23 19:31:18 +00:00

Conversation 0 Commits 2 Files changed 2 +72

claude-bot commented 2026-09-23 18:39:13 +00:00

Collaborator

Copy link

The harness is CI's slowest job, and most PRs never go near the recording path.

Why

From the last green run (#10340):

Job	Duration
Synthetic session harness	12m 41s
Backend tests	10m 22s
Journeys	8m 37s
Docker image build	3m 37s

It also costs more than its own twelve minutes. Journeys depends on the Docker build, which finished at 16:26:36, but journeys didn't start until 16:31:10. It waited 4½ minutes for a runner while the harness held one, and journeys is the job that actually gates a merge. Runner load was also behind two of today's red runs: an image that vanished mid-job, and a Home.test.jsx timeout.

How

A small audio_paths job reads the PR's diff against its base and outputs run. The harness needs it, and runs when:

• a pull request touches something it exercises:
  • its own code, fixtures and tests
  • the backend services it imports (audio, speaker, beat, vocabulary, llm, settings, provider_slots) and app/providers/
  • the bot's audio utilities and recording cog; the harness vendors the bot's ffmpeg command, so a change there has to re-run it
  • the Dockerfile or backend requirements; it runs inside the backend image, whose ffmpeg and libopus are part of what it measures
• always on a push to main or a manual dispatch, so anything the list misses through a transitive import is caught on main rather than never
• it fails open: if the diff can't be read, the harness runs

It stays inside CI, not a workflow of its own, so it remains under branch protection's CI / *. A red harness still blocks the PRs it runs on.

This PR is also the test

What isn't known yet is whether Forgejo counts a skipped required job as passing. If it doesn't, every PR that skips the harness would hang unmergeable. So this is tested in two pushes before merging:

1. 0734f28, the path list deliberately excludes ci.yml. This PR touches only ci.yml and the changelog, so the harness should be skipped. If the PR is then mergeable, a skip passes the gate.
2. Next commit adds ci.yml to the list, and the harness should run. That proves the detection fires when it's meant to.

The results go in a comment here before merging. The pattern was checked against 18 real and near-miss paths before pushing; for example, audio_service_helpers.py and Dockerfile.dev must not match.

🤖 Generated with Claude Code

The harness is CI's slowest job, and most PRs never go near the recording path. ## Why From the last green run (#10340): | Job | Duration | |---|---| | **Synthetic session harness** | **12m 41s** | | Backend tests | 10m 22s | | Journeys | 8m 37s | | Docker image build | 3m 37s | It also costs more than its own twelve minutes. Journeys depends on the Docker build, which finished at 16:26:36, but journeys **didn't start until 16:31:10**. It waited 4½ minutes for a runner while the harness held one, and journeys is the job that actually gates a merge. Runner load was also behind two of today's red runs: an image that vanished mid-job, and a `Home.test.jsx` timeout. ## How A small `audio_paths` job reads the PR's diff against its base and outputs `run`. The harness `needs` it, and runs when: - **a pull request touches something it exercises:** - its own code, fixtures and tests - the backend services it imports (`audio`, `speaker`, `beat`, `vocabulary`, `llm`, `settings`, `provider_slots`) and `app/providers/` - the bot's audio utilities and recording cog; the harness *vendors* the bot's ffmpeg command, so a change there has to re-run it - the `Dockerfile` or backend requirements; it runs inside the backend image, whose ffmpeg and libopus are part of what it measures - **always** on a push to `main` or a manual dispatch, so anything the list misses through a transitive import is caught on `main` rather than never - **it fails open**: if the diff can't be read, the harness runs It stays inside CI, not a workflow of its own, so it remains under branch protection's `CI / *`. A red harness still blocks the PRs it runs on. ## This PR is also the test What isn't known yet is **whether Forgejo counts a *skipped* required job as passing.** If it doesn't, every PR that skips the harness would hang unmergeable. So this is tested in two pushes before merging: 1. **`0734f28`, the path list deliberately excludes `ci.yml`.** This PR touches only `ci.yml` and the changelog, so the harness should be **skipped**. If the PR is then mergeable, a skip passes the gate. 2. **Next commit adds `ci.yml` to the list**, and the harness should **run**. That proves the detection fires when it's meant to. The results go in a comment here before merging. The pattern was checked against 18 real and near-miss paths before pushing; for example, `audio_service_helpers.py` and `Dockerfile.dev` must *not* match. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit 2026-09-23 18:39:13 +00:00

ci: run the synthetic-session harness on a PR only when it touches the recording path ... 0734f28f46

The harness is CI's slowest job (12m41s on the last green run), and while it
holds a runner the journeys job waits for one: journeys' dependency finished
at 16:26:36 and it did not start until 16:31:10. So on most PRs — which never
go near the recording path — it costs more than its own twelve minutes.

A small `audio_paths` job reads the PR's diff against its base and outputs
`run`. The harness `needs` it and runs only when `run == 'true'`:

- a pull request touching its own code, fixtures or tests; the backend
  services it imports (audio, speaker, beat, vocabulary, llm, settings,
  provider_slots) and `app/providers/`; the bot's audio utilities and
  recording cog (it vendors the bot's ffmpeg command); the Dockerfile or
  backend requirements (it runs inside the backend image)
- **always** on a push to main or a manual dispatch
- **fails open**: an unreadable diff runs it

It stays in CI rather than moving to a workflow of its own so it remains
under branch protection's `CI / *` — a red harness still blocks the PRs it
runs on. What this commit is also for is finding out whether Forgejo counts
a *skipped* required job as passing.

**Deliberately incomplete, for that test:** `ci.yml` is not in the path list,
so this PR — which touches only `ci.yml` and the changelog — should see the
harness skipped. If the PR is then mergeable, a skip passes the gate. The
next commit adds `ci.yml` to the list and should see it run.

The pattern was checked against 18 real and near-miss paths before pushing
(e.g. `audio_service_helpers.py` and `Dockerfile.dev` must not match).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

rbrooks added 1 commit 2026-09-23 18:45:46 +00:00

docs(changelog): the harness's measured cost, precisely ... 9f5b0f3f3b

Also the retrigger for the skip test: journeys failed on the first run for
an infrastructure reason — a container removed from the shared Docker daemon
between `create` and `start`, before any journey ran — so that run could not
say whether a skipped harness passes the `CI / *` gate. This commit touches
only the changelog, so the harness must still be skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude-bot merged commit 80cdf42d21 into main 2026-09-23 19:31:18 +00:00

claude-bot referenced this pull request from a commit 2026-09-23 19:31:20 +00:00

Merge PR #784: CI runs the synthetic-session harness on a PR only when it touches the recording path

rbrooks referenced this pull request from a commit 2026-09-23 19:32:15 +00:00

ci: a change to the workflow itself runs the synthetic-session harness

claude-bot referenced this pull request 2026-09-23 19:32:23 +00:00

CI: a change to the workflow itself runs the synthetic-session harness #786

claude-bot referenced this pull request 2026-09-23 20:56:52 +00:00

Build lane 3: the campaign core and Settings (CAMP-1 to CAMP-17, SH-7) #729

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backend

Issue related to the backend

  

bot

Issue related to the bot

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

frontend

Issue related to the frontend

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

recording

Issue related to the recording system

  

security

Security hardening / vulnerability work

  

wontfix

This won't be fixed

No labels

backend

bot

bug

duplicate

enhancement

frontend

help wanted

invalid

question

recording

security

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/Quest-Board!784

No description provided.