The seven journeys as end-to-end tests: the Playwright harness #726

Closed
opened 2026-09-22 00:12:18 +00:00 by claude-bot · 5 comments
Collaborator

Part of the v4.5.0 build plan (docs/design/v4.5.0-build-plan.md §4; umbrella #366). The brief's seven journeys (§2) are the acceptance bar of every build lane. This issue is the harness they run in; each lane then adds or extends the specs that cross its surfaces.

What it is

A Playwright suite under webapp/frontend/e2e/, one spec per journey (j1-first-campaign.spec.ts … j7-a-character.spec.ts), written against roles and the brief's vocabulary (getByRole('button', { name: 'Send my votes' })), never against CSS classes. The fiction is the prototype's (Dana, Priya, The Amberwake Accord, Session 14), so a test asserts what the mockup shows.

The stack it runs against

  • The app: the Compose stack CI already builds for the synthetic-session job (db, redis, migrate, backend, worker, frontend), started once per run from a docker-compose.e2e.yml overlay. The bot container is not started.
  • Sign-in: a small fake OIDC issuer container, so the real login and callback code runs and each test picks a seeded identity. No test-only login endpoint in the product.
  • Seed: a script that creates the fiction through the API before the run; each test creates what it mutates under a unique name.
  • The seams: a fake bot HTTP server that records what it was asked to post and serves it back to the tests; a mail catcher the tests query; the stub ASR provider (providers/stub.py) and the eval harness's stub LLM enabled for the worker by an environment setting this stack alone sets. Foundry and Stripe are not exercised.
  • J4 starts where the bot hands audio over: the test posts the committed synthetic-session clips to the bot audio endpoint and follows the pipeline through transcript, summary, review and approval (J5) without a GPU.

Three projects per journey

Desktop viewport; an emulated phone; keyboard-only (Tab, Enter, Space, no pointer). That is the brief's "passes on desktop and on a phone" and #367's "every core journey completable by keyboard alone" as executable checks. @axe-core/playwright scans every page a journey visits and fails on a WCAG 2.2 AA violation.

In CI

One job, journeys, after the image build; traces, videos and screenshots kept as artefacts on failure. The same suite runs locally against the Docker loop with npx playwright test.

Acceptance

  • The overlay, the fake issuer, the mail catcher, the fake bot server and the seed script, documented in webapp/frontend/e2e/README.md
  • J1 and J2 green in all three projects, introduced with lane 1's shell
  • The journeys job in ci.yml with artefacts on failure; a red journey blocks merge like any other job
  • Each later lane's issue lists the journeys it must keep or add
**Part of the v4.5.0 build plan** (`docs/design/v4.5.0-build-plan.md` §4; umbrella #366). The brief's seven journeys (§2) are the acceptance bar of every build lane. This issue is the harness they run in; each lane then adds or extends the specs that cross its surfaces. ## What it is A Playwright suite under `webapp/frontend/e2e/`, one spec per journey (`j1-first-campaign.spec.ts` … `j7-a-character.spec.ts`), written against roles and the brief's vocabulary (`getByRole('button', { name: 'Send my votes' })`), never against CSS classes. The fiction is the prototype's (Dana, Priya, The Amberwake Accord, Session 14), so a test asserts what the mockup shows. ## The stack it runs against - **The app**: the Compose stack CI already builds for the synthetic-session job (db, redis, migrate, backend, worker, frontend), started once per run from a `docker-compose.e2e.yml` overlay. The bot container is not started. - **Sign-in**: a small fake OIDC issuer container, so the real login and callback code runs and each test picks a seeded identity. No test-only login endpoint in the product. - **Seed**: a script that creates the fiction through the API before the run; each test creates what it mutates under a unique name. - **The seams**: a fake bot HTTP server that records what it was asked to post and serves it back to the tests; a mail catcher the tests query; the stub ASR provider (`providers/stub.py`) and the eval harness's stub LLM enabled for the worker by an environment setting this stack alone sets. Foundry and Stripe are not exercised. - **J4** starts where the bot hands audio over: the test posts the committed synthetic-session clips to the bot audio endpoint and follows the pipeline through transcript, summary, review and approval (J5) without a GPU. ## Three projects per journey Desktop viewport; an emulated phone; keyboard-only (Tab, Enter, Space, no pointer). That is the brief's "passes on desktop and on a phone" and #367's "every core journey completable by keyboard alone" as executable checks. `@axe-core/playwright` scans every page a journey visits and fails on a WCAG 2.2 AA violation. ## In CI One job, `journeys`, after the image build; traces, videos and screenshots kept as artefacts on failure. The same suite runs locally against the Docker loop with `npx playwright test`. ## Acceptance - [ ] The overlay, the fake issuer, the mail catcher, the fake bot server and the seed script, documented in `webapp/frontend/e2e/README.md` - [ ] J1 and J2 green in all three projects, introduced with lane 1's shell - [ ] The `journeys` job in `ci.yml` with artefacts on failure; a red journey blocks merge like any other job - [ ] Each later lane's issue lists the journeys it must keep or add
Author
Collaborator

Starting the harness (2026-09-22), in parallel with lane 1's server change (#727). First PR: the Compose overlay, the fake OIDC issuer, the mail catcher, the fake bot server, the seed script, the Playwright configuration with its three projects and the axe scan, the journeys CI job with artefacts on failure, and one smoke journey (sign in through the fake issuer, land on Home) so the pipe is proven end to end. J1 and J2 follow once the new shell is in the app, since the specs are written against its roles and names.

Starting the harness (2026-09-22), in parallel with lane 1's server change (#727). First PR: the Compose overlay, the fake OIDC issuer, the mail catcher, the fake bot server, the seed script, the Playwright configuration with its three projects and the axe scan, the `journeys` CI job with artefacts on failure, and one smoke journey (sign in through the fake issuer, land on Home) so the pipe is proven end to end. J1 and J2 follow once the new shell is in the app, since the specs are written against its roles and names.
Author
Collaborator

The harness is built and runs green locally, end to end, in one command (make journeys / scripts/journeys.sh): the stack from docker-compose.e2e.yml, the fake OIDC issuer, Mailpit, a fake bot, a fake AI server, the seed (Dana runs The Amberwake Accord, Priya plays in it, Session 13 completed, Session 14 a week ahead), then Playwright in the three projects with the axe scan. Nine smoke tests pass; the seed also proves that product mail reaches the catcher and that a session_confirmed event reaches the fake bot off the Redis stream.

Departures from this issue's text, all deliberate:

  • JavaScript, not TypeScript. The repo has no TypeScript toolchain, and Playwright strips types without checking them, so .ts files would have looked typed and checked nothing. Renaming later is mechanical.
  • A fake AI server, not the stub provider adapters. providers/stub.py says in its own docstring why the stub ASR is kept out of the registry (an operator-selectable provider that invents text is a foot-gun), and there is no stub LLM at the provider interface at all. The endpoint URL is already configuration, so the harness runs a small server that speaks the bundled WhisperX and llama.cpp shapes and the real adapters run against it. Strictly more coverage, and zero product code changed.
  • The fake bot also consumes the Redis stream, because that is how a campaign with a guild is told things; an HTTP-only fake would have asserted against silence.
  • The axe gate ships with an owned baseline: the pre-redesign pages J0 visits have real contrast failures (/login's guide links; /dashboard's New Campaign button, role pills and countdown link). They are listed per page, per node, with the lane that removes them (#727, #728); anything else fails, including a new violation of the same rule on the same page. QB_E2E_STRICT_A11Y=1 ignores the baseline so a lane can ask whether its page is clean.
  • The keyboard project is enforced: in it, any pointer call throws, and J0 asserts the guard itself.

Housekeeping the harness needed: .gitattributes forcing LF on shell scripts (a CRLF init.sh stops Postgres on a Windows checkout), PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD in the frontend images and CI job, ruff over e2e/, a gitleaks allowlist for the obviously fake .env.e2e, exact pins for @playwright/test, @axe-core/playwright and the Playwright image, which move together.

One thing that cannot be verified from here: whether the Forgejo runner's bind mounts reach the workspace (the synthetic-session job already works around docker-in-docker for the same reason). The script probes it right after up and fails with a message naming the remedy; if the first CI run dies there, the fix is to bake e2e/ into the fake and seed images as that job does.

Found on the way: #743 (campaign create silently drops eight declared fields). Next: rebase onto the shell (#742) once it merges, rewrite J0 against the new frame (the account menu and the Contents instead of the old nav bar), unify the fiction on the prototype's "Dana Whitlock", then the PR.

The harness is built and runs green locally, end to end, in one command (`make journeys` / `scripts/journeys.sh`): the stack from `docker-compose.e2e.yml`, the fake OIDC issuer, Mailpit, a fake bot, a fake AI server, the seed (Dana runs The Amberwake Accord, Priya plays in it, Session 13 completed, Session 14 a week ahead), then Playwright in the three projects with the axe scan. Nine smoke tests pass; the seed also proves that product mail reaches the catcher and that a `session_confirmed` event reaches the fake bot off the Redis stream. Departures from this issue's text, all deliberate: - **JavaScript, not TypeScript.** The repo has no TypeScript toolchain, and Playwright strips types without checking them, so `.ts` files would have looked typed and checked nothing. Renaming later is mechanical. - **A fake AI server, not the stub provider adapters.** `providers/stub.py` says in its own docstring why the stub ASR is kept out of the registry (an operator-selectable provider that invents text is a foot-gun), and there is no stub LLM at the provider interface at all. The endpoint URL is already configuration, so the harness runs a small server that speaks the bundled WhisperX and llama.cpp shapes and the real adapters run against it. Strictly more coverage, and zero product code changed. - **The fake bot also consumes the Redis stream**, because that is how a campaign with a guild is told things; an HTTP-only fake would have asserted against silence. - **The axe gate ships with an owned baseline**: the pre-redesign pages J0 visits have real contrast failures (`/login`'s guide links; `/dashboard`'s New Campaign button, role pills and countdown link). They are listed per page, per node, with the lane that removes them (#727, #728); anything else fails, including a new violation of the same rule on the same page. `QB_E2E_STRICT_A11Y=1` ignores the baseline so a lane can ask whether its page is clean. - **The keyboard project is enforced**: in it, any pointer call throws, and J0 asserts the guard itself. Housekeeping the harness needed: `.gitattributes` forcing LF on shell scripts (a CRLF `init.sh` stops Postgres on a Windows checkout), `PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD` in the frontend images and CI job, ruff over `e2e/`, a gitleaks allowlist for the obviously fake `.env.e2e`, exact pins for `@playwright/test`, `@axe-core/playwright` and the Playwright image, which move together. One thing that cannot be verified from here: whether the Forgejo runner's bind mounts reach the workspace (the `synthetic-session` job already works around docker-in-docker for the same reason). The script probes it right after `up` and fails with a message naming the remedy; if the first CI run dies there, the fix is to bake `e2e/` into the fake and seed images as that job does. Found on the way: #743 (campaign create silently drops eight declared fields). Next: rebase onto the shell (#742) once it merges, rewrite J0 against the new frame (the account menu and the Contents instead of the old nav bar), unify the fiction on the prototype's "Dana Whitlock", then the PR.
Author
Collaborator

#746 is merged on green CI, with the journeys job green on the runner twice over. What is on main: docker-compose.e2e.yml with no host bind mount (a backend test refuses one), the fake OIDC issuer, Mailpit, the fake bot that also reads the Redis stream, the fake AI server, the seed in the prototype's fiction, Playwright in three projects with the enforced keyboard guard and the WCAG 2.2 AA axe gate with its owned baseline, scripts/journeys.sh / make journeys, the journeys CI job with artefacts on failure (upload-artifact@v3; v4 is not supported on Forgejo), and webapp/frontend/e2e/README.md.

Acceptance so far: the overlay, fakes, seed and README are done; the CI job blocks a PR like any other; each lane issue names its journeys (the build plan's table). Still open here: J1 and J2 green in all three projects, which arrive with lane 1's Home PR (in progress), so this stays open until then.

On its first day the gate found #745 (fixed) and the seed found #743 (fixed).

#746 is merged on green CI, with the `journeys` job green on the runner twice over. What is on main: `docker-compose.e2e.yml` with no host bind mount (a backend test refuses one), the fake OIDC issuer, Mailpit, the fake bot that also reads the Redis stream, the fake AI server, the seed in the prototype's fiction, Playwright in three projects with the enforced keyboard guard and the WCAG 2.2 AA axe gate with its owned baseline, `scripts/journeys.sh` / `make journeys`, the `journeys` CI job with artefacts on failure (`upload-artifact@v3`; v4 is not supported on Forgejo), and `webapp/frontend/e2e/README.md`. Acceptance so far: the overlay, fakes, seed and README are done; the CI job blocks a PR like any other; each lane issue names its journeys (the build plan's table). Still open here: J1 and J2 green in all three projects, which arrive with lane 1's Home PR (in progress), so this stays open until then. On its first day the gate found #745 (fixed) and the seed found #743 (fixed).
Author
Collaborator

With #749 merged, J1 (a first-time GM sees the two ways in; Dana starts a campaign from nothing and lands on Home with it first) and J2 (Priya is asked whether she is in, on Home and in the bell) are green in all three projects on the runner, and /dashboard and /campaigns/new pass the strict axe gate; /login is the only tolerated entry left, owned by lane 2. That completes this issue's acceptance: the harness with its fakes, seed and README; J1 and J2 introduced with lane 1; the journeys job blocking merges; each later lane naming its journeys. Later lanes extend the specs from here. Closing.

With #749 merged, J1 (a first-time GM sees the two ways in; Dana starts a campaign from nothing and lands on Home with it first) and J2 (Priya is asked whether she is in, on Home and in the bell) are green in all three projects on the runner, and `/dashboard` and `/campaigns/new` pass the strict axe gate; `/login` is the only tolerated entry left, owned by lane 2. That completes this issue's acceptance: the harness with its fakes, seed and README; J1 and J2 introduced with lane 1; the `journeys` job blocking merges; each later lane naming its journeys. Later lanes extend the specs from here. Closing.
Author
Collaborator

J6 has landed in #770, and it surfaced two things about the harness itself

Neither is urgent; both are the kind of thing that costs a day when somebody meets it cold.

The suite is now visibly paying the sign-in limiter

J6 costs three sign-ins (one per project), taking the suite to six per project, eighteen per run, against a limit of 10/minute per IP.

signInAs waits the 429 out, so every run is green and nothing is broken. But one test per run now takes ~23s instead of ~3s — and which test moves between runs: [phone] J1 in one, [phone] J2 in the next. That wandering is the signature of a rate-limit window being waited out, not of a slow page, and it is exactly the shape of thing somebody will otherwise spend an afternoon profiling.

J3, J4 and J5 will make it two or three tests. Worth deciding before then whether the e2e stack should raise the limit for its own OIDC fake, or whether the suite should share a signed-in state between projects.

The axe gate only ever sees light mode

Chromium's default prefers-color-scheme is light, so every scan in the suite is a light-mode scan. The brief (§6, constraint 6) makes both themes first-class on every page.

This is not theoretical: #770 fixed a real 1.4.3 contrast failure whose dark half is computed, not measured, because nothing in the suite can measure it. A colorScheme: "dark" pass, or a fourth project, would close it — at the cost of a third or a quarter more run time, which is why it is a decision for whoever owns this issue rather than something to slip into a lane's PR.

Also

  • identities.json carried the GM's retired surname ("sub": "e2e-dana-okafor") — in the file this issue's README calls "the one place their names are written down". Fixed in #770.
  • a11y-baseline.js cannot describe a campaign-scoped page at all — filed as #771. It matters to this issue because from lane 3 onward, "tolerate this one rule on this one page" is not expressible; the only key that matches a UUID address is "*", which quietens the rule everywhere.
  • Two new fixtures, openContents() and goVia(name, url), documented in the README's fixture table. The Contents is a rail on a desktop and a sheet behind a button on a phone; every lane from here walks it, and one spelling beats three.

🤖 Generated with Claude Code

## J6 has landed in #770, and it surfaced two things about the harness itself Neither is urgent; both are the kind of thing that costs a day when somebody meets it cold. ### The suite is now visibly paying the sign-in limiter J6 costs three sign-ins (one per project), taking the suite to **six per project, eighteen per run**, against a limit of **10/minute per IP**. `signInAs` waits the 429 out, so every run is green and nothing is broken. But **one test per run now takes ~23s instead of ~3s** — and *which* test moves between runs: `[phone] J1` in one, `[phone] J2` in the next. That wandering is the signature of a rate-limit window being waited out, not of a slow page, and it is exactly the shape of thing somebody will otherwise spend an afternoon profiling. J3, J4 and J5 will make it two or three tests. Worth deciding before then whether the e2e stack should raise the limit for its own OIDC fake, or whether the suite should share a signed-in state between projects. ### The axe gate only ever sees light mode Chromium's default `prefers-color-scheme` is light, so **every scan in the suite is a light-mode scan**. The brief (§6, constraint 6) makes both themes first-class on every page. This is not theoretical: #770 fixed a real 1.4.3 contrast failure whose **dark half is computed, not measured**, because nothing in the suite can measure it. A `colorScheme: "dark"` pass, or a fourth project, would close it — at the cost of a third or a quarter more run time, which is why it is a decision for whoever owns this issue rather than something to slip into a lane's PR. ### Also - **`identities.json` carried the GM's retired surname** (`"sub": "e2e-dana-okafor"`) — in the file this issue's README calls "the one place their names are written down". Fixed in #770. - **`a11y-baseline.js` cannot describe a campaign-scoped page at all** — filed as #771. It matters to this issue because from lane 3 onward, "tolerate this one rule on this one page" is not expressible; the only key that matches a UUID address is `"*"`, which quietens the rule everywhere. - Two new fixtures, `openContents()` and `goVia(name, url)`, documented in the README's fixture table. The Contents is a rail on a desktop and a sheet behind a button on a phone; every lane from here walks it, and one spelling beats three. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign in to join this conversation.
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/Quest-Board#726
No description provided.