feat(evals): synthetic-session harness — an invented 30-minute session that exercises transcription end to end (#433) #497

Merged

claude-bot merged 16 commits from feat/433-synthetic-session into main 2026-09-05 06:01:59 +00:00

Conversation 0 Commits 16 Files changed 257 +20933 -1

claude-bot commented 2026-09-05 03:51:34 +00:00

Collaborator

Copy link

Closes #433 (build stages 1–4 and 6–8 of the plan; the LLM leg of stage 5 and its band calibration run during dev validation — see "Known gaps").

The per-speaker transcription change (#342) had only ever run against unit mocks. This is a committed, re-runnable session built from invented audio: no real voices, no real content.

What

• Scene (evals/synthetic_session/scene.py): 218 utterances, 1160 s of speech in 1800 s, six tracks + a presence-only member — GM, two linked players, a late joiner at 360 s, an unlinked guest, a quiet player (~68 s, mostly out-of-game) and an all-zeros silent track. Every line carries a register (IC 899 s / OOC 169 s / META 72 s / PROPOSED 20 s → table talk 20.7 %), 16.5 % overlap, the four seeded episodes, the proposed-but-unperformed plan, a repeated line by two speakers, twelve scripted silences. Timing sampled from statistics with a fixed seed; nothing from a real session. One documented deviation: the plan's speech shares were mutually inconsistent, so the majors' ratios and the quiet player's absolute figure were kept (GM lands at 44 %).
• Marker screening: 66 candidates × 2 carrier trials × the Opus chain × the live WhisperX server → 24 kept (36 % survival, matching the spike). Two findings recorded: survival is a property of the channel (the −20 dB echoed track needed 25 candidates for 6), and screening is itself noisy because piper has no seed.
• Clips: 218 committed .opus, 9.07 MB (within the carve-out budget), each verified by transcription (mean WER 4.8 %); the renderer may retry a synthesis failure but is forbidden from retrying a missing marker.
• Assembler + seeding: manifest + clips → the bot's own directory contract in full and ic_only assemblies; track_sha256 over PCM frames (a file hash would break on an ffmpeg upgrade for a non-audio reason); the resample step runs the bot's convert_to_whisper_wav command rather than importing the bot's numpy resampler into the backend. seed_db --i-know-this-writes --verify seeds all three resolve_speakers label paths and proves each resolves to the manifest's label.
• Golden transcript derived through the production renderer and byte-checked.
• Scorers: the harness's first precision metric (OOC leakage), marker partition (foreign = 0, per-speaker floor, order, position), word-boundary matching, numeral normalisation, VAD kept, WER, onset drift. Bands calibrated from three identical live ASR runs (large-v3-turbo, CUDA): session WER 3.73 % full / 3.32 % ic-only, marker detection 72 % / 65 %, foreign markers 0, onset drift median 0.07 s, the quiet player above the 30 s threshold in both assemblies. Exact assertions are exact; bands carry the measured value and the rationale.
• Whisper shim + negative fixtures: each asserts its mutation took (truncated WAV, zeroed tail, empty-for-one-file above and below the threshold, all-zero track with the has_audible_speech precondition, #344 snowflake-as-value on the unlinked owner, #342 symmetric swap).
• CI: the no-GPU parts run on every push with QB_SYNTHETIC_REQUIRE_AUDIO=1 so a silently skipped subject fails rather than passing. make synthetic-session. README with what it proves, the five caveats, and how to author and re-render.

Rebased onto main (changelog and Makefile conflicts; both sides kept — a follow-up commit repairs the Makefile after the first resolution went wrong).

Known gaps (also in the README)

• The full live run through POST /api/bot/sessions/{id}/audio with a real LLM, the full vs ic_only summary comparison, and leg 3 (live LLM on the ASR transcript) are not wired — the curl and the seeding/assembly are validated for it; the orchestrator runs it during dev validation.
• #432 is exercised directly but not pinned end-to-end.
• Marker screening is per-carrier and over-predicts survival by ~28 points in a full session (Marlgate passed 2/2 and was lost at all four occurrences) — the highest-value follow-on.

Verification

• Full backend suite 1856 passed (327 s, audio tests included); harness files alone 78 passed. ruff clean. Version sync OK. Stub eval leg 1.000 on all four metrics.

🤖 Generated with Claude Code

Closes #433 (build stages 1–4 and 6–8 of the plan; the LLM leg of stage 5 and its band calibration run during dev validation — see "Known gaps"). The per-speaker transcription change (#342) had only ever run against unit mocks. This is a committed, re-runnable session built from invented audio: no real voices, no real content. ## What - **Scene** (`evals/synthetic_session/scene.py`): 218 utterances, 1160 s of speech in 1800 s, six tracks + a presence-only member — GM, two linked players, a late joiner at 360 s, an unlinked guest, a quiet player (~68 s, mostly out-of-game) and an all-zeros silent track. Every line carries a register (IC 899 s / OOC 169 s / META 72 s / PROPOSED 20 s → table talk 20.7 %), 16.5 % overlap, the four seeded episodes, the proposed-but-unperformed plan, a repeated line by two speakers, twelve scripted silences. Timing sampled from statistics with a fixed seed; nothing from a real session. One documented deviation: the plan's speech shares were mutually inconsistent, so the majors' ratios and the quiet player's absolute figure were kept (GM lands at 44 %). - **Marker screening**: 66 candidates × 2 carrier trials × the Opus chain × the live WhisperX server → 24 kept (36 % survival, matching the spike). Two findings recorded: survival is a property of the channel (the −20 dB echoed track needed 25 candidates for 6), and screening is itself noisy because piper has no seed. - **Clips**: 218 committed `.opus`, 9.07 MB (within the carve-out budget), each verified by transcription (mean WER 4.8 %); the renderer may retry a synthesis failure but is forbidden from retrying a missing marker. - **Assembler + seeding**: manifest + clips → the bot's own directory contract in `full` and `ic_only` assemblies; `track_sha256` over PCM frames (a file hash would break on an ffmpeg upgrade for a non-audio reason); the resample step runs the bot's `convert_to_whisper_wav` command rather than importing the bot's numpy resampler into the backend. `seed_db --i-know-this-writes --verify` seeds all three `resolve_speakers` label paths and proves each resolves to the manifest's label. - **Golden transcript** derived through the production renderer and byte-checked. - **Scorers**: the harness's first precision metric (OOC leakage), marker partition (foreign = 0, per-speaker floor, order, position), word-boundary matching, numeral normalisation, VAD kept, WER, onset drift. **Bands calibrated from three identical live ASR runs** (`large-v3-turbo`, CUDA): session WER 3.73 % full / 3.32 % ic-only, marker detection 72 % / 65 %, **foreign markers 0**, onset drift median 0.07 s, the quiet player above the 30 s threshold in both assemblies. Exact assertions are exact; bands carry the measured value and the rationale. - **Whisper shim + negative fixtures**: each asserts its mutation took (truncated WAV, zeroed tail, empty-for-one-file above and below the threshold, all-zero track with the `has_audible_speech` precondition, #344 snowflake-as-value on the unlinked owner, #342 symmetric swap). - **CI**: the no-GPU parts run on every push with `QB_SYNTHETIC_REQUIRE_AUDIO=1` so a silently skipped subject fails rather than passing. `make synthetic-session`. README with what it proves, the five caveats, and how to author and re-render. Rebased onto main (changelog and Makefile conflicts; both sides kept — a follow-up commit repairs the Makefile after the first resolution went wrong). ## Known gaps (also in the README) - The full live run through `POST /api/bot/sessions/{id}/audio` with a real LLM, the `full` vs `ic_only` summary comparison, and leg 3 (live LLM on the ASR transcript) are not wired — the curl and the seeding/assembly are validated for it; the orchestrator runs it during dev validation. - #432 is exercised directly but not pinned end-to-end. - Marker screening is per-carrier and over-predicts survival by ~28 points in a full session (`Marlgate` passed 2/2 and was lost at all four occurrences) — the highest-value follow-on. ## Verification - Full backend suite **1856 passed** (327 s, audio tests included); harness files alone 78 passed. ruff clean. Version sync OK. Stub eval leg 1.000 on all four metrics. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot scheduled this pull request to auto merge when all checks succeed 2026-09-05 03:51:41 +00:00

claude-bot referenced this pull request 2026-09-05 04:37:17 +00:00

[Backend] Evaluate ASR and LLM providers against the golden corpus before committing #360

rbrooks force-pushed feat/433-synthetic-session from b916e6920e to ceb6d51bfc 2026-09-05 05:20:23 +00:00 Compare

rbrooks added 1 commit 2026-09-05 05:45:47 +00:00

ci: point the stub eval leg at the harness's eval fixture directory (#433) ... f48f5b2109

The fixture loader globs every *.json in the directory it is given; the
synthetic session's root holds manifest, calibration and screening files
that are not eval fixtures. The Makefile target already used the eval/
subdirectory; the CI step did not.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot referenced this pull request 2026-09-05 05:51:02 +00:00

[Backend] Follow-ups from the provider evaluation: usage rows misclassify sniffed managed runs, managed ASR is unpriceable, and the live tooling cannot reach the managed adapter (#360) #502

claude-bot merged commit 985ef7971a into main 2026-09-05 06:01:59 +00:00

claude-bot deleted branch feat/433-synthetic-session 2026-09-05 06:02:00 +00:00

claude-bot referenced this pull request from a commit 2026-09-05 06:02:00 +00:00

Merge PR #497: synthetic-session transcription harness (#433)

claude-bot referenced this pull request 2026-09-05 06:02:51 +00:00

[Evals] Synthetic-audio end-to-end harness for the per-speaker transcription pipeline #433

claude-bot referenced this pull request 2026-09-05 06:04:34 +00:00

chore(release): v4.2.0 — AI Provider Abstraction & BYO-AI #504

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backend

Issue related to the backend

  

bot

Issue related to the bot

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

frontend

Issue related to the frontend

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

recording

Issue related to the recording system

  

security

Security hardening / vulnerability work

  

wontfix

This won't be fixed

No labels

backend

bot

bug

duplicate

enhancement

frontend

help wanted

invalid

question

recording

security

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/Quest-Board!497

No description provided.