feat(evals): synthetic-session harness — an invented 30-minute session that exercises transcription end to end (#433) #497

Merged

claude-bot merged 16 commits from feat/433-synthetic-session into main 2026-09-05 06:01:59 +00:00

Conversation 0 Commits 16 Files changed 257 +20933 -1

claude-bot commented 2026-09-05 03:51:34 +00:00

Contributor

Copy link

Closes #433 (build stages 1–4 and 6–8 of the plan; the LLM leg of stage 5 and its band calibration run during dev validation — see "Known gaps").

The per-speaker transcription change (#342) had only ever run against unit mocks. This is a committed, re-runnable session built from invented audio: no real voices, no real content.

What

• Scene (evals/synthetic_session/scene.py): 218 utterances, 1160 s of speech in 1800 s, six tracks + a presence-only member — GM, two linked players, a late joiner at 360 s, an unlinked guest, a quiet player (~68 s, mostly out-of-game) and an all-zeros silent track. Every line carries a register (IC 899 s / OOC 169 s / META 72 s / PROPOSED 20 s → table talk 20.7 %), 16.5 % overlap, the four seeded episodes, the proposed-but-unperformed plan, a repeated line by two speakers, twelve scripted silences. Timing sampled from statistics with a fixed seed; nothing from a real session. One documented deviation: the plan's speech shares were mutually inconsistent, so the majors' ratios and the quiet player's absolute figure were kept (GM lands at 44 %).
• Marker screening: 66 candidates × 2 carrier trials × the Opus chain × the live WhisperX server → 24 kept (36 % survival, matching the spike). Two findings recorded: survival is a property of the channel (the −20 dB echoed track needed 25 candidates for 6), and screening is itself noisy because piper has no seed.
• Clips: 218 committed .opus, 9.07 MB (within the carve-out budget), each verified by transcription (mean WER 4.8 %); the renderer may retry a synthesis failure but is forbidden from retrying a missing marker.
• Assembler + seeding: manifest + clips → the bot's own directory contract in full and ic_only assemblies; track_sha256 over PCM frames (a file hash would break on an ffmpeg upgrade for a non-audio reason); the resample step runs the bot's convert_to_whisper_wav command rather than importing the bot's numpy resampler into the backend. seed_db --i-know-this-writes --verify seeds all three resolve_speakers label paths and proves each resolves to the manifest's label.
• Golden transcript derived through the production renderer and byte-checked.
• Scorers: the harness's first precision metric (OOC leakage), marker partition (foreign = 0, per-speaker floor, order, position), word-boundary matching, numeral normalisation, VAD kept, WER, onset drift. Bands calibrated from three identical live ASR runs (large-v3-turbo, CUDA): session WER 3.73 % full / 3.32 % ic-only, marker detection 72 % / 65 %, foreign markers 0, onset drift median 0.07 s, the quiet player above the 30 s threshold in both assemblies. Exact assertions are exact; bands carry the measured value and the rationale.
• Whisper shim + negative fixtures: each asserts its mutation took (truncated WAV, zeroed tail, empty-for-one-file above and below the threshold, all-zero track with the has_audible_speech precondition, #344 snowflake-as-value on the unlinked owner, #342 symmetric swap).
• CI: the no-GPU parts run on every push with QB_SYNTHETIC_REQUIRE_AUDIO=1 so a silently skipped subject fails rather than passing. make synthetic-session. README with what it proves, the five caveats, and how to author and re-render.

Rebased onto main (changelog and Makefile conflicts; both sides kept — a follow-up commit repairs the Makefile after the first resolution went wrong).

Known gaps (also in the README)

• The full live run through POST /api/bot/sessions/{id}/audio with a real LLM, the full vs ic_only summary comparison, and leg 3 (live LLM on the ASR transcript) are not wired — the curl and the seeding/assembly are validated for it; the orchestrator runs it during dev validation.
• #432 is exercised directly but not pinned end-to-end.
• Marker screening is per-carrier and over-predicts survival by ~28 points in a full session (Marlgate passed 2/2 and was lost at all four occurrences) — the highest-value follow-on.

Verification

• Full backend suite 1856 passed (327 s, audio tests included); harness files alone 78 passed. ruff clean. Version sync OK. Stub eval leg 1.000 on all four metrics.

🤖 Generated with Claude Code

Closes #433 (build stages 1–4 and 6–8 of the plan; the LLM leg of stage 5 and its band calibration run during dev validation — see "Known gaps"). The per-speaker transcription change (#342) had only ever run against unit mocks. This is a committed, re-runnable session built from invented audio: no real voices, no real content. ## What - **Scene** (`evals/synthetic_session/scene.py`): 218 utterances, 1160 s of speech in 1800 s, six tracks + a presence-only member — GM, two linked players, a late joiner at 360 s, an unlinked guest, a quiet player (~68 s, mostly out-of-game) and an all-zeros silent track. Every line carries a register (IC 899 s / OOC 169 s / META 72 s / PROPOSED 20 s → table talk 20.7 %), 16.5 % overlap, the four seeded episodes, the proposed-but-unperformed plan, a repeated line by two speakers, twelve scripted silences. Timing sampled from statistics with a fixed seed; nothing from a real session. One documented deviation: the plan's speech shares were mutually inconsistent, so the majors' ratios and the quiet player's absolute figure were kept (GM lands at 44 %). - **Marker screening**: 66 candidates × 2 carrier trials × the Opus chain × the live WhisperX server → 24 kept (36 % survival, matching the spike). Two findings recorded: survival is a property of the channel (the −20 dB echoed track needed 25 candidates for 6), and screening is itself noisy because piper has no seed. - **Clips**: 218 committed `.opus`, 9.07 MB (within the carve-out budget), each verified by transcription (mean WER 4.8 %); the renderer may retry a synthesis failure but is forbidden from retrying a missing marker. - **Assembler + seeding**: manifest + clips → the bot's own directory contract in `full` and `ic_only` assemblies; `track_sha256` over PCM frames (a file hash would break on an ffmpeg upgrade for a non-audio reason); the resample step runs the bot's `convert_to_whisper_wav` command rather than importing the bot's numpy resampler into the backend. `seed_db --i-know-this-writes --verify` seeds all three `resolve_speakers` label paths and proves each resolves to the manifest's label. - **Golden transcript** derived through the production renderer and byte-checked. - **Scorers**: the harness's first precision metric (OOC leakage), marker partition (foreign = 0, per-speaker floor, order, position), word-boundary matching, numeral normalisation, VAD kept, WER, onset drift. **Bands calibrated from three identical live ASR runs** (`large-v3-turbo`, CUDA): session WER 3.73 % full / 3.32 % ic-only, marker detection 72 % / 65 %, **foreign markers 0**, onset drift median 0.07 s, the quiet player above the 30 s threshold in both assemblies. Exact assertions are exact; bands carry the measured value and the rationale. - **Whisper shim + negative fixtures**: each asserts its mutation took (truncated WAV, zeroed tail, empty-for-one-file above and below the threshold, all-zero track with the `has_audible_speech` precondition, #344 snowflake-as-value on the unlinked owner, #342 symmetric swap). - **CI**: the no-GPU parts run on every push with `QB_SYNTHETIC_REQUIRE_AUDIO=1` so a silently skipped subject fails rather than passing. `make synthetic-session`. README with what it proves, the five caveats, and how to author and re-render. Rebased onto main (changelog and Makefile conflicts; both sides kept — a follow-up commit repairs the Makefile after the first resolution went wrong). ## Known gaps (also in the README) - The full live run through `POST /api/bot/sessions/{id}/audio` with a real LLM, the `full` vs `ic_only` summary comparison, and leg 3 (live LLM on the ASR transcript) are not wired — the curl and the seeding/assembly are validated for it; the orchestrator runs it during dev validation. - #432 is exercised directly but not pinned end-to-end. - Marker screening is per-carrier and over-predicts survival by ~28 points in a full session (`Marlgate` passed 2/2 and was lost at all four occurrences) — the highest-value follow-on. ## Verification - Full backend suite **1856 passed** (327 s, audio tests included); harness files alone 78 passed. ruff clean. Version sync OK. Stub eval leg 1.000 on all four metrics. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 10 commits 2026-09-05 03:51:34 +00:00

feat(evals): scene manifest — the authored session and the marker screening rig (#433) ... badfff852f

The per-speaker transcription change (#342) is the riskiest thing in v4.0.0 and
has never run against a real Whisper server outside unit mocks. Neither has
speaker resolution against real member rows. The existing eval harness starts
from a *transcript*, so nothing in it exercises transcription at all.

This is the first half of closing that gap: a fully invented thirty-minute
session, written for the fixture, from which everything else is derived.

**scene.py** is the only file here that needs authoring judgement. It holds 218
turns across six speakers, each carrying a register — `ic` / `ooc` / `meta` /
`proposed` — because every table-talk metric is scored against those labels and
not against prose style. About a fifth of the session is table talk, in four
seeded episodes chosen to separate failures that would otherwise blur together:
a late arrival whose recap accurately describes real in-fiction events in an
out-of-game register, a joke that spirals across dense overlapping turns, a
rules lookup full of character names and action verbs that reads exactly like an
event, and a snack break that is a genuine low-activity region and not the end
of the session. A plan is proposed in character and never performed, which is
the one exclusion `beat_service.py`'s prompt states outright and nothing tests.

**Markers are selected, not chosen.** The spike found three of eight invented
proper nouns survived ASR on a word boundary, and the failures followed no rule
anyone could state in advance — `Thistledown` was perfect, `Copperkettle` came
back *copperkittle*. So `screen_markers.py` renders each candidate in two
carrier sentences with its owner's voice, pushes them through the same Opus
chain the clips will use, transcribes them on the real server, and keeps only
what survives both trials as a whole word. Measured: **24 of 66 candidates,
36.4%**, in line with the spike.

Two findings from that run are recorded in `screening.json` and shape the rest:
survival is a property of the *channel* as much as the word (the 20 dB-down
track needed 25 candidates to yield 6, where clean channels yielded 5 from 11),
and screening is itself noisy — the same word passed 2/2 on one run and 1/2 on
the next, because piper has no `--seed`.

`text_norm.py` matches on **whole tokens**. The spike scored survival with a
substring test and overstated it by two of eight: `Varnholt` "survived" inside
*varnholtz*. That is not a rounding error, it is a metric that points the wrong
way — a mangled word usually keeps the intended word as a prefix, so a
substring-matched floor drifts *upward* as the ASR gets *worse*.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(evals): rendered clips — 218 committed Opus utterances (#433) ... 77593ac367

The audio. 218 clips, 9.07 MB, 1 160 seconds of speech across five voices, under
the ~10 MB budget `CLAUDE.md`'s committed-audio carve-out sets.

**Why this is committed rather than generated.** Three reasons, and all three
are in the carve-out: piper has no `--seed`, so a re-render is not the fixture
the scorers were calibrated against; the Opus encode *is* the degradation
mechanism, and decode is bit-exact per RFC 6716, so a committed lossy file
decodes identically everywhere; and the voice models are 63 MB each, which would
put 190 MB of downloads in CI for no gain.

**Every clip was transcribed after encoding and scored against the line it was
supposed to say.** Mean per-clip WER 4.76%, no clip needed a re-render. That
step is the point of `render.py`, not a flourish: the spike lost an afternoon to
`--download-dir` — a flag belonging to piper's voice downloader, not its
synthesiser — being swallowed as input text and spoken aloud, at exit 0,
producing perfectly valid audio of entirely the wrong words. Every check that
asked "does the clip exist?" passed. This project's recurring failure mode,
appearing inside its own test harness.

`render.py` may re-render for a *synthesis* failure — no audio, an implausibly
short clip, a word error rate too high for the render to be of the right
sentence. It is explicitly forbidden from re-rendering because a **marker**
failed. Re-rolling until markers survive would select the fixture toward audio
the ASR happens to find easy, and the per-speaker marker floor — which exists to
fail when the assertion loses its teeth — would end up measuring the retry loop.

The signal chain gives each speaker a distinct voice, level, mic profile and
bitrate: 32k for a bad connection, 64k baseline, 96k for a good client, and one
speaker 20 dB down so the -45 dB VAD floor is stressed rather than sitting
unused. Two things measured rather than assumed — that channel still yields 100%
VAD kept at the production floor, and the one 0 dB track clips at 147 ppm, which
is a hot headset rather than distortion.

Each utterance is bedded on 200 ms of -55 dB tone rather than butted against
digital zero. That is a false-red guard: a one-sample cliff makes
`silencedetect` find silence early and hands Whisper an abrupt attack, which it
is well documented to answer by clipping the first word.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(evals): assembler and seeding — clips to the bot's own directory contract (#433) ... 4b0a500f18

`manifest.json` is the ground truth for everything scoreable in the fixture, and
`assemble.py` turns it plus the committed clips into exactly the directory the
bot hands the backend: `{discord_user_id}.wav` per captured speaker at 16 kHz
mono, all spanning the whole session from one t0, plus `speakers.json` and
`presence.json`.

Content is authored; **timing is sampled**, with a fixed seed, from six
statistics that live next to the scene. Nothing per-line from a real session is
read at build time or committed, which is what keeps the repository
unambiguously clean of anything session-derived. Clip durations are measured
rather than sampled — they are whatever piper produced, and the timeline is laid
out around them.

Realised: 218 utterances, 1 160 s of speech in a 1 800 s session (64% duty
cycle), 20.7% table talk, 16.5% of utterances overlapping another track, the
late joiner's first audio at exactly 360.0 s, twelve scripted silences of 8 s or
more, and 84 s of tail padding after the last line.

**`track_sha256` covers the PCM frames, not the file bytes.** ffmpeg stamps its
own version string into a `LIST` chunk of every WAV it writes, so a file hash
would break on an ffmpeg upgrade for a reason that has nothing to do with the
audio — turning a genuinely useful drift alarm into one nobody trusts. Hashing
the samples keeps the alarm pointed at the resampler, which is the only part of
the chain that can plausibly move: Opus decode is normatively bit-exact.

The resample step runs the bot's own command from
`bot/questboard_bot/utils/audio.py`. One caveat, named in the module rather than
glossed: since #176 the live capture path does not use it — the sink resamples
as audio arrives with the windowed-sinc filter in `utils/resample.py`. Using
that would mean adding numpy to the backend for one filter, so the harness takes
ffmpeg's and says so.

`seed_db.py` writes the campaign, members, platform links and characters through
the app's own models, so `resolve_speakers` can be checked against **real rows**
on all three label paths — GM, linked player with a character, and an unlinked
guest who genuinely has no `PlatformLink` (with a linked owner the backend
recovers the name from the database and #344's path is never taken). Ids are
UUID5 under a fixture namespace so they cannot collide with real data, and it
still refuses to run without `--i-know-this-writes`, because "cannot collide" is
a claim about this code and not about the database it is aimed at.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(evals): golden transcript — derived through the production renderer (#433) ... 6482e97b3e

The manifest knows exactly who said what and when, so the transcript a perfect
transcription would produce is computable. It is computed by feeding manifest
segments through the pipeline's own three functions — `order_segments` →
`mark_overlapping` → `merge_attributed_transcript` — and the result is committed
and byte-compared.

**Why the byte comparison earns its place.** `beat_service` parses
`[HH:MM:SS] Speaker:` lines back out of the rendered transcript to verify that a
claimed actor really spoke a cited line, so the line format is load-bearing
rather than cosmetic. Deriving the golden file through the real renderer means a
change to that format breaks loudly, in the same commit that makes the change,
instead of silently desynchronising a hand-written fixture nobody re-derives.

It is worth being clear about what this does *not* buy: the golden file is not
independent evidence about the renderer, because it is the renderer's own
output. It cannot catch a renderer bug. What it catches is **drift** — the
renderer changing while the fixture, the scorers and the committed numbers all
assume it did not.

Both assemblies are derived from the same manifest and the same clips: `full`
(218 lines) and `ic_only` (155), with the out-of-game and rules-chatter lines
omitted and the timeline otherwise identical. Difference the two summaries and
you get the number that matters — what the table talk cost.

`derive()` calls the three functions separately *and* asserts the count of
rendered timestamp ranges matches the overlap index set. That is not redundancy:
it is how the harness gets the overlap indices to report on, and if the two ever
disagreed the assertion would fail, which is the point.

Segments are timed from `speech_start_s`/`speech_end_s`, not from clip
placement. A clip carries 200 ms of -55 dB tone at each edge, and a perfect
transcriber would time the speech, not the bed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(evals): scorers, the harness's first precision metric, and bands from a measured run (#433) ... 3bf00947dd

`scoring.py` measures coverage, attribution, chronology and beat validation —
all four are recall or correctness measures, so **nothing in the harness could
penalise a summary for containing something it should not**. Three prompts in
this codebase already instruct against table talk (`beat_service.py:456`,
`speaker_service.py:381`, and the lore prompt in `audio_service.py:154`) and not
one of them has a test. `score_table_talk` is the missing metric.

**The assertion this deliberately does not make.** "Every segment's
`speaker_label` matches the identity for its `track_owner_id`" is a tautology —
`transcribe_track` sets both fields from the same `Track` — so it can only fail
if `resolve_speakers` is broken, which `seed_db.verify_labels` checks directly
against real rows. Written as an attribution test it would pass forever while
proving nothing, and that is the exact shape of test this project has been
bitten by repeatedly.

What replaces it is the marker partition, and it is asserted **exactly zero**
rather than against a threshold. The exactness is licensed by an asymmetry that
is now measured twice rather than argued: ASR error can *lose* a marker but
cannot manufacture a foreign one. Zero foreign markers in both assemblies, with
28% of scripted marker occurrences lost.

Marker occurrences are matched to transcript hits **one to one, nearest pair
first**. A speaker's lexicon cycles, so the same word appears several times
across the session, and a plain nearest-match let one surviving hit answer for
every occurrence of that word — reporting six occurrences the ASR had simply
lost as "detected, in the wrong place". That inverts a recognition failure into
a position failure and corrupts the one measurement that catches #320's shape.
With the constraint, out-of-position is 0 in both assemblies and can be exact.

**Every band constant comes from a measured run**, recorded in
`calibration.json`; none was chosen in advance. Inventing a ceiling before
observing a distribution is the same trap this harness exists to get out of, in
a new costume. Measured, `full` / `ic_only`, against `large-v3-turbo` on CUDA:

    session WER            3.73%  / 3.32%       band ≤ 0.08
    worst per-speaker WER  6.16%  / 5.56%       band ≤ 0.12
    marker detection       72.2%  / 64.8%       band ≥ 0.55
    worst speaker          50.0%  / 41.7%       band ≥ 0.30
    onset drift, median    0.068s / 0.070s      band ≤ 0.50
    onset drift, p95       2.107s / 0.574s      band ≤ 5.0
    foreign markers        0      / 0           exact
    out of position        0      / 0           exact
    marker order           ok     / ok          exact
    before the late join   0      / 0           exact
    inside scripted silence 0     / 0           exact
    label mismatches       0      / 0           exact

Three independent runs produced these numbers to four decimal places, which is
what makes the tight exact assertions defensible rather than optimistic.

`asr_leg.py` runs the *production* functions — `drop_silent_tracks`,
`check_tracks_cover_session`, `transcribe_with_optional_vad`,
`check_transcript_covers_session`, `merge_attributed_transcript` — so the guards
run and pass rather than being skipped, and `_offset_segments` (where #320's
shape lives, and which is only reachable through the VAD branch) is in the path.

`build_fixture.py` makes the scene an ordinary eval fixture as well, so the LLM
half can be scored on the golden transcript and, later, on the ASR transcript,
with leg 3 minus leg 2 read as ASR-induced degradation rather than model
variance. It scores 1.000 on all four metrics with the stub provider, which is
the right baseline: this fixture's adversarial content is the table talk, and
what scores that is the precision metric above.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

test(evals): the Whisper shim and the negative fixtures (#433) ... 80386a6c65

A guard nobody has watched fire is a guard nobody knows works. #432 was exactly
that — found by reading, not by running.

**Every negative fixture asserts that its mutation took**, before asserting the
outcome. That is not ceremony: the recurring failure in this repository is a
fixture that collapsed before it reached the code under test. Truncate *all* the
tracks and `derive_session_duration` shrinks with them, the ratio stays 1.0, the
coverage guard never fires, and the test goes green having proved nothing. So
the truncation test truncates one track and then asserts the derived duration
did *not* move.

`shim.py` is an in-process fake of the Whisper wire format, and it is the piece
that unlocks the most. It patches the `httpx.AsyncClient` *constructor* rather
than binding a port (flaky in CI, racy in parallel) or reaching into the
provider's internals (which would stop testing the code that actually runs the
moment the provider abstraction changes shape). It also makes something
assertable that several guard docstrings claim and nothing verified: **which
filenames reached the server, and how many times.**

The fixtures, and what each one is careful about:

- `check_tracks_cover_session` — one track truncated, and **zero** Whisper
  requests asserted. This guard's whole value is running before a byte reaches
  the GPU, and until the shim existed that was an unverified sentence.
- `check_transcript_covers_session` (#431) — written as "this input is fine and
  the pipeline destroys it". Track lengths preserved, speech zeroed from
  halfway: the shape a table that played for two hours and forgot to stop for
  another fifty minutes produces. The audible extent is asserted to have moved
  before the guard is called, or it is not the #431 shape.
- `check_every_speaker_was_transcribed`, fatal branch — victim's VAD **kept**
  seconds measured and asserted ≥ 30 s first. Not speech seconds:
  `compute_speech_spans` pads every span by 300 ms a side and folds in every gap
  under 2 s, so which branch a speaker takes is decided by how their audio
  chunks.
- …and the warn branch, which needs its own mutation. The quiet player is
  scripted comfortably above the threshold (70.7 s kept) *because* the fixture
  has to prove a barely-audible player survives, so reaching the warn branch
  means silencing most of her track — measured before and after.
- `drop_silent_tracks` — `has_audible_speech(path, 32) is False` asserted as a
  precondition. A dither floor above amplitude 32 leaves the track undropped and
  the fixture green and worthless. The pairing with
  `narrow_speakers_to_captured` is checked too, since that pairing is where
  #432's defect actually lived.
- #344 snowflake-as-value — uses the **unlinked** owner, with `role == unknown`
  asserted first. With a linked member the backend recovers the name from the
  database and the broken path is never taken.
- #342 symmetric swap — the milestone's namesake, end to end for the first time.
  A server that permutes its echoed labels must change **nothing**, and the
  assertion is that attribution is bit-identical to the clean run rather than
  merely still plausible.

Also here: the scorers exercised on a clean input (they must not cry wolf) and
on inputs broken in exactly the way each metric exists to catch — a symmetric
label swap, every marker mangled, a 400-second timestamp shift with ownership
intact, a reordered marker sequence, a hallucination inside a scripted silence,
a summary full of table talk, and a summary that credits the recap's speaker
instead of the person who acted.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

ci: run the synthetic-session harness on every push, without a GPU (#433) ... c3fdac1ddf

A new job rather than steps inside the backend one, for two reasons that are
both about honesty.

**It needs ffmpeg.** Everything here decodes Opus and runs the bot's own
resample command over a thirty-minute six-track session. Adding an apt install
to the backend job would slow every backend run for tests that are not that
job's subject, and the step verifies libopus specifically — Opus decode being
bit-exact (RFC 6716) is the property every committed track hash rests on, so an
ffmpeg without the decoder should fail early and clearly rather than late and
obscurely.

**It must not be able to skip silently.** The audio tests carry a `skipif` on
ffmpeg so they behave sensibly on a developer machine without it. In *this* job
a skip would be a green run that proved nothing — precisely the failure mode the
whole harness exists to stop — so `QB_SYNTHETIC_REQUIRE_AUDIO=1` turns the skip
into a strict xfail that fails the job.

Five steps, roughly four minutes: the manifest and scene invariants plus every
scorer (no audio decoded); both assemblies with all twelve `track_sha256` values
and both golden transcripts byte-compared; the guards, each watched firing on
real audio with the Whisper wire format faked in process; the harness's own
`verify` entry point, so a broken CLI is caught rather than only a broken
library; and the stub eval leg over the scene as an ordinary fixture.

Nothing here touches the live Whisper server or an LLM. Those legs are
`python -m evals.synthetic_session asr-leg` and `python -m evals --provider
live`, run by hand against the dev stack: there is no GPU on a CI runner, and
pinning accuracy numbers to a model that can be swapped underneath us would be a
flaky gate measuring the wrong thing.

The stub leg's thresholds are loose on purpose — this catches a broken harness,
an import error, a scorer silently returning zero — not a model regression.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs: what the synthetic session harness proves, and what it does not (#433) ... b2b4b11907

The harness README carries the measured numbers, the screening table, the
re-authoring order, and — first, before any of that — the four things a green
run does **not** prove. That ordering is deliberate. A test instrument that
overstates its own reach is worse than no instrument, because it retires the
worry it cannot actually answer:

1. Nothing about capture. The fixture *is* the output of the bot's capture
   stage; gap fill, DTX, tail padding and decoder reset are assumed correct by
   construction, and #320 lived in exactly that code.
2. Nothing about whether the bot puts the right user's audio in
   `{user_id}.wav`. A hand-built directory asserts the mapping it created, which
   is the other half of #342 and still has no test.
3. Nothing about bleed. Synthetic tracks are perfectly isolated; in production
   at least one player is usually on open speakers, carrying everyone.
4. Accuracy numbers are not accuracy numbers. TTS pronunciation is
   dictionary-correct, so every word error rate here is a tripwire against
   previous runs of this fixture and never a quality bar.

A fifth caveat is specific to this build and is named rather than buried: the
assembler runs the bot's ffmpeg resample command, but since #176 the live
capture path uses the windowed-sinc filter in `utils/resample.py` instead. The
two differ in the stopband, not in anything Whisper's mel front end can see — but
"not measurably different" is not "identical".

Also recorded, because both are findings rather than trivia: the plan's five
speech-share figures cannot all hold simultaneously (the arithmetic is written
out), and **a marker that survives a carrier sentence can still be lost in the
session** — `Marlgate` passed screening 2/2 and was lost at all four of its
occurrences in the full assembly, because Whisper has far more context to reshape
an invented name with over thirty minutes than over one sentence. That is why the
assertion rests on a per-speaker aggregate floor and why re-rendering to chase a
marker is forbidden.

A "Known gaps" section names the four things this build did not finish, so the
next person does not have to discover them: no automated live LLM leg, no
end-to-end `full` versus `ic_only` summary comparison, #432 not yet pinned as a
failing test, and marker screening that is per-carrier rather than per-session.

Plus a `make synthetic-session` target, a section in `docs/DEVELOPMENT.md`, and
one operator-facing CHANGELOG entry that says plainly what this costs the
repository in bytes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

fix(evals): make the seeding script run, and the CLI report what it measured (#433) ... af39329141

Three things found by running the parts of the harness the tests do not reach.

**`seed_db` named a scheduling mode that does not exist.** `SchedulingMode` has
`vote`, `direct` and `tentative`; the script asked for `fixed` and died on the
session insert. Caught by pointing it at a real migrated database, which is the
only way it could have been — the script is deliberately not run by the test
suite, because it writes.

With that fixed it does what it exists for: seeded against a real schema, every
speaker resolves to exactly the label the manifest predicts, on all three paths
(GM, linked player with a character, unlinked guest). Re-running is idempotent,
and the write guard refuses without `--i-know-this-writes`.

**`--print-sql` claimed more independence than it has.** It opens no connection,
but importing the module still runs `app.config.Settings()` validation, so the
usual seven environment variables must be present even though nothing reads
them. Said plainly in the docstring rather than left to be discovered.

**`asr-leg` measured everything and asserted nothing.** It printed a JSON block
and exited 0 regardless, so `scorers.check` — every exact assertion and every
calibrated band — was only ever consulted by the test suite. That is backwards:
this is the command somebody runs against a server they have just changed, and
it is the moment the thresholds would have told them something. It now checks,
prints each failure with `EXACT` or `BAND` in front of it, and exits non-zero.
`--record` still writes `calibration.json` first, because a run that broke a
band is exactly the run whose numbers you want written down.

Also a pointer from `evals/README.md` to the sibling harness, naming the one
thing it has that this one does not: a precision metric.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

fix(make): keep both the conformance and synthetic-session targets after the rebase (#433) ... d9447b8080

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed 2026-09-05 03:51:41 +00:00

rbrooks added 1 commit 2026-09-05 04:02:42 +00:00

ci: let grep read all of ffmpeg's decoder list, not just the first match (#433) ... d30ca8cf58

`grep -q` exits on the first "opus", ffmpeg takes SIGPIPE writing the rest,
and the runner's pipefail turns exit 141 into a failed step before a single
harness test runs. Read to EOF and discard instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

rbrooks added 1 commit 2026-09-05 04:14:43 +00:00

ci: run the synthetic-session harness inside the backend image (#433) ... b916e6920e

The committed track hashes pin the toolchain the backend ships with. The
runner's Ubuntu ffmpeg 6.1 decoded and resampled every speech track to
different samples than the image's Debian ffmpeg, so the first run failed
the hash check for a reason that had nothing to do with the audio. Build
backend-dev, copy the checkout into a throwaway image, and run there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot referenced this pull request 2026-09-05 04:37:17 +00:00

[Backend] Evaluate ASR and LLM providers against the golden corpus before committing #360

rbrooks force-pushed feat/433-synthetic-session from b916e6920e to ceb6d51bfc 2026-09-05 05:20:23 +00:00 Compare

rbrooks added 1 commit 2026-09-05 05:45:47 +00:00

ci: point the stub eval leg at the harness's eval fixture directory (#433) ... f48f5b2109

The fixture loader globs every *.json in the directory it is given; the
synthetic session's root holds manifest, calibration and screening files
that are not eval fixtures. The Makefile target already used the eval/
subdirectory; the CI step did not.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot referenced this pull request 2026-09-05 05:51:02 +00:00

[Backend] Follow-ups from the provider evaluation: usage rows misclassify sniffed managed runs, managed ASR is unpriceable, and the live tooling cannot reach the managed adapter (#360) #502

claude-bot merged commit 985ef7971a into main 2026-09-05 06:01:59 +00:00

claude-bot deleted branch feat/433-synthetic-session 2026-09-05 06:02:00 +00:00

claude-bot referenced this pull request from a commit 2026-09-05 06:02:00 +00:00

Merge PR #497: synthetic-session transcription harness (#433)

claude-bot referenced this pull request 2026-09-05 06:02:51 +00:00

[Evals] Synthetic-audio end-to-end harness for the per-speaker transcription pipeline #433

claude-bot referenced this pull request 2026-09-05 06:04:34 +00:00

chore(release): v4.2.0 — AI Provider Abstraction & BYO-AI #504

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backend

Issue related to the backend

  

bot

Issue related to the bot

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

frontend

Issue related to the frontend

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

recording

Issue related to the recording system

  

security

Security hardening / vulnerability work

  

wontfix

This won't be fixed

No labels

backend

bot

bug

duplicate

enhancement

frontend

help wanted

invalid

question

recording

security

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/Quest-Board!497

No description provided.