[Evals] Synthetic-audio end-to-end harness for the per-speaker transcription pipeline

claude-bot commented

2026-08-28 22:45:09 +00:00

Collaborator

Why

The per-speaker transcription change (#342) is the riskiest thing in v4.0.0 and has never run against a real Whisper server outside unit mocks. Neither has speaker resolution against real member rows, nor beats → validation → compose → anchors against a real LLM. The eval harness starts from a transcript, so nothing in it exercises transcription at all.

Every real recording has been destroyed and is unrecoverable, so this is built from invented audio: no real voices, no real content, committable, re-runnable against any provider indefinitely.

Design

Full plan and rationale: https://claude.ai/code/artifact/d8bd7647-9fc2-4c25-a3ad-375c977cdb2f

Settled:

30-minute scene. Below ~25 min the transcript fits a single extraction window — the regime #423's own sweep measured as worst, and one production never uses. 30 min buys two windows and puts the quiet player past the 30 s threshold naturally.
Manifest + committed per-utterance Opus clips + deterministic assembler. TTS is a one-time authoring step, never a test dependency. The Opus encode is the Discord-codec simulation, and decode is bit-exact per RFC 6716, so committing lossy audio is safe. Requires a carve-out in CLAUDE.md's "do not commit audio files" rule.
Entry via POST /api/bot/sessions/{id}/audio with force=true — the production intake path. Admin retry (admin.py:155) 409s unless the session is already failed, so it cannot be used for a fresh synthetic session.
Timing sampled from statistics with a fixed seed — turn shares, gap histogram, 13.1% overlap rate, late join, quiet-player share. Nothing per-line from a real session is read or committed.
~20% of speech is table talk, in four seeded episodes (arrival greetings, a joke that derails, a rules lookup, a snack-break lull) plus a proposed-but-unperformed plan. Every utterance carries a register: ic / ooc / meta / proposed.
The scene assembles two ways from one manifest — full and ic_only — so the cost of the table talk is measurable rather than assumed.

What this measures that nothing currently does

Three prompts already instruct against table talk (beat_service.py:456, speaker_service.py:381, audio_service.py:154) and nothing tests any of them. scoring.py has coverage, attribution, chronology and beat-validation — all recall or correctness. There is no precision metric in the harness at all, so nothing can penalise a summary for containing material that shouldn't be there. This adds the first one.

Key design constraints

The obvious attribution assertion is a tautology. transcribe_track sets speaker_label and track_owner_id from the same Track object, so asserting they agree tests nothing. Replaced with a per-speaker marker lexicon, asserting foreign-marker count is exactly zero — licensed by the asymmetry that ASR error can only miss a marker, never manufacture a foreign one. Markers are sequence-numbered so the assertion also catches timestamp corruption with attribution intact (the #320 shape).
The quiet player's branch is decided by VAD kept seconds, not speech seconds. compute_speech_spans pads each span 300 ms per side and folds gaps under 2 s, so fragmented speech inflates kept well above real speech time. Measure and assert it; do not infer it from the script.
An over-aggressive table-talk filter is a new way to lose a quiet player. Wren speaks ~50 s, mostly out-of-game, with ~12 s of genuine in-fiction action. Those two actions must reach the summary.
Every negative fixture must assert its mutation took, not merely that the outcome occurred. The recurring failure in this repo is a fixture that collapsed before reaching the code under test.

Acceptance criteria

CLAUDE.md carve-out for synthetic TTS clips, written before the first audio commit.
30-minute scene manifest with registers, marker lexicons and seeded episodes; clips rendered and committed.
Deterministic assembler producing both assemblies, with track_sha256 asserted.
DB seeding covering all three resolve_speakers label paths (GM, linked player with character, unlinked guest).
Golden true transcript derived through production code and byte-checked.
First live run on dev against real Whisper + LLM, both assemblies, numbers recorded.
Scorers and bands calibrated from that run, not invented ahead of it — including the table-talk leakage metric.
Whisper HTTP shim and the negative fixtures (including the #342 symmetric-swap case, currently untested end-to-end).
CI wiring for the no-GPU parts: schema, hashes, assembly, golden byte-equality, stub eval leg.

#431 and #432 were both found while designing this, by reading the pipeline. #432's assertion is written into the harness as a deliberately failing check until it is fixed.
Follow-on worth having: a stub Whisper server replaying manifest-derived segments would let the whole process_audio path run in CI with no GPU.

## Why The per-speaker transcription change (#342) is the riskiest thing in v4.0.0 and has never run against a real Whisper server outside unit mocks. Neither has speaker resolution against real member rows, nor beats → validation → compose → anchors against a real LLM. The eval harness starts from a *transcript*, so nothing in it exercises transcription at all. Every real recording has been destroyed and is unrecoverable, so this is built from invented audio: no real voices, no real content, committable, re-runnable against any provider indefinitely. ## Design Full plan and rationale: **https://claude.ai/code/artifact/d8bd7647-9fc2-4c25-a3ad-375c977cdb2f** Settled: - **30-minute scene.** Below ~25 min the transcript fits a single extraction window — the regime #423's own sweep measured as worst, and one production never uses. 30 min buys two windows and puts the quiet player past the 30 s threshold naturally. - **Manifest + committed per-utterance Opus clips + deterministic assembler.** TTS is a one-time authoring step, never a test dependency. The Opus encode *is* the Discord-codec simulation, and decode is bit-exact per RFC 6716, so committing lossy audio is safe. Requires a carve-out in `CLAUDE.md`'s "do not commit audio files" rule. - **Entry via `POST /api/bot/sessions/{id}/audio` with `force=true`** — the production intake path. Admin retry (`admin.py:155`) 409s unless the session is already `failed`, so it cannot be used for a fresh synthetic session. - **Timing sampled from statistics with a fixed seed** — turn shares, gap histogram, 13.1% overlap rate, late join, quiet-player share. Nothing per-line from a real session is read or committed. - **~20% of speech is table talk**, in four seeded episodes (arrival greetings, a joke that derails, a rules lookup, a snack-break lull) plus a proposed-but-unperformed plan. Every utterance carries a register: `ic` / `ooc` / `meta` / `proposed`. - **The scene assembles two ways** from one manifest — `full` and `ic_only` — so the cost of the table talk is measurable rather than assumed. ## What this measures that nothing currently does Three prompts already instruct against table talk (`beat_service.py:456`, `speaker_service.py:381`, `audio_service.py:154`) and **nothing tests any of them**. `scoring.py` has coverage, attribution, chronology and beat-validation — all recall or correctness. There is **no precision metric in the harness at all**, so nothing can penalise a summary for containing material that shouldn't be there. This adds the first one. ## Key design constraints - **The obvious attribution assertion is a tautology.** `transcribe_track` sets `speaker_label` and `track_owner_id` from the same `Track` object, so asserting they agree tests nothing. Replaced with a per-speaker marker lexicon, asserting foreign-marker count is exactly zero — licensed by the asymmetry that ASR error can only *miss* a marker, never manufacture a foreign one. Markers are sequence-numbered so the assertion also catches timestamp corruption with attribution intact (the #320 shape). - **The quiet player's branch is decided by VAD `kept` seconds, not speech seconds.** `compute_speech_spans` pads each span 300 ms per side and folds gaps under 2 s, so fragmented speech inflates `kept` well above real speech time. Measure and assert it; do not infer it from the script. - **An over-aggressive table-talk filter is a new way to lose a quiet player.** Wren speaks ~50 s, mostly out-of-game, with ~12 s of genuine in-fiction action. Those two actions must reach the summary. - **Every negative fixture must assert its mutation took**, not merely that the outcome occurred. The recurring failure in this repo is a fixture that collapsed before reaching the code under test. ## Acceptance criteria - [ ] `CLAUDE.md` carve-out for synthetic TTS clips, written before the first audio commit. - [ ] 30-minute scene manifest with registers, marker lexicons and seeded episodes; clips rendered and committed. - [ ] Deterministic assembler producing both assemblies, with `track_sha256` asserted. - [ ] DB seeding covering all three `resolve_speakers` label paths (GM, linked player with character, unlinked guest). - [ ] Golden true transcript derived through production code and byte-checked. - [ ] First live run on dev against real Whisper + LLM, both assemblies, numbers recorded. - [ ] Scorers and bands calibrated from that run, not invented ahead of it — including the table-talk leakage metric. - [ ] Whisper HTTP shim and the negative fixtures (including the #342 symmetric-swap case, currently untested end-to-end). - [ ] CI wiring for the no-GPU parts: schema, hashes, assembly, golden byte-equality, stub eval leg. ## Related - #431 and #432 were both found while designing this, by reading the pipeline. #432's assertion is written into the harness as a deliberately failing check until it is fixed. - Follow-on worth having: a stub Whisper server replaying manifest-derived segments would let the whole `process_audio` path run in CI with no GPU.

claude-bot commented

2026-08-28 23:02:17 +00:00

Author

Collaborator

Spike complete — the premise holds, one design change needed

Ten utterances, three piper voices, eight invented proper nouns of varied difficulty. Rendered, put through the full Opus chain (libopus -b:a {32,64,96}k -vbr on -application voip -frame_duration 20, decoded, then the bot's own resample command verbatim), assembled onto one 72-second clock, transcribed by the production Whisper server — large-v3-turbo on CUDA.

Results

Question	Answer
Is TTS clean enough that failures are attributable to the pipeline?	WER 6.4% session (4.1 / 7.5 / 9.3 per speaker)
Does anything leak across tracks?	0 of 8 markers landed on a foreign track
Do invented proper nouns survive ASR?	3 of 8 on an exact word boundary
Is placement on the shared clock accurate?	< 1 s — scheduled 2.0/11.5/28.5/63.0 → transcribed 2/11/28/63
Does cross-track overlap materialise?	Yes — two tracks interleave at 63–65 s

The asymmetry the exact attribution assertion rests on — ASR error can only lose a marker, never manufacture a foreign one — is now measured rather than argued. Five of eight markers were mangled and not one appeared on another speaker's track.

Design change: markers must be screened, not chosen

Only 3/8 survived exactly. The failures follow no clean rule:

Marker	Result	ASR produced
`Quillfeather`, `Thistledown`, `Bramblewick`	survived	—
`Varnholt`	mangled	varnholtz
`Ashgrove`	mangled	ashgrover (boundary bled into the next word)
`Marrowgate`	lost	marovate
`Copperkettle`	lost	copperkittle — two very common words, still failed
`Zhaltirmek`	lost	chaltramek — non-English phonology, as predicted

So: screen marker candidates empirically before committing them (render → transcribe → keep what survives, using the spike as the rig), and lean the assertion on a per-speaker aggregate floor rather than any individual marker. At ~2 markers/minute over 30 minutes there is ample redundancy for a ~40% loss rate, provided no single marker is load-bearing.

Match on word boundaries, not substrings. Substring matching passed Varnholt inside varnholtz and Ashgrove inside ashgrover — overstating survival by 2/8. Worse, it drifts upward as ASR gets worse, since mangled words keep the marker as a prefix.

Five things for the build

piper has no --seed. Re-rendering is not reproducible. This independently settles the commit-the-clips decision: the clip must be the artifact, because it cannot be regenerated.
A wrong flag produced valid audio of the wrong words at exit 0. --download-dir belongs to piper.download_voices, not the synthesiser; passed to the synthesiser it was swallowed as input text and spoken aloud ("Download ire slash work slash voices"). Caught only by transcribing the clip. The assembler must verify clip content, not just existence — this is the repo's recurring failure mode appearing inside its own test harness.
Number normalisation is inconsistent within one session. "fifteen" → 15, "twenty" → twenty, same voice, seconds apart. The scorer needs a numeral table or it reports false errors.
Whisper re-segments. Ten authored utterances became eighteen segments. Timing assertions must be written 1:N, not 1:1.
Scripted disfluencies get mangled and that is fine. "Uh, hang on" → "At Hanon". Realistic, but it costs WER, so the disfluency budget and the error-rate ceiling have to be set together.

Sizing

Committed Opus totalled ~354 KB for ~48 s of speech, extrapolating to ~10 MB for the 30-minute scene. In line with the estimate in the plan.

Environment notes

Voice models are 63 MB each (190 MB for three) — confirms they belong in authoring, never as a test-time dependency.
The GPU box is reachable directly from a dev workstation, so the spike needed no dev-host access at all.
Wire contract verified against the live server, not the code: POST /transcribe with audio + speaker + optional language; also exposes /health and /transcribe/session.

Plan updated: https://claude.ai/code/artifact/d8bd7647-9fc2-4c25-a3ad-375c977cdb2f (§7 revised, §11 added)

## Spike complete — the premise holds, one design change needed Ten utterances, three piper voices, eight invented proper nouns of varied difficulty. Rendered, put through the full Opus chain (`libopus -b:a {32,64,96}k -vbr on -application voip -frame_duration 20`, decoded, then the bot's own resample command verbatim), assembled onto one 72-second clock, transcribed by the production Whisper server — `large-v3-turbo` on CUDA. ### Results | Question | Answer | |---|---| | Is TTS clean enough that failures are attributable to the pipeline? | **WER 6.4%** session (4.1 / 7.5 / 9.3 per speaker) | | Does anything leak across tracks? | **0 of 8** markers landed on a foreign track | | Do invented proper nouns survive ASR? | **3 of 8** on an exact word boundary | | Is placement on the shared clock accurate? | **< 1 s** — scheduled 2.0/11.5/28.5/63.0 → transcribed 2/11/28/63 | | Does cross-track overlap materialise? | Yes — two tracks interleave at 63–65 s | The asymmetry the exact attribution assertion rests on — *ASR error can only lose a marker, never manufacture a foreign one* — is now **measured rather than argued**. Five of eight markers were mangled and not one appeared on another speaker's track. ### Design change: markers must be screened, not chosen Only 3/8 survived exactly. The failures follow no clean rule: | Marker | Result | ASR produced | |---|---|---| | `Quillfeather`, `Thistledown`, `Bramblewick` | survived | — | | `Varnholt` | mangled | *varnholtz* | | `Ashgrove` | mangled | *ashgrover* (boundary bled into the next word) | | `Marrowgate` | lost | *marovate* | | `Copperkettle` | lost | *copperkittle* — two very common words, still failed | | `Zhaltirmek` | lost | *chaltramek* — non-English phonology, as predicted | So: **screen marker candidates empirically before committing them** (render → transcribe → keep what survives, using the spike as the rig), and lean the assertion on a **per-speaker aggregate floor** rather than any individual marker. At ~2 markers/minute over 30 minutes there is ample redundancy for a ~40% loss rate, provided no single marker is load-bearing. **Match on word boundaries, not substrings.** Substring matching passed `Varnholt` inside *varnholtz* and `Ashgrove` inside *ashgrover* — overstating survival by 2/8. Worse, it drifts *upward* as ASR gets worse, since mangled words keep the marker as a prefix. ### Five things for the build - **piper has no `--seed`.** Re-rendering is not reproducible. This independently settles the commit-the-clips decision: the clip must *be* the artifact, because it cannot be regenerated. - **A wrong flag produced valid audio of the wrong words at exit 0.** `--download-dir` belongs to `piper.download_voices`, not the synthesiser; passed to the synthesiser it was swallowed as input text and spoken aloud ("Download ire slash work slash voices"). Caught only by transcribing the clip. **The assembler must verify clip content, not just existence** — this is the repo's recurring failure mode appearing inside its own test harness. - **Number normalisation is inconsistent within one session.** "fifteen" → *15*, "twenty" → *twenty*, same voice, seconds apart. The scorer needs a numeral table or it reports false errors. - **Whisper re-segments.** Ten authored utterances became eighteen segments. Timing assertions must be written 1:N, not 1:1. - **Scripted disfluencies get mangled and that is fine.** "Uh, hang on" → "At Hanon". Realistic, but it costs WER, so the disfluency budget and the error-rate ceiling have to be set together. ### Sizing Committed Opus totalled ~354 KB for ~48 s of speech, extrapolating to **~10 MB** for the 30-minute scene. In line with the estimate in the plan. ### Environment notes - Voice models are 63 MB each (190 MB for three) — confirms they belong in authoring, never as a test-time dependency. - The GPU box is reachable directly from a dev workstation, so the spike needed no dev-host access at all. - Wire contract verified against the live server, not the code: `POST /transcribe` with `audio` + `speaker` + optional `language`; also exposes `/health` and `/transcribe/session`. Plan updated: https://claude.ai/code/artifact/d8bd7647-9fc2-4c25-a3ad-375c977cdb2f (§7 revised, §11 added)

metric	`full`	`ic_only`
submit → ready	10.4 min	10.8 min
audio submitted / transcribed after VAD	9,000 s / 1,162 s (87 % cut)	9,000 s / 921 s (90 % cut)
ASR realtime factor	7.9×	8.5×
LLM calls / prompt / completion tokens	5 / 35,829 / 15,299	7 / 35,786 / 19,201
markers detected (vocabulary from 5 seeded characters only)	38/54	34/54
foreign markers / out of position	0 / 0	0 / 0
session WER	3.62 %	3.21 %
OOC leak probes fired	1/7 (`leak_rules_numbers`)	0/7
IC coverage (required probes)	4/5	5/5
beats stored / with problems	27 / 3	28 / 3
word timestamps	3,463 words, 0 null, 0 interpolated	—

Rows
Columns

[Evals] Synthetic-audio end-to-end harness for the per-speaker transcription pipeline #433

Why