Synthetic-session harness: marker scorer denominator, vocabulary-gap severity, hotwords

claude-bot commented

2026-09-05 06:56:33 +00:00

Collaborator

Follow-ups surfaced by the vocabulary-bias measurement (PR #505, #355, #360). None of them change the numbers already reported, but each should be fixed before the harness is used for a cross-assembly or cross-provider comparison.

score_markers uses the wrong denominator for the in-character-only assembly. The scorer divides by the full marker count (54) for both assemblies, but only 50 of the markers occur in the ic_only script, so that assembly's detection rate is understated by ~7 % relative. Around evals/synthetic_session/scorers.py:308. Count the markers that actually occur in the scored script instead.
GAP_NO_VOCABULARY is probably under-severe. The capability gap for "this provider ignores the vocabulary list" is currently low severity. The measurement shows the list is worth 9–24 pp of proper-noun recall, which is the largest single effect measured in v4.2.0, so SEVERITY_MEDIUM fits better. Also check the copy on the Admin capability panel says what the GM loses.
hotwords is unexplored. WhisperX API v2 honours both initial_prompt (what the vocabulary list becomes today) and hotwords. Run the same three-condition matrix with hotwords, and with both, and record which one the adapter should prefer.
Leg 3 (LLM on the ASR transcript) is still run by hand. The harness automates the audio and ASR legs and the LLM-on-golden-transcript leg, but scoring the summary produced from the ASR transcript still needs a manual step. Automate it so a provider swap can be scored end to end in one command.

Follow-ups surfaced by the vocabulary-bias measurement (PR #505, #355, #360). None of them change the numbers already reported, but each should be fixed before the harness is used for a cross-assembly or cross-provider comparison. 1. **`score_markers` uses the wrong denominator for the in-character-only assembly.** The scorer divides by the full marker count (54) for both assemblies, but only 50 of the markers occur in the `ic_only` script, so that assembly's detection rate is understated by ~7 % relative. Around `evals/synthetic_session/scorers.py:308`. Count the markers that actually occur in the scored script instead. 2. **`GAP_NO_VOCABULARY` is probably under-severe.** The capability gap for "this provider ignores the vocabulary list" is currently low severity. The measurement shows the list is worth 9–24 pp of proper-noun recall, which is the largest single effect measured in v4.2.0, so `SEVERITY_MEDIUM` fits better. Also check the copy on the Admin capability panel says what the GM loses. 3. **`hotwords` is unexplored.** WhisperX API v2 honours both `initial_prompt` (what the vocabulary list becomes today) and `hotwords`. Run the same three-condition matrix with hotwords, and with both, and record which one the adapter should prefer. 4. **Leg 3 (LLM on the ASR transcript) is still run by hand.** The harness automates the audio and ASR legs and the LLM-on-golden-transcript leg, but scoring the summary produced from the *ASR* transcript still needs a manual step. Automate it so a provider swap can be scored end to end in one command.

claude-bot commented

2026-09-05 07:31:58 +00:00

Author

Collaborator

5. seed_db.py creates PlatformLink rows without verified_at, so the attendance path is never exercised. Found validating v4.2.0 on dev: the end-to-end run logged attendance autofill {'spoke': 0, 'in_channel_silent': 0, 'unmatched': 5, 'auto_applied': 0} and wrote zero attendance_proposals. Root cause is that _resolve_discord_ids_to_members (attendance_service.py, around line 150) deliberately requires verified_at IS NOT NULL — an unclaimed link must never drive attendance — while seed_db.py (around line 216) never sets it. Speaker resolution does not require verification, which is why the transcript labels were still right; only attendance went blind. The product behaviour is correct, the fixture is wrong.

Fix: set verified_at on the seeded links, then assert in the harness's check that the presence-only member (Kesh) and the silent-track owner (Dunmar) are reported as uncaptured, and that the five speakers match. That turns the #114 uncaptured-member path into something the harness actually tests.

**5. `seed_db.py` creates `PlatformLink` rows without `verified_at`, so the attendance path is never exercised.** Found validating v4.2.0 on dev: the end-to-end run logged `attendance autofill {'spoke': 0, 'in_channel_silent': 0, 'unmatched': 5, 'auto_applied': 0}` and wrote zero `attendance_proposals`. Root cause is that `_resolve_discord_ids_to_members` (`attendance_service.py`, around line 150) deliberately requires `verified_at IS NOT NULL` — an unclaimed link must never drive attendance — while `seed_db.py` (around line 216) never sets it. Speaker resolution does not require verification, which is why the transcript labels were still right; only attendance went blind. The product behaviour is correct, the fixture is wrong. Fix: set `verified_at` on the seeded links, then assert in the harness's check that the presence-only member (Kesh) and the silent-track owner (Dunmar) are reported as uncaptured, and that the five speakers match. That turns the #114 uncaptured-member path into something the harness actually tests.

claude-bot referenced this issue

2026-09-05 07:34:17 +00:00

chore(release): v4.2.0 — AI Provider Abstraction & BYO-AI #504

claude-bot referenced this issue

2026-09-05 07:34:28 +00:00

[Evals] Synthetic-audio end-to-end harness for the per-speaker transcription pipeline #433

claude-bot commented

2026-09-05 15:00:07 +00:00

Author

Collaborator

Picking up items 1, 2 and 5 for v4.2.1 (the scorer denominator, the GAP_NO_VOCABULARY severity, and seed_db setting verified_at). Items 3 (hotwords) and 4 (leg 3 automation) are measurement and harness work rather than fixes, and stay open here.

Picking up items **1, 2 and 5** for v4.2.1 (the scorer denominator, the `GAP_NO_VOCABULARY` severity, and `seed_db` setting `verified_at`). Items 3 (hotwords) and 4 (leg 3 automation) are measurement and harness work rather than fixes, and stay open here.

rbrooks referenced this issue from a commit

2026-09-05 15:43:38 +00:00

fix(evals): score marker detection against the assembly's own script (#506)

rbrooks referenced this issue from a commit

2026-09-05 15:43:38 +00:00

fix(webapp): rank the missing vocabulary gap by what it was measured to cost (#506)

rbrooks referenced this issue from a commit

2026-09-05 15:43:38 +00:00

fix(evals): seed the fixture's Discord links verified (#506)

claude-bot referenced this issue

2026-09-05 15:43:55 +00:00

fix: record the model the bundled transcriber reports, and three harness fixes from the v4.2.0 validation #510

rbrooks referenced this issue from a commit

2026-09-05 16:03:00 +00:00

chore(release): v4.2.1

claude-bot referenced this issue

2026-09-05 16:03:15 +00:00

chore(release): v4.2.1 #511

claude-bot commented

2026-09-05 16:03:36 +00:00

Author

Collaborator

Items 1, 2 and 5 are merged in PR #510 and ship in v4.2.1. Items 3 (hotwords) and 4 (leg 3 automation) remain open here.

1. score_markers denominator. ic_only renders only the ic and proposed registers, and four of the manifest's 54 marker occurrences (all 24 distinct markers still appear in IC lines) are spoken exclusively in out-of-character lines: astrid_040 Silverbeck, bryn_114 Ravenstoke, bryn_176 Nettleford, wren_196 Blackwillow. They were being scored as markers the transcriber lost, so a flawless ic_only transcript could not score above 50/54. score_markers now takes the assembly and filters expected occurrences by its registers, exactly as score_wer does, and returns the excluded occurrences alongside numerator and denominator so a denominator that moves is visible rather than inferred. full keeps every register and is unchanged at 54. Foreign markers are deliberately still checked across the whole lexicon; the order check's expected side is filtered with everything else.

The recorded figures in calibration.json and the harness README are annotated, not restated: the numerator is not re-derivable from the record (it stores counts, not segments) and re-measuring needs the GPU box. What can be said without a rerun is bounded. Detection per marker is min(occurrences, hits), so the correction can only lower a numerator, which puts the control's ic_only between 31/50 and 35/50 (62.0–70.0 %) and the worst speaker between 4/11 and 5/11. Both bands hold at the pessimistic end, so the floors stay where the measurement put them. On the next GPU rerun, rewrite the tables from the new run rather than patching; calibration.json carries a note saying so.

2. GAP_NO_VOCABULARY severity. Raised low → medium. The #355 A/B measured +9 to +24 points of proper-noun recall with WER flat, the largest single effect in v4.2.0; ranking that below "the provider was guessed from the URL" told operators the wrong thing to fix first. The consequence copy now says what is lost (character and place names coming back mis-heard, unrepairable downstream because everything after transcription reads the transcript and not the audio) with the measured range in plain words. The OPERATIONS gap table is updated; the frontend renders severity generically, so nothing there changed. Runs already stamped low keep it, which is what a provenance record is for.

5. seed_db never set verified_at. attendance_service resolves Discord ids through verified links only, on purpose, so the fixture seeded a roster the attendance path could not see and the uncaptured-member check (#114) was unreachable through it. Links are now seeded with a fixed timestamp, --print-sql renders it, and --verify grew a cheap second check that every seeded link is verified.

Confirmed against a real database rather than argued: seeded → labels and links both clean → {'spoke': 4, 'in_channel_silent': 2, 'unmatched': 1, 'auto_applied': 0} with unmatched: ['Corvin']. That is 4 spoke rather than 5 because Corvin is the deliberately unlinked guest (#344's third label path); Dunmar reaches in_channel_silent via the dropped silent track and Marn via presence. Nulling verified_at on the same rows reproduces the old {'spoke': 0, 'in_channel_silent': 0, 'unmatched': 5} exactly, so this was the cause and not a symptom. The expected counts are tabulated in the harness README's live-run section.

Backend suite 2323 passed / 13 skipped, ruff clean.

Items 1, 2 and 5 are merged in PR #510 and ship in v4.2.1. Items 3 (hotwords) and 4 (leg 3 automation) remain open here. **1. `score_markers` denominator.** `ic_only` renders only the `ic` and `proposed` registers, and four of the manifest's 54 marker *occurrences* (all 24 distinct markers still appear in IC lines) are spoken exclusively in out-of-character lines: `astrid_040` Silverbeck, `bryn_114` Ravenstoke, `bryn_176` Nettleford, `wren_196` Blackwillow. They were being scored as markers the transcriber lost, so a flawless `ic_only` transcript could not score above 50/54. `score_markers` now takes the assembly and filters expected occurrences by its registers, exactly as `score_wer` does, and returns the excluded occurrences alongside numerator and denominator so a denominator that moves is visible rather than inferred. `full` keeps every register and is unchanged at 54. Foreign markers are deliberately still checked across the whole lexicon; the order check's expected side is filtered with everything else. The recorded figures in `calibration.json` and the harness README are **annotated, not restated**: the numerator is not re-derivable from the record (it stores counts, not segments) and re-measuring needs the GPU box. What can be said without a rerun is bounded. Detection per marker is `min(occurrences, hits)`, so the correction can only lower a numerator, which puts the control's `ic_only` between 31/50 and 35/50 (62.0–70.0 %) and the worst speaker between 4/11 and 5/11. Both bands hold at the pessimistic end, so the floors stay where the measurement put them. On the next GPU rerun, rewrite the tables from the new run rather than patching; `calibration.json` carries a note saying so. **2. `GAP_NO_VOCABULARY` severity.** Raised low → medium. The #355 A/B measured +9 to +24 points of proper-noun recall with WER flat, the largest single effect in v4.2.0; ranking that below "the provider was guessed from the URL" told operators the wrong thing to fix first. The consequence copy now says what is lost (character and place names coming back mis-heard, unrepairable downstream because everything after transcription reads the transcript and not the audio) with the measured range in plain words. The OPERATIONS gap table is updated; the frontend renders severity generically, so nothing there changed. Runs already stamped `low` keep it, which is what a provenance record is for. **5. `seed_db` never set `verified_at`.** `attendance_service` resolves Discord ids through verified links only, on purpose, so the fixture seeded a roster the attendance path could not see and the uncaptured-member check (#114) was unreachable through it. Links are now seeded with a fixed timestamp, `--print-sql` renders it, and `--verify` grew a cheap second check that every seeded link is verified. Confirmed against a real database rather than argued: seeded → labels and links both clean → `{'spoke': 4, 'in_channel_silent': 2, 'unmatched': 1, 'auto_applied': 0}` with `unmatched: ['Corvin']`. That is 4 spoke rather than 5 because Corvin is the deliberately unlinked guest (#344's third label path); Dunmar reaches `in_channel_silent` via the dropped silent track and Marn via presence. Nulling `verified_at` on the same rows reproduces the old `{'spoke': 0, 'in_channel_silent': 0, 'unmatched': 5}` exactly, so this was the cause and not a symptom. The expected counts are tabulated in the harness README's live-run section. Backend suite 2323 passed / 13 skipped, ruff clean.

claude-bot referenced this issue

2026-09-06 00:31:42 +00:00

[Backend] Evaluate ASR and LLM providers against the golden corpus before committing #360

claude-bot added this to the v4.7.0 - Browse the Rules milestone

2026-10-27 17:38:57 +00:00

claude-bot commented

2026-10-30 19:29:53 +00:00

Author

Collaborator

Not blocked (2026-10-30): the dev host docker-test reaches the GPU box. WhisperX http://10.3.0.28:8091/health answers 200, and the LLM is on :8090. So items 3 (the hotwords matrix) and 4 (automating leg 3) can both be run from there. Queued in v4.7.0 after the current fixes.

**Not blocked (2026-10-30):** the dev host `docker-test` reaches the GPU box. WhisperX `http://10.3.0.28:8091/health` answers 200, and the LLM is on `:8090`. So items 3 (the hotwords matrix) and 4 (automating leg 3) can both be run from there. Queued in v4.7.0 after the current fixes.

claude-bot commented

2026-11-02 05:42:04 +00:00

Author

Collaborator

Picking up items 4 (automate leg 3) and 3 (the hotwords matrix) for 4.7.0, in that order, so the hotwords conditions can be scored end to end with the same command. Both run from docker-test against the GPU box (WhisperX :8091, LLM :8090).

rbrooks referenced this issue from a commit

2026-11-02 08:02:49 +00:00

wip: leg 3 summary-leg command and vocabulary mechanism switch (#506)

rbrooks referenced this issue from a commit

2026-11-02 08:02:49 +00:00

wip: leg 3 tests, changelog fragment (#506)

rbrooks referenced this issue from a commit

2026-11-02 08:02:49 +00:00

feat(repo): leg 3 in one command and the hotwords mechanism matrix (#506)

rbrooks referenced this issue from a commit

2026-11-02 08:02:49 +00:00

style: ruff format the leg 3 tests (#506)

claude-bot referenced this issue from a pull request that will close it,

2026-11-02 08:03:00 +00:00

feat(repo): eval harness leg 3 in one command, and the hotwords matrix (#506) #1313

claude-bot commented

2026-11-02 08:03:04 +00:00

Author

Collaborator

Items 3 and 4 are done in PR #1313. Details and the full table are on the PR.

Item 4: summary-leg runs audio → ASR → summary → scores in one command, reusing leg 2's code, with 14 offline tests.
Item 3: with a realistic 100-term vocabulary, hotwords beats initial_prompt:
- name recall 88.9% against 81.5% on full, and 90.0% against 80.0% on ic_only;
- WER is lower too;
- at 24 terms the two are the same.
- Sending both fields at 100 terms gets HTTP 500 from WhisperX. That's a bug in the iac-repo WhisperX wrapper, not Quest Board.
Recommendation: prefer hotwords when declared, fall back to initial_prompt, never send both. The product default is left alone: whether to switch it is a question for the owner.

Caveats:

one synthetic scene, one run per cell;
the matrix runs weren't --recorded;
the leg-3 summary scores are single samples and too noisy to rank conditions, so a --repeat flag is the follow-up.

Items 3 and 4 are done in PR #1313. Details and the full table are on the PR. - **Item 4:** `summary-leg` runs audio → ASR → summary → scores in one command, reusing leg 2's code, with 14 offline tests. - **Item 3:** with a realistic 100-term vocabulary, `hotwords` beats `initial_prompt`: - name recall 88.9% against 81.5% on `full`, and 90.0% against 80.0% on `ic_only`; - WER is lower too; - at 24 terms the two are the same. - Sending both fields at 100 terms gets HTTP 500 from WhisperX. That's a bug in the iac-repo WhisperX wrapper, not Quest Board. - **Recommendation:** prefer `hotwords` when declared, fall back to `initial_prompt`, never send both. The product default is left alone: whether to switch it is a question for the owner. **Caveats:** - one synthetic scene, one run per cell; - the matrix runs weren't `--record`ed; - the leg-3 summary scores are single samples and too noisy to rank conditions, so a `--repeat` flag is the follow-up.

claude-bot referenced this issue

2026-11-02 17:26:43 +00:00

Transcription: send campaign names as hotwords by default, and check it on dev with a real recording #1315

claude-bot referenced this issue

2026-11-02 17:31:38 +00:00

feat(webapp): campaign names go to the transcriber as hotwords by default (#1315) #1316

rbrooks referenced this issue from a commit

2026-11-02 17:31:40 +00:00

feat(webapp): campaign names go to the transcriber as hotwords by default (#1315)

claude-bot closed this issue

2026-11-02 18:49:23 +00:00

claude-bot referenced this issue from a commit