fix: record the model the bundled transcriber reports, and three harness fixes from the v4.2.0 validation #510

Merged
claude-bot merged 5 commits from fix/508-506-usage-model-and-harness into main 2026-09-05 16:02:14 +00:00
Contributor

Closes #508. Refs #506 (items 1, 2 and 5; items 3 and 4 stay open).

Four independent fixes from the first live validation of v4.2.0, one commit each.

  • #508 session_usage.asr_model was empty on every self-hosted run. The bundled adapter takes no model parameter, so the only source was the configured string, which is blank by default. The server names its model in /health on both API versions and discovery was already parsing that body, so AsrCapabilities now carries it and resolve_asr_identity falls back configured → requested id → reported model → None. No new request on the hot path: process_audio re-resolves identity right after the vocabulary step has probed (cached per endpoint) and before transcription. A test pins that a populated model on local_whisper still yields no dollar figure. Admin → AI Usage shows the model beside the transcription time.
  • #506 item 1: the marker scorer's denominator. ic_only drops the ooc/meta registers, and four of the 54 marker occurrences are spoken only there, so a flawless ic_only transcript could not score above 50/54. score_markers now filters expected occurrences by the assembly's registers the way score_wer does and reports the excluded occurrences. full is unchanged at 54. Recorded calibration numbers are annotated with bounds rather than restated, since re-measuring needs the GPU box.
  • #506 item 2: GAP_NO_VOCABULARY low → medium, with the consequence copy rewritten around what the GM loses and the measured 10–25 point proper-noun effect from #355.
  • #506 item 5: seed_db now seeds verified links, so the attendance path is reachable through the harness. Proven against a real database: seeded → {'spoke': 4, 'in_channel_silent': 2, 'unmatched': 1} (Corvin is the deliberately unlinked guest); nulling verified_at reproduces the old all-unmatched result exactly. --verify checks the links too.

Backend suite 2323 passed / 13 skipped, frontend 505 passed, ruff and eslint clean. No migration, no contract change.

🤖 Generated with Claude Code

Closes #508. Refs #506 (items 1, 2 and 5; items 3 and 4 stay open). Four independent fixes from the first live validation of v4.2.0, one commit each. - **#508 `session_usage.asr_model` was empty on every self-hosted run.** The bundled adapter takes no model parameter, so the only source was the configured string, which is blank by default. The server names its model in `/health` on both API versions and discovery was already parsing that body, so `AsrCapabilities` now carries it and `resolve_asr_identity` falls back configured → requested id → reported model → `None`. No new request on the hot path: `process_audio` re-resolves identity right after the vocabulary step has probed (cached per endpoint) and before transcription. A test pins that a populated model on `local_whisper` still yields no dollar figure. Admin → AI Usage shows the model beside the transcription time. - **#506 item 1: the marker scorer's denominator.** `ic_only` drops the `ooc`/`meta` registers, and four of the 54 marker occurrences are spoken only there, so a flawless `ic_only` transcript could not score above 50/54. `score_markers` now filters expected occurrences by the assembly's registers the way `score_wer` does and reports the excluded occurrences. `full` is unchanged at 54. Recorded calibration numbers are annotated with bounds rather than restated, since re-measuring needs the GPU box. - **#506 item 2: `GAP_NO_VOCABULARY` low → medium**, with the consequence copy rewritten around what the GM loses and the measured 10–25 point proper-noun effect from #355. - **#506 item 5: `seed_db` now seeds verified links**, so the attendance path is reachable through the harness. Proven against a real database: seeded → `{'spoke': 4, 'in_channel_silent': 2, 'unmatched': 1}` (Corvin is the deliberately unlinked guest); nulling `verified_at` reproduces the old all-unmatched result exactly. `--verify` checks the links too. Backend suite 2323 passed / 13 skipped, frontend 505 passed, ruff and eslint clean. No migration, no contract change. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Every run through the bundled WhisperX server wrote `asr_model` empty while
`llm_model` was populated, because the bundled adapter takes no model
parameter: the server loads one at startup and there is nothing configured to
record. That left "which model transcribed this session" unanswerable for
exactly the deployments where the model changes without a setting being edited.

The server does say — `/health` reports it on both API versions, and capability
discovery already reads that body — so `AsrCapabilities` now carries the
reported model and `resolve_asr_identity` falls back to it when nothing is
configured and the adapter has no model to request. The read is of the last
probed answer, not a probe, so nothing new goes on the wire; `process_audio`
resolves identity a second time after the vocabulary step has probed, which is
where a cold worker first learns the answer, and still before transcription so
a run that dies mid-ASR records what was serving it.

Unknown stays None rather than becoming "": an endpoint nobody could reach has
told us nothing. A configured model still wins. Pricing is untouched —
`estimate_asr_cost` short-circuits on the self-hosted adapter names before it
looks a rate up — so naming a model cannot put a dollar figure on somebody's
own GPU.

Admin → AI Usage now shows it beside the transcription time, the way the LLM
column has always named its own model.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`score_markers` counted every one of the manifest's 54 marker occurrences for
both assemblies. `ic_only` renders only the `ic` and `proposed` registers, and
four of those occurrences are spoken exclusively in out-of-character lines it
does not contain — so they were scored as markers the transcriber lost. That
understated the leg by roughly six points and, worse, turned part of the gap
between the two assemblies into arithmetic while it read as a measurement of
the ASR.

The scorer now takes the assembly, filters expected occurrences by its
registers exactly as `score_wer` does, and returns the excluded occurrences in
the report so a denominator that moved is visible rather than inferred from a
rate that changed. `full` keeps every register, so its denominator is still 54
and every committed `full` figure means what it said. The order check's
expected side is filtered with it — an `ic_only` track did not speak the table
talk. Foreign markers deliberately are not: a marker on a track whose audio
never said it is a finding under either assembly.

The recorded `ic_only` numbers in `calibration.json` and the README are
annotated rather than restated. Re-deriving the numerator needs the segments,
which the record does not store, and re-running the leg needs the GPU server;
what can be said without either is that the control lands between 31/50 and
35/50 and the worst speaker between 4/11 and 5/11, so both floors still hold.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`asr_no_vocabulary` was severity low, which put it below "the provider was
guessed from the URL" in the panel an operator opens to decide what to fix. The
#355 A/B measured the vocabulary list at +9 to +24 percentage points of
proper-noun recall on identical audio against the same server, with word error
rate flat or falling — the largest single effect anything in v4.2.0 measured.
Low was a judgement made before there was a measurement; it is medium now.

The consequence copy said "invented proper nouns are transcribed phonetically",
which is the mechanism. It now says what the GM loses: their own characters' and
places' names coming back mis-heard, spelled differently each time, and
unrepairable downstream because the summariser, the beat extractor and the wiki
proposer all read the transcript rather than the audio.

Rows already stamped with the old severity keep it. A run's `capability_gaps`
record what was believed when it ran, and re-deriving history is what the
provenance record exists not to do.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`attendance_service._resolve_discord_ids_to_members` requires
`verified_at IS NOT NULL` on purpose — an unclaimed link must never drive
attendance — and `seed_db` created its links without it. So the harness seeded
five members the attendance path then treated as strangers: a live run returned
`{'spoke': 0, 'in_channel_silent': 0, 'unmatched': 5}` whatever the pipeline
did, and the uncaptured-member check (#114), one of the things this fixture
exists to exercise, was unreachable through it.

Seeded with a fixed timestamp rather than `now()`, in keeping with the file's
uuid5 ids: two seeds of the same fixture should differ in nothing a query can
see. `--print-sql` renders it too, so the two descriptions of what seeding does
still agree.

`--verify` grew a second, cheap check — every seeded link exists and is
verified — and prints what a live run's attendance proposals should then say.
Confirmed against a real database: 4 spoke (GM, Ilsa, Rook, Petra), 2
in_channel_silent (Dunmar's dropped silent track and Marn's presence-only
membership), 1 unmatched (Corvin, the deliberately unlinked guest), 0
auto-applied on the default `propose` mode. Nulling `verified_at` on the same
rows reproduces the old all-strangers result exactly, which is the evidence
that this was the cause and not a symptom.

The README's live-run section carries the same table, because the half that
needs a processed session cannot be checked by `--verify`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
chore(changelog): one heading style in the unreleased section
All checks were successful
CI / Bot/backend version sync (pull_request) Successful in 28s
CI / Backend lint (ruff) (pull_request) Successful in 33s
CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 1m8s
CI / Bot tests and audit (pull_request) Successful in 1m23s
CI / Frontend tests, audit, and build (pull_request) Successful in 1m41s
CI / Docker image build (pull_request) Successful in 4m38s
CI / Backend migration, tests, and audit (pull_request) Successful in 8m19s
CI / Synthetic session harness (no GPU, no LLM) (pull_request) Successful in 17m33s
194a3507a3
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
claude-bot scheduled this pull request to auto merge when all checks succeed 2026-09-05 15:44:16 +00:00
claude-bot deleted branch fix/508-506-usage-model-and-harness 2026-09-05 16:02:15 +00:00
Sign in to join this conversation.
No description provided.