evals: the ASR leg can name its provider, model and vocabulary, and we measured what the vocabulary is worth #505

Merged

claude-bot merged 3 commits from feat/502-asr-leg-vocabulary into main

2026-09-05 07:11:21 +00:00

claude-bot commented

2026-09-05 06:55:42 +00:00

Contributor

Refs #502 (item 5), #355 (the measured effect of vocabulary bias), #360 (experiment 1 from the provider evaluation).

The synthetic-session harness's ASR leg gained --asr-provider, --asr-model and --vocabulary so a run can be pointed at a named provider and handed a term list, and the run record carries what was used. With that in place the effect of vocabulary bias on the invented campaign's proper nouns was measured against the bundled WhisperX server (API v2, large-v3-turbo):

condition	marker detection	WER
control (no vocabulary), reproduces the calibration	72.2 %	unchanged
the 24 markers as the list (ceiling)	96.3 %	unchanged
24 markers diluted in a 100-term list (realistic)	81.5 %	unchanged

No WER regression, no fabricated names, and the control reproduced the calibration run bit-identically, so the harness is stable enough to carry this kind of comparison. Evals-only change, no runtime code, no migration.

🤖 Generated with Claude Code

Refs #502 (item 5), #355 (the measured effect of vocabulary bias), #360 (experiment 1 from the provider evaluation). The synthetic-session harness's ASR leg gained `--asr-provider`, `--asr-model` and `--vocabulary` so a run can be pointed at a named provider and handed a term list, and the run record carries what was used. With that in place the effect of vocabulary bias on the invented campaign's proper nouns was measured against the bundled WhisperX server (API v2, large-v3-turbo): | condition | marker detection | WER | foreign markers | |---|---|---|---| | control (no vocabulary), reproduces the calibration | 72.2 % | unchanged | 0 | | the 24 markers as the list (ceiling) | 96.3 % | unchanged | 0 | | 24 markers diluted in a 100-term list (realistic) | 81.5 % | unchanged | 0 | No WER regression, no fabricated names, and the control reproduced the calibration run bit-identically, so the harness is stable enough to carry this kind of comparison. Evals-only change, no runtime code, no migration. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 3 commits

2026-09-05 06:55:42 +00:00

feat(evals): the ASR leg can name its provider, model and vocabulary (#502 ) 81263171f0

`asr_leg.run_leg` called `transcribe_with_optional_vad` with four positional
arguments and stopped there, so the three settings a production caller settles
before transcription -- the configured adapter, its model, and the campaign's
proper nouns (#355) -- were left at their defaults and nothing recorded that
they had been. Every figure in `calibration.json`, including the 72.2% marker
detection the `session_marker_detection` band cites, was therefore measured
with vocabulary bias *off* against a server whose `/health` reports
`initial_prompt: true`. That is a fine number; it was just not labelled as one
configuration among several, which is what makes an A/B impossible to ask for.

So the leg now takes all three and forwards them, and the CLI can set them:

  --asr-provider / --asr-model   default from QB_SYNTH_ASR_PROVIDER and
                                 QB_SYNTH_ASR_MODEL; blank keeps today's
                                 behaviour of sniffing from the endpoint URL
  --vocabulary none|markers|file:<path>

`markers` is the union of the manifest's per-speaker marker lists,
deduplicated, in manifest order -- the union and not per-track lists, because
that is the only shape production can produce: `vocabulary_service` assembles
what the wiki knows about the *campaign* and cannot know who is about to say
what. It also leaves the `foreign_markers == 0` assertion something to catch:
with every speaker's markers in every speaker's prompt, a bias that
manufactures a name on the wrong track finally has the opportunity to.

The terms are passed as a list and formatted by the adapter, not here.
`LocalWhisperProvider.submit` comma-joins and truncates at 100; reproducing
that in the harness would measure the harness's idea of the prompt shape
rather than the pipeline's. A test pins the fixture's 24 markers under that
cap, because a truncated list would bias the model toward a prefix while the
report claimed the whole lexicon.

Recording, both additive:

* `calibration.json` gains a `vocabulary_legs` slot and a `recorded_at` stamp
  per slot. `legs` stays the control, because every band constant cites it and
  `scorers._calibrated_kept` reads it -- recording an experiment into `legs`
  would re-derive the thresholds from the experiment.
* `--record` now also appends one line per assembly to the eval harness's
  `results/history.jsonl`, carrying the vocabulary mode, term count, provider
  and model beside the detection rate and WER. Same file as the summariser's
  history on purpose: "did turning bias on help?" is the question that file
  exists for, and a second history is a second place to forget to look.

Artifacts from a biased run are written under `<assembly>.vocab-<mode>` so they
cannot overwrite the control they are being compared against.

No measurement in this commit; it only makes one askable.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

chore(evals): record marker detection with vocabulary bias on (#355 , #360 ) 72547101a4

The measurement #502 made askable. Six live runs against the bundled server
(`large-v3-turbo` on CUDA, `/health` reporting `initial_prompt: true`), both
assemblies, three vocabularies.

The control first, because the comparison is worth nothing without it: with
`--vocabulary none` the leg reproduced the calibrated record **bit-identically**
on every scored field -- 72.2% / 64.8% marker detection, 3.73% / 3.32% WER,
every per-speaker figure, every kept second. That is now the fourth identical
run of this leg, so the band constants are untouched: nothing about the control
moved and there is nothing to re-derive.

Marker detection, scripted occurrences recovered as whole words on the right
track, out of 54:

                        full          ic_only
  none (control)   39  72.2%     35  64.8%
  markers (24)     52  96.3%     48  88.9%     +24.1 pp, both
  diluted (100)    44  81.5%     40  74.1%     +9.3 pp, both

`markers` is the 24 screened proper nouns and nothing else -- the ceiling of
the mechanism, since the list names exactly the words being scored. `diluted`
answers the obvious objection by hiding those 24 among 76 campaign names that
are never spoken, which is the shape `vocabulary_service` actually produces:
about half the lift survives. Both are worth having and neither should be
quoted as the other, so `vocabulary_legs` is keyed by mode.

Three findings beyond the headline:

* **WER did not suffer.** The stated risk was that a prompt induces list-like
  output; it did not appear. Session WER fell in five of six legs and was flat
  in the sixth (`full` + `markers`, 3.73% -> 3.70%), and no per-speaker figure
  rose by more than 0.4 pp. Nor is there any prompt echo in the transcripts.
* **Nothing was manufactured.** Foreign markers stayed 0 and out-of-position
  stayed 0 in all six legs -- and this time that means something, because every
  speaker's markers were in every speaker's prompt, so the model was offered
  the chance to put Corvin's names on Petra's track. Stronger still: counting
  raw whole-word hits, no marker appeared *more often than it was scripted* in
  any run, and of the 76 diluted-list names that are never spoken, not one
  reached a transcript. Biasing toward a name does not conjure it.
* **The residual loss is one word.** With `markers`, the only occurrences still
  missed anywhere are two of the GM's four `Marlgate` -- the marker README.md
  already names as the one that passed screening 2/2 and was lost at all four
  occurrences. Bias recovers half of it.

Side effects worth knowing: Whisper returns ~17% fewer, longer segments with a
prompt (367 -> 303 on `full`), which improves onset p95 (2.107 s -> 0.574 s) and
costs three utterances their own segment (`onset_unmatched` 1 -> 4, band ≤ 6).
The diluted list shows neither effect as strongly. VAD kept seconds are
identical everywhere, as they must be -- ffmpeg decides them before a byte is
sent, and that is what makes these runs a comparison rather than six sessions.

Every run is in `results/history.jsonl` under its own `QB_EVAL_GIT_SHA`,
including the repeat of `markers`, which came back identical on every scored
field: run-to-run spread on this server is zero.

Two tests pin the record rather than the rerun, which needs a GPU: that `legs`
is still the un-biased run the bands cite, and that the with-bias legs still
show the lift, the flat WER and the zero foreign markers they are quoted for.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs(evals): what vocabulary bias is worth, and what the bands describe (#355 , #502 )

CI / Bot/backend version sync (pull_request) Successful in 45s

Details

CI / Backend lint (ruff) (pull_request) Successful in 55s

Details

CI / Docker image build (pull_request) Successful in 18s

Details

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 1m30s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 1m55s

Details

CI / Bot tests and audit (pull_request) Successful in 2m9s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 9m34s

Details

CI / Synthetic session harness (no GPU, no LLM) (pull_request) Successful in 14m39s

Details

05ffec29c0

The harness README's measured table gains a `+vocab` column for both assemblies
and a new "Vocabulary bias, measured" section carrying the three-way comparison,
the four risks that did not materialise (WER, foreign markers, over-generation,
fabricated names), the two side effects that did (fewer/longer segments, three
utterances losing a segment of their own), and the reason `markers` is a ceiling
rather than a forecast.

"What screening does not tell you" gets its other half. It has said since #433
that a marker surviving a carrier sentence can still be lost in the session, and
named `Marlgate` as the case in point. What it could not say was that most of
that loss is recoverable and the pipeline already owns the mechanism.

The bands section and `scorers.py`'s `CALIBRATION` docstring now say out loud
that every measured figure is the **control**. Before #502 that was true by
inability -- the leg could not pass a vocabulary. It is now a choice, and a
choice nobody wrote down is a choice somebody undoes: re-recording a biased run
into `legs` would move `session_marker_detection`'s cited figure from 64.8% to
88.9% and re-derive a floor meant to catch a broken pipeline from a run with the
mitigation on. Two band rationales gain the specific with-bias numbers so that
trap is visible where the constant is, and `onset_unmatched` notes that its
headroom is 4-of-6 rather than 1-of-6 once a prompt is in play.

`docs/OPERATIONS.md` "What changes when you update" now quotes the figure to
operators, with the ceiling and the diluted number both, because "names are
spelled consistently instead of phonetically" is the kind of claim a self-hoster
has no way to size. `docs/DEVELOPMENT.md` gains the A/B commands.

No CHANGELOG entry: nothing user-facing changed. The capability panel still
rates `No vocabulary hints` as low severity, which this measurement is an
argument against -- but that lives in `capability_service.py` and is a separate
change with its own user-visible consequence.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed

2026-09-05 06:56:07 +00:00

claude-bot referenced this pull request

2026-09-05 06:56:15 +00:00

[Backend] Feed campaign proper nouns to the ASR as vocabulary bias #355

claude-bot referenced this pull request

2026-09-05 06:56:21 +00:00

[Backend] Evaluate ASR and LLM providers against the golden corpus before committing #360

claude-bot referenced this pull request

2026-09-05 06:56:23 +00:00

[Backend] Follow-ups from the provider evaluation: usage rows misclassify sniffed managed runs, managed ASR is unpriceable, and the live tooling cannot reach the managed adapter (#360) #502

claude-bot referenced this pull request