Synthetic-session harness: marker scorer denominator, vocabulary-gap severity, hotwords #506
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Follow-ups surfaced by the vocabulary-bias measurement (PR #505, #355, #360). None of them change the numbers already reported, but each should be fixed before the harness is used for a cross-assembly or cross-provider comparison.
score_markersuses the wrong denominator for the in-character-only assembly. The scorer divides by the full marker count (54) for both assemblies, but only 50 of the markers occur in theic_onlyscript, so that assembly's detection rate is understated by ~7 % relative. Aroundevals/synthetic_session/scorers.py:308. Count the markers that actually occur in the scored script instead.GAP_NO_VOCABULARYis probably under-severe. The capability gap for "this provider ignores the vocabulary list" is currently low severity. The measurement shows the list is worth 9–24 pp of proper-noun recall, which is the largest single effect measured in v4.2.0, soSEVERITY_MEDIUMfits better. Also check the copy on the Admin capability panel says what the GM loses.hotwordsis unexplored. WhisperX API v2 honours bothinitial_prompt(what the vocabulary list becomes today) andhotwords. Run the same three-condition matrix with hotwords, and with both, and record which one the adapter should prefer.5.
seed_db.pycreatesPlatformLinkrows withoutverified_at, so the attendance path is never exercised. Found validating v4.2.0 on dev: the end-to-end run loggedattendance autofill {'spoke': 0, 'in_channel_silent': 0, 'unmatched': 5, 'auto_applied': 0}and wrote zeroattendance_proposals. Root cause is that_resolve_discord_ids_to_members(attendance_service.py, around line 150) deliberately requiresverified_at IS NOT NULL— an unclaimed link must never drive attendance — whileseed_db.py(around line 216) never sets it. Speaker resolution does not require verification, which is why the transcript labels were still right; only attendance went blind. The product behaviour is correct, the fixture is wrong.Fix: set
verified_aton the seeded links, then assert in the harness's check that the presence-only member (Kesh) and the silent-track owner (Dunmar) are reported as uncaptured, and that the five speakers match. That turns the #114 uncaptured-member path into something the harness actually tests.Picking up items 1, 2 and 5 for v4.2.1 (the scorer denominator, the
GAP_NO_VOCABULARYseverity, andseed_dbsettingverified_at). Items 3 (hotwords) and 4 (leg 3 automation) are measurement and harness work rather than fixes, and stay open here.Items 1, 2 and 5 are merged in PR #510 and ship in v4.2.1. Items 3 (hotwords) and 4 (leg 3 automation) remain open here.
1.
score_markersdenominator.ic_onlyrenders only theicandproposedregisters, and four of the manifest's 54 marker occurrences (all 24 distinct markers still appear in IC lines) are spoken exclusively in out-of-character lines:astrid_040Silverbeck,bryn_114Ravenstoke,bryn_176Nettleford,wren_196Blackwillow. They were being scored as markers the transcriber lost, so a flawlessic_onlytranscript could not score above 50/54.score_markersnow takes the assembly and filters expected occurrences by its registers, exactly asscore_werdoes, and returns the excluded occurrences alongside numerator and denominator so a denominator that moves is visible rather than inferred.fullkeeps every register and is unchanged at 54. Foreign markers are deliberately still checked across the whole lexicon; the order check's expected side is filtered with everything else.The recorded figures in
calibration.jsonand the harness README are annotated, not restated: the numerator is not re-derivable from the record (it stores counts, not segments) and re-measuring needs the GPU box. What can be said without a rerun is bounded. Detection per marker ismin(occurrences, hits), so the correction can only lower a numerator, which puts the control'sic_onlybetween 31/50 and 35/50 (62.0–70.0 %) and the worst speaker between 4/11 and 5/11. Both bands hold at the pessimistic end, so the floors stay where the measurement put them. On the next GPU rerun, rewrite the tables from the new run rather than patching;calibration.jsoncarries a note saying so.2.
GAP_NO_VOCABULARYseverity. Raised low → medium. The #355 A/B measured +9 to +24 points of proper-noun recall with WER flat, the largest single effect in v4.2.0; ranking that below "the provider was guessed from the URL" told operators the wrong thing to fix first. The consequence copy now says what is lost (character and place names coming back mis-heard, unrepairable downstream because everything after transcription reads the transcript and not the audio) with the measured range in plain words. The OPERATIONS gap table is updated; the frontend renders severity generically, so nothing there changed. Runs already stampedlowkeep it, which is what a provenance record is for.5.
seed_dbnever setverified_at.attendance_serviceresolves Discord ids through verified links only, on purpose, so the fixture seeded a roster the attendance path could not see and the uncaptured-member check (#114) was unreachable through it. Links are now seeded with a fixed timestamp,--print-sqlrenders it, and--verifygrew a cheap second check that every seeded link is verified.Confirmed against a real database rather than argued: seeded → labels and links both clean →
{'spoke': 4, 'in_channel_silent': 2, 'unmatched': 1, 'auto_applied': 0}withunmatched: ['Corvin']. That is 4 spoke rather than 5 because Corvin is the deliberately unlinked guest (#344's third label path); Dunmar reachesin_channel_silentvia the dropped silent track and Marn via presence. Nullingverified_aton the same rows reproduces the old{'spoke': 0, 'in_channel_silent': 0, 'unmatched': 5}exactly, so this was the cause and not a symptom. The expected counts are tabulated in the harness README's live-run section.Backend suite 2323 passed / 13 skipped, ruff clean.