Drop Whisper hallucination segments before the transcript is merged #569
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem. Whisper produces text for near-silent audio, and nothing in the pipeline removes it. Tonight's session (2026-09-09, six speakers, VAD on at -45 dB / 2000 ms, large-v3-turbo) stored 1,778 segments of which 143 are not speech:
Plus a smaller number of single-word artefacts repeated across speakers ("Fulfillment", "Mimic", "Outro", "S.T.", "S.T.T.") and foreign-language fragments on quiet stretches. They cost tokens, they are cited as evidence by the beat extractor, and they mislead a summariser that takes "Thank you" literally.
Design. A deterministic, conservative filter in
audio_serviceapplied to the flat segment list aftertranscribe_with_optional_vadand beforecheck_transcript_covers_session/merge_attributed_transcript(the seam isreminder_tasks.process_audioaround line 2790). Pure function, returns(kept, dropped), no model, no network:thank youruns,thanks for watching,sous-titrage…,closed caption(ing|s)…,subtitles by…,amara.org,red bee media,s.t.,s.t.t.,outro,fulfillment,mimic, and the lone-punctuation case (no letters or digits at all).no_speech_prob,avg_logprob) use it as a fourth, optional signal; do not require it.Never log the dropped text, only counts (per track and total), matching the module's rule that transcript content stays out of logs. Record the total on the run so a GM can see it: the usage/telemetry record from #357 or the summarisation run record, whichever already carries per-run counts. The eval harness (#349, #433) must go through the same function so a change to the list is measured, not assumed.
Not in scope here. Correcting misheard names ("Adani", "Parahark") against the vocabulary list. That is a separate, riskier step; file it if the filter proves out.
Tests. A fixture list of segments in the shapes above (these are Whisper artefacts, not anyone's speech), asserting each rule drops what it should and keeps ordinary short lines ("Yes.", "Okay.", "Bark, bark, bark." spoken by the GM), plus one end-to-end test through
process_audio's seam showing the merged transcript no longer contains them and the count reaches the record.Correction to the design after checking the vocabulary the ASR received. The worker sent 36 terms (from 42 candidates, limit 100, style
initial_prompt) for this session. Three of the "artefacts" listed above are those terms echoing back: "Fulfillment" is the wiki entry Office of Prophetic Fulfillment, "The Miller, The Miller" is The Miller, "Mimic" is Mimic's Slime. Whisper regurgitates its bias prompt on near-silent audio, which is a known failure mode ofinitial_prompt.So the filter must not hard-code campaign words. Instead it gets a fourth rule, prompt echo: a segment whose whole normalised text equals one of the vocabulary terms sent for the run (or that term repeated) is dropped unless the term is a character or NPC name, since "Theo!" alone is plausible speech and "Office of Prophetic Fulfillment" alone is not. Counted separately from the generic artefacts.
Two follow-ups this points at, not for this issue.
vocabulary_serviceshould prefer named entities (characters, NPCs, places, factions) and skip concept/item titles that are ordinary words.hotwordsas well asinitial_prompt; whether hotwords echo less on silence is worth measuring on the dev replay of this session (audio already copied to dev).Correction to my previous comment's first follow-up: the generic titles I listed (Greenery, Mushrooms, Foggy Sunlight, Cave Lore, Fate, Cavern Flora) are all
conceptentries, andPROPER_NOUN_LORE_TYPESalready excludes that type, so they were never in the 36 terms. The dilution claim is withdrawn.The echoed terms are legitimate proper nouns the list should carry: Office of Prophetic Fulfillment (organization), The Miller (npc), Mimic's Slime (artifact). So the remaining follow-up is only the second one: the echo is a property of
initial_prompton silent audio, and the server also offershotwords. Measuring both on the dev replay is the next step; the prompt-echo rule in this issue covers the transcript either way.Measured on dev, same audio, before and after #570. The transcript side is deterministic (the baseline replay reproduced prod's 1,778 segments and every junk count exactly), so this is a clean before/after:
hallucination_dropson the run:{"artefact": 154, "repetition": 5, "prompt_echo": 1}; the worker logged the same split, counts only.One gap left, filed as #576. The echo rule matches a segment that equals a whole vocabulary term, but Whisper echoes a fragment: "Fulfillment" is the last word of Office of Prophetic Fulfillment, "Mimic" the first word of Mimic's Slime. "The Miller" is kept on purpose (an NPC name is plausible speech).
Side effect worth knowing: with 160 fewer lines the transcript fell from 5 extraction windows to 4, and this run had no failed windows and validated 23 of 53 beats in 3 passes, against 27 of 99 in 4 passes with two windows lost on the baseline. Same model, same router; too early to call that the filter's doing rather than luck, but it is the direction #572 predicts.