Drop Whisper hallucination segments before the transcript is merged

claude-bot commented

2026-09-09 05:04:08 +00:00

Collaborator

Problem. Whisper produces text for near-silent audio, and nothing in the pipeline removes it. Tonight's session (2026-09-09, six speakers, VAD on at -45 dB / 2000 ms, large-v3-turbo) stored 1,778 segments of which 143 are not speech:

shape	count
caption credits ("Sous-titrage ST' 501", "Closed Captioning by …", "Red Bee Media", "Société Radio-Canada")	41
bare "Thank you." / "Thank you. Thank you."	78
a lone dot	24

Plus a smaller number of single-word artefacts repeated across speakers ("Fulfillment", "Mimic", "Outro", "S.T.", "S.T.T.") and foreign-language fragments on quiet stretches. They cost tokens, they are cited as evidence by the beat extractor, and they mislead a summariser that takes "Thank you" literally.

Design. A deterministic, conservative filter in audio_service applied to the flat segment list after transcribe_with_optional_vad and before check_transcript_covers_session / merge_attributed_transcript (the seam is reminder_tasks.process_audio around line 2790). Pure function, returns (kept, dropped), no model, no network:

Whole-segment match, after lower-casing and stripping punctuation and whitespace, against a small list of known Whisper hallucination phrases and regex families: thank you runs, thanks for watching, sous-titrage…, closed caption(ing|s)…, subtitles by…, amara.org, red bee media, s.t., s.t.t., outro, fulfillment, mimic, and the lone-punctuation case (no letters or digits at all).
Repetition loops: one token repeated six or more times with nothing else in the segment.
Script mismatch: when the campaign language is set and is Latin-script, a segment whose letters are mostly outside Latin script (CJK, Cyrillic, Arabic…) is dropped. Latin-extended (á, é) stays, so Icelandic-looking fragments are not caught by this rule; that is acceptable.
If the provider returned per-segment confidence (no_speech_prob, avg_logprob) use it as a fourth, optional signal; do not require it.

Never log the dropped text, only counts (per track and total), matching the module's rule that transcript content stays out of logs. Record the total on the run so a GM can see it: the usage/telemetry record from #357 or the summarisation run record, whichever already carries per-run counts. The eval harness (#349, #433) must go through the same function so a change to the list is measured, not assumed.

Not in scope here. Correcting misheard names ("Adani", "Parahark") against the vocabulary list. That is a separate, riskier step; file it if the filter proves out.

Tests. A fixture list of segments in the shapes above (these are Whisper artefacts, not anyone's speech), asserting each rule drops what it should and keeps ordinary short lines ("Yes.", "Okay.", "Bark, bark, bark." spoken by the GM), plus one end-to-end test through process_audio's seam showing the merged transcript no longer contains them and the count reaches the record.

**Problem.** Whisper produces text for near-silent audio, and nothing in the pipeline removes it. Tonight's session (2026-09-09, six speakers, VAD on at -45 dB / 2000 ms, large-v3-turbo) stored 1,778 segments of which 143 are not speech: | shape | count | |---|---| | caption credits ("Sous-titrage ST' 501", "Closed Captioning by …", "Red Bee Media", "Société Radio-Canada") | 41 | | bare "Thank you." / "Thank you. Thank you." | 78 | | a lone dot | 24 | Plus a smaller number of single-word artefacts repeated across speakers ("Fulfillment", "Mimic", "Outro", "S.T.", "S.T.T.") and foreign-language fragments on quiet stretches. They cost tokens, they are cited as evidence by the beat extractor, and they mislead a summariser that takes "Thank you" literally. **Design.** A deterministic, conservative filter in `audio_service` applied to the flat segment list after `transcribe_with_optional_vad` and before `check_transcript_covers_session` / `merge_attributed_transcript` (the seam is `reminder_tasks.process_audio` around line 2790). Pure function, returns `(kept, dropped)`, no model, no network: 1. Whole-segment match, after lower-casing and stripping punctuation and whitespace, against a small list of known Whisper hallucination phrases and regex families: `thank you` runs, `thanks for watching`, `sous-titrage…`, `closed caption(ing|s)…`, `subtitles by…`, `amara.org`, `red bee media`, `s.t.`, `s.t.t.`, `outro`, `fulfillment`, `mimic`, and the lone-punctuation case (no letters or digits at all). 2. Repetition loops: one token repeated six or more times with nothing else in the segment. 3. Script mismatch: when the campaign language is set and is Latin-script, a segment whose letters are mostly outside Latin script (CJK, Cyrillic, Arabic…) is dropped. Latin-extended (á, é) stays, so Icelandic-looking fragments are not caught by this rule; that is acceptable. 4. If the provider returned per-segment confidence (`no_speech_prob`, `avg_logprob`) use it as a fourth, optional signal; do not require it. Never log the dropped text, only counts (per track and total), matching the module's rule that transcript content stays out of logs. Record the total on the run so a GM can see it: the usage/telemetry record from #357 or the summarisation run record, whichever already carries per-run counts. The eval harness (#349, #433) must go through the same function so a change to the list is measured, not assumed. **Not in scope here.** Correcting misheard names ("Adani", "Parahark") against the vocabulary list. That is a separate, riskier step; file it if the filter proves out. **Tests.** A fixture list of segments in the shapes above (these are Whisper artefacts, not anyone's speech), asserting each rule drops what it should and keeps ordinary short lines ("Yes.", "Okay.", "Bark, bark, bark." spoken by the GM), plus one end-to-end test through `process_audio`'s seam showing the merged transcript no longer contains them and the count reaches the record.

claude-bot commented

2026-09-09 05:06:40 +00:00

Author

Collaborator

Correction to the design after checking the vocabulary the ASR received. The worker sent 36 terms (from 42 candidates, limit 100, style initial_prompt) for this session. Three of the "artefacts" listed above are those terms echoing back: "Fulfillment" is the wiki entry Office of Prophetic Fulfillment, "The Miller, The Miller" is The Miller, "Mimic" is Mimic's Slime. Whisper regurgitates its bias prompt on near-silent audio, which is a known failure mode of initial_prompt.

So the filter must not hard-code campaign words. Instead it gets a fourth rule, prompt echo: a segment whose whole normalised text equals one of the vocabulary terms sent for the run (or that term repeated) is dropped unless the term is a character or NPC name, since "Theo!" alone is plausible speech and "Office of Prophetic Fulfillment" alone is not. Counted separately from the generic artefacts.

Two follow-ups this points at, not for this issue.

The vocabulary list is diluted by generic wiki titles the lore pipeline auto-created: Greenery, Mushrooms, Foggy Sunlight, Cave Lore, Fate, Underground Cavern. Those are not proper nouns and bias nothing useful, while giving Whisper more to echo. vocabulary_service should prefer named entities (characters, NPCs, places, factions) and skip concept/item titles that are ordinary words.
The server advertises hotwords as well as initial_prompt; whether hotwords echo less on silence is worth measuring on the dev replay of this session (audio already copied to dev).

**Correction to the design after checking the vocabulary the ASR received.** The worker sent 36 terms (from 42 candidates, limit 100, style `initial_prompt`) for this session. Three of the "artefacts" listed above are those terms echoing back: "Fulfillment" is the wiki entry *Office of Prophetic Fulfillment*, "The Miller, The Miller" is *The Miller*, "Mimic" is *Mimic's Slime*. Whisper regurgitates its bias prompt on near-silent audio, which is a known failure mode of `initial_prompt`. So the filter must not hard-code campaign words. Instead it gets a fourth rule, **prompt echo**: a segment whose whole normalised text equals one of the vocabulary terms sent for the run (or that term repeated) is dropped unless the term is a character or NPC name, since "Theo!" alone is plausible speech and "Office of Prophetic Fulfillment" alone is not. Counted separately from the generic artefacts. **Two follow-ups this points at, not for this issue.** - The vocabulary list is diluted by generic wiki titles the lore pipeline auto-created: *Greenery*, *Mushrooms*, *Foggy Sunlight*, *Cave Lore*, *Fate*, *Underground Cavern*. Those are not proper nouns and bias nothing useful, while giving Whisper more to echo. `vocabulary_service` should prefer named entities (characters, NPCs, places, factions) and skip concept/item titles that are ordinary words. - The server advertises `hotwords` as well as `initial_prompt`; whether hotwords echo less on silence is worth measuring on the dev replay of this session (audio already copied to dev).

claude-bot commented

2026-09-09 05:09:32 +00:00

Author

Collaborator

Correction to my previous comment's first follow-up: the generic titles I listed (Greenery, Mushrooms, Foggy Sunlight, Cave Lore, Fate, Cavern Flora) are all concept entries, and PROPER_NOUN_LORE_TYPES already excludes that type, so they were never in the 36 terms. The dilution claim is withdrawn.

The echoed terms are legitimate proper nouns the list should carry: Office of Prophetic Fulfillment (organization), The Miller (npc), Mimic's Slime (artifact). So the remaining follow-up is only the second one: the echo is a property of initial_prompt on silent audio, and the server also offers hotwords. Measuring both on the dev replay is the next step; the prompt-echo rule in this issue covers the transcript either way.

Correction to my previous comment's first follow-up: the generic titles I listed (*Greenery*, *Mushrooms*, *Foggy Sunlight*, *Cave Lore*, *Fate*, *Cavern Flora*) are all `concept` entries, and `PROPER_NOUN_LORE_TYPES` already excludes that type, so they were never in the 36 terms. The dilution claim is withdrawn. The echoed terms are legitimate proper nouns the list should carry: *Office of Prophetic Fulfillment* (organization), *The Miller* (npc), *Mimic's Slime* (artifact). So the remaining follow-up is only the second one: the echo is a property of `initial_prompt` on silent audio, and the server also offers `hotwords`. Measuring both on the dev replay is the next step; the prompt-echo rule in this issue covers the transcript either way.

rbrooks referenced this issue from a commit

2026-09-09 05:48:02 +00:00

fix(backend): drop Whisper's hallucinations before the transcript is merged (#569)

rbrooks referenced this issue from a commit

2026-09-09 05:48:02 +00:00

docs: the recording architecture lists the hallucination filter step (#569)

claude-bot referenced this issue from a pull request that will close it,

2026-09-09 05:48:16 +00:00

fix(backend): drop Whisper's hallucinations before the transcript is merged (#569) #570

claude-bot closed this issue

2026-09-09 06:10:34 +00:00

claude-bot referenced this issue from a commit

2026-09-09 06:10:34 +00:00

Merge PR #570: fix(backend): drop Whisper's hallucinations before the transcript is merged (#569)

claude-bot referenced this issue

2026-09-09 06:43:46 +00:00

Vocabulary cleaning strips a closing parenthesis and a trailing period from terms sent to Whisper #574

claude-bot commented

2026-09-09 07:27:57 +00:00

Author

Collaborator

Measured on dev, same audio, before and after #570. The transcript side is deterministic (the baseline replay reproduced prod's 1,778 segments and every junk count exactly), so this is a clean before/after:

	before	after
segments stored	1,778	1,618
caption credits	41	0
bare "Thank you."	78	0
lone dots	24	0
vocabulary echoes (Fulfillment / Mimic / The Miller)	10	9

hallucination_drops on the run: {"artefact": 154, "repetition": 5, "prompt_echo": 1}; the worker logged the same split, counts only.

One gap left, filed as #576. The echo rule matches a segment that equals a whole vocabulary term, but Whisper echoes a fragment: "Fulfillment" is the last word of Office of Prophetic Fulfillment, "Mimic" the first word of Mimic's Slime. "The Miller" is kept on purpose (an NPC name is plausible speech).

Side effect worth knowing: with 160 fewer lines the transcript fell from 5 extraction windows to 4, and this run had no failed windows and validated 23 of 53 beats in 3 passes, against 27 of 99 in 4 passes with two windows lost on the baseline. Same model, same router; too early to call that the filter's doing rather than luck, but it is the direction #572 predicts.

**Measured on dev, same audio, before and after #570.** The transcript side is deterministic (the baseline replay reproduced prod's 1,778 segments and every junk count exactly), so this is a clean before/after: | | before | after | |---|---|---| | segments stored | 1,778 | 1,618 | | caption credits | 41 | 0 | | bare "Thank you." | 78 | 0 | | lone dots | 24 | 0 | | vocabulary echoes (Fulfillment / Mimic / The Miller) | 10 | 9 | `hallucination_drops` on the run: `{"artefact": 154, "repetition": 5, "prompt_echo": 1}`; the worker logged the same split, counts only. **One gap left, filed as #576.** The echo rule matches a segment that equals a whole vocabulary term, but Whisper echoes a *fragment*: "Fulfillment" is the last word of *Office of Prophetic Fulfillment*, "Mimic" the first word of *Mimic's Slime*. "The Miller" is kept on purpose (an NPC name is plausible speech). Side effect worth knowing: with 160 fewer lines the transcript fell from 5 extraction windows to 4, and this run had no failed windows and validated 23 of 53 beats in 3 passes, against 27 of 99 in 4 passes with two windows lost on the baseline. Same model, same router; too early to call that the filter's doing rather than luck, but it is the direction #572 predicts.

claude-bot referenced this issue

2026-09-09 07:28:05 +00:00

Prompt-echo rule should catch fragments of a vocabulary term, not only the whole term #576

claude-bot referenced this issue

2026-09-09 08:25:02 +00:00

Send the campaign vocabulary as Whisper hotwords instead of an initial_prompt, behind a switch, and measure the echo rate #577

claude-bot referenced this issue

2026-09-09 10:49:21 +00:00

Snap misheard character names to GM-defined "heard as" aliases after transcription #579

claude-bot referenced this issue

2026-09-11 23:47:14 +00:00

chore(release): v4.4.0 #612