fix(backend): drop Whisper's hallucinations before the transcript is merged (#569) #570
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/569-hallucination-filter"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #569.
What. A pure, deterministic filter (
audio_service.drop_hallucinations) over the flat segment list, applied inprocess_audioafter the coverage guard and before the merge, so the merged transcript, the stored segment rows, beat evidence and highlights all see one filtered set. Five rules, each counted separately: known Whisper artefacts (caption credits, "Thank you." runs, "thanks for watching", the Japanese sign-off, lone punctuation), repetition loops, prompt echo (a segment that is nothing but one of the vocabulary terms this run sent to the ASR, unless that term is a person's name), script mismatch when the campaign has a pinned Latin-script language, and a confidence rule that stays dormant until an adapter passes per-segment scores through. Text is never logged, only counts.Where the counts go.
summarisation_runs.hallucination_drops(JSONB{reason: count}, migration3e4f5a6b7c8d, raw SQL, nullable and not backfilled:NULLmeans not measured,{}means measured and clean), surfaced on the summarisation-run API ashallucinations_droppedandhallucination_drops. The eval harness's ASR leg goes through the same function.Two calls a reviewer should see. The filter runs after
check_transcript_covers_session, not before as the issue first said: a hallucinated line at 03:12 is still evidence the transcriber processed 03:12, and judging coverage on the filtered list would fail a session whose last stretch is open mics with no speech, destroying a paid-for transcript (#431). Pinned by a test. And person names are resolved from the database (characters, NPC cards, PC/NPC/deity wiki titles and aliases) unioned with the session's identities, so an absent player's character still protects its name; when they cannot be resolved the echo rule is disabled rather than run with an empty allow-list.Motivation and numbers are on #569: on the 2026-09-09 session, 143 of 1,778 stored lines were not speech even with VAD on, and three "artefacts" turned out to be the vocabulary prompt echoing back.
Tests: 77 in
test_hallucination_filter.py; full backend suite 2,964 passed; migration upgraded, downgraded and re-upgraded on an empty database.🤖 Generated with Claude Code
Whisper writes text for near-silent audio and nothing in the pipeline removed it. One real six-speaker session stored 1,778 segments of which 143 were not speech: 41 caption credits ("Sous-titrage ST' 501", "Closed Captioning by …", "Red Bee Media"), 78 bare "Thank you.", 24 lone dots, decoder loops, and — via the vocabulary bias built in #355 — the campaign's own wiki titles handed straight back out of the initial_prompt. They cost tokens in a context that already has to be windowed, the beat extractor cites them as evidence, and a summariser reading "Thank you." seventy-eight times writes about how polite everybody was. `audio_service.drop_hallucinations` is a pure `(kept, dropped)` filter with five rules, every one of them whole-segment so the cost of a false positive is one contentless line: the artefact list, a one-token loop repeated six or more times, a prompt echo, a script mismatch against a *pinned* Latin-script language, and the provider's own confidence as an optional fourth signal that nothing populates today. The prompt-echo rule is the one that could take real speech, so it has an allow-list: a segment that is exactly a vocabulary term (or that term repeated — the transcript had "The Miller, The Miller") is dropped unless the term names a person, because "Wilhelmina?" is a whole utterance and "Office of Prophetic Fulfillment" is not. `vocabulary_service.campaign_person_names` sources that from characters, NPC cards and the wiki's person entries, and `resolve_person_names` returns None rather than an empty set when it cannot, so a failed query disables the rule instead of running it without its exceptions. Deliberately *after* `check_transcript_covers_session`, which the issue put it before. That guard asks whether transcription came back covering the audio it was sent, and a hallucinated segment at 03:12:00 is real evidence the transcriber processed the span at 03:12:00 — measuring coverage on the filtered list would raise on a session whose last forty minutes are open mics and no speech, destroying a correct transcript that was already paid for, which is #431's mistake exactly. A tail the filter empties is logged instead. Counts are recorded per rule on `summarisation_runs.hallucination_drops` (migration `3e4f5a6b7c8d`, raw DDL, nullable and not backfilled so an older run reads as "not measured" rather than "measured zero"), totalled by a derived property, and exposed on the summarisation-run API. Never the text: the log line carries counts by rule and by speaker only. The #349 eval harness runs the same function in the same position, and records what it removed alongside the numbers it scored, so a filter that starts eating invented dialogue shows up as a coverage loss on a fixture whose manifest says what was said. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>