fix: a misdetected language on a 3-second clip must not cost a session (#347) #420
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/vad-span-resilience"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Hotfix for the 2026-08-26 production incident. Session
3563d67e— 77 minutes, six speakers — ended asfailedwith no transcript.What happened
Three speech spans out of 77, on one track of six, returned 500 from WhisperX. From the service log:
Three seconds of English, confidently identified as Icelandic, and WhisperX ships no wav2vec2 aligner for Icelandic. Spans with enough audio to be recognisable reported
en (1.00)in the same log.Diagnosis, before the log was available: each failing span was replayed in isolation as the first request of a connection, three times each — 500 every time, while controls returned 200 and
/healthstayed green throughout. Content-dependent and fully reproducible. Two hypotheses were tested against the real audio and falsified: clipping (spans that clip harder transcribe fine) and "no speech found" (synthetic silence, noise, tones and clicks all return 200). Pinning the language made all three failing spans return 200, which the log then confirmed.Two fixes
1. The language is established once per session, not guessed per span (#347).
VAD created the conditions: it cuts a track into 1-3 second clips and transcribes each alone, discarding exactly the context detection depends on. The language is now detected once from the longest span available and pinned for the rest.
Best evidence wins regardless of track order — a track inherits a session language only when that answer rests on a longer probe than its own, otherwise the quiet player whose longest utterance is three seconds pins their bad guess for everyone. The probe is a real transcription and its result is reused, so detection costs no extra call.
transcribe_session_vad/transcribe_with_optional_vadtake an optionallanguagethat skips detection entirely. Nothing passes it yet — that is #419.2. One bad span no longer destroys a session.
Independently of the cause, the pipeline's response to a clip it could not transcribe was to lose everything: the exception propagated out of the per-span loop, discarding 42 already-transcribed spans and aborting before five of six speakers were touched at all. Spans are now retried, then skipped individually, with the lost timestamps logged.
Still fails loudly above 25% of a track's spans — handing back three-quarters of a session as though it were whole is the same silent loss in a different costume. A share rather than a count, because 3 of 77 is a bad clip and 3 of 4 is an outage.
Why this is worse than the crash suggests
The 500 was the lucky outcome. WhisperX has aligners for German, Spanish, Dutch, French and more; had the misdetection landed on one of those, the clip would have returned confident, silently wrong text straight into the transcript, summary and wiki. #419 tracks the per-campaign language override that closes that off.
Testing
930 backend tests pass;
ruff checkandruff format --checkclean. 16 new tests acrosstest_vad_language_pinning.pyandtest_vad_span_resilience.py.Two bugs the tests caught that review did not: the warning reporting data loss had eight format placeholders and seven arguments — it would have raised at runtime, in the code whose only job is making loss visible — and the first draft of the resilience tests keyed its fakes off call count, which a retry shifts, so they asserted against the wrong spans while passing.
Deployment
Session
3563d67eis fully recoverable: the audio directory is intact by design, and with the language pinned all three failing spans transcribe, so nothing is lost. Reprocess after deploy.Closes #347.
Production, 2026-08-26. A 77-minute six-speaker session ended as `failed` with no transcript because three clips out of 77, on one track of six, returned 500 from Whisper. Whisper was not down. It answered /health throughout, transcribed the other 74 spans on that track, and returned 500 for those three every time they were replayed — in isolation, as the first request of a connection, three attempts each. Something about that audio; the service is otherwise healthy. The pipeline's response was to lose everything: transcribe_track_vad called Whisper once per span and let the exception propagate, so 42 already-transcribed spans were discarded and the remaining five speakers were never read at all. For an unattended pipeline that a self-hoster is meant to leave running, a handful of bad clips cannot be a session-ending event. Each span is now retried, then skipped on its own, and the run continues. What was lost is counted and logged with the timestamps, because a transcript quietly missing a passage is precisely the failure this pipeline exists to prevent. Silence is not success, though: past a quarter of a track's spans this stops looking like bad clips and starts looking like an outage, and handing back three-quarters of a session as if it were whole would be the same silent loss in a new costume. Above that share it raises and the GM reprocesses. The threshold is a share rather than a count on purpose — 3 of 77 is a bad clip, 3 of 4 is not. Two bugs the tests caught that reading did not: * The warning that reports the loss had eight format placeholders and seven arguments — it would have raised at runtime, in the code whose entire job is to make data loss visible. * My first draft of the tests keyed their fakes off call count, which a retry shifts, so they were quietly asserting against the wrong spans. They key off the span index now. Does not address why those clips fail: two hypotheses were tested against the real audio and both falsified. Full-scale clipping is not it (spans that clip harder transcribe fine), and "the model found no speech" is not it (synthetic silence, noise, tones and clicks all return 200). That needs the Whisper service's own log, which is not reachable from the app host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Root cause of tonight's lost session, from the WhisperX log: whisperx.asr - WARNING - Audio is shorter than 30s, language detection may be inaccurate whisperx.asr - INFO - Detected language: is (0.95) in first 30s of audio whisperx.alignment - ERROR - No default alignment model for language: is ValueError: No default align-model for language: is Three seconds of English, confidently identified as Icelandic, and WhisperX ships no wav2vec2 aligner for Icelandic. Spans with enough audio to be recognisable reported `en (1.00)` in the same log, and all three failing spans return 200 the moment a language is pinned. VAD created the conditions. It cuts a track into 1-3 second clips and transcribes each alone, which throws away exactly the context detection depends on — a 77-minute recording is unambiguous, "yeah, fair" is not. So the language is now established once per session, from the longest span available, and pinned for every other span. The best evidence wins regardless of track order: a track uses a language established elsewhere in the session only when that answer rests on a longer probe than the track can offer itself. Otherwise the quiet player whose longest utterance is three seconds — the one whose track failed tonight — would pin their own bad guess for everybody. The probe is a real transcription and its result is reused, so detection costs no extra call. transcribe_session_vad and transcribe_with_optional_vad take an optional `language`, which when supplied is treated as certain and skips detection entirely. Nothing passes it yet; a per-campaign setting is the obvious next step, and worth doing, because this bug is worse than the crash suggests. The 500 was the *lucky* outcome — it was loud, and it stopped the pipeline. WhisperX has aligners for German, Spanish, Dutch, French and more. Had the misdetection landed on any of those instead of Icelandic, the same clip would have returned a confident, silently wrong transcription and gone straight into the transcript, the summary and the wiki with nothing to mark it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>