[Backend] Keep self-hosted ASR first-class — VAD-cut with forced alignment #352
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found in the August 2026 session lifecycle review (#319). Self-hosted is a first-class target, not a fallback — it is the configuration the project owner will run for the foreseeable future.
Why
The bundled self-hosted path is plain Whisper, which carries exactly the failure modes this review exists to fix: long-form timestamp drift and hallucination on silence. Once the capture fix lands and tracks become mostly silence, that is the documented worst case for Whisper.
The good news is that the fix costs nothing but engineering: WhisperX-style VAD-cut-and-merge with forced phoneme alignment addresses both, and produces the word-level timestamps the provider contract normalises on. A self-hoster should get a genuinely accurate pipeline, not a degraded one.
Proposed fix
Move the bundled path from plain Whisper webservice to faster-whisper or WhisperX with VAD-cut-and-merge plus forced alignment. Emit word-level timestamps so the local adapter satisfies the same contract as a managed one, with no downstream branching.
Verify against the golden corpus (see the eval harness in v4.0.0) so "self-hosted is first-class" is a measured claim rather than an aspiration.
Acceptance criteria
Picking this up. Grounding first, because the premise turned out to be half wrong in a useful way.
What the bundled server actually is
It is not "plain Whisper webservice".
whisper_config.endpoint_urlpoints athttp://10.3.0.28:8091, a ~130-line FastAPI wrapper around WhisperX 3.8.5 (large-v3-turbo, CUDA, faster-whisper 1.2.1) inRhoving/iac-repo(whisperx-ansible/roles/whisperx/files/api/main.py). It already runs WhisperX's VAD-cut transcription andwhisperx.alignforced alignment on every request — and then threw the word timings away at the API boundary, returning only{start, end, text, speaker}. It also had noinitial_promptfield, so the vocabulary bias from #355 was silently dropped.So the first two criteria were true server-side and false at the interface, and the fix was an API change, not an engine change.
Server half — done and deployed (Rhoving/iac-repo#396, PR #397)
API v2, backward compatible:
segments[].words[] {word, start, end, score}(nullwhere WhisperX could not align a word;wordsmay be empty for a segment), optionalinitial_prompt/hotwordsper request (applied by swappingpipeline.optionsunder a lock, WhisperX's own idiom), one lock around transcribe+align withconcurrency_limit: 1advertised, and/healthnow carriesapi_version: 2andcapabilities: {word_timestamps, initial_prompt, hotwords, vad, alignment, concurrency_limit}computed from the live pipeline. Requirements pinned to what was running. Deployed to ai-host 2026-09-05 02:25 UTC (20 ok / 5 changed / 0 failed) after confirming prod had nothing in flight.Verified live from the dev box:
/health→api_version: 2, all six capabilities true/1; a silent-WAVPOST /transcribewithinitial_prompt→ 200,{"segments": [], "language": "en", "api_version": 2}in 7.4 s including model load.Client half — next, after #357 lands
LocalWhisperProviderwill read/health.capabilities(cached) and declareword_timestamps=Trueagainst a v2 server, consumewordsas real spans (interpolated=False) with interpolation only fornulltimings, keep the current interpolation for a v1 server, and declareconcurrency_limitfrom the server. Words get persisted on transcript segments (one nullable JSONB column — sequenced after #357's migration to avoid a second Alembic head) and used byanchor_highlightsto tighten anchors when present, so the capability is not moot. Upgrade path for self-hosters goes in OPERATIONS. Accuracy against the golden corpus is #433's live leg.Client half landed in PR #499 (merged 2026-09-05, CI green; 2087 backend tests). With the server half (Rhoving/iac-repo#396, deployed 02:25 UTC) this closes the issue.
LocalWhisperProviderdiscovers the server's capabilities from/health(cached per endpoint, 10 min) and declaresword_timestamps=Trueagainst v2; real spans are parsed withinterpolated=False,null-timed words placed proportionally between aligned neighbours and flagged, emptywordsfalling back to whole-segment interpolation. Against a v1 or unreachable server it declares no word timestamps and no vocabulary support, so #355 skips the query and the #358 panel says "update the bundled server" rather than pretending.transcript_segments.words(migrationc7d8e9fa0b1c), and a test proves the local v2 adapter and the stub adapter produce identical rows from identical timings.anchor_highlightsnow anchors to the first measured word of a matched phrase.Live: a real
submitwith a vocabulary returned{"word": "Bramblewick", "start": 0.031, "end": 0.253}— the v2 parser on a real body and the server honouringinitial_promptin one request.