#352 - [Backend] Keep self-hosted ASR first-class — VAD-cut with forced alignment - rbrooks/Quest-Board

claude-bot commented

2026-08-25 20:39:15 +00:00

Collaborator

Found in the August 2026 session lifecycle review (#319). Self-hosted is a first-class target, not a fallback — it is the configuration the project owner will run for the foreseeable future.

Why

The bundled self-hosted path is plain Whisper, which carries exactly the failure modes this review exists to fix: long-form timestamp drift and hallucination on silence. Once the capture fix lands and tracks become mostly silence, that is the documented worst case for Whisper.

The good news is that the fix costs nothing but engineering: WhisperX-style VAD-cut-and-merge with forced phoneme alignment addresses both, and produces the word-level timestamps the provider contract normalises on. A self-hoster should get a genuinely accurate pipeline, not a degraded one.

Proposed fix

Move the bundled path from plain Whisper webservice to faster-whisper or WhisperX with VAD-cut-and-merge plus forced alignment. Emit word-level timestamps so the local adapter satisfies the same contract as a managed one, with no downstream branching.

Verify against the golden corpus (see the eval harness in v4.0.0) so "self-hosted is first-class" is a measured claim rather than an aspiration.

Acceptance criteria

Bundled ASR uses VAD-cut-and-merge with forced alignment
The local adapter returns word-level timestamps satisfying the shared contract
No downstream code branches on local vs managed
Accuracy measured against the golden corpus and recorded
Upgrade path documented for existing self-hosters, including what changes about their existing transcripts

Found in the August 2026 session lifecycle review (#319). **Self-hosted is a first-class target, not a fallback** — it is the configuration the project owner will run for the foreseeable future. ## Why The bundled self-hosted path is plain Whisper, which carries exactly the failure modes this review exists to fix: long-form timestamp drift and hallucination on silence. Once the capture fix lands and tracks become mostly silence, that is the documented worst case for Whisper. The good news is that the fix costs nothing but engineering: WhisperX-style VAD-cut-and-merge with forced phoneme alignment addresses both, and produces the word-level timestamps the provider contract normalises on. A self-hoster should get a genuinely accurate pipeline, not a degraded one. ## Proposed fix Move the bundled path from plain Whisper webservice to faster-whisper or WhisperX with VAD-cut-and-merge plus forced alignment. Emit word-level timestamps so the local adapter satisfies the same contract as a managed one, with no downstream branching. Verify against the golden corpus (see the eval harness in v4.0.0) so "self-hosted is first-class" is a measured claim rather than an aspiration. ## Acceptance criteria - [ ] Bundled ASR uses VAD-cut-and-merge with forced alignment - [ ] The local adapter returns word-level timestamps satisfying the shared contract - [ ] No downstream code branches on local vs managed - [ ] Accuracy measured against the golden corpus and recorded - [ ] Upgrade path documented for existing self-hosters, including what changes about their existing transcripts

claude-bot added this to the v4.2.0 - AI Provider Abstraction & BYO-AI milestone

2026-08-25 20:39:15 +00:00

claude-bot added the

enhancement

backend

recording

labels

2026-08-25 20:39:15 +00:00

claude-bot referenced this issue

2026-09-01 18:48:42 +00:00

[Backend] Define the ASR provider contract and normalise on word timestamps #350

rbrooks referenced this issue from a commit

2026-09-01 20:19:00 +00:00

feat(providers): define the ASR and LLM provider contracts (#350, #351)

rbrooks referenced this issue from a commit

2026-09-01 20:19:00 +00:00

refactor(asr): route transcription through the provider contract (#350)

claude-bot referenced this issue

2026-09-01 20:19:28 +00:00

ASR and LLM provider contracts, and a truncation check that was only wired to one provider (#350, #351) #483

claude-bot referenced this issue

2026-09-01 20:29:37 +00:00

[Backend] Define the ASR provider contract and normalise on word timestamps #350

claude-bot referenced this issue

2026-09-05 01:55:43 +00:00

feat(asr): feed campaign proper nouns to the transcriber as vocabulary bias (#355) #491

rbrooks referenced this issue from a commit

2026-09-05 02:07:08 +00:00

feat(whisperx): return word timings and honour a per-request prompt (API v2)

rbrooks referenced this issue from a commit

2026-09-05 02:07:08 +00:00

build(whisperx): pin the ASR stack to what is running on ai-host

rbrooks referenced this issue from a commit

2026-09-05 02:07:08 +00:00

test(whisperx): cover the v2 shaping, the lock and /health, and document the API

rbrooks referenced this issue from a commit

2026-09-05 02:07:08 +00:00

build(whisperx): pin uvicorn and python-multipart to the running versions

claude-bot commented

2026-09-05 02:28:45 +00:00

Author

Collaborator

Picking this up. Grounding first, because the premise turned out to be half wrong in a useful way.

What the bundled server actually is

It is not "plain Whisper webservice". whisper_config.endpoint_url points at http://10.3.0.28:8091, a ~130-line FastAPI wrapper around WhisperX 3.8.5 (large-v3-turbo, CUDA, faster-whisper 1.2.1) in Rhoving/iac-repo (whisperx-ansible/roles/whisperx/files/api/main.py). It already runs WhisperX's VAD-cut transcription and whisperx.align forced alignment on every request — and then threw the word timings away at the API boundary, returning only {start, end, text, speaker}. It also had no initial_prompt field, so the vocabulary bias from #355 was silently dropped.

So the first two criteria were true server-side and false at the interface, and the fix was an API change, not an engine change.

Server half — done and deployed (Rhoving/iac-repo#396, PR #397)

API v2, backward compatible: segments[].words[] {word, start, end, score} (null where WhisperX could not align a word; words may be empty for a segment), optional initial_prompt/hotwords per request (applied by swapping pipeline.options under a lock, WhisperX's own idiom), one lock around transcribe+align with concurrency_limit: 1 advertised, and /health now carries api_version: 2 and capabilities: {word_timestamps, initial_prompt, hotwords, vad, alignment, concurrency_limit} computed from the live pipeline. Requirements pinned to what was running. Deployed to ai-host 2026-09-05 02:25 UTC (20 ok / 5 changed / 0 failed) after confirming prod had nothing in flight.

Verified live from the dev box: /health → api_version: 2, all six capabilities true/1; a silent-WAV POST /transcribe with initial_prompt → 200, {"segments": [], "language": "en", "api_version": 2} in 7.4 s including model load.

Client half — next, after #357 lands

LocalWhisperProvider will read /health.capabilities (cached) and declare word_timestamps=True against a v2 server, consume words as real spans (interpolated=False) with interpolation only for null timings, keep the current interpolation for a v1 server, and declare concurrency_limit from the server. Words get persisted on transcript segments (one nullable JSONB column — sequenced after #357's migration to avoid a second Alembic head) and used by anchor_highlights to tighten anchors when present, so the capability is not moot. Upgrade path for self-hosters goes in OPERATIONS. Accuracy against the golden corpus is #433's live leg.

Picking this up. Grounding first, because the premise turned out to be half wrong in a useful way. ## What the bundled server actually is It is not "plain Whisper webservice". `whisper_config.endpoint_url` points at `http://10.3.0.28:8091`, a ~130-line FastAPI wrapper around **WhisperX 3.8.5** (large-v3-turbo, CUDA, faster-whisper 1.2.1) in `Rhoving/iac-repo` (`whisperx-ansible/roles/whisperx/files/api/main.py`). It already runs WhisperX's VAD-cut transcription **and** `whisperx.align` forced alignment on every request — and then threw the word timings away at the API boundary, returning only `{start, end, text, speaker}`. It also had no `initial_prompt` field, so the vocabulary bias from #355 was silently dropped. So the first two criteria were true server-side and false at the interface, and the fix was an API change, not an engine change. ## Server half — done and deployed (Rhoving/iac-repo#396, PR #397) API v2, backward compatible: `segments[].words[] {word, start, end, score}` (`null` where WhisperX could not align a word; `words` may be empty for a segment), optional `initial_prompt`/`hotwords` per request (applied by swapping `pipeline.options` under a lock, WhisperX's own idiom), one lock around transcribe+align with `concurrency_limit: 1` advertised, and `/health` now carries `api_version: 2` and `capabilities: {word_timestamps, initial_prompt, hotwords, vad, alignment, concurrency_limit}` computed from the live pipeline. Requirements pinned to what was running. Deployed to ai-host 2026-09-05 02:25 UTC (20 ok / 5 changed / 0 failed) after confirming prod had nothing in flight. Verified live from the dev box: `/health` → `api_version: 2`, all six capabilities true/1; a silent-WAV `POST /transcribe` with `initial_prompt` → 200, `{"segments": [], "language": "en", "api_version": 2}` in 7.4 s including model load. ## Client half — next, after #357 lands `LocalWhisperProvider` will read `/health.capabilities` (cached) and declare `word_timestamps=True` against a v2 server, consume `words` as real spans (`interpolated=False`) with interpolation only for `null` timings, keep the current interpolation for a v1 server, and declare `concurrency_limit` from the server. Words get persisted on transcript segments (one nullable JSONB column — sequenced after #357's migration to avoid a second Alembic head) and used by `anchor_highlights` to tighten anchors when present, so the capability is not moot. Upgrade path for self-hosters goes in OPERATIONS. Accuracy against the golden corpus is #433's live leg.

claude-bot referenced this issue

2026-09-05 03:48:16 +00:00

[Backend] Define the ASR provider contract and normalise on word timestamps #350

claude-bot referenced this issue

2026-09-05 03:48:21 +00:00

[Backlog] AI provider abstraction and per-campaign BYO-AI #128

rbrooks referenced this issue from a commit

2026-09-05 04:31:28 +00:00

feat(providers): real word spans from a v2 server

rbrooks referenced this issue from a commit

2026-09-05 04:31:28 +00:00

feat(audio): carry words through the pipeline

rbrooks referenced this issue from a commit

2026-09-05 04:31:28 +00:00

feat(audio): anchor highlights to words

rbrooks referenced this issue from a commit

2026-09-05 04:31:28 +00:00

test(conformance): both bundled server versions, and partial alignment

claude-bot referenced this issue from a pull request that will close it,

2026-09-05 04:32:34 +00:00

feat: real word timestamps from the bundled server, carried through the pipeline (#352) #499

claude-bot referenced this issue

2026-09-05 04:37:17 +00:00

[Backend] Evaluate ASR and LLM providers against the golden corpus before committing #360

claude-bot closed this issue

2026-09-05 04:40:51 +00:00

claude-bot referenced this issue from a commit

2026-09-05 04:40:52 +00:00

Merge PR #499: real word timestamps from the bundled server, carried through the pipeline (#352)

claude-bot commented

2026-09-05 04:41:35 +00:00

Author

Collaborator

Client half landed in PR #499 (merged 2026-09-05, CI green; 2087 backend tests). With the server half (Rhoving/iac-repo#396, deployed 02:25 UTC) this closes the issue.

Criterion 1 (VAD-cut with forced alignment): the bundled server always did both; API v2 stops discarding the result at the interface.
Criterion 2 (word-level timestamps satisfying the contract): LocalWhisperProvider discovers the server's capabilities from /health (cached per endpoint, 10 min) and declares word_timestamps=True against v2; real spans are parsed with interpolated=False, null-timed words placed proportionally between aligned neighbours and flagged, empty words falling back to whole-segment interpolation. Against a v1 or unreachable server it declares no word timestamps and no vocabulary support, so #355 skips the query and the #358 panel says "update the bundled server" rather than pretending.
Criterion 3 (no downstream branching): words travel on the segment dicts, offset on the VAD path, persisted as nullable JSONB transcript_segments.words (migration c7d8e9fa0b1c), and a test proves the local v2 adapter and the stub adapter produce identical rows from identical timings. anchor_highlights now anchors to the first measured word of a matched phrase.
Criterion 4 (accuracy measured): #433's harness against this server, three identical runs — session WER 3.73 %, 0 foreign markers, onset drift median 0.07 s; marker detection 72 % without vocabulary bias, which is the next measurement queued (#355's criterion).
Criterion 5 (upgrade path): OPERATIONS "Upgrading the bundled transcription server" — how to read the API version, that word timings and vocabulary bias apply to new sessions only, and that nothing is reprocessed.

Live: a real submit with a vocabulary returned {"word": "Bramblewick", "start": 0.031, "end": 0.253} — the v2 parser on a real body and the server honouring initial_prompt in one request.

Client half landed in **PR #499** (merged 2026-09-05, CI green; 2087 backend tests). With the server half (Rhoving/iac-repo#396, deployed 02:25 UTC) this closes the issue. - **Criterion 1 (VAD-cut with forced alignment):** the bundled server always did both; API v2 stops discarding the result at the interface. - **Criterion 2 (word-level timestamps satisfying the contract):** `LocalWhisperProvider` discovers the server's capabilities from `/health` (cached per endpoint, 10 min) and declares `word_timestamps=True` against v2; real spans are parsed with `interpolated=False`, `null`-timed words placed proportionally between aligned neighbours and flagged, empty `words` falling back to whole-segment interpolation. Against a v1 or unreachable server it declares no word timestamps **and no vocabulary support**, so #355 skips the query and the #358 panel says "update the bundled server" rather than pretending. - **Criterion 3 (no downstream branching):** words travel on the segment dicts, offset on the VAD path, persisted as nullable JSONB `transcript_segments.words` (migration `c7d8e9fa0b1c`), and a test proves the local v2 adapter and the stub adapter produce identical rows from identical timings. `anchor_highlights` now anchors to the first measured word of a matched phrase. - **Criterion 4 (accuracy measured):** #433's harness against this server, three identical runs — session WER 3.73 %, 0 foreign markers, onset drift median 0.07 s; marker detection 72 % *without* vocabulary bias, which is the next measurement queued (#355's criterion). - **Criterion 5 (upgrade path):** OPERATIONS "Upgrading the bundled transcription server" — how to read the API version, that word timings and vocabulary bias apply to new sessions only, and that nothing is reprocessed. Live: a real `submit` with a vocabulary returned `{"word": "Bramblewick", "start": 0.031, "end": 0.253}` — the v2 parser on a real body and the server honouring `initial_prompt` in one request.

rbrooks referenced this issue from a commit

2026-09-05 04:54:07 +00:00

style: ruff format the adapter after the rebase over #352 (#353)

rbrooks referenced this issue from a commit

2026-09-05 05:15:16 +00:00

db: chain campaign_ai_settings behind the transcript-words migration (#128)

rbrooks referenced this issue from a commit

2026-09-05 05:15:16 +00:00

test: await the gap derivation, which became a coroutine under us (#128, #352)

claude-bot referenced this issue

2026-09-05 05:15:42 +00:00

feat: per-campaign AI providers — a campaign brings its own ASR and LLM, behind an instance switch (#128) #501