[Backend] Define the ASR provider contract and normalise on word timestamps #350
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found in the August 2026 session lifecycle review (#319). Core of the provider abstraction; the shape everything else in this milestone depends on.
Why
The backend currently speaks one bespoke dialect to one out-of-repo WhisperX server, whose behaviour it cannot verify — file/speaker zip order, internal VAD, per-file error handling and whether it ever re-labels a speaker are all unknown from inside this repo. That is an unacceptable dependency for a product that must run on self-hosted CPU, self-hosted GPU, or any of several managed APIs without a rewrite.
Proposed contract
The adapter surface, kept deliberately narrow:
track_idpassed through and returned. Association is by id, never by ordering.words: [{text, start_ms, end_ms, confidence}], plus segments derived from words rather than trusted separately.vocabulary: list[str]input, mapped per provider to keyterms, keyword boosting, or an initial prompt.Word-level timestamps are the normalisation target because they are the only representation that supports every downstream need — segment rendering, beat evidence citations, and any future interjection-level interleaving — without a second round trip.
Note that speaker identity never comes from the ASR. Quest Board knows it from Discord track ownership, which is a genuine structural advantage: overlapping speech is already separated onto clean per-speaker tracks, so no provider diarization is needed or wanted.
Acceptance criteria
track_idpassthroughvocabularymaps correctly for at least the local adapter and one managed adapterPicking this up alongside #351 on
feat/350-351-provider-contracts— the two contracts land together, since chunk sizing on the LLM side reads a declared window and neither shape is safe to fix in isolation.Grounding: what the ASR path actually is today
One bespoke dialect, in
transcribe_track(audio_service.py:502):POST {url}/transcribe, multipart formaudio+speaker+ optionallanguage. No submit/poll, no webhook.{"segments": [{start, end, text, speaker}], "language"}— segment-level only. There are no word timestamps anywhere in the pipeline.vocabularyinput.Two things this issue asks for are already true, and should not be rebuilt:
transcribe_tracksends one file per request and stampsspeaker/track_owner_idfrom theTrackobject it was handed. The server's speaker echo is read and deliberately discarded — #342 already fixed that, and the comment at:556says so explicitly. The adapter contract needs to preserve this property, not introduce it.compute_speech_spans→transcribe_track_vad→_transcribe_span_resilientlydoes span cutting and offset remapping above the transport, which is exactly where this issue says it belongs.The word-timestamp entanglement
The acceptance criterion "results normalise to word-level timestamps regardless of provider" cannot be satisfied while the only implementation is the bundled server, which is segment-only and out of repo. That is #352's work.
Rather than block this issue behind a server change, I am building to the declare-and-degrade design this issue already describes: a provider declares
word_timestamps: bool, and the local adapter declaresFalseand synthesises word spans by interpolating across the segment, flagged as degraded. Downstream code reads words either way and does not branch on provider; #358 surfaces the gap to the user. When #352 lands, the local adapter flips the flag and the interpolation stops being reached.This keeps the criterion honest — normalisation is real, the precision is declared — instead of asserting a word-level guarantee the pipeline cannot currently make.
Async shape
Contract is submit/poll per this issue's spec. The bundled server is synchronous, so its adapter resolves immediately on submit and
pollreturns the completed result. That keeps a managed async provider (AssemblyAI-shaped) from needing a second code path, without making the local path pay for polling it does not need.Landed in PR #483 (merged, CI green on all 7 jobs). Leaving this open — five of six criteria are met and the sixth genuinely needs a second adapter.
Criteria
track_idpassthrough —AsrProviderinapp/providers/asr.py. Synchronous providers resolve on submit andpollreturns what it holds, so an async managed provider needs no second code path.AsrCapabilities, withneeds_chunkingderived from the declared limits.vocabularymaps correctly for at least the local adapter and one managed adapter — local only.LocalWhisperProvidermaps it to Whisper'sinitial_prompt, truncated to a declared 100-term limit. A managed adapter needs #360 to pick one first.transcribe_trackcarries the parameter through so #355 is a call-site change, not a plumbing one.TranscriptionResulthas no speaker field. There is a test asserting its absence, so adding one to make an adapter's life easier fails and sends the author to #342.The word-timestamp caveat, restated now it is real
LocalWhisperProviderdeclaresword_timestamps=Falseand interpolates spans across each segment by character length — proportional, so "a" and "extraordinarily" do not get equal time — marking every wordinterpolated=Trueand the resultword_timestamps_interpolated.Nothing downstream consumes words yet. On this provider they would add no precision over the segments they came from, and pushing a new shape through
merge_attributed_transcriptand thetranscript_segmentsrows is a change worth making when #352 makes the words real. So the normalisation is in place and unexercised, which is the honest state.Wiring
transcribe_tracknow resolves an adapter and submits through it. Wire format is byte-for-byte unchanged and the returned dict shape is unchanged, sotranscribe_session,transcribe_track_vad,_transcribe_span_resilientlyand the merge are untouched; the VAD path comes along because it delegates here.Mutation-checked rather than assumed: dropping the segment conversion fails 7 of the 54 transcription tests, and pointing the adapter at a wrong path fails 1. Worth recording that only one of those 54 asserts the request URL — the contract tests cover the adapter directly, but that is a thin spot in the pipeline suite if someone changes the endpoint later.
The last open criterion —
vocabularymaps correctly for a managed adapter — is met by PR #496 (merged 2026-09-05):OpenAiCompatibleAsrProvider(app/providers/asr_openai.py, registered asopenai, serving OpenAI, Groq and any/v1/audio/transcriptionsserver) maps vocabulary to thepromptfield, 40 terms, asserted on the wire by the conformance suite. It also returns real word spans, declares the 25 MiB cap (enforced client-side, with chunk-and-remap staying in the caller), and is selectable explicitly in Admin → Bot Settings.Recap of the whole issue now that it closes:
AsrProvidercontract with submit/poll/normalise andtrack_idpassthrough (PR #483); declared limits actually enforced and typed errors (PR #493); a second real adapter so the contract is no longer a description of one implementation (PR #496); and the bundled WhisperX server's API v2 (Rhoving/iac-repo#396) makes the local adapter's word timestamps real rather than interpolated — the client side of that is in flight on #352.