feat: real word timestamps from the bundled server, carried through the pipeline (#352) #499

Merged

claude-bot merged 7 commits from feat/352-word-timestamps-client into main 2026-09-05 04:40:50 +00:00

Conversation 0 Commits 7 Files changed 21 +2736 -141

claude-bot commented 2026-09-05 04:32:34 +00:00

Contributor

Copy link

Closes #352 (client half; the server half is Rhoving/iac-repo#396, deployed and verified live).

The bundled WhisperX server already did VAD-cut transcription and forced alignment and threw the word timings away; API v2 returns them. This makes the local adapter say what it can do, consume the words as real spans, carry them through the pipeline, and use them.

What

• Capability discovery — probe_local_whisper_capabilities does a 5 s GET /health, cached process-wide per endpoint for 10 minutes (a fresh adapter is built per call, so an instance cache would probe once per VAD span). A v2 server declares word_timestamps=True, vocabulary via initial_prompt, and the server's concurrency_limit; a v1 server (or an unreachable one) declares no word timestamps and VocabularyStyle.NONE — a v1 server ignores initial_prompt, so the honest declaration makes #355 skip the query and the #358 panel show asr_no_vocabulary with "update the bundled server". capabilities is a property returning the last-known value; the probe is awaited in submit and at the read sites that decide something. Deliberately not probed at the #356 slot decision — a queued session must not touch the provider.
• Real word spans — parse_aligned_segments: aligned words interpolated=False with the alignment score as confidence; null-timed words placed proportionally between their aligned neighbours and flagged; "words": [] falls back to whole-segment interpolation; the parser is chosen by the declaration, never by sniffing the body. A 500 invalidates the cached probe (a replaced server); a 429 does not.
• Carried through — segment dicts carry words; _offset_segments offsets them on the VAD path; migration c7d8e9fa0b1c (after b6c7d8e9fa0b, single head, up/down/up verified) adds nullable JSONB transcript_segments.words, written with the rows. A test drives identical timings through the local v2 adapter and the stub adapter and asserts identical persisted rows — one code path for both.
• Used — anchor_highlights anchors to the first non-interpolated word of the matched phrase, else the segment start as before; and it now matches across consecutive same-track segments, because compact_transcript joins those lines with a space so a quote the model actually read can straddle a boundary while having exactly one person to credit. Cross-speaker stitches are still dropped.
• Conformance — the Whisper wire fake serves v2 (/health capabilities, words with a null-timed word and an empty-words segment) with a v1 variant as its own case. One case was restated: a word-timestamping provider must have measured at least one word and its degraded flag must match any(word.interpolated) in both directions — no forced-alignment server can promise every word.
• Docs: OPERATIONS "Upgrading the bundled transcription server" (how to tell the API version, what changes for new sessions only, the panel gap until you update).

Live check against 10.3.0.28:8091: probe reads v2; a real submit with 4 s of non-speech and a vocabulary returned {"word":"Bramblewick","start":0.031,"end":0.253,"score":0.007} — the parser handling a real body and the server honouring initial_prompt, in one request.

Rebased over #350 (kept both sides: retry_after_seconds, the adapter's model=, OpenAiAsrCase beside LocalWhisperV1Case).

Verification

• Backend: 2087 passed, 13 skipped. ruff clean. alembic heads → one head. Version sync OK; no bot contract change.

🤖 Generated with Claude Code

Closes #352 (client half; the server half is Rhoving/iac-repo#396, deployed and verified live). The bundled WhisperX server already did VAD-cut transcription and forced alignment and threw the word timings away; API v2 returns them. This makes the local adapter *say* what it can do, consume the words as real spans, carry them through the pipeline, and use them. ## What - **Capability discovery** — `probe_local_whisper_capabilities` does a 5 s `GET /health`, cached process-wide per endpoint for 10 minutes (a fresh adapter is built per call, so an instance cache would probe once per VAD span). A v2 server declares `word_timestamps=True`, vocabulary via `initial_prompt`, and the server's `concurrency_limit`; a v1 server (or an unreachable one) declares no word timestamps **and `VocabularyStyle.NONE`** — a v1 server ignores `initial_prompt`, so the honest declaration makes #355 skip the query and the #358 panel show `asr_no_vocabulary` with "update the bundled server". `capabilities` is a property returning the last-known value; the probe is awaited in `submit` and at the read sites that decide something. Deliberately **not** probed at the #356 slot decision — a queued session must not touch the provider. - **Real word spans** — `parse_aligned_segments`: aligned words `interpolated=False` with the alignment score as confidence; `null`-timed words placed proportionally between their aligned neighbours and flagged; `"words": []` falls back to whole-segment interpolation; the parser is chosen by the *declaration*, never by sniffing the body. A 500 invalidates the cached probe (a replaced server); a 429 does not. - **Carried through** — segment dicts carry `words`; `_offset_segments` offsets them on the VAD path; migration **`c7d8e9fa0b1c`** (after `b6c7d8e9fa0b`, single head, up/down/up verified) adds nullable JSONB `transcript_segments.words`, written with the rows. A test drives identical timings through the local v2 adapter and the stub adapter and asserts identical persisted rows — one code path for both. - **Used** — `anchor_highlights` anchors to the first non-interpolated word of the matched phrase, else the segment start as before; and it now matches across consecutive same-track segments, because `compact_transcript` joins those lines with a space so a quote the model actually read can straddle a boundary while having exactly one person to credit. Cross-speaker stitches are still dropped. - **Conformance** — the Whisper wire fake serves v2 (`/health` capabilities, `words` with a null-timed word and an empty-words segment) with a v1 variant as its own case. One case was restated: a word-timestamping provider must have measured *at least one* word and its degraded flag must match `any(word.interpolated)` in both directions — no forced-alignment server can promise every word. - Docs: OPERATIONS "Upgrading the bundled transcription server" (how to tell the API version, what changes for new sessions only, the panel gap until you update). **Live check** against `10.3.0.28:8091`: probe reads v2; a real `submit` with 4 s of non-speech and a vocabulary returned `{"word":"Bramblewick","start":0.031,"end":0.253,"score":0.007}` — the parser handling a real body and the server honouring `initial_prompt`, in one request. Rebased over #350 (kept both sides: `retry_after_seconds`, the adapter's `model=`, `OpenAiAsrCase` beside `LocalWhisperV1Case`). ## Verification - Backend: **2087 passed, 13 skipped**. ruff clean. `alembic heads` → one head. Version sync OK; no bot contract change. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 7 commits 2026-09-05 04:32:34 +00:00

feat(providers): discover the bundled server's capabilities ... 0addb6e06e

Two servers now answer to `local_whisper`. The pre-#352 build returns
segments only and silently ignores `initial_prompt`; the VAD-cut
forced-alignment build reports `api_version: 2` from `/health` and
returns per-word spans. A self-hoster upgrades their images when they
choose, so the adapter has to ask rather than assert.

`LocalWhisperProvider.capabilities` becomes a read of the last probed
answer for that endpoint, cached process-wide for ten minutes and keyed
by URL — `resolve_asr_provider` builds a fresh adapter per call, so
anything held on the instance would probe once per VAD span. `submit`
probes ahead of every request, and any async caller whose behaviour
turns on the answer resolves it through `resolve_capabilities`: the
vocabulary decision (#355), the provenance stamp and the Admin panel
(#358).

Deliberately *not* the concurrency-slot decision (#356). A session
waiting for a slot must cost nothing on the wire, and a probe there
would put a request in front of the decision not to make one. It reads
the last known answer instead, which is at worst a cap of 1 on the first
session a worker handles — the pre-#352 behaviour, and what the bundled
server reports anyway.

Failure is pessimistic: an old server, a dead one, a proxy serving an
HTML error page, unreadable JSON — all land on no word timings *and*
`VocabularyStyle.NONE`. The vocabulary half is the one that matters to a
GM. A v1 server accepts `initial_prompt` and does nothing with it, so
declaring the capability would spend a wiki query per session on a
request that changes nothing and would hide the reason the campaign's
proper nouns keep coming back spelled phonetically. Declared honestly,

A transcription failure invalidates the cached probe; a 429 does not. A
500 is the most common way a replaced server announces itself, while a
429 only says the one that is there is busy.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(providers): real word spans from a v2 server ... d72dc4c008

The #352 server carries `words` on every segment. Aligned words are
taken as measured — `interpolated=False`, the alignment score carried
through as confidence — and only what alignment could not place is
invented.

Forced alignment is not all-or-nothing, which is the part worth getting
right. WhisperX gives up on individual words — a number spoken over
laughter, a proper noun its phoneme model has never met — and writes
`null` into that word's start, end and score. Those are placed
proportionally in the gap between their aligned neighbours (or the
segment bounds at either end) and marked `interpolated=True`; the
neighbours are untouched, because one word nobody could place must not
downgrade the twenty around it. A segment whose alignment failed
entirely arrives with `"words": []` and gets the whole-segment
interpolation the pre-#352 server always got.

`word_timestamps_interpolated` is therefore read off the words rather
than off the declaration: it is a claim about one track, where the
capability is a forecast about an endpoint, and on an alignment server
the two genuinely differ.

Which parser runs is decided by the declaration, never by sniffing the
body, so what the adapter says and what it returns cannot disagree. The
one case that costs something — `/transcribe` reachable behind a proxy
while `/health` is not, so real words arrive and are interpolated
anyway — logs a warning naming the fix, because nobody reaches
"my timings are estimated because of a proxy rule" unaided.

A v1 response is byte for byte what it has always been: the same
interpolation, the same segment bounds, both degradation codes.

Verified against the live server at 10.3.0.28:8091 — the probe reads
api_version 2, the vocabulary reaches `initial_prompt` (and visibly
biases the output), a real word parses with its score, and silence
returns no segments and no degradation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(audio): carry words through the pipeline ... 90bf4014b1

`transcribe_track` puts `words` on every segment dict — `{word, start,
end, interpolated}` — and the VAD path shifts their times with the
segment they belong to.

They travel; nothing branches on them. The merge, the overlap marking,
the line format and the coverage checks are untouched, and every
provider fills the key: the #352 server with measured spans, a v1 server
and the interpolating path with estimates that say so. That is what lets
the two consumers that want words — highlight anchoring, and the
persisted rows — ask one question instead of asking which adapter ran.

`_offset_segments` is the half that cannot merely pass them through. A
span cut at two hours is transcribed as its own file starting at zero,
so a word left in span-relative time would be a *measured* timestamp
pointing two hours from the words it belongs to — worse than the
interpolated one it replaced, because nothing would mark it wrong.

`confidence` is dropped on the way into the dict. Nothing reads it, and
an alignment score and a managed provider's per-word confidence are on
different scales, so carrying it would only guarantee that two runs of
one session on two providers stored visibly different things.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

db: transcript segments carry their per-word timings ... a62c105046

Migration `c7d8e9fa0b1c` adds a nullable JSONB `words` column to
`transcript_segments`, and `process_audio` writes what the adapter
measured.

Additive and nullable is the whole upgrade story. Existing rows keep the
segment-level times they were written with and read back as NULL —
"this run recorded no words", not "this utterance had none". Nothing is
backfilled: those words were never measured, and inventing them would
put an estimate in the one column whose purpose is to tell measured from
estimated. A GM who wants word timings on an old session reprocesses it.

JSONB rather than a `transcript_segment_words` table. A session is
1,500-2,500 segments and 25,000-30,000 words, so a row per word would
multiply the largest table in the schema by fifteen for data that is
only ever read as "all the words of this segment, in order" — never
joined, filtered or aggregated.

The shape is identical across providers, which is asserted rather than
assumed: the same session driven through the bundled v2 adapter and
through the stub — two wire formats, two confidence scales — persists
byte-identical rows. Confidence is not stored, which is what makes that
possible and costs nothing, since nothing reads it.

Verified up/down/up against Postgres 16 from `b6c7d8e9fa0b`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(audio): anchor highlights to words ... e7af009824

A highlight's timestamp is now the start of the word the quote begins
on, when that word was measured.

A segment is an utterance, not a sentence. "and then — hold on, let me
check something first — I cast fireball" is one segment, and captioning
the fireball line with the segment start sends a reader twenty seconds
early to hear somebody deciding what to do. #352 makes the word
positions real, so the anchor can be the word.

Only when it was measured. An interpolated span is a guess at a position
inside a segment whose *bounds* were measured, so preferring it would
trade a real number for an invented one — which is the whole reason the
`interpolated` flag is carried per word rather than per result. A
segment with no words, a v1 server's estimates, or words that do not
reconstruct their own text all fall back to the segment start, exactly
as before.

A quote spanning two segments of one track now anchors instead of being
dropped. The transcript the model reads is compacted (#341), so a
same-speaker run renders as a single line and a quote lifted from it can
straddle a segment boundary while still having exactly one person to
credit. Runs are consecutive, same-track, and capped at the same 30 s
`compact_transcript` uses, so the joined text is text the model could
actually have seen. A quote spanning two *speakers* is still dropped:
that is #346's rule and the reason it exists.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

test(conformance): both bundled server versions, and partial alignment ... df1b079198

The Whisper wire fake now answers `GET /health` as well as
`POST /transcribe`, in either API version, and the v2 body carries
`words` — including a word alignment could not place and a segment whose
alignment failed entirely. Without the health route every case would
have silently exercised the v1 branch, because the adapter would have
been probing a hostname that does not resolve.

`local_whisper_v1` becomes its own case rather than a variant, so the
whole file runs against the server a self-hoster may not have upgraded
yet. The two disagree on exactly the declarations #352 changed, which is
what makes running everything against both worth the case.

`test_the_word_timestamp_declaration_matches_the_words` is restated. Its
old form required every word from a word-timestamping provider to be
measured, which no forced-alignment server can promise — WhisperX gives
up on a number spoken over laughter several times an evening. So the
declaration is a claim about the endpoint, `interpolated` is a fact
about each word, and what must agree is the result's degradation code
with the words in it — asserted both ways, so an adapter cannot pass by
marking everything degraded. The overclaim it existed to catch is still
caught: declaring word timestamps and measuring none of them fails.

One case added: a word that could not be placed is marked rather than
hidden, carries no confidence, and still sits in order inside its own
segment.

`requests_seen` counts transcription requests only. The capability probe
carries no audio and happens whether or not the input is refused, so
counting it would turn "no oversized upload was attempted" into "no
packet was sent".

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs: upgrading the bundled transcription server ... ce0db95162

OPERATIONS.md gains "Upgrading the bundled transcription server": how to
tell which API version yours runs (`curl /health` — no `api_version`
means v1), what the two gaps in the capability panel mean until you
update, and what changes when you do.

The part worth writing down is what *does not* change. Word timings and
vocabulary bias apply to sessions recorded after the update; nothing is
reprocessed, older transcripts keep their sentence-level times, and
their word column stays empty rather than being filled with estimates by
a column whose whole purpose is telling measured from estimated. A GM
who wants an old session upgraded reprocesses it, which is only possible
while its audio is still retained.

Also corrects the capability panel's own description: it now says the
ASR side asks the endpoint one cached question, because "nothing here
sends a request" stopped being true when the two server versions had to
be told apart.

CHANGELOG: one Added entry leading with the migration id.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed 2026-09-05 04:33:25 +00:00

claude-bot merged commit c2709f9ed6 into main 2026-09-05 04:40:50 +00:00

claude-bot deleted branch feat/352-word-timestamps-client 2026-09-05 04:40:51 +00:00

claude-bot referenced this pull request from a commit 2026-09-05 04:40:52 +00:00

Merge PR #499: real word timestamps from the bundled server, carried through the pipeline (#352)

claude-bot referenced this pull request 2026-09-05 04:41:35 +00:00

[Backend] Keep self-hosted ASR first-class — VAD-cut with forced alignment #352

claude-bot referenced this pull request 2026-09-05 06:04:34 +00:00

chore(release): v4.2.0 — AI Provider Abstraction & BYO-AI #504

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backend

Issue related to the backend

  

bot

Issue related to the bot

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

frontend

Issue related to the frontend

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

recording

Issue related to the recording system

  

security

Security hardening / vulnerability work

  

wontfix

This won't be fixed

No labels

backend

bot

bug

duplicate

enhancement

frontend

help wanted

invalid

question

recording

security

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/Quest-Board!499

No description provided.