feat: OpenAI-compatible transcription adapter and an ASR provider select (#350, #128) #496

Merged

claude-bot merged 7 commits from feat/350-managed-asr-adapter into main 2026-09-05 03:47:45 +00:00

Conversation 0 Commits 7 Files changed 22 +2842 -116

claude-bot commented 2026-09-05 03:41:37 +00:00

Contributor

Copy link

Closes #350 (its last open criterion: vocabulary on a managed adapter). Delivers the managed-ASR half of #128; the per-campaign BYO-AI half follows separately.

You can now transcribe with OpenAI or Groq, or any server speaking POST /v1/audio/transcriptions, instead of the bundled WhisperX — and say so explicitly rather than having it guessed from the URL.

What

• OpenAiCompatibleAsrProvider (app/providers/asr_openai.py, registered as openai — the wire shape, not the vendor): multipart file/model/response_format=verbose_json/timestamp_granularities[] (word + segment)/language/prompt for vocabulary (40 terms, since the 224-token prompt window drops the tail silently), Bearer auth. Real words[] spans (word_timestamps=True); interpolation only if a server omits words. Declares max_bytes = 25 MiB (identical on OpenAI and Groq), no concurrency limit, and a new enforces_own_rate_limit capability so the conformance suite stops demanding a concurrency figure from a managed synchronous adapter. 429 → ProviderRateLimitError(retry_after), 413 → ProviderInputTooLargeError. Model defaults per host (whisper-1, whisper-large-v3-turbo); a third-party server must name its model or gets a clear configuration error. Language codes coerced both ways (~40 languages; an unrecognised value is dropped so per-request detection takes over rather than failing every span). URL building tolerates Groq's /openai/v1 base.
• Chunking honoured by the caller: transcribe_with_optional_vad resolves the provider up front and, when needs_chunking and a track exceeds a declared limit, takes the span-cutting path even with trimming off; provider_max_span_seconds converts the byte cap to ~802 s at the cut format; split_oversized_spans windows a still-oversized span with 2 s overlap and a midpoint seam rule so nothing is doubled or dropped; >25 % refused windows raise rather than return a partial.
• Settings/UI: asr_provider / asr_provider_resolved / asr_provider_choices / asr_model on the bot-settings API (blank preserves, auto clears, mirroring the LLM select); a select and model field in the Whisper block; the Test button speaks the chosen dialect. whisper_provider renamed to asr_provider — it shipped in v4.1.1 with no UI and nothing had set it. asr_model wired through process_audio with the model asserted on the wire on both the plain and VAD paths, plus an AST guard on the call site because the task has no harness.
• Conformance: openai case + verbose_json fake — all 15 ASR cases pass, including vocabulary-on-the-wire and byte-limit-with-zero-requests.
• Docs: OPERATIONS "Choosing a transcription provider" — including that audio leaves your server with a managed provider and what that means for a group that self-hosts for that reason.

No live call against OpenAI or Groq was possible (no keys); the fake follows the documented verbose_json shape, cited in the adapter docstring; the monthly canary is what catches drift. Rebased over #357 and #356 (one conflict: the transcription call's try/finally from #356 plus this branch's one keyword).

Verification

• Backend: 1979 passed, 12 skipped, 1 xfailed before the final rebase; the covering suites (test_asr_openai, test_asr_chunking, test_provider_concurrency) re-run green after it. Frontend 486 passed, eslint clean apart from the pre-existing warning. ruff clean. No migration; no bot contract change.

🤖 Generated with Claude Code

Closes #350 (its last open criterion: `vocabulary` on a managed adapter). Delivers the managed-ASR half of #128; the per-campaign BYO-AI half follows separately. You can now transcribe with OpenAI or Groq, or any server speaking `POST /v1/audio/transcriptions`, instead of the bundled WhisperX — and say so explicitly rather than having it guessed from the URL. ## What - **`OpenAiCompatibleAsrProvider`** (`app/providers/asr_openai.py`, registered as `openai` — the wire shape, not the vendor): multipart `file`/`model`/`response_format=verbose_json`/`timestamp_granularities[]` (word + segment)/`language`/**`prompt`** for vocabulary (40 terms, since the 224-token prompt window drops the tail silently), Bearer auth. Real `words[]` spans (`word_timestamps=True`); interpolation only if a server omits words. Declares `max_bytes = 25 MiB` (identical on OpenAI and Groq), no concurrency limit, and a new **`enforces_own_rate_limit`** capability so the conformance suite stops demanding a concurrency figure from a managed synchronous adapter. 429 → `ProviderRateLimitError(retry_after)`, 413 → `ProviderInputTooLargeError`. Model defaults per host (`whisper-1`, `whisper-large-v3-turbo`); a third-party server must name its model or gets a clear configuration error. Language codes coerced both ways (~40 languages; an unrecognised value is dropped so per-request detection takes over rather than failing every span). URL building tolerates Groq's `/openai/v1` base. - **Chunking honoured by the caller**: `transcribe_with_optional_vad` resolves the provider up front and, when `needs_chunking` and a track exceeds a declared limit, takes the span-cutting path even with trimming off; `provider_max_span_seconds` converts the byte cap to ~802 s at the cut format; `split_oversized_spans` windows a still-oversized span with 2 s overlap and a midpoint seam rule so nothing is doubled or dropped; >25 % refused windows raise rather than return a partial. - **Settings/UI**: `asr_provider` / `asr_provider_resolved` / `asr_provider_choices` / `asr_model` on the bot-settings API (blank preserves, `auto` clears, mirroring the LLM select); a select and model field in the Whisper block; the Test button speaks the chosen dialect. `whisper_provider` renamed to `asr_provider` — it shipped in v4.1.1 with no UI and nothing had set it. `asr_model` wired through `process_audio` with the model asserted on the wire on both the plain and VAD paths, plus an AST guard on the call site because the task has no harness. - **Conformance**: `openai` case + `verbose_json` fake — all 15 ASR cases pass, including vocabulary-on-the-wire and byte-limit-with-zero-requests. - Docs: OPERATIONS "Choosing a transcription provider" — including that audio leaves your server with a managed provider and what that means for a group that self-hosts for that reason. No live call against OpenAI or Groq was possible (no keys); the fake follows the documented `verbose_json` shape, cited in the adapter docstring; the monthly canary is what catches drift. Rebased over #357 and #356 (one conflict: the transcription call's try/finally from #356 plus this branch's one keyword). ## Verification - Backend: **1979 passed, 12 skipped, 1 xfailed** before the final rebase; the covering suites (`test_asr_openai`, `test_asr_chunking`, `test_provider_concurrency`) re-run green after it. Frontend **486 passed**, eslint clean apart from the pre-existing warning. ruff clean. No migration; no bot contract change. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 7 commits 2026-09-05 03:41:37 +00:00

feat(providers): OpenAI-compatible ASR adapter ... 11c35ecc66

#350 left one criterion unmet — "vocabulary maps correctly for at least the
local adapter and one managed adapter" — and it could not be met, because
there was no managed adapter. A contract with one implementation is a
description of that implementation, and "swap your provider without a rewrite"
was a claim nothing tested.

So: POST /v1/audio/transcriptions, which is what OpenAI documents, what Groq
implements verbatim, and what most self-hostable Whisper servers speak. Named
for the wire shape rather than the vendor, because they are all one adapter.

It takes the other branch of the contract everywhere it is allowed to. Words
are measured rather than interpolated, so this is the first configuration in
which a quote citation lands on the word; vocabulary is spelled as `prompt`
rather than `initial_prompt`, truncated to 40 terms because OpenAI's prompt
window is 224 tokens and overflow drops the tail silently; and it declares a
real 25 MB per-request limit where the bundled server declares none, which is
what makes caller-owned chunking something the code has to actually do.

`concurrency_limit=None` needed a new declaration to be honest rather than an
omission. `AsrCapabilities` gains `enforces_own_rate_limit`: a self-hosted
server has a small fixed slot count and no way to say it is full, so an adapter
for one must name a number; a managed API knows the caller's real quota — which
varies by tier and by model and cannot be guessed from in here — and says so
with a 429 and a Retry-After. #356's scheduler pre-throttles the first and
reacts to the second.

Two things the API's own asymmetries force:

- `language` is an ISO-639-1 code on the way in and an English name on the way
  back, and `_detect_track_language` feeds one straight into the other. Both
  directions are mapped and an unrecognised value is dropped rather than
  forwarded — dropping degrades to per-request detection, which the pipeline
  handles, while forwarding a token the API rejects would fail every span of
  the track it had just detected.
- a server that returns no `words` (some compatible builds, and OpenAI's newer
  transcription models) falls back to the interpolation the local adapter uses
  and marks the *result* degraded. The capability declaration does not move:
  it describes the configuration, which #358 reads before anything runs, while
  `Word.interpolated` describes what came back.

No live call was made against OpenAI or Groq — there are no keys here. The
fields relied on are cited in the module docstring from the documented
verbose_json shape, and the monthly canary is what would catch that
documentation drifting from the server.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(providers): register and sniff the OpenAI-compatible ASR adapter ... 87154bee44

`ASR_PROVIDERS` gains one entry, `openai`, and it is the only one it will ever
need for this dialect: OpenAI, Groq and every compatible server select it, and
three entries pointing at one class would imply differences that do not exist
and would need a fourth the next time somebody ships a clone.

`resolve_asr_provider` gains the sniff the LLM side has had since #351, so an
unset provider is guessed rather than assumed. Deliberately tighter than
`sniff_llm_provider`: it matches on the parsed *host*, and only on
api.openai.com and api.groq.com. A URL substring match also matches a path and
a query string, and what is being decided here is which API an audio file and
a key are sent to. Everything else is the bundled server, which is what an
unset provider has always meant, so no existing deployment changes on upgrade.

A self-hosted OpenAI-compatible server on a custom domain cannot be detected
from a URL and has to be selected explicitly. That is the same visible exit the
LLM sniff has, and the reason there is a select at all.

`WhisperConfig` gains `model`, read from and written to the existing
`whisper_config` JSON. No migration: it is one string on one settings row, and
buying a rehearsal, a backup and a rollback plan for that would be a poor
trade. Existing rows have no such key and read back blank, which is exactly
what the per-host defaults are for.

`model` is passed to every adapter, including the bundled one, which has no use
for it — the WhisperX server loads its own model and its form has no field to
override it. Uniform construction is the point: the alternative is a registry
that knows which adapter takes which argument, which is the per-provider
knowledge this module exists to hold in one table.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(audio): honour needs_chunking in the caller, not in the adapter ... 32f1bc3ccc

The contract puts cut-and-remap on the caller — an adapter that chunked
internally would return timestamps in its own private frame, which is the #320
misattribution again. That was free to say while no adapter declared a limit.
It is not free now.

25 MB is about 13.6 minutes of the 16 kHz mono 16-bit WAV the bot writes, and a
per-speaker track from a three-hour session is an order of magnitude past it.
With silence trimming off, every track would be refused by
`enforce_input_limits` before a byte was sent: a self-hoster who switched to a
managed provider would find transcription had simply stopped.

So `transcribe_with_optional_vad` now resolves the provider first and takes the
span-cutting path when the provider needs chunking and a track would not fit,
whatever the toggle says — with a warning naming why, and using the default
thresholds. It is not the same thing the toggle asks for: trimming is a cost
optimisation, this is a hard requirement of the endpoint, and conflating them
would leave an operator unable to turn off something they never turned on. It
does mean ffmpeg is on this path, which it was not before; OPERATIONS.md says
so.

A speech span is not guaranteed to fit either. The realistic case is not a
quarter-hour monologue but a noise floor set high enough that `silencedetect`
finds nothing and the whole track comes back as one span. `split_oversized_spans`
cuts those into windows sized from the declared limit, overlapping by two
seconds so the model hears a word straddling a cut from both sides.

The seam is the part worth reading. Both windows transcribe the overlap, and
each keeps only the output whose *midpoint* falls on its side of the middle of
it. Testing the start would keep a straddling segment twice; testing
containment would drop it from both, and a transcript quietly missing a
sentence at every boundary is the failure this pipeline exists to prevent. By
midpoint, every second of audio is exactly one window's responsibility.

Also: `_transcribe_span_resiliently` no longer retries `ProviderInputTooLargeError`.
The next attempt would send the identical file to the identical limit; three
of those is just a slower way to reach the same skip.

`asr_model` is threaded through the ASR path alongside `asr_provider`. The one
remaining wire — passing `whisper_cfg.model` at the `process_audio` call site —
is deliberately not in this branch: `app/tasks/` is being edited by #356 and
#357 concurrently. Until it lands, a blank model still resolves through the
per-host defaults, so OpenAI and Groq work end to end and only a third-party
compatible server needs the field.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(admin): an ASR provider select, and a Test button that speaks its dialect ... fa89b38b1c

Mirrors the LLM select #351 added, with the same three-state convention: blank
preserves, `auto` clears back to guessing, an unknown name is a 400 at the form
rather than a stored value that fails hours later in a worker.

Renames the request/response field `whisper_provider` to `asr_provider`. It
shipped in v4.1.1 with no UI at all — there was one adapter to choose between —
so nothing has ever set it through the product, and the name should match
`ASR_PROVIDERS` / `resolve_asr_provider` / the `asr_provider` argument the
pipeline already carries. What is being chosen is a speech-to-text service, not
a build of Whisper.

Adds `asr_provider_resolved` (so the "Detect from the URL" option can name the
guess instead of leaving the operator to infer it), `asr_provider_choices` from
the registry, and `asr_model`.

A blank model on a host with no default is rejected at the *form*, by
constructing the adapter that would be stored and asking it to resolve its
model — so the per-host defaults and the wording live in one place. Caught here
because the alternative is a settings page that saves happily and a session,
recorded hours later, that dies in a Celery worker nobody is watching: by then
the recording exists and the evening is over. This is the one moment the
operator is looking at the field the message names.

The Test button now sends its second of silence through the configured
adapter's own `submit` instead of hand-building `POST /transcribe`. That was
testing a wire format rather than a configuration: against an OpenAI-compatible
endpoint the old request 404s, so the button would have reported "unreachable"
for a service that works perfectly. Exactly the mistake #488 fixed on the LLM
side — a Test button is worth having only if it makes the call the product
makes. It is bounded at 30 s here rather than by the adapter's ten-minute
track timeout, because someone is watching it.

The UI copy says the thing that actually matters: with a managed provider your
session audio leaves your machine, and if that is why you self-host, this is
the setting that undoes it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

test(conformance): the openai case, and a verbose_json wire fake ... a05be83537

Adding an adapter to the suite is a case and a fake, and no test changes —
that was the design, and this is the first time it has been exercised by
something other than the adapter it was written around. All fifteen ASR cases
run against the new adapter and pass.

It is the case that makes the suite a test of the contract rather than of the
bundled server: it takes the opposite branch wherever the contract allows one.
Words measured where the local adapter interpolates, a declared byte limit
where the local one declares none, vocabulary under a different field name, a
model that has to be resolved. The stub already covered some of that
in-process; this covers it over a transport, so httpx encodes the body and the
assertions are about bytes that reached one.

The fake is built from the documented verbose_json shape — there is no OpenAI
or Groq key here, so nothing has been checked against a live response, and the
docstring says so rather than implying otherwise. It carries the scoring fields
the real API returns, so ignoring them is demonstrated rather than assumed, and
a `speaker` the API does not send but several compatible servers add when
diarization is on: the contract has nowhere to put it, and that is what proves
it is dropped rather than merely unused.

`test_the_asr_adapter_declares_a_concurrency_limit` needed its condition
corrected. It required any synchronous adapter to name a limit, using
`async_completion` as a stand-in for "self-hosted" — which held while every
adapter was the bundled server or the stub. Transcription has no asynchronous
job API anywhere, so this adapter is synchronous *and* managed, which is the
case the test's own docstring already said `None` was right for. It now reads
`enforces_own_rate_limit`, which is the declaration that actually decides it.

There is deliberately no conformance case for a server that omits `words`. The
adapter still declares `word_timestamps=True` — a statement about the
configuration — while marking that individual result degraded, and the
conformance case asserts the stricter per-run correspondence. Making it a case
would assert that the declaration is wrong. It is tested as adapter behaviour
in tests/test_asr_openai.py, alongside the URL forms, the per-host model
defaults, the language coercion in both directions, and the 413/429/5xx
mapping.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs: choosing a transcription provider, and what it costs you ... 72b54fda86

The decision this feature asks an operator to make is not mainly a quality one,
so the docs lead with the part that is irreversible: with a managed provider,
every per-speaker track of every recording is uploaded to a third party. A
group that self-hosts Quest Board precisely so their table's audio stays on
their own machine should read that sentence and stop, and OPERATIONS.md says so
before it says anything about accuracy.

For everyone else it covers what they need to actually do it: the two options
and what the model field means on each, that detection matches on the host and
only recognises the two managed ones, the 25 MB cap and what the pipeline does
about it — including that trimming is forced on when a provider needs it and
that ffmpeg is therefore on that path — and the exact fields for pointing at
Groq, whose base URL contains its own /openai/v1.

DEVELOPMENT.md's "adding a provider adapter" gains the naming rule the registry
now depends on: name the entry for the wire shape, not the vendor.

The changelog entry is one item because it is one decision, and it is written
for the person who has to make it rather than for the person who built it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(tasks): pass the configured transcription model to the pipeline ... 6328006cfb

One line at the `process_audio` call site, held back from the earlier commits
because `app/tasks/` was being edited concurrently by #356 and #357. #357 has
landed; this is the wire it was waiting on.

Without it a blank `asr_model` still reached the adapter, which meant OpenAI and
Groq worked through their per-host defaults and only a self-hosted
OpenAI-compatible server did not — the configuration with no default and
therefore the one that needs the field most. A missing argument here would have
been invisible until a session had already been recorded.

`process_audio` has no test harness: it is a Celery task carrying a database
session, a directory of WAVs and a notify callback, and every test in the
repository mocks it. So the wiring is covered from two directions instead. Two
tests run a real track through `transcribe_with_optional_vad` — both branches,
because the trimmed one is dozens of separate requests and the provider holds
no state between them — with the real registry, the real adapter and a
MockTransport as the only non-production part, and assert the model arrives as
the `model` multipart field. A third reads the call's shape out of the task's
own AST, which is resilient to reformatting and to arguments being added (#356
is about to add some) and fails if the provider or the model stops being
passed. Both were mutation-checked: deleting either wiring point fails them.

The AST test is a stopgap for the missing harness and says so.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed 2026-09-05 03:41:49 +00:00

claude-bot merged commit 17e257a741 into main 2026-09-05 03:47:45 +00:00

claude-bot deleted branch feat/350-managed-asr-adapter 2026-09-05 03:47:45 +00:00

claude-bot referenced this pull request from a commit 2026-09-05 03:47:46 +00:00

Merge PR #496: OpenAI-compatible transcription adapter and ASR provider select (#350, #128)

claude-bot referenced this pull request 2026-09-05 03:48:16 +00:00

[Backend] Define the ASR provider contract and normalise on word timestamps #350

claude-bot referenced this pull request 2026-09-05 03:48:21 +00:00

[Backlog] AI provider abstraction and per-campaign BYO-AI #128

claude-bot referenced this pull request 2026-09-05 05:15:42 +00:00

feat: per-campaign AI providers — a campaign brings its own ASR and LLM, behind an instance switch (#128) #501

claude-bot referenced this pull request 2026-09-05 05:16:29 +00:00

[Backend] Follow-ups from the provider evaluation: usage rows misclassify sniffed managed runs, managed ASR is unpriceable, and the live tooling cannot reach the managed adapter (#360) #502

claude-bot referenced this pull request 2026-09-05 05:25:03 +00:00

[Backlog] AI provider abstraction and per-campaign BYO-AI #128

claude-bot referenced this pull request 2026-09-05 06:04:34 +00:00

chore(release): v4.2.0 — AI Provider Abstraction & BYO-AI #504

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

backend

Issue related to the backend

  

bot

Issue related to the bot

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

frontend

Issue related to the frontend

  

help wanted

Need some help

  

invalid

Something is wrong

  

question

More information is needed

  

recording

Issue related to the recording system

  

security

Security hardening / vulnerability work

  

wontfix

This won't be fixed

No labels

backend

bot

bug

duplicate

enhancement

frontend

help wanted

invalid

question

recording

security

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

rbrooks/Quest-Board!496

No description provided.