[Backend] Evaluate ASR and LLM providers against the golden corpus before committing #360
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found in the August 2026 session lifecycle review (#319). Explicitly a decision-support task — no provider commitment is being made in this milestone.
Why
The review produced a costed recommendation (AssemblyAI Universal async for ASR, Claude Sonnet 5 for the LLM, roughly $0.95/session and $4.11/customer-month at weekly play). That recommendation is deliberately not being adopted as a commitment. The abstraction lands first; the choice is made afterwards, with measurements rather than vendor claims.
Several figures in that analysis are flagged unverified, and several providers price promotionally — Deepgram's published rate is promotional, AssemblyAI's keyterms feature is free only while in beta, and Gemini Flash pricing is scheduled to double at the start of 2027. Any decision should model list rates, not headline rates.
Proposed fix
Once the abstraction and the golden corpus exist, run the same corpus through each candidate adapter and record: word error rate, proper-noun accuracy with and without vocabulary bias, timestamp accuracy on long audio, beat validation rate, chronology and attribution scores, cost per session at list rates, and data-retention terms.
Include the self-hosted configurations as candidates on equal footing — the point is to find out where the real quality gap is, not to assume one.
Bring the results back as a decision, with the retention and privacy terms of each managed provider stated explicitly, since a privacy-sensitive self-hosting audience is part of the product's identity.
Acceptance criteria
Evaluated, not committed — first pass written 2026-09-05 as
docs/.internal/ai-provider-evaluation-2026-09.md(internal, gitignored by convention; the full report will be attached here once the self-hosted LLM cells are filled from #353/#354). It prices seven ASR and seven LLM candidates at list rates, states each one's retention terms, and marks every accuracy cell measured, not run — no key, or pending.Does the August recommendation survive? Partly. The $0.95/session figure reconstructs as 4.9 h × $0.15/hr (AssemblyAI Universal-2) + ~$0.20 LLM — it was priced on the older tier. At Universal-3.5 Pro's current $0.21/hr the same session is $1.23, and $5.33/customer-month at weekly play. The VAD assumption underneath it (28 % kept) is over twice the only measured figure (12.9 %, from the #433 synthetic session), so the model's error bars are wider than the differences it is being used to decide between.
New, from reading the pipeline rather than the rate cards. With VAD on, spans are transcribed individually; the synthetic session yields 207 spans, median 5.4 s, 92 % under 10 s. Groq's 10-second per-request minimum therefore bills 1.77× the audio — an effective $0.071/hr, not $0.04 — and a 3.5 h session is ~1,450 requests, which is a rate-limit question for every managed candidate.
Recommendation: keep self-hosted as the default; make managed a per-campaign opt-in (#128). At 60k/8k tokens the LLM is under a tenth of the bill, so choose the LLM on the four eval metrics, not price; cost discipline belongs on the ASR side. The counterargument is written at equal length: a GPU is the real adoption barrier, the self-hosted accuracy evidence is TTS audio its own README calls a tripwire, and managed at $5.33/month beats a purpose-bought GPU above a 24.7 W average draw.
The measurement that would most change the answer costs nothing: the 72.2 % marker-detection figure was taken with vocabulary bias off (
asr_leg.pypasses novocabulary=). Re-running the ASR leg with the fixture's 24 screened markers needs no key and no adapter — queued right after #433 merges, and it doubles as #355's "measured effect" criterion.Last criterion, enforced: PR #498 adds
tests/test_provider_isolation.py— no vendor hostname in code and no vendor SDK import outsideapp/providers/. It also names what it cannot catch:audio_service.pystill owns six per-vendor HTTP transports with vendor headers and hardcoded model defaults (#443 territory).Code gaps the arithmetic found (to be filed as follow-ups):
ASR_HOURLY_USDis empty and keyed on the adapter name, so OpenAI, Groq and a self-hosted compatible server would price identically; a sniffed managed run is recorded with the configured provider string and so reads as self-hosted in usage rollups;TOKEN_RATESlacks Sonnet 5 / Opus 5 (which OPERATIONS suggests);--liveconformance cannot reach the managed ASR adapter; the synthetic ASR leg cannot be pointed at one.Left open until the self-hosted GPU and CPU LLM numbers land.
Full evaluation, part 1 of 2 (sections 1–5; part 2 carries privacy, recommendation, code findings, next runs and sources). Written 2026-09-05; self-hosted LLM cells filled from #353/#354 the same day.
ASR and LLM Providers — Evaluation Before Commitment
Date: 2026-09-05 · Issue: #360 · Status: Decision support. No provider is chosen here, and nothing in this document changes a default.
Question: The August review recommended AssemblyAI Universal async for ASR and Claude Sonnet 5 for the LLM at roughly $0.95 a session and $4.11 a customer-month. Several of its figures were flagged unverified. Does the recommendation survive being checked against list rates, published retention terms, and the accuracy the corpus can actually measure?
1. Purpose, and what is not being decided
The review that opened this milestone put a managed stack at "roughly $0.95 a session, dominated by ASR, with a VAD pre-pass worth about 72% of the audio — every one of those numbers is a guess, because nothing in the product had ever recorded what a session cost" (
webapp/backend/app/models/session_usage.py:5). This document is the next step after that admission: the numbers that can be checked, checked; the numbers that cannot, named as such.Not decided here: which ASR provider Quest Board ships pointed at (it stays the bundled WhisperX server;
app/providers/registry.py:83); which LLM a hosted tier would use; whether a hosted tier exists at all.Decided here: what each candidate costs at list, what each one does with your audio, what the corpus has actually measured, and which single measurement would most change the answer.
Only list rates are modelled. Three of the seven ASR candidates price promotionally right now: Deepgram's headline rate is a 50% discount that requires enrolling in a programme that keeps your audio for training, AssemblyAI's keyterm boosting is free only during beta, and Gemini Flash's current rate is marked as expiring on 2026-12-31. A commitment made against a promotional price is a commitment to renegotiate later from a weaker position.
Retention terms are a first-class column, not a footnote. A privacy-sensitive self-hosting audience is part of this product's identity, and
docs/OPERATIONS.mdalready tells that audience that "with a managed provider, your session audio leaves your machine … if you self-host Quest Board because your table's recordings should not go to a cloud service, this is the setting that undoes that".2. Method
2.1 The corpus
evals/fixtures/synthetic/evals/fixtures/synthetic_session/(#433)evals/fixtures/private/2.2 The metrics
Summary side (
evals/README.md): coverage, attribution, chronology (Kendall's tau-b), beat validation rate (through the realbeat_service.validate_beats). ASR side (evals/synthetic_session/scorers.py): session and per-speaker WER, marker detection, foreign markers, onset drift, and exact structural assertions.2.3 What could not be run here
There are no API keys in this environment. Every managed candidate's accuracy column reads not run, with the exact command that produces it once a key exists (§9.2). Consequences: (1) no managed provider's accuracy on this corpus is known — not worse, not better — unknown; (2) the self-hosted numbers are not an absolute bar either — the synthetic session is TTS (§3.3).
2.4 Where the rates come from
Gathered by web search on 2026-09-05. [V] vendor page, [3P] third-party aggregator. Every LLM token rate in this document is [3P]. That alone bars a commitment.
3. ASR candidates
3.1 The table (rates per hour of audio submitted — see §5)
AsrProviderbut is unwrittenwhisper-1verbose_jsonwords)prompt, 224 tokensOpenAiCompatibleAsrProvidergpt-4o-transcribejsononly, words interpolatedpromptgpt-4o-mini-transcribepromptwhisper-large-v3-turbopromptinitial_prompt, 100 terms3.2 What "adapter today" costs
Only two rows work without new code. AssemblyAI and Deepgram both need an adapter written against
AsrProvider— the contract anticipates them (AsrJob/poll/async_completion) — but "the contract supports it" is not "it is written and conformance-tested".3.3 Self-hosted ASR, measured (PR #497 / #433: WhisperX
large-v3-turbo, RTX 3090, three identical runs)fullic_onlyThe README's own caveat: "Accuracy numbers are not accuracy numbers. TTS pronunciation is dictionary-correct. Treat every word error rate here as a tripwire against previous runs of this fixture, never as an absolute quality bar." Also: nothing about bleed, nothing about capture. What is transferable: 0 foreign markers across both assemblies, zero segments inside scripted silence, zero before the late joiner. Those are the properties a managed replacement would have to match.
3.4 The measured marker figure was taken without vocabulary bias
asr_leg.pycallstranscribe_with_optional_vadwith novocabulary=. 72.2% is the un-hinted floor on a provider that declaresINITIAL_PROMPTsupport. The single cheapest experiment in this document — no key, no adapter, one argument — is to re-run that leg with the fixture's 24 screened markers and read the lift.3.5 The request shape the pipeline actually produces
With VAD on (the default),
transcribe_session_vadtranscribes spans individually — one request per span, on every provider. Derived from the manifest: 207 spans for 5 speaking tracks in 30 minutes (1 208.9 kept seconds vs 1 162.4 measured — within 4%), median 5.4 s, 190 of 207 (92%) under 10 s.4. LLM candidates
4.1 The table (per 1M tokens; window and schema as the adapter declares them)
⚠ Sonnet 5's $2/$10 is the intro rate the notes say ended 2026-08-31; §5.4 shows the $3/$15 case. ⚠ Gemini modelled at the 2027 figure because the current one expires 2026-12-31; the notes are inconsistent on the step ($0.76 vs $1.50) — the higher is used.
4.2 Two structural facts
Gemini cannot be selected at all today (
LLM_PROVIDERShas four entries; an unknown name raises). Self-hosted's schema guarantee is stronger, not weaker: on llama.cpp and Ollama the schema becomes a GBNF grammar, so the model cannot emit malformed JSON; on Anthropic and OpenAI it is structured output.4.3 The self-hosted LLM cells (filled 2026-09-05 from #353/#354)
Four synthetic five-minute fixtures, one run each,
history.jsonltags47c31d7-353-gpu/47c31d7-353-cpu. Caveat: the private hand-verified session was not available, so neither run exercised windowing and the coverage figures describe a short transcript. Only coverage degraded on the small model; chronology, attribution and grammar-validated beats held — the CPU summary is shorter, not wronger. A generic QEMU vCPU without AVX2 measured 0.3 tok/s and is the documented floor.5. The cost model
5.1 The session, and the three audio scenarios
3.5 hours, five speakers = 17.5 raw track-hours; what is submitted depends on the VAD pre-pass:
5.2 ASR per session, at list
whisper-1/gpt-4o-transcribe, $0.36/hrgpt-4o-mini-transcribe, $0.18/hr5.3 LLM per session, at list (assumption: ~60 000 prompt + ~8 000 completion tokens — unmeasured until one production
session_usagerow exists)5.4 Reproducing $0.95, and where it moves
$0.95 = ~$0.75 ASR + ~$0.20 LLM, and × 4.33 sessions/month = $4.11 (weekly play). The ASR term reproduces as 4.9 h at $0.15/hr (Universal-2) — the $0.95 was priced on AssemblyAI's older tier. At the current Pro rate the same session is $1.23 and the customer-month $5.33, 30% above the figure the recommendation rests on.
The LLM choice is not a cost decision at this token volume. The whole spread between LLMs is smaller than the spread between two VAD assumptions; choose the LLM on the four metrics, and put cost discipline on the ASR side.
5.5 Whole stacks, per customer-month at weekly play
whisper-1+ GPT-55.6 What self-hosted actually costs
The synthetic run transcribed 1 162 s of kept audio in 95 s (12.2× real time), so scenario C's 8 127 kept seconds take ≈ 11 minutes of GPU; at 350 W that is 0.065 kWh ≈ $0.02 at $0.30/kWh. The LLM leg measured 30–49 s of RTX 3090 time per five-minute fixture and adds well under a cent. The crossover: managed at $5.33/month buys 17.8 kWh — a 24.7 W average draw. A GPU that exists only for Quest Board and idles 719 hours a month will not beat that; for the actual audience, whose GPU exists anyway, the marginal cost is the 11 minutes.
Full evaluation, part 2 of 2 (sections 6–10).
6. Privacy and retention
6.1 Retention defaults, per provider
/v1/audio/transcriptionsis ZDR-eligibleaudio_tempand the database hold — governed by the product, kept until GM approval, cascaded on deletionThe AssemblyAI disagreement is not a nitpick. If AssemblyAI is ever adopted, an explicit post-transcription DELETE is part of the adapter, not a follow-up.
6.2 Promotional pricing that costs your audio
Deepgram's widely-quoted rate is 50% off conditional on enrolment in the Model Improvement Partnership Program, under which audio is retained and used to improve their models. For this product that is not a discount, it is a sale. The $0.258/hr in §3.1 is the opted-out rate. AssemblyAI's keyterm boosting is free during beta; a commitment now prices the feature at zero and may be repricing later.
6.3 What ZDR actually requires
OpenAI — an approval process per organisation (the transcription endpoint is eligible). Anthropic — an enterprise arrangement. AssemblyAI — Streaming only. Deepgram — enterprise. None is available to a self-hoster on a credit card. With any managed ASR provider, the table's audio is on someone else's disk for up to 30 days, and on one candidate possibly indefinitely.
6.4 Two gaps, named
Groq's terms were not researched, and Groq is the cheapest candidate by an order of magnitude. Gemini's terms were not researched either; free and paid tiers have historically differed, so a trial on a free key is not evidence about the paid one.
7. Recommendation, and the counterargument
7.1 Recommendation
Do not commit to a managed provider in this milestone. Keep the bundled WhisperX server and a self-hosted LLM as the defaults, and make managed providers a per-campaign opt-in — which is what #128's BYO-AI builds. Three reasons about evidence rather than preference: no managed candidate's accuracy on this corpus is known, while the self-hosted path has measured numbers with zero invented proper nouns; the recommendation's own economics have moved ($0.95 was priced on the older $0.15/hr tier; the same session is $1.23 today, and the VAD assumption is over twice the only measured figure, so the model's error bars exceed the differences it is deciding between); and every managed candidate puts the table's audio on a third party's disk for up to 30 days — indefinitely on one reading of AssemblyAI's docs — which is the exact thing this audience self-hosts to avoid.
If a managed leg is nonetheless needed for a hosted tier, the defensible starting pair is AssemblyAI Universal-3.5 Pro (per-second billing, keyterms, a real async contract the adapter interface anticipates) and Claude Haiku 4.5 rather than Sonnet 5 — at 60k/8k tokens the LLM is under a tenth of the bill and should follow the four metrics, not the price list. Groq is the cost floor by a wide margin and should be reconsidered the moment its retention terms are read.
7.2 The counterargument, made properly
A GPU is the real barrier to adoption, and $5.33 a customer-month is not. Every managed stack lands between $0.83 and $8.32 per customer-month at list, inside any plausible subscription, and removes the hardest prerequisite the product has. Evaluating providers only against people who already own a 3090 is evaluating for the audience the product already has.
The self-hosted accuracy evidence is thinner than it looks. 3.73% WER is TTS audio on perfectly isolated tracks, and its own README calls it a tripwire. 72.2% marker detection means more than a quarter of a session's invented proper nouns are lost — the single most product-damaging error mode for a fantasy campaign — and nobody has measured whether a managed keyterm-boosting provider recovers them.
"Self-hosted is free" is false in the case that matters. Managed at $5.33/month equals a 25 W average draw; a box bought for this purpose loses on electricity alone before hardware amortisation.
7.3 What measurement would change the answer, per unit of effort
session_usagerow — replaces the token and VAD assumptions with measurements.8. What this evaluation found in the code
ASR_HOURLY_USDis empty, and it is keyed on the adapter name, which is a wire shape: one"openai"entry would price OpenAI ($0.36), Groq ($0.04) and a self-hosted compatible server ($0) identically. The rate must key on host and model.reminder_tasks.pyrecords the configured provider string; an operator on "Detect from the URL" with a Groq URL storesNone, andusage_servicetreats""as self-hosted. Recordprovider.capabilities.nameinstead.claude-sonnet-5, noclaude-opus-5, nogemini;PRICING_AS_OF = 2026-05-01.--liveconformance cannot reach the managed ASR adapter —LiveWhisperCasehardwiresLocalWhisperProvider; needsQB_CONFORMANCE_ASR_PROVIDERrouted through the registry.asr_leg.pypasses noasr_provider=/asr_model=/vocabulary=; the prerequisite for measurement 2 above.9. Open questions and the exact next runs
Commands (from
webapp/backend, inside the backend image): managed LLM —QB_EVAL_PROVIDER=anthropic QB_EVAL_ENDPOINT=https://api.anthropic.com/v1/messages QB_EVAL_MODEL=claude-sonnet-5 QB_EVAL_CONTEXT_TOKENS=200000 QB_EVAL_API_KEY=… python -m evals --fixtures evals/fixtures/private --provider live --record(and the OpenAI equivalent withgpt-5, 128000); self-hosted —QB_EVAL_PROVIDER=llamacpp QB_EVAL_ENDPOINT=http://<gpu-host>:8080/v1/chat/completions QB_EVAL_MODEL=qwen3.5 QB_EVAL_CONTEXT_TOKENS=32768 python -m evals --fixtures evals/fixtures/private --provider live --record; ASR conformance —QB_CONFORMANCE_ASR_URL=… QB_CONFORMANCE_API_KEY=… make conformance-live; the synthetic ASR leg —QB_SYNTH_ASR_ENDPOINT=http://<whisper-host>:8091 python -m evals.synthetic_session asr-leg --assembly full --record, and the same with the 24 screened markers as vocabulary (the zero-key experiment that should happen first).10. Sources (gathered 2026-09-05; [V] vendor, [3P] aggregator)
ASR pricing: [V] assemblyai.com/blog/speech-recognition-cost; [3P] costbench.com/software/ai-transcription-apis/assemblyai/; [3P] convertaudiototext.com/blog/deepgram-nova-3-explained; [3P] diyai.io/ai-tools/speech-to-text/openai-whisper-api-pricing-2026/; [3P] cloudzero.com/blog/openai-pricing/; [3P] cloudzero.com/blog/groq-pricing/; [3P] openrouter.ai/openai/whisper-large-v3-turbo.
LLM pricing (all third-party): cloudzero.com/blog/claude-pricing/; benchlm.ai/anthropic/api-pricing; pricepertoken.com; morphllm.com/openai-api-pricing; finout.io/blog/anthropic-api-pricing; cloudzero.com/blog/gemini-pricing/; morphllm.com/gemini-api-pricing; benchlm.ai/google/api-pricing.
Retention (all vendor): support.assemblyai.com/articles/2240096256; assemblyai.com/docs/faq/what-is-your-data-retention-policy; developers.deepgram.com/docs/the-deepgram-model-improvement-partnership-program; deepgram.com/privacy; developers.openai.com/api/docs/guides/your-data; platform.claude.com/docs/en/manage-claude/api-and-data-retention.
Measured in this repository:
evals/synthetic_session/README.md(#433/#497);evals/fixtures/synthetic_session/manifest.json(§3.5 derivation);evals/results/history.jsonl(#353/#354 runs);app/services/ai_pricing.py,app/services/usage_service.py,app/providers/*,tests/conformance/cases.py.Experiment 1 (vocabulary bias) is done — PR #505, full numbers on #355.
Marker detection on the self-hosted default goes from 72.2 % with no vocabulary to 96.3 % with the session's own markers as the list, and 81.5 % with those markers diluted in a realistic 100-term campaign vocabulary. WER unchanged, zero fabricated names, zero markers leaking from the other assembly, zero run-to-run spread.
What this changes in the evaluation:
--asr-provider/--asr-model/--vocabularyflags are there so the comparison is like-for-like.Still open on this issue: the managed cells that need API keys, and the LLM-on-ASR-transcript leg (leg 3), which is still run by hand.
Moved from v4.2.0 to v4.6.0 Release Readiness so the v4.2.0 milestone can close. The decision this issue was opened to support is made and written up (self-hosted stays the default; managed providers are a per-campaign opt-in), and the abstraction, harness and vocabulary measurement that make the remaining cells fillable have all shipped. What is left is filling the managed-provider cells, which needs paid API keys and is a spend decision rather than engineering, plus the leg-3 automation tracked on #506. Both belong with the release-readiness pass, where the numbers get re-checked against list prices anyway.