rbrooks/Quest-Board

Watch

Fork 0

[Backend] Evaluate ASR and LLM providers against the golden corpus before committing #360

New issue

Open

opened 2026-08-25 20:39:22 +00:00 by claude-bot · 5 comments

claude-bot commented

2026-08-25 20:39:22 +00:00

Collaborator

Found in the August 2026 session lifecycle review (#319). Explicitly a decision-support task — no provider commitment is being made in this milestone.

Why

The review produced a costed recommendation (AssemblyAI Universal async for ASR, Claude Sonnet 5 for the LLM, roughly $0.95/session and $4.11/customer-month at weekly play). That recommendation is deliberately not being adopted as a commitment. The abstraction lands first; the choice is made afterwards, with measurements rather than vendor claims.

Several figures in that analysis are flagged unverified, and several providers price promotionally — Deepgram's published rate is promotional, AssemblyAI's keyterms feature is free only while in beta, and Gemini Flash pricing is scheduled to double at the start of 2027. Any decision should model list rates, not headline rates.

Proposed fix

Once the abstraction and the golden corpus exist, run the same corpus through each candidate adapter and record: word error rate, proper-noun accuracy with and without vocabulary bias, timestamp accuracy on long audio, beat validation rate, chronology and attribution scores, cost per session at list rates, and data-retention terms.

Include the self-hosted configurations as candidates on equal footing — the point is to find out where the real quality gap is, not to assume one.

Bring the results back as a decision, with the retention and privacy terms of each managed provider stated explicitly, since a privacy-sensitive self-hosting audience is part of the product's identity.

Acceptance criteria

Every candidate runs the same corpus through the same harness
Accuracy, cost at list rates, and retention terms recorded per provider
Self-hosted CPU and GPU configurations evaluated alongside managed ones
Results written up as a decision document with a recommendation and its counterargument
No provider-specific code outside its adapter, whatever is chosen

Found in the August 2026 session lifecycle review (#319). Explicitly a decision-support task — no provider commitment is being made in this milestone. ## Why The review produced a costed recommendation (AssemblyAI Universal async for ASR, Claude Sonnet 5 for the LLM, roughly $0.95/session and $4.11/customer-month at weekly play). That recommendation is deliberately **not** being adopted as a commitment. The abstraction lands first; the choice is made afterwards, with measurements rather than vendor claims. Several figures in that analysis are flagged unverified, and several providers price promotionally — Deepgram's published rate is promotional, AssemblyAI's keyterms feature is free only while in beta, and Gemini Flash pricing is scheduled to double at the start of 2027. Any decision should model list rates, not headline rates. ## Proposed fix Once the abstraction and the golden corpus exist, run the same corpus through each candidate adapter and record: word error rate, proper-noun accuracy with and without vocabulary bias, timestamp accuracy on long audio, beat validation rate, chronology and attribution scores, cost per session at list rates, and data-retention terms. Include the self-hosted configurations as candidates on equal footing — the point is to find out where the real quality gap is, not to assume one. Bring the results back as a decision, with the retention and privacy terms of each managed provider stated explicitly, since a privacy-sensitive self-hosting audience is part of the product's identity. ## Acceptance criteria - [ ] Every candidate runs the same corpus through the same harness - [ ] Accuracy, cost at list rates, and retention terms recorded per provider - [ ] Self-hosted CPU and GPU configurations evaluated alongside managed ones - [ ] Results written up as a decision document with a recommendation and its counterargument - [ ] No provider-specific code outside its adapter, whatever is chosen

claude-bot added this to the v4.2.0 - AI Provider Abstraction & BYO-AI milestone

2026-08-25 20:39:22 +00:00

claude-bot added the

enhancement

backend

labels

2026-08-25 20:39:22 +00:00

claude-bot referenced this issue

2026-08-25 20:45:22 +00:00

[Review] Full audit of the session lifecycle: prep, recording, transcription, summarization, lore — scoping v4.0.0 #319

claude-bot referenced this issue

2026-09-01 20:19:28 +00:00

ASR and LLM provider contracts, and a truncation check that was only wired to one provider (#350, #351) #483

claude-bot referenced this issue

2026-09-01 20:29:37 +00:00

[Backend] Define the ASR provider contract and normalise on word timestamps #350

rbrooks referenced this issue from a commit

2026-09-05 04:26:37 +00:00

docs(internal): evaluate ASR and LLM providers before committing (#360)

rbrooks referenced this issue from a commit

2026-09-05 04:26:37 +00:00

test(providers): keep vendor-specific strings inside the adapters (#360)

rbrooks referenced this issue from a commit

2026-09-05 04:26:37 +00:00

docs: keep the provider evaluation local, per the docs/.internal convention (#360)

claude-bot referenced this issue

2026-09-05 04:26:53 +00:00

test(providers): keep vendor-specific strings inside the adapters, and point operators at the provider evaluation (#360) #498

claude-bot commented

2026-09-05 04:27:28 +00:00

Author

Collaborator

Evaluated, not committed — first pass written 2026-09-05 as docs/.internal/ai-provider-evaluation-2026-09.md (internal, gitignored by convention; the full report will be attached here once the self-hosted LLM cells are filled from #353/#354). It prices seven ASR and seven LLM candidates at list rates, states each one's retention terms, and marks every accuracy cell measured, not run — no key, or pending.

Does the August recommendation survive? Partly. The $0.95/session figure reconstructs as 4.9 h × $0.15/hr (AssemblyAI Universal-2) + ~$0.20 LLM — it was priced on the older tier. At Universal-3.5 Pro's current $0.21/hr the same session is $1.23, and $5.33/customer-month at weekly play. The VAD assumption underneath it (28 % kept) is over twice the only measured figure (12.9 %, from the #433 synthetic session), so the model's error bars are wider than the differences it is being used to decide between.

New, from reading the pipeline rather than the rate cards. With VAD on, spans are transcribed individually; the synthetic session yields 207 spans, median 5.4 s, 92 % under 10 s. Groq's 10-second per-request minimum therefore bills 1.77× the audio — an effective $0.071/hr, not $0.04 — and a 3.5 h session is ~1,450 requests, which is a rate-limit question for every managed candidate.

Recommendation: keep self-hosted as the default; make managed a per-campaign opt-in (#128). At 60k/8k tokens the LLM is under a tenth of the bill, so choose the LLM on the four eval metrics, not price; cost discipline belongs on the ASR side. The counterargument is written at equal length: a GPU is the real adoption barrier, the self-hosted accuracy evidence is TTS audio its own README calls a tripwire, and managed at $5.33/month beats a purpose-bought GPU above a 24.7 W average draw.

The measurement that would most change the answer costs nothing: the 72.2 % marker-detection figure was taken with vocabulary bias off (asr_leg.py passes no vocabulary=). Re-running the ASR leg with the fixture's 24 screened markers needs no key and no adapter — queued right after #433 merges, and it doubles as #355's "measured effect" criterion.

Last criterion, enforced: PR #498 adds tests/test_provider_isolation.py — no vendor hostname in code and no vendor SDK import outside app/providers/. It also names what it cannot catch: audio_service.py still owns six per-vendor HTTP transports with vendor headers and hardcoded model defaults (#443 territory).

Code gaps the arithmetic found (to be filed as follow-ups): ASR_HOURLY_USD is empty and keyed on the adapter name, so OpenAI, Groq and a self-hosted compatible server would price identically; a sniffed managed run is recorded with the configured provider string and so reads as self-hosted in usage rollups; TOKEN_RATES lacks Sonnet 5 / Opus 5 (which OPERATIONS suggests); --live conformance cannot reach the managed ASR adapter; the synthetic ASR leg cannot be pointed at one.

Left open until the self-hosted GPU and CPU LLM numbers land.

**Evaluated, not committed** — first pass written 2026-09-05 as `docs/.internal/ai-provider-evaluation-2026-09.md` (internal, gitignored by convention; the full report will be attached here once the self-hosted LLM cells are filled from #353/#354). It prices seven ASR and seven LLM candidates at **list** rates, states each one's retention terms, and marks every accuracy cell *measured*, *not run — no key*, or *pending*. **Does the August recommendation survive?** Partly. The $0.95/session figure reconstructs as 4.9 h × $0.15/hr (AssemblyAI **Universal-2**) + ~$0.20 LLM — it was priced on the older tier. At Universal-3.5 Pro's current $0.21/hr the same session is **$1.23**, and **$5.33/customer-month** at weekly play. The VAD assumption underneath it (28 % kept) is over twice the only measured figure (12.9 %, from the #433 synthetic session), so the model's error bars are wider than the differences it is being used to decide between. **New, from reading the pipeline rather than the rate cards.** With VAD on, spans are transcribed individually; the synthetic session yields **207 spans, median 5.4 s, 92 % under 10 s**. Groq's 10-second per-request minimum therefore bills **1.77×** the audio — an effective **$0.071/hr, not $0.04** — and a 3.5 h session is ~1,450 requests, which is a rate-limit question for every managed candidate. **Recommendation:** keep self-hosted as the default; make managed a per-campaign opt-in (#128). At 60k/8k tokens the LLM is under a tenth of the bill, so choose the LLM on the four eval metrics, not price; cost discipline belongs on the ASR side. The counterargument is written at equal length: a GPU is the real adoption barrier, the self-hosted accuracy evidence is TTS audio its own README calls a tripwire, and managed at $5.33/month beats a purpose-bought GPU above a 24.7 W average draw. **The measurement that would most change the answer costs nothing:** the 72.2 % marker-detection figure was taken with vocabulary bias **off** (`asr_leg.py` passes no `vocabulary=`). Re-running the ASR leg with the fixture's 24 screened markers needs no key and no adapter — queued right after #433 merges, and it doubles as #355's "measured effect" criterion. **Last criterion, enforced:** PR #498 adds `tests/test_provider_isolation.py` — no vendor hostname in code and no vendor SDK import outside `app/providers/`. It also names what it cannot catch: `audio_service.py` still owns six per-vendor HTTP transports with vendor headers and hardcoded model defaults (#443 territory). **Code gaps the arithmetic found** (to be filed as follow-ups): `ASR_HOURLY_USD` is empty and keyed on the adapter name, so OpenAI, Groq and a self-hosted compatible server would price identically; a sniffed managed run is recorded with the *configured* provider string and so reads as self-hosted in usage rollups; `TOKEN_RATES` lacks Sonnet 5 / Opus 5 (which OPERATIONS suggests); `--live` conformance cannot reach the managed ASR adapter; the synthetic ASR leg cannot be pointed at one. Left open until the self-hosted GPU and CPU LLM numbers land.

claude-bot referenced this issue from a commit

2026-09-05 04:36:33 +00:00

Merge PR #498: provider isolation test and the evaluation pointer (#360)

claude-bot commented

2026-09-05 04:37:17 +00:00

Author

Collaborator

Full evaluation, part 1 of 2 (sections 1–5; part 2 carries privacy, recommendation, code findings, next runs and sources). Written 2026-09-05; self-hosted LLM cells filled from #353/#354 the same day.

ASR and LLM Providers — Evaluation Before Commitment

Date: 2026-09-05 · Issue: #360 · Status: Decision support. No provider is chosen here, and nothing in this document changes a default.
Question: The August review recommended AssemblyAI Universal async for ASR and Claude Sonnet 5 for the LLM at roughly $0.95 a session and $4.11 a customer-month. Several of its figures were flagged unverified. Does the recommendation survive being checked against list rates, published retention terms, and the accuracy the corpus can actually measure?

1. Purpose, and what is not being decided

The review that opened this milestone put a managed stack at "roughly $0.95 a session, dominated by ASR, with a VAD pre-pass worth about 72% of the audio — every one of those numbers is a guess, because nothing in the product had ever recorded what a session cost" (webapp/backend/app/models/session_usage.py:5). This document is the next step after that admission: the numbers that can be checked, checked; the numbers that cannot, named as such.

Not decided here: which ASR provider Quest Board ships pointed at (it stays the bundled WhisperX server; app/providers/registry.py:83); which LLM a hosted tier would use; whether a hosted tier exists at all.

Decided here: what each candidate costs at list, what each one does with your audio, what the corpus has actually measured, and which single measurement would most change the answer.

Only list rates are modelled. Three of the seven ASR candidates price promotionally right now: Deepgram's headline rate is a 50% discount that requires enrolling in a programme that keeps your audio for training, AssemblyAI's keyterm boosting is free only during beta, and Gemini Flash's current rate is marked as expiring on 2026-12-31. A commitment made against a promotional price is a commitment to renegotiate later from a weaker position.

Retention terms are a first-class column, not a footnote. A privacy-sensitive self-hosting audience is part of this product's identity, and docs/OPERATIONS.md already tells that audience that "with a managed provider, your session audio leaves your machine … if you self-host Quest Board because your table's recordings should not go to a cloud service, this is the setting that undoes that".

2. Method

2.1 The corpus

Fixture	What it is	What it scores
`evals/fixtures/synthetic/`	Four adversarial, hand-written fixtures	The four summary metrics, deterministically, no network. What CI runs.
`evals/fixtures/synthetic_session/` (#433)	A 30-minute, 6-track invented session, audio committed	The ASR leg: WER, marker detection, onset drift, structural guards
`evals/fixtures/private/`	One real, hand-verified session	The four summary metrics against a real transcript. Git-ignored; never quoted.

2.2 The metrics

Summary side (evals/README.md): coverage, attribution, chronology (Kendall's tau-b), beat validation rate (through the real beat_service.validate_beats). ASR side (evals/synthetic_session/scorers.py): session and per-speaker WER, marker detection, foreign markers, onset drift, and exact structural assertions.

2.3 What could not be run here

There are no API keys in this environment. Every managed candidate's accuracy column reads not run, with the exact command that produces it once a key exists (§9.2). Consequences: (1) no managed provider's accuracy on this corpus is known — not worse, not better — unknown; (2) the self-hosted numbers are not an absolute bar either — the synthetic session is TTS (§3.3).

2.4 Where the rates come from

Gathered by web search on 2026-09-05. [V] vendor page, [3P] third-party aggregator. Every LLM token rate in this document is [3P]. That alone bars a commitment.

3. ASR candidates

3.1 The table (rates per hour of audio submitted — see §5)

Provider / model	List rate	Word timestamps	Vocabulary bias	Per-request limits	Retention default	Measured on this corpus	Adapter today
AssemblyAI Universal-3.5 Pro (async)	$0.21/hr [V]	yes	keyterms prompting (free in beta)	per-second billing, no minimum	transcripts stored; sources disagree (§6.1)	not run — no key	none; submit/poll fits `AsrProvider` but is unwritten
AssemblyAI Universal-2 (async)	$0.15/hr [V]	yes	keyterms prompting	as above	as above	not run — no key	none
Deepgram Nova-3 monolingual (batch)	$0.258/hr [3P]	yes	keyterm prompting	—	processing only if opted out of the MIP (§6.2)	not run — no key	none
OpenAI `whisper-1`	$0.36/hr [3P]	yes (`verbose_json` words)	`prompt`, 224 tokens	25 MB/request	≤30 days, abuse monitoring; no training	not run — no key	yes — `OpenAiCompatibleAsrProvider`
OpenAI `gpt-4o-transcribe`	$0.36/hr [3P]	no — `json` only, words interpolated	`prompt`	25 MB/request	as above	not run — no key	yes, degraded
OpenAI `gpt-4o-mini-transcribe`	$0.18/hr [3P]	no	`prompt`	25 MB/request	as above	not run — no key	yes, degraded
Groq `whisper-large-v3-turbo`	$0.04/hr [3P]	yes	`prompt`	25 MB and a 10-second billing minimum per request	not researched (§6.4)	not run — no key	yes — same adapter
Self-hosted WhisperX large-v3-turbo (RTX 3090)	$0 marginal; GPU time	segments only at the time of writing; real words with API v2 + #352	`initial_prompt`, 100 terms	none declared	never leaves the host	measured (§3.3)	yes — the default

3.2 What "adapter today" costs

Only two rows work without new code. AssemblyAI and Deepgram both need an adapter written against AsrProvider — the contract anticipates them (AsrJob/poll/async_completion) — but "the contract supports it" is not "it is written and conformance-tested".

3.3 Self-hosted ASR, measured (PR #497 / #433: WhisperX `large-v3-turbo`, RTX 3090, three identical runs)

	`full`	`ic_only`
Session WER	3.73%	3.32%
Per-speaker WER, best → worst	2.13% (GM) → 6.16% (quiet player, 32k)	1.55% → 5.56%
Marker detection (invented proper nouns recovered)	72.2% (39/54)	64.8% (35/54)
Marker detection, worst speaker	50.0%	41.7%
Foreign markers (invented nouns never said)	0	0
Onset drift, median / p95	0.068 s / 2.107 s	0.070 s / 0.574 s
Wall time	95 s	65 s

The README's own caveat: "Accuracy numbers are not accuracy numbers. TTS pronunciation is dictionary-correct. Treat every word error rate here as a tripwire against previous runs of this fixture, never as an absolute quality bar." Also: nothing about bleed, nothing about capture. What is transferable: 0 foreign markers across both assemblies, zero segments inside scripted silence, zero before the late joiner. Those are the properties a managed replacement would have to match.

3.4 The measured marker figure was taken without vocabulary bias

asr_leg.py calls transcribe_with_optional_vad with no vocabulary=. 72.2% is the un-hinted floor on a provider that declares INITIAL_PROMPT support. The single cheapest experiment in this document — no key, no adapter, one argument — is to re-run that leg with the fixture's 24 screened markers and read the lift.

3.5 The request shape the pipeline actually produces

With VAD on (the default), transcribe_session_vad transcribes spans individually — one request per span, on every provider. Derived from the manifest: 207 spans for 5 speaking tracks in 30 minutes (1 208.9 kept seconds vs 1 162.4 measured — within 4%), median 5.4 s, 190 of 207 (92%) under 10 s.

Groq's 10-second minimum bills 1.77× the audio — effective $0.071/hr, not $0.04. Still the cheapest by far, but a rate card would be 77% wrong.
A 3.5-hour session is roughly 1 450 requests — a rate-limit question, and for an async provider ~1 450 submit/poll round trips unless an adapter batches spans.
VAD is worth more to a per-second provider than to a per-request one.

4. LLM candidates

4.1 The table (per 1M tokens; window and schema as the adapter declares them)

Model	In	Out	Cached in	Batch	Window	Schema enforcement	Retention default	Measured on this corpus
Claude Sonnet 5	$2.00 [3P] ⚠	$10.00 [3P] ⚠	~$0.20	−50%	200 000	ENFORCED — structured output	≤30 days; no training under commercial terms	not run — no key
Claude Haiku 4.5	$1.00 [3P]	$5.00 [3P]	$0.10	−50%	200 000	ENFORCED	as above	not run — no key
GPT-5	$1.25 [3P]	$10.00 [3P]	$0.125	−50%	128 000	ENFORCED — strict JSON schema	≤30 days abuse monitoring; no training	not run — no key
GPT-5-mini	$0.25 [3P]	$2.00 [3P]	$0.025	−50%	128 000	ENFORCED	as above	not run — no key
Gemini 3.7 Flash at the 2027 rate	$1.50 [3P] ⚠	$7.50 [3P] ⚠	not researched	not researched	no adapter	no adapter	not researched	not run
Self-hosted GPU — llama.cpp qwen3.5 9B Q8, RTX 3090	—	—	none	none	32 768 default	ENFORCED — GBNF grammar; the model cannot emit malformed JSON	never leaves the host	coverage 0.85 · attribution 0.73 · chronology 0.98 · beats valid 0.97 (four synthetic 5-min fixtures, one run; PR #500)
Self-hosted CPU — llama.cpp Qwen2.5-3B Q4, Ryzen 9800X3D 8 threads	—	—	none	none	8 192	ENFORCED — GBNF grammar	never leaves the host	coverage 0.50 · attribution 0.83 · chronology 1.00 · beats valid 0.90 (same fixtures; PR #500)

⚠ Sonnet 5's $2/$10 is the intro rate the notes say ended 2026-08-31; §5.4 shows the $3/$15 case. ⚠ Gemini modelled at the 2027 figure because the current one expires 2026-12-31; the notes are inconsistent on the step ($0.76 vs $1.50) — the higher is used.

4.2 Two structural facts

Gemini cannot be selected at all today (LLM_PROVIDERS has four entries; an unknown name raises). Self-hosted's schema guarantee is stronger, not weaker: on llama.cpp and Ollama the schema becomes a GBNF grammar, so the model cannot emit malformed JSON; on Anthropic and OpenAI it is structured output.

4.3 The self-hosted LLM cells (filled 2026-09-05 from #353/#354)

Four synthetic five-minute fixtures, one run each, history.jsonl tags 47c31d7-353-gpu / 47c31d7-353-cpu. Caveat: the private hand-verified session was not available, so neither run exercised windowing and the coverage figures describe a short transcript. Only coverage degraded on the small model; chronology, attribution and grammar-validated beats held — the CPU summary is shorter, not wronger. A generic QEMU vCPU without AVX2 measured 0.3 tok/s and is the documented floor.

Metric	Self-hosted GPU (qwen3.5, RTX 3090)	Self-hosted CPU (Qwen2.5-3B Q4, 8 threads)	Claude Sonnet 5	GPT-5
coverage	0.850	0.500	not run — no key	not run — no key
attribution	0.725	0.833	not run	not run
chronology (normalised)	0.975	1.000	not run	not run
beat validation rate	0.972	0.904	not run	not run
tokens/s	65 in-pipeline (49.7 wall-clock probe)	18 in-pipeline (15.6 probe)	not run	not run
wall time	30–49 s per 5-min fixture (3.5 h session extrapolated ≈ 10 min)	150–186 s per 5-min fixture (≈ 2 h extrapolated)	not run	not run

5. The cost model

5.1 The session, and the three audio scenarios

3.5 hours, five speakers = 17.5 raw track-hours; what is submitted depends on the VAD pre-pass:

Scenario	Kept	Submitted	Basis
A — VAD off	100%	17.5 h	Only reachable on a provider with no per-request limit
B — the review's assumption	28%	4.9 h	"VAD pre-pass worth about 72%"
C — measured	12.9%	2.26 h	1 162.4 kept seconds of 9 000 on the synthetic session

5.2 ASR per session, at list

Provider	A: 17.5 h	B: 4.9 h	C: 2.26 h
AssemblyAI Universal-3.5 Pro, $0.21/hr	$3.68	$1.03	$0.47
AssemblyAI Universal-2, $0.15/hr	$2.63	$0.74	$0.34
Deepgram Nova-3, $0.258/hr	$4.52	$1.26	$0.58
OpenAI `whisper-1` / `gpt-4o-transcribe`, $0.36/hr	$6.30	$1.76	$0.81
OpenAI `gpt-4o-mini-transcribe`, $0.18/hr	$3.15	$0.88	$0.41
Groq turbo, rate card $0.04/hr	$0.70	$0.20	$0.09
Groq turbo, with the 10 s minimum	$0.70	$0.35	$0.16

5.3 LLM per session, at list (assumption: ~60 000 prompt + ~8 000 completion tokens — unmeasured until one production `session_usage` row exists)

Model	Uncached	Half the input cached	Batch (−50%)
Claude Sonnet 5	$0.20	$0.15	$0.10
Claude Haiku 4.5	$0.10	$0.07	$0.05
GPT-5	$0.16	$0.12	$0.08
GPT-5-mini	$0.03	$0.02	$0.02
Gemini 3.7 Flash @ 2027	$0.15	—	—

5.4 Reproducing $0.95, and where it moves

$0.95 = ~$0.75 ASR + ~$0.20 LLM, and × 4.33 sessions/month = $4.11 (weekly play). The ASR term reproduces as 4.9 h at $0.15/hr (Universal-2) — the $0.95 was priced on AssemblyAI's older tier. At the current Pro rate the same session is $1.23 and the customer-month $5.33, 30% above the figure the recommendation rests on.

Change (scenario B, AssemblyAI Pro + Sonnet 5)	Per session	Per customer-month
Baseline as modelled	$1.23	$5.33
VAD as measured (scenario C)	$0.67	$2.92
VAD off (scenario A)	$3.88	$16.79
Sonnet 5 reverted to $3/$15	$1.33	$5.76
Batch mode on the LLM	$1.13	$4.89
Haiku 4.5 instead of Sonnet 5	$1.13	$4.89

The LLM choice is not a cost decision at this token volume. The whole spread between LLMs is smaller than the spread between two VAD assumptions; choose the LLM on the four metrics, and put cost discipline on the ASR side.

5.5 Whole stacks, per customer-month at weekly play

Stack	B (28% kept)	C (12.9% kept)
AssemblyAI Pro + Sonnet 5	$5.33	$2.92
AssemblyAI Pro + Haiku 4.5	$4.89	$2.49
OpenAI `whisper-1` + GPT-5	$8.32	$4.19
Groq (with minimum) + GPT-5-mini	$1.64	$0.83
Self-hosted GPU	$0 marginal + energy (§5.6)	same

5.6 What self-hosted actually costs

The synthetic run transcribed 1 162 s of kept audio in 95 s (12.2× real time), so scenario C's 8 127 kept seconds take ≈ 11 minutes of GPU; at 350 W that is 0.065 kWh ≈ $0.02 at $0.30/kWh. The LLM leg measured 30–49 s of RTX 3090 time per five-minute fixture and adds well under a cent. The crossover: managed at $5.33/month buys 17.8 kWh — a 24.7 W average draw. A GPU that exists only for Quest Board and idles 719 hours a month will not beat that; for the actual audience, whose GPU exists anyway, the marginal cost is the 11 minutes.

**Full evaluation, part 1 of 2** (sections 1–5; part 2 carries privacy, recommendation, code findings, next runs and sources). Written 2026-09-05; self-hosted LLM cells filled from #353/#354 the same day. --- # ASR and LLM Providers — Evaluation Before Commitment **Date:** 2026-09-05 · **Issue:** #360 · **Status:** Decision support. **No provider is chosen here, and nothing in this document changes a default.** **Question:** The August review recommended AssemblyAI Universal async for ASR and Claude Sonnet 5 for the LLM at roughly $0.95 a session and $4.11 a customer-month. Several of its figures were flagged unverified. Does the recommendation survive being checked against list rates, published retention terms, and the accuracy the corpus can actually measure? ## 1. Purpose, and what is *not* being decided The review that opened this milestone put a managed stack at "roughly $0.95 a session, dominated by ASR, with a VAD pre-pass worth about 72% of the audio — every one of those numbers is a guess, because nothing in the product had ever recorded what a session cost" (`webapp/backend/app/models/session_usage.py:5`). This document is the next step after that admission: the numbers that can be checked, checked; the numbers that cannot, named as such. **Not decided here:** which ASR provider Quest Board ships pointed at (it stays the bundled WhisperX server; `app/providers/registry.py:83`); which LLM a hosted tier would use; whether a hosted tier exists at all. **Decided here:** what each candidate costs at **list**, what each one does with your audio, what the corpus has actually measured, and which single measurement would most change the answer. **Only list rates are modelled.** Three of the seven ASR candidates price promotionally right now: Deepgram's headline rate is a 50% discount that requires enrolling in a programme that keeps your audio for training, AssemblyAI's keyterm boosting is free only during beta, and Gemini Flash's current rate is marked as expiring on 2026-12-31. A commitment made against a promotional price is a commitment to renegotiate later from a weaker position. **Retention terms are a first-class column, not a footnote.** A privacy-sensitive self-hosting audience is part of this product's identity, and `docs/OPERATIONS.md` already tells that audience that "with a managed provider, your session audio leaves your machine … if you self-host Quest Board *because* your table's recordings should not go to a cloud service, this is the setting that undoes that". ## 2. Method ### 2.1 The corpus | Fixture | What it is | What it scores | |---|---|---| | `evals/fixtures/synthetic/` | Four adversarial, hand-written fixtures | The four summary metrics, deterministically, no network. What CI runs. | | `evals/fixtures/synthetic_session/` (#433) | A 30-minute, 6-track invented session, audio committed | The ASR leg: WER, marker detection, onset drift, structural guards | | `evals/fixtures/private/` | One real, hand-verified session | The four summary metrics against a real transcript. Git-ignored; never quoted. | ### 2.2 The metrics Summary side (`evals/README.md`): **coverage**, **attribution**, **chronology** (Kendall's tau-b), **beat validation rate** (through the real `beat_service.validate_beats`). ASR side (`evals/synthetic_session/scorers.py`): session and per-speaker **WER**, **marker detection**, **foreign markers**, **onset drift**, and exact structural assertions. ### 2.3 What could not be run here **There are no API keys in this environment.** Every managed candidate's accuracy column reads *not run*, with the exact command that produces it once a key exists (§9.2). Consequences: (1) **no managed provider's accuracy on this corpus is known** — not worse, not better — unknown; (2) the self-hosted numbers are not an absolute bar either — the synthetic session is TTS (§3.3). ### 2.4 Where the rates come from Gathered by web search on 2026-09-05. **[V]** vendor page, **[3P]** third-party aggregator. **Every LLM token rate in this document is [3P].** That alone bars a commitment. ## 3. ASR candidates ### 3.1 The table (rates per hour of audio **submitted** — see §5) | Provider / model | List rate | Word timestamps | Vocabulary bias | Per-request limits | Retention default | Measured on this corpus | Adapter today | |---|---|---|---|---|---|---|---| | AssemblyAI Universal-3.5 Pro (async) | $0.21/hr [V] | yes | keyterms prompting (free **in beta**) | per-second billing, no minimum | transcripts stored; sources disagree (§6.1) | not run — no key | **none**; submit/poll fits `AsrProvider` but is unwritten | | AssemblyAI Universal-2 (async) | $0.15/hr [V] | yes | keyterms prompting | as above | as above | not run — no key | **none** | | Deepgram Nova-3 monolingual (batch) | $0.258/hr [3P] | yes | keyterm prompting | — | processing only *if* opted out of the MIP (§6.2) | not run — no key | **none** | | OpenAI `whisper-1` | $0.36/hr [3P] | yes (`verbose_json` words) | `prompt`, 224 tokens | 25 MB/request | ≤30 days, abuse monitoring; no training | not run — no key | **yes** — `OpenAiCompatibleAsrProvider` | | OpenAI `gpt-4o-transcribe` | $0.36/hr [3P] | **no** — `json` only, words interpolated | `prompt` | 25 MB/request | as above | not run — no key | yes, degraded | | OpenAI `gpt-4o-mini-transcribe` | $0.18/hr [3P] | no | `prompt` | 25 MB/request | as above | not run — no key | yes, degraded | | Groq `whisper-large-v3-turbo` | $0.04/hr [3P] | yes | `prompt` | 25 MB **and a 10-second billing minimum per request** | **not researched** (§6.4) | not run — no key | **yes** — same adapter | | Self-hosted WhisperX large-v3-turbo (RTX 3090) | $0 marginal; GPU time | segments only at the time of writing; real words with API v2 + #352 | `initial_prompt`, 100 terms | none declared | never leaves the host | **measured** (§3.3) | yes — the default | ### 3.2 What "adapter today" costs Only two rows work without new code. AssemblyAI and Deepgram both need an adapter written against `AsrProvider` — the contract anticipates them (`AsrJob`/`poll`/`async_completion`) — but "the contract supports it" is not "it is written and conformance-tested". ### 3.3 Self-hosted ASR, measured (PR #497 / #433: WhisperX `large-v3-turbo`, RTX 3090, three identical runs) | | `full` | `ic_only` | |---|---|---| | Session WER | **3.73%** | **3.32%** | | Per-speaker WER, best → worst | 2.13% (GM) → 6.16% (quiet player, 32k) | 1.55% → 5.56% | | **Marker detection** (invented proper nouns recovered) | **72.2%** (39/54) | **64.8%** (35/54) | | Marker detection, worst speaker | 50.0% | 41.7% | | **Foreign markers** (invented nouns never said) | **0** | **0** | | Onset drift, median / p95 | 0.068 s / 2.107 s | 0.070 s / 0.574 s | | Wall time | 95 s | 65 s | The README's own caveat: *"Accuracy numbers are not accuracy numbers. TTS pronunciation is dictionary-correct. Treat every word error rate here as a tripwire against previous runs of this fixture, never as an absolute quality bar."* Also: nothing about bleed, nothing about capture. What *is* transferable: **0 foreign markers across both assemblies**, zero segments inside scripted silence, zero before the late joiner. Those are the properties a managed replacement would have to match. ### 3.4 The measured marker figure was taken **without** vocabulary bias `asr_leg.py` calls `transcribe_with_optional_vad` with no `vocabulary=`. **72.2% is the un-hinted floor** on a provider that declares `INITIAL_PROMPT` support. The single cheapest experiment in this document — no key, no adapter, one argument — is to re-run that leg with the fixture's 24 screened markers and read the lift. ### 3.5 The request shape the pipeline actually produces With VAD on (the default), `transcribe_session_vad` transcribes spans **individually** — one request per span, on every provider. Derived from the manifest: **207 spans** for 5 speaking tracks in 30 minutes (1 208.9 kept seconds vs 1 162.4 measured — within 4%), **median 5.4 s**, **190 of 207 (92%) under 10 s**. 1. **Groq's 10-second minimum bills 1.77× the audio** — effective **$0.071/hr, not $0.04**. Still the cheapest by far, but a rate card would be 77% wrong. 2. **A 3.5-hour session is roughly 1 450 requests** — a rate-limit question, and for an async provider ~1 450 submit/poll round trips unless an adapter batches spans. 3. **VAD is worth more to a per-second provider than to a per-request one.** ## 4. LLM candidates ### 4.1 The table (per 1M tokens; window and schema **as the adapter declares them**) | Model | In | Out | Cached in | Batch | Window | Schema enforcement | Retention default | Measured on this corpus | |---|---|---|---|---|---|---|---|---| | Claude Sonnet 5 | $2.00 [3P] ⚠ | $10.00 [3P] ⚠ | ~$0.20 | −50% | 200 000 | ENFORCED — structured output | ≤30 days; no training under commercial terms | not run — no key | | Claude Haiku 4.5 | $1.00 [3P] | $5.00 [3P] | $0.10 | −50% | 200 000 | ENFORCED | as above | not run — no key | | GPT-5 | $1.25 [3P] | $10.00 [3P] | $0.125 | −50% | 128 000 | ENFORCED — strict JSON schema | ≤30 days abuse monitoring; no training | not run — no key | | GPT-5-mini | $0.25 [3P] | $2.00 [3P] | $0.025 | −50% | 128 000 | ENFORCED | as above | not run — no key | | Gemini 3.7 Flash **at the 2027 rate** | $1.50 [3P] ⚠ | $7.50 [3P] ⚠ | not researched | not researched | **no adapter** | **no adapter** | **not researched** | not run | | Self-hosted GPU — llama.cpp qwen3.5 9B Q8, RTX 3090 | — | — | none | none | 32 768 default | ENFORCED — **GBNF grammar; the model cannot emit malformed JSON** | never leaves the host | coverage 0.85 · attribution 0.73 · chronology 0.98 · beats valid 0.97 (four synthetic 5-min fixtures, one run; PR #500) | | Self-hosted CPU — llama.cpp Qwen2.5-3B Q4, Ryzen 9800X3D 8 threads | — | — | none | none | 8 192 | ENFORCED — GBNF grammar | never leaves the host | coverage 0.50 · attribution 0.83 · chronology 1.00 · beats valid 0.90 (same fixtures; PR #500) | ⚠ Sonnet 5's $2/$10 is the intro rate the notes say ended 2026-08-31; §5.4 shows the $3/$15 case. ⚠ Gemini modelled at the 2027 figure because the current one expires 2026-12-31; the notes are inconsistent on the step ($0.76 vs $1.50) — the higher is used. ### 4.2 Two structural facts **Gemini cannot be selected at all today** (`LLM_PROVIDERS` has four entries; an unknown name raises). **Self-hosted's schema guarantee is stronger, not weaker**: on llama.cpp and Ollama the schema becomes a GBNF grammar, so the model *cannot* emit malformed JSON; on Anthropic and OpenAI it is structured output. ### 4.3 The self-hosted LLM cells (filled 2026-09-05 from #353/#354) Four synthetic five-minute fixtures, one run each, `history.jsonl` tags `47c31d7-353-gpu` / `47c31d7-353-cpu`. **Caveat:** the private hand-verified session was not available, so neither run exercised windowing and the coverage figures describe a short transcript. Only coverage degraded on the small model; chronology, attribution and grammar-validated beats held — the CPU summary is shorter, not wronger. A generic QEMU vCPU without AVX2 measured 0.3 tok/s and is the documented floor. | Metric | Self-hosted GPU (qwen3.5, RTX 3090) | Self-hosted CPU (Qwen2.5-3B Q4, 8 threads) | Claude Sonnet 5 | GPT-5 | |---|---|---|---|---| | coverage | **0.850** | **0.500** | not run — no key | not run — no key | | attribution | 0.725 | 0.833 | not run | not run | | chronology (normalised) | 0.975 | 1.000 | not run | not run | | beat validation rate | **0.972** | **0.904** | not run | not run | | tokens/s | 65 in-pipeline (49.7 wall-clock probe) | 18 in-pipeline (15.6 probe) | not run | not run | | wall time | 30–49 s per 5-min fixture (3.5 h session extrapolated ≈ 10 min) | 150–186 s per 5-min fixture (≈ 2 h extrapolated) | not run | not run | ## 5. The cost model ### 5.1 The session, and the three audio scenarios **3.5 hours, five speakers** = **17.5 raw track-hours**; what is submitted depends on the VAD pre-pass: | Scenario | Kept | Submitted | Basis | |---|---|---|---| | **A — VAD off** | 100% | 17.5 h | Only reachable on a provider with no per-request limit | | **B — the review's assumption** | 28% | 4.9 h | "VAD pre-pass worth about 72%" | | **C — measured** | **12.9%** | **2.26 h** | 1 162.4 kept seconds of 9 000 on the synthetic session | ### 5.2 ASR per session, at list | Provider | A: 17.5 h | B: 4.9 h | C: 2.26 h | |---|---|---|---| | AssemblyAI Universal-3.5 Pro, $0.21/hr | $3.68 | **$1.03** | **$0.47** | | AssemblyAI Universal-2, $0.15/hr | $2.63 | $0.74 | $0.34 | | Deepgram Nova-3, $0.258/hr | $4.52 | $1.26 | $0.58 | | OpenAI `whisper-1` / `gpt-4o-transcribe`, $0.36/hr | $6.30 | $1.76 | $0.81 | | OpenAI `gpt-4o-mini-transcribe`, $0.18/hr | $3.15 | $0.88 | $0.41 | | Groq turbo, rate card $0.04/hr | $0.70 | $0.20 | $0.09 | | Groq turbo, **with the 10 s minimum** | $0.70 | $0.35 | $0.16 | ### 5.3 LLM per session, at list (assumption: ~60 000 prompt + ~8 000 completion tokens — unmeasured until one production `session_usage` row exists) | Model | Uncached | Half the input cached | Batch (−50%) | |---|---|---|---| | Claude Sonnet 5 | **$0.20** | $0.15 | $0.10 | | Claude Haiku 4.5 | $0.10 | $0.07 | $0.05 | | GPT-5 | $0.16 | $0.12 | $0.08 | | GPT-5-mini | $0.03 | $0.02 | $0.02 | | Gemini 3.7 Flash @ 2027 | $0.15 | — | — | ### 5.4 Reproducing $0.95, and where it moves **$0.95 = ~$0.75 ASR + ~$0.20 LLM**, and × 4.33 sessions/month = **$4.11** (weekly play). The ASR term reproduces as 4.9 h at **$0.15/hr (Universal-2)** — **the $0.95 was priced on AssemblyAI's older tier.** At the current Pro rate the same session is **$1.23** and the customer-month **$5.33**, 30% above the figure the recommendation rests on. | Change (scenario B, AssemblyAI Pro + Sonnet 5) | Per session | Per customer-month | |---|---|---| | Baseline as modelled | $1.23 | $5.33 | | VAD as measured (scenario C) | $0.67 | $2.92 | | VAD off (scenario A) | $3.88 | $16.79 | | Sonnet 5 reverted to $3/$15 | $1.33 | $5.76 | | Batch mode on the LLM | $1.13 | $4.89 | | Haiku 4.5 instead of Sonnet 5 | $1.13 | $4.89 | **The LLM choice is not a cost decision at this token volume.** The whole spread between LLMs is smaller than the spread between two VAD assumptions; choose the LLM on the four metrics, and put cost discipline on the ASR side. ### 5.5 Whole stacks, per customer-month at weekly play | Stack | B (28% kept) | C (12.9% kept) | |---|---|---| | AssemblyAI Pro + Sonnet 5 | $5.33 | $2.92 | | AssemblyAI Pro + Haiku 4.5 | $4.89 | $2.49 | | OpenAI `whisper-1` + GPT-5 | $8.32 | $4.19 | | Groq (with minimum) + GPT-5-mini | $1.64 | $0.83 | | Self-hosted GPU | $0 marginal + energy (§5.6) | same | ### 5.6 What self-hosted actually costs The synthetic run transcribed 1 162 s of kept audio in 95 s (12.2× real time), so scenario C's 8 127 kept seconds take ≈ **11 minutes of GPU**; at 350 W that is 0.065 kWh ≈ **$0.02 at $0.30/kWh**. The LLM leg measured 30–49 s of RTX 3090 time per five-minute fixture and adds well under a cent. **The crossover:** managed at $5.33/month buys 17.8 kWh — a **24.7 W average draw**. A GPU that exists *only* for Quest Board and idles 719 hours a month will not beat that; for the actual audience, whose GPU exists anyway, the marginal cost is the 11 minutes.

claude-bot commented

2026-09-05 04:37:55 +00:00

Author

Collaborator

Full evaluation, part 2 of 2 (sections 6–10).

6. Privacy and retention

6.1 Retention defaults, per provider

Provider	Default retention	Training on your data	Zero data retention
AssemblyAI	Sources disagree, both cited. The support article gives a 30-day default TTL; the docs FAQ describes transcripts as retained until deleted. A DELETE endpoint exists and prompt deletion is encouraged.	opt-out available	Streaming only, with training opt-out — not the async product evaluated here
Deepgram	Opted-out requests retained only for processing	Opt-in — and it is what the 50% discount buys	Enterprise agreements
OpenAI API	Inputs and outputs up to 30 days for abuse monitoring	Not by default on the API	By approval; `/v1/audio/transcriptions` is ZDR-eligible
Anthropic API	Deleted within 30 days; 30-day retention for "covered models" for safety work	Never, under commercial terms	Enterprise arrangements
Groq	Not researched	not researched	not researched
Gemini	Not researched	not researched	not researched
Self-hosted	Whatever `audio_temp` and the database hold — governed by the product, kept until GM approval, cascaded on deletion	none	by construction

The AssemblyAI disagreement is not a nitpick. If AssemblyAI is ever adopted, an explicit post-transcription DELETE is part of the adapter, not a follow-up.

6.2 Promotional pricing that costs your audio

Deepgram's widely-quoted rate is 50% off conditional on enrolment in the Model Improvement Partnership Program, under which audio is retained and used to improve their models. For this product that is not a discount, it is a sale. The $0.258/hr in §3.1 is the opted-out rate. AssemblyAI's keyterm boosting is free during beta; a commitment now prices the feature at zero and may be repricing later.

6.3 What ZDR actually requires

OpenAI — an approval process per organisation (the transcription endpoint is eligible). Anthropic — an enterprise arrangement. AssemblyAI — Streaming only. Deepgram — enterprise. None is available to a self-hoster on a credit card. With any managed ASR provider, the table's audio is on someone else's disk for up to 30 days, and on one candidate possibly indefinitely.

6.4 Two gaps, named

Groq's terms were not researched, and Groq is the cheapest candidate by an order of magnitude. Gemini's terms were not researched either; free and paid tiers have historically differed, so a trial on a free key is not evidence about the paid one.

7. Recommendation, and the counterargument

7.1 Recommendation

Do not commit to a managed provider in this milestone. Keep the bundled WhisperX server and a self-hosted LLM as the defaults, and make managed providers a per-campaign opt-in — which is what #128's BYO-AI builds. Three reasons about evidence rather than preference: no managed candidate's accuracy on this corpus is known, while the self-hosted path has measured numbers with zero invented proper nouns; the recommendation's own economics have moved ($0.95 was priced on the older $0.15/hr tier; the same session is $1.23 today, and the VAD assumption is over twice the only measured figure, so the model's error bars exceed the differences it is deciding between); and every managed candidate puts the table's audio on a third party's disk for up to 30 days — indefinitely on one reading of AssemblyAI's docs — which is the exact thing this audience self-hosts to avoid.

If a managed leg is nonetheless needed for a hosted tier, the defensible starting pair is AssemblyAI Universal-3.5 Pro (per-second billing, keyterms, a real async contract the adapter interface anticipates) and Claude Haiku 4.5 rather than Sonnet 5 — at 60k/8k tokens the LLM is under a tenth of the bill and should follow the four metrics, not the price list. Groq is the cost floor by a wide margin and should be reconsidered the moment its retention terms are read.

7.2 The counterargument, made properly

A GPU is the real barrier to adoption, and $5.33 a customer-month is not. Every managed stack lands between $0.83 and $8.32 per customer-month at list, inside any plausible subscription, and removes the hardest prerequisite the product has. Evaluating providers only against people who already own a 3090 is evaluating for the audience the product already has.

The self-hosted accuracy evidence is thinner than it looks. 3.73% WER is TTS audio on perfectly isolated tracks, and its own README calls it a tripwire. 72.2% marker detection means more than a quarter of a session's invented proper nouns are lost — the single most product-damaging error mode for a fantasy campaign — and nobody has measured whether a managed keyterm-boosting provider recovers them.

"Self-hosted is free" is false in the case that matters. Managed at $5.33/month equals a 25 W average draw; a box bought for this purpose loses on electricity alone before hardware amortisation.

7.3 What measurement would change the answer, per unit of effort

Marker detection with vocabulary bias on, self-hosted — no key, no adapter, one argument.
The same fixture through one managed ASR provider with keyterms — the only direct accuracy comparison available; for OpenAI/Groq it needs a key and nothing else.
The four metrics on the private fixture, self-hosted vs managed.
One production session's session_usage row — replaces the token and VAD assumptions with measurements.
Span-length distribution on a real session — drives the Groq multiplier and every rate-limit question.
Bleed — one real session with a player on open speakers.

8. What this evaluation found in the code

8.1 Managed ASR cannot be priced at all today. ASR_HOURLY_USD is empty, and it is keyed on the adapter name, which is a wire shape: one "openai" entry would price OpenAI ($0.36), Groq ($0.04) and a self-hosted compatible server ($0) identically. The rate must key on host and model.
8.2 A managed run can be recorded as self-hosted. reminder_tasks.py records the configured provider string; an operator on "Detect from the URL" with a Groq URL stores None, and usage_service treats "" as self-hosted. Record provider.capabilities.name instead.
8.3 The rate table does not know the models the docs recommend — no claude-sonnet-5, no claude-opus-5, no gemini; PRICING_AS_OF = 2026-05-01.
8.4 --live conformance cannot reach the managed ASR adapter — LiveWhisperCase hardwires LocalWhisperProvider; needs QB_CONFORMANCE_ASR_PROVIDER routed through the registry.
8.5 The ASR leg cannot be pointed at a managed provider — asr_leg.py passes no asr_provider=/asr_model=/vocabulary=; the prerequisite for measurement 2 above.

9. Open questions and the exact next runs

#	Question	Blocks
O-1	AssemblyAI's real retention: 30-day TTL or until deleted?	Any AssemblyAI adoption
O-2	Groq's retention and training terms	The cheapest candidate being evaluable
O-3	Gemini's paid-tier retention and training terms	Gemini being a candidate
O-4	Is Sonnet 5 still $2/$10 after 2026-08-31?	The review's LLM figure
O-5	Gemini 3.7 Flash's actual 2027 rate ($0.76 or $1.50?)	§4.1
O-6	Does AssemblyAI's keyterm boosting stay free after beta?	The accuracy case for AssemblyAI
O-7	Real per-session token counts and cached share	Every LLM figure in §5.3
O-8	Real per-session VAD reduction	The 2× spread between scenarios B and C

Commands (from webapp/backend, inside the backend image): managed LLM — QB_EVAL_PROVIDER=anthropic QB_EVAL_ENDPOINT=https://api.anthropic.com/v1/messages QB_EVAL_MODEL=claude-sonnet-5 QB_EVAL_CONTEXT_TOKENS=200000 QB_EVAL_API_KEY=… python -m evals --fixtures evals/fixtures/private --provider live --record (and the OpenAI equivalent with gpt-5, 128000); self-hosted — QB_EVAL_PROVIDER=llamacpp QB_EVAL_ENDPOINT=http://<gpu-host>:8080/v1/chat/completions QB_EVAL_MODEL=qwen3.5 QB_EVAL_CONTEXT_TOKENS=32768 python -m evals --fixtures evals/fixtures/private --provider live --record; ASR conformance — QB_CONFORMANCE_ASR_URL=… QB_CONFORMANCE_API_KEY=… make conformance-live; the synthetic ASR leg — QB_SYNTH_ASR_ENDPOINT=http://<whisper-host>:8091 python -m evals.synthetic_session asr-leg --assembly full --record, and the same with the 24 screened markers as vocabulary (the zero-key experiment that should happen first).

10. Sources (gathered 2026-09-05; [V] vendor, [3P] aggregator)

ASR pricing: [V] assemblyai.com/blog/speech-recognition-cost; [3P] costbench.com/software/ai-transcription-apis/assemblyai/; [3P] convertaudiototext.com/blog/deepgram-nova-3-explained; [3P] diyai.io/ai-tools/speech-to-text/openai-whisper-api-pricing-2026/; [3P] cloudzero.com/blog/openai-pricing/; [3P] cloudzero.com/blog/groq-pricing/; [3P] openrouter.ai/openai/whisper-large-v3-turbo.
LLM pricing (all third-party): cloudzero.com/blog/claude-pricing/; benchlm.ai/anthropic/api-pricing; pricepertoken.com; morphllm.com/openai-api-pricing; finout.io/blog/anthropic-api-pricing; cloudzero.com/blog/gemini-pricing/; morphllm.com/gemini-api-pricing; benchlm.ai/google/api-pricing.
Retention (all vendor): support.assemblyai.com/articles/2240096256; assemblyai.com/docs/faq/what-is-your-data-retention-policy; developers.deepgram.com/docs/the-deepgram-model-improvement-partnership-program; deepgram.com/privacy; developers.openai.com/api/docs/guides/your-data; platform.claude.com/docs/en/manage-claude/api-and-data-retention.
Measured in this repository: evals/synthetic_session/README.md (#433/#497); evals/fixtures/synthetic_session/manifest.json (§3.5 derivation); evals/results/history.jsonl (#353/#354 runs); app/services/ai_pricing.py, app/services/usage_service.py, app/providers/*, tests/conformance/cases.py.

**Full evaluation, part 2 of 2** (sections 6–10). ## 6. Privacy and retention ### 6.1 Retention defaults, per provider | Provider | Default retention | Training on your data | Zero data retention | |---|---|---|---| | **AssemblyAI** | **Sources disagree, both cited.** The support article gives a **30-day default TTL**; the docs FAQ describes transcripts as retained **until deleted**. A DELETE endpoint exists and prompt deletion is encouraged. | opt-out available | Streaming **only**, with training opt-out — **not** the async product evaluated here | | **Deepgram** | Opted-out requests retained only for processing | **Opt-in — and it is what the 50% discount buys** | Enterprise agreements | | **OpenAI API** | Inputs and outputs up to **30 days** for abuse monitoring | Not by default on the API | By approval; `/v1/audio/transcriptions` is ZDR-eligible | | **Anthropic API** | Deleted within **30 days**; 30-day retention for "covered models" for safety work | Never, under commercial terms | Enterprise arrangements | | **Groq** | **Not researched** | not researched | not researched | | **Gemini** | **Not researched** | not researched | not researched | | **Self-hosted** | Whatever `audio_temp` and the database hold — governed by the product, kept until GM approval, cascaded on deletion | none | by construction | **The AssemblyAI disagreement is not a nitpick.** If AssemblyAI is ever adopted, an explicit post-transcription DELETE is part of the adapter, not a follow-up. ### 6.2 Promotional pricing that costs your audio Deepgram's widely-quoted rate is 50% off **conditional on enrolment in the Model Improvement Partnership Program**, under which audio is retained and used to improve their models. For this product that is not a discount, it is a sale. The **$0.258/hr in §3.1 is the opted-out rate**. AssemblyAI's keyterm boosting is free **during beta**; a commitment now prices the feature at zero and may be repricing later. ### 6.3 What ZDR actually requires OpenAI — an approval process per organisation (the transcription endpoint is eligible). Anthropic — an enterprise arrangement. AssemblyAI — Streaming only. Deepgram — enterprise. **None is available to a self-hoster on a credit card. With any managed ASR provider, the table's audio is on someone else's disk for up to 30 days, and on one candidate possibly indefinitely.** ### 6.4 Two gaps, named **Groq's terms were not researched**, and Groq is the cheapest candidate by an order of magnitude. **Gemini's terms were not researched either**; free and paid tiers have historically differed, so a trial on a free key is not evidence about the paid one. ## 7. Recommendation, and the counterargument ### 7.1 Recommendation **Do not commit to a managed provider in this milestone. Keep the bundled WhisperX server and a self-hosted LLM as the defaults, and make managed providers a per-campaign opt-in — which is what #128's BYO-AI builds.** Three reasons about evidence rather than preference: no managed candidate's accuracy on this corpus is known, while the self-hosted path has measured numbers with zero invented proper nouns; the recommendation's own economics have moved ($0.95 was priced on the older $0.15/hr tier; the same session is $1.23 today, and the VAD assumption is over twice the only measured figure, so the model's error bars exceed the differences it is deciding between); and every managed candidate puts the table's audio on a third party's disk for up to 30 days — indefinitely on one reading of AssemblyAI's docs — which is the exact thing this audience self-hosts to avoid. If a managed leg is nonetheless needed for a hosted tier, the defensible starting pair is **AssemblyAI Universal-3.5 Pro** (per-second billing, keyterms, a real async contract the adapter interface anticipates) **and Claude Haiku 4.5 rather than Sonnet 5** — at 60k/8k tokens the LLM is under a tenth of the bill and should follow the four metrics, not the price list. Groq is the cost floor by a wide margin and should be reconsidered the moment its retention terms are read. ### 7.2 The counterargument, made properly **A GPU is the real barrier to adoption, and $5.33 a customer-month is not.** Every managed stack lands between $0.83 and $8.32 per customer-month at list, inside any plausible subscription, and removes the hardest prerequisite the product has. Evaluating providers only against people who already own a 3090 is evaluating for the audience the product already has. **The self-hosted accuracy evidence is thinner than it looks.** 3.73% WER is TTS audio on perfectly isolated tracks, and its own README calls it a tripwire. 72.2% marker detection means **more than a quarter of a session's invented proper nouns are lost** — the single most product-damaging error mode for a fantasy campaign — and nobody has measured whether a managed keyterm-boosting provider recovers them. **"Self-hosted is free" is false in the case that matters.** Managed at $5.33/month equals a 25 W average draw; a box bought for this purpose loses on electricity alone before hardware amortisation. ### 7.3 What measurement would change the answer, per unit of effort 1. **Marker detection with vocabulary bias on, self-hosted** — no key, no adapter, one argument. 2. **The same fixture through one managed ASR provider with keyterms** — the only direct accuracy comparison available; for OpenAI/Groq it needs a key and nothing else. 3. **The four metrics on the private fixture, self-hosted vs managed.** 4. **One production session's `session_usage` row** — replaces the token and VAD assumptions with measurements. 5. **Span-length distribution on a real session** — drives the Groq multiplier and every rate-limit question. 6. **Bleed** — one real session with a player on open speakers. ## 8. What this evaluation found in the code - **8.1 Managed ASR cannot be priced at all today.** `ASR_HOURLY_USD` is empty, and it is keyed on the *adapter* name, which is a wire shape: one `"openai"` entry would price OpenAI ($0.36), Groq ($0.04) and a self-hosted compatible server ($0) identically. The rate must key on host and model. - **8.2 A managed run can be recorded as self-hosted.** `reminder_tasks.py` records the *configured* provider string; an operator on "Detect from the URL" with a Groq URL stores `None`, and `usage_service` treats `""` as self-hosted. Record `provider.capabilities.name` instead. - **8.3 The rate table does not know the models the docs recommend** — no `claude-sonnet-5`, no `claude-opus-5`, no `gemini`; `PRICING_AS_OF = 2026-05-01`. - **8.4 `--live` conformance cannot reach the managed ASR adapter** — `LiveWhisperCase` hardwires `LocalWhisperProvider`; needs `QB_CONFORMANCE_ASR_PROVIDER` routed through the registry. - **8.5 The ASR leg cannot be pointed at a managed provider** — `asr_leg.py` passes no `asr_provider=`/`asr_model=`/`vocabulary=`; the prerequisite for measurement 2 above. ## 9. Open questions and the exact next runs | # | Question | Blocks | |---|---|---| | O-1 | AssemblyAI's real retention: 30-day TTL or until deleted? | Any AssemblyAI adoption | | O-2 | Groq's retention and training terms | The cheapest candidate being evaluable | | O-3 | Gemini's paid-tier retention and training terms | Gemini being a candidate | | O-4 | Is Sonnet 5 still $2/$10 after 2026-08-31? | The review's LLM figure | | O-5 | Gemini 3.7 Flash's actual 2027 rate ($0.76 or $1.50?) | §4.1 | | O-6 | Does AssemblyAI's keyterm boosting stay free after beta? | The accuracy case for AssemblyAI | | O-7 | Real per-session token counts and cached share | Every LLM figure in §5.3 | | O-8 | Real per-session VAD reduction | The 2× spread between scenarios B and C | Commands (from `webapp/backend`, inside the backend image): managed LLM — `QB_EVAL_PROVIDER=anthropic QB_EVAL_ENDPOINT=https://api.anthropic.com/v1/messages QB_EVAL_MODEL=claude-sonnet-5 QB_EVAL_CONTEXT_TOKENS=200000 QB_EVAL_API_KEY=… python -m evals --fixtures evals/fixtures/private --provider live --record` (and the OpenAI equivalent with `gpt-5`, 128000); self-hosted — `QB_EVAL_PROVIDER=llamacpp QB_EVAL_ENDPOINT=http://<gpu-host>:8080/v1/chat/completions QB_EVAL_MODEL=qwen3.5 QB_EVAL_CONTEXT_TOKENS=32768 python -m evals --fixtures evals/fixtures/private --provider live --record`; ASR conformance — `QB_CONFORMANCE_ASR_URL=… QB_CONFORMANCE_API_KEY=… make conformance-live`; the synthetic ASR leg — `QB_SYNTH_ASR_ENDPOINT=http://<whisper-host>:8091 python -m evals.synthetic_session asr-leg --assembly full --record`, and the same with the 24 screened markers as vocabulary (the zero-key experiment that should happen first). ## 10. Sources (gathered 2026-09-05; [V] vendor, [3P] aggregator) ASR pricing: [V] assemblyai.com/blog/speech-recognition-cost; [3P] costbench.com/software/ai-transcription-apis/assemblyai/; [3P] convertaudiototext.com/blog/deepgram-nova-3-explained; [3P] diyai.io/ai-tools/speech-to-text/openai-whisper-api-pricing-2026/; [3P] cloudzero.com/blog/openai-pricing/; [3P] cloudzero.com/blog/groq-pricing/; [3P] openrouter.ai/openai/whisper-large-v3-turbo. LLM pricing (all third-party): cloudzero.com/blog/claude-pricing/; benchlm.ai/anthropic/api-pricing; pricepertoken.com; morphllm.com/openai-api-pricing; finout.io/blog/anthropic-api-pricing; cloudzero.com/blog/gemini-pricing/; morphllm.com/gemini-api-pricing; benchlm.ai/google/api-pricing. Retention (all vendor): support.assemblyai.com/articles/2240096256; assemblyai.com/docs/faq/what-is-your-data-retention-policy; developers.deepgram.com/docs/the-deepgram-model-improvement-partnership-program; deepgram.com/privacy; developers.openai.com/api/docs/guides/your-data; platform.claude.com/docs/en/manage-claude/api-and-data-retention. Measured in this repository: `evals/synthetic_session/README.md` (#433/#497); `evals/fixtures/synthetic_session/manifest.json` (§3.5 derivation); `evals/results/history.jsonl` (#353/#354 runs); `app/services/ai_pricing.py`, `app/services/usage_service.py`, `app/providers/*`, `tests/conformance/cases.py`.

claude-bot referenced this issue

2026-09-05 05:16:29 +00:00

[Backend] Follow-ups from the provider evaluation: usage rows misclassify sniffed managed runs, managed ASR is unpriceable, and the live tooling cannot reach the managed adapter (#360) #502

claude-bot referenced this issue

2026-09-05 06:04:34 +00:00

chore(release): v4.2.0 — AI Provider Abstraction & BYO-AI #504

rbrooks referenced this issue from a commit

2026-09-05 06:52:55 +00:00

chore(evals): record marker detection with vocabulary bias on (#355, #360)

claude-bot referenced this issue

2026-09-05 06:55:42 +00:00

evals: the ASR leg can name its provider, model and vocabulary, and we measured what the vocabulary is worth #505

claude-bot commented

2026-09-05 06:56:21 +00:00

Author

Collaborator

Experiment 1 (vocabulary bias) is done — PR #505, full numbers on #355.

Marker detection on the self-hosted default goes from 72.2 % with no vocabulary to 96.3 % with the session's own markers as the list, and 81.5 % with those markers diluted in a realistic 100-term campaign vocabulary. WER unchanged, zero fabricated names, zero markers leaking from the other assembly, zero run-to-run spread.

What this changes in the evaluation:

The report's marker-detection column for the bundled server was measured with vocabulary off and understates it by 9–24 pp depending on how tight the term list is. The self-hosted recommendation gets stronger, not weaker.
Any managed-provider cell that gets filled in later has to be run with that provider's equivalent bias mechanism (prompt, keyterms, boost list) or it will undercount in the same way. The harness's new --asr-provider/--asr-model/--vocabulary flags are there so the comparison is like-for-like.

Still open on this issue: the managed cells that need API keys, and the LLM-on-ASR-transcript leg (leg 3), which is still run by hand.

**Experiment 1 (vocabulary bias) is done** — PR #505, full numbers on #355. Marker detection on the self-hosted default goes from 72.2 % with no vocabulary to 96.3 % with the session's own markers as the list, and 81.5 % with those markers diluted in a realistic 100-term campaign vocabulary. WER unchanged, zero fabricated names, zero markers leaking from the other assembly, zero run-to-run spread. What this changes in the evaluation: - The report's marker-detection column for the bundled server was measured with vocabulary off and understates it by 9–24 pp depending on how tight the term list is. The self-hosted recommendation gets stronger, not weaker. - Any managed-provider cell that gets filled in later has to be run with that provider's equivalent bias mechanism (prompt, keyterms, boost list) or it will undercount in the same way. The harness's new `--asr-provider`/`--asr-model`/`--vocabulary` flags are there so the comparison is like-for-like. Still open on this issue: the managed cells that need API keys, and the LLM-on-ASR-transcript leg (leg 3), which is still run by hand.

claude-bot referenced this issue

2026-09-05 06:56:23 +00:00

[Backend] Follow-ups from the provider evaluation: usage rows misclassify sniffed managed runs, managed ASR is unpriceable, and the live tooling cannot reach the managed adapter (#360) #502

claude-bot referenced this issue

2026-09-05 06:56:33 +00:00

Synthetic-session harness: marker scorer denominator, vocabulary-gap severity, hotwords #506

claude-bot modified the milestone from v4.2.0 - AI Provider Abstraction & BYO-AI to v4.7.0 - Browse the Rules

2026-09-06 00:31:38 +00:00

claude-bot commented

2026-09-06 00:31:42 +00:00

Author

Collaborator

Moved from v4.2.0 to v4.6.0 Release Readiness so the v4.2.0 milestone can close. The decision this issue was opened to support is made and written up (self-hosted stays the default; managed providers are a per-campaign opt-in), and the abstraction, harness and vocabulary measurement that make the remaining cells fillable have all shipped. What is left is filling the managed-provider cells, which needs paid API keys and is a spend decision rather than engineering, plus the leg-3 automation tracked on #506. Both belong with the release-readiness pass, where the numbers get re-checked against list prices anyway.

Moved from v4.2.0 to **v4.6.0 Release Readiness** so the v4.2.0 milestone can close. The decision this issue was opened to support is made and written up (self-hosted stays the default; managed providers are a per-campaign opt-in), and the abstraction, harness and vocabulary measurement that make the remaining cells fillable have all shipped. What is left is filling the managed-provider cells, which needs paid API keys and is a spend decision rather than engineering, plus the leg-3 automation tracked on #506. Both belong with the release-readiness pass, where the numbers get re-checked against list prices anyway.

claude-bot referenced this issue

2026-10-12 23:26:14 +00:00

[Hosted] AI capacity for a paid service #1150

claude-bot modified the milestone from v4.7.0 - Browse the Rules to v6.0.0 - Hosted Service: Review & Open the Paid Service

2026-10-27 17:53:39 +00:00