[Backend] Follow-ups from the provider evaluation: usage rows misclassify sniffed managed runs, managed ASR is unpriceable, and the live tooling cannot reach the managed adapter (#360)

claude-bot commented

2026-09-05 05:16:29 +00:00

Collaborator

Five gaps found while doing the #360 arithmetic against the real modules (full detail in §8 of the evaluation posted on #360). Filed together because they are small, related, and two of them make the new usage panel (#357) wrong for exactly the campaigns #128 lets bring their own provider. Items 1–4 should land before v4.2.0 ships; item 5 after the #433 harness merges.

1. A sniffed managed run is recorded as self-hosted. reminder_tasks.py records asr_provider=(whisper_cfg.provider or None) — the configured string, not the resolved adapter. An operator who pasted a Groq URL and left the provider on "Detect from the URL" (which OPERATIONS tells them to do) stores None, and usage_service._is_self_hosted treats that as self-hosted (SELF_HOSTED_ASR = {"local_whisper", ""}). Their managed spend is reported as free compute time. Record the resolved provider.capabilities.name (and the same check for the LLM side: llm_cfg.provider vs llm_provider_for(target).capabilities.name). Test: a sniffed OpenAI-compatible endpoint yields a row classified managed.
2. Managed ASR cannot be priced. ASR_HOURLY_USD is empty, so estimate_asr_cost returns None for every provider including the adapter #496 shipped. Worse, it is keyed on the adapter name, which is a wire shape: one "openai" entry would price OpenAI ($0.36/hr), Groq ($0.04/hr) and a self-hosted compatible server ($0) identically. Key on host + model (the usage row already stores provider and model; add the host if needed), with list rates for api.openai.com (whisper-1/gpt-4o-transcribe $0.36/hr, gpt-4o-mini-transcribe $0.18/hr) and api.groq.com (whisper-large-v3-turbo $0.04/hr, and note the 10-second per-request minimum that bills ~1.77× on VAD-cut spans — either model it from the request count or state the estimate is a floor). Unknown host → None, marked estimate.
3. The rate table does not know the models the docs recommend. TOKEN_RATES has no claude-sonnet-5 ($2/$10 in, cached at 10 %), claude-opus-5 ($5/$25), gemini; OPERATIONS offers claude-opus-5 as an example and it produces no estimate. Add the current list rates for the Claude 5 family (Sonnet 5, Opus 5, Fable 5.1 $10/$50) and bump PRICING_AS_OF to the date they were read (2026-09-05, from aggregators — mark as such).
4. --live conformance cannot reach the managed ASR adapter. LiveWhisperCase.provider() hardwires LocalWhisperProvider; add QB_CONFORMANCE_ASR_PROVIDER (and QB_CONFORMANCE_ASR_MODEL) routed through resolve_asr_provider, mirroring QB_CONFORMANCE_LLM_PROVIDER.
5. The synthetic ASR leg cannot be pointed at a managed provider and passes no vocabulary. asr_leg.py calls transcribe_with_optional_vad without asr_provider=, asr_model= or vocabulary=. Thread them from environment variables (QB_SYNTH_ASR_PROVIDER, QB_SYNTH_ASR_MODEL, and a --vocabulary / --markers flag that passes the fixture's screened markers). This is the prerequisite for the zero-key experiment the evaluation ranks first: marker detection with vocabulary bias, which is also #355's unmet "measured effect" criterion. Do this after #433 merges.

Five gaps found while doing the #360 arithmetic against the real modules (full detail in §8 of the evaluation posted on #360). Filed together because they are small, related, and two of them make the new usage panel (#357) wrong for exactly the campaigns #128 lets bring their own provider. Items 1–4 should land before v4.2.0 ships; item 5 after the #433 harness merges. - [ ] **1. A sniffed managed run is recorded as self-hosted.** `reminder_tasks.py` records `asr_provider=(whisper_cfg.provider or None)` — the *configured* string, not the resolved adapter. An operator who pasted a Groq URL and left the provider on "Detect from the URL" (which OPERATIONS tells them to do) stores `None`, and `usage_service._is_self_hosted` treats that as self-hosted (`SELF_HOSTED_ASR = {"local_whisper", ""}`). Their managed spend is reported as free compute time. Record the resolved `provider.capabilities.name` (and the same check for the LLM side: `llm_cfg.provider` vs `llm_provider_for(target).capabilities.name`). Test: a sniffed OpenAI-compatible endpoint yields a row classified managed. - [ ] **2. Managed ASR cannot be priced.** `ASR_HOURLY_USD` is empty, so `estimate_asr_cost` returns `None` for every provider including the adapter #496 shipped. Worse, it is keyed on the adapter name, which is a wire shape: one `"openai"` entry would price OpenAI ($0.36/hr), Groq ($0.04/hr) and a self-hosted compatible server ($0) identically. Key on host + model (the usage row already stores provider and model; add the host if needed), with list rates for `api.openai.com` (`whisper-1`/`gpt-4o-transcribe` $0.36/hr, `gpt-4o-mini-transcribe` $0.18/hr) and `api.groq.com` (`whisper-large-v3-turbo` $0.04/hr, and note the 10-second per-request minimum that bills ~1.77× on VAD-cut spans — either model it from the request count or state the estimate is a floor). Unknown host → `None`, marked estimate. - [ ] **3. The rate table does not know the models the docs recommend.** `TOKEN_RATES` has no `claude-sonnet-5` ($2/$10 in, cached at 10 %), `claude-opus-5` ($5/$25), `gemini`; OPERATIONS offers `claude-opus-5` as an example and it produces no estimate. Add the current list rates for the Claude 5 family (Sonnet 5, Opus 5, Fable 5.1 $10/$50) and bump `PRICING_AS_OF` to the date they were read (2026-09-05, from aggregators — mark as such). - [ ] **4. `--live` conformance cannot reach the managed ASR adapter.** `LiveWhisperCase.provider()` hardwires `LocalWhisperProvider`; add `QB_CONFORMANCE_ASR_PROVIDER` (and `QB_CONFORMANCE_ASR_MODEL`) routed through `resolve_asr_provider`, mirroring `QB_CONFORMANCE_LLM_PROVIDER`. - [ ] **5. The synthetic ASR leg cannot be pointed at a managed provider and passes no vocabulary.** `asr_leg.py` calls `transcribe_with_optional_vad` without `asr_provider=`, `asr_model=` or `vocabulary=`. Thread them from environment variables (`QB_SYNTH_ASR_PROVIDER`, `QB_SYNTH_ASR_MODEL`, and a `--vocabulary` / `--markers` flag that passes the fixture's screened markers). This is the prerequisite for the zero-key experiment the evaluation ranks first: marker detection *with* vocabulary bias, which is also #355's unmet "measured effect" criterion. Do this after #433 merges.

claude-bot added the

backend

bug

labels

2026-09-05 05:16:42 +00:00

claude-bot added this to the v4.2.0 - AI Provider Abstraction & BYO-AI milestone

2026-09-05 05:16:43 +00:00

rbrooks referenced this issue from a commit

2026-09-05 05:41:26 +00:00

fix(usage): record the resolved provider, not the configured string (#502)

rbrooks referenced this issue from a commit

2026-09-05 05:41:26 +00:00

feat(pricing): price managed ASR by host and model, and know the Claude 5 family (#502)

rbrooks referenced this issue from a commit

2026-09-05 05:41:26 +00:00

test(conformance): --live can drive the managed ASR adapter (#502)

rbrooks referenced this issue from a commit

2026-09-05 05:41:26 +00:00

docs: what a run cost, which endpoint it cost it on, and how to drive live mode (#502)

rbrooks referenced this issue from a commit

2026-09-05 05:41:26 +00:00

db: chain the usage-identity columns behind #501's campaign settings (#502)

claude-bot referenced this issue

2026-09-05 05:41:44 +00:00

fix(usage): record the resolved provider, price managed ASR by host and model, and let live conformance reach the managed adapter (#502) #503

claude-bot referenced this issue from a commit

2026-09-05 05:50:29 +00:00

Merge PR #503: usage rows record the resolved provider; managed ASR priced by host and model; live conformance reaches the managed adapter (#502)

claude-bot commented

2026-09-05 05:51:02 +00:00

Author

Collaborator

Items 1–4 landed in PR #503 (merged 2026-09-05, CI green; 2213 backend tests).

1. Usage rows now record the resolved adapter for both sides via usage_service.resolve_asr_identity() / resolve_llm_identity() (the LLM side was already right on successful runs through record_llm_call; the ASR fallback and the pre-first-LLM-call failure path were the wrong ones). Migration e9fa0b1c2d3e adds asr_model, asr_endpoint_host, llm_endpoint_host; the ASR model goes through the adapter's own default resolution so a blank model on Groq is billed as whisper-large-v3-turbo.
2. ASR_HOURLY_USD is keyed {host: {model-prefix: rate}} with the shared longest-prefix rule; self-hosted is checked first so a vendor-looking hostname never gets a vendor's rate. Groq's 10-second minimum is documented as making the estimate a floor rather than modelled (the row has no request count); whisper-large-v3 omitted until its rate is confirmed.
3. TOKEN_RATES knows Sonnet 5, Opus 5, Fable 5.1 (and Sonnet 4.6); PRICING_AS_OF = 2026-09-05 with provenance stated.
4. QB_CONFORMANCE_ASR_PROVIDER / QB_CONFORMANCE_ASR_MODEL drive the managed adapter in --live mode; seven wiring tests run without --live so a mis-pointed harness cannot fail silently.

Item 5 (the synthetic ASR leg: provider, model and vocabulary threading, then the marker-detection-with-bias measurement) is queued behind PR #497.

Items 1–4 landed in **PR #503** (merged 2026-09-05, CI green; 2213 backend tests). - **1.** Usage rows now record the *resolved* adapter for both sides via `usage_service.resolve_asr_identity()` / `resolve_llm_identity()` (the LLM side was already right on successful runs through `record_llm_call`; the ASR fallback and the pre-first-LLM-call failure path were the wrong ones). Migration `e9fa0b1c2d3e` adds `asr_model`, `asr_endpoint_host`, `llm_endpoint_host`; the ASR model goes through the adapter's own default resolution so a blank model on Groq is billed as `whisper-large-v3-turbo`. - **2.** `ASR_HOURLY_USD` is keyed `{host: {model-prefix: rate}}` with the shared longest-prefix rule; self-hosted is checked first so a vendor-looking hostname never gets a vendor's rate. Groq's 10-second minimum is documented as making the estimate a floor rather than modelled (the row has no request count); `whisper-large-v3` omitted until its rate is confirmed. - **3.** `TOKEN_RATES` knows Sonnet 5, Opus 5, Fable 5.1 (and Sonnet 4.6); `PRICING_AS_OF` = 2026-09-05 with provenance stated. - **4.** `QB_CONFORMANCE_ASR_PROVIDER` / `QB_CONFORMANCE_ASR_MODEL` drive the managed adapter in `--live` mode; seven wiring tests run without `--live` so a mis-pointed harness cannot fail silently. Item 5 (the synthetic ASR leg: provider, model and vocabulary threading, then the marker-detection-with-bias measurement) is queued behind PR #497.

rbrooks referenced this issue from a commit

2026-09-05 06:52:55 +00:00

feat(evals): the ASR leg can name its provider, model and vocabulary (#502)

rbrooks referenced this issue from a commit

2026-09-05 06:52:55 +00:00

chore(evals): record marker detection with vocabulary bias on (#355, #360)

rbrooks referenced this issue from a commit

2026-09-05 06:52:55 +00:00

docs(evals): what vocabulary bias is worth, and what the bands describe (#355, #502)

claude-bot referenced this issue

2026-09-05 06:55:42 +00:00

evals: the ASR leg can name its provider, model and vocabulary, and we measured what the vocabulary is worth #505

claude-bot commented

2026-09-05 06:56:23 +00:00

Author

Collaborator

Item 5 (the ASR leg can name its provider, model and vocabulary) is in PR #505, auto-merging on green. That is the last item on this list; closing once it lands. The vocabulary measurement it enabled is written up on #355 and #360.

claude-bot commented

2026-09-05 07:11:39 +00:00

Author

Collaborator

All five items are merged: items 1–4 in PR #503 (shipped in v4.2.0), item 5 in PR #505 (evals only, no runtime change, rides along with the next deploy). Closing.

claude-bot closed this issue

2026-09-05 07:11:40 +00:00

claude-bot referenced this issue

2026-09-05 07:31:53 +00:00

[Backend] session_usage.asr_model is left empty while llm_model is populated (#357) #508

Rows
Columns

[Backend] Follow-ups from the provider evaluation: usage rows misclassify sniffed managed runs, managed ASR is unpriceable, and the live tooling cannot reach the managed adapter (#360) #502