[Backend] Follow-ups from the provider evaluation: usage rows misclassify sniffed managed runs, managed ASR is unpriceable, and the live tooling cannot reach the managed adapter (#360) #502
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Five gaps found while doing the #360 arithmetic against the real modules (full detail in §8 of the evaluation posted on #360). Filed together because they are small, related, and two of them make the new usage panel (#357) wrong for exactly the campaigns #128 lets bring their own provider. Items 1–4 should land before v4.2.0 ships; item 5 after the #433 harness merges.
reminder_tasks.pyrecordsasr_provider=(whisper_cfg.provider or None)— the configured string, not the resolved adapter. An operator who pasted a Groq URL and left the provider on "Detect from the URL" (which OPERATIONS tells them to do) storesNone, andusage_service._is_self_hostedtreats that as self-hosted (SELF_HOSTED_ASR = {"local_whisper", ""}). Their managed spend is reported as free compute time. Record the resolvedprovider.capabilities.name(and the same check for the LLM side:llm_cfg.providervsllm_provider_for(target).capabilities.name). Test: a sniffed OpenAI-compatible endpoint yields a row classified managed.ASR_HOURLY_USDis empty, soestimate_asr_costreturnsNonefor every provider including the adapter #496 shipped. Worse, it is keyed on the adapter name, which is a wire shape: one"openai"entry would price OpenAI ($0.36/hr), Groq ($0.04/hr) and a self-hosted compatible server ($0) identically. Key on host + model (the usage row already stores provider and model; add the host if needed), with list rates forapi.openai.com(whisper-1/gpt-4o-transcribe$0.36/hr,gpt-4o-mini-transcribe$0.18/hr) andapi.groq.com(whisper-large-v3-turbo$0.04/hr, and note the 10-second per-request minimum that bills ~1.77× on VAD-cut spans — either model it from the request count or state the estimate is a floor). Unknown host →None, marked estimate.TOKEN_RATEShas noclaude-sonnet-5($2/$10 in, cached at 10 %),claude-opus-5($5/$25),gemini; OPERATIONS offersclaude-opus-5as an example and it produces no estimate. Add the current list rates for the Claude 5 family (Sonnet 5, Opus 5, Fable 5.1 $10/$50) and bumpPRICING_AS_OFto the date they were read (2026-09-05, from aggregators — mark as such).--liveconformance cannot reach the managed ASR adapter.LiveWhisperCase.provider()hardwiresLocalWhisperProvider; addQB_CONFORMANCE_ASR_PROVIDER(andQB_CONFORMANCE_ASR_MODEL) routed throughresolve_asr_provider, mirroringQB_CONFORMANCE_LLM_PROVIDER.asr_leg.pycallstranscribe_with_optional_vadwithoutasr_provider=,asr_model=orvocabulary=. Thread them from environment variables (QB_SYNTH_ASR_PROVIDER,QB_SYNTH_ASR_MODEL, and a--vocabulary/--markersflag that passes the fixture's screened markers). This is the prerequisite for the zero-key experiment the evaluation ranks first: marker detection with vocabulary bias, which is also #355's unmet "measured effect" criterion. Do this after #433 merges.Items 1–4 landed in PR #503 (merged 2026-09-05, CI green; 2213 backend tests).
usage_service.resolve_asr_identity()/resolve_llm_identity()(the LLM side was already right on successful runs throughrecord_llm_call; the ASR fallback and the pre-first-LLM-call failure path were the wrong ones). Migratione9fa0b1c2d3eaddsasr_model,asr_endpoint_host,llm_endpoint_host; the ASR model goes through the adapter's own default resolution so a blank model on Groq is billed aswhisper-large-v3-turbo.ASR_HOURLY_USDis keyed{host: {model-prefix: rate}}with the shared longest-prefix rule; self-hosted is checked first so a vendor-looking hostname never gets a vendor's rate. Groq's 10-second minimum is documented as making the estimate a floor rather than modelled (the row has no request count);whisper-large-v3omitted until its rate is confirmed.TOKEN_RATESknows Sonnet 5, Opus 5, Fable 5.1 (and Sonnet 4.6);PRICING_AS_OF= 2026-09-05 with provenance stated.QB_CONFORMANCE_ASR_PROVIDER/QB_CONFORMANCE_ASR_MODELdrive the managed adapter in--livemode; seven wiring tests run without--liveso a mis-pointed harness cannot fail silently.Item 5 (the synthetic ASR leg: provider, model and vocabulary threading, then the marker-detection-with-bias measurement) is queued behind PR #497.
Item 5 (the ASR leg can name its provider, model and vocabulary) is in PR #505, auto-merging on green. That is the last item on this list; closing once it lands. The vocabulary measurement it enabled is written up on #355 and #360.
All five items are merged: items 1–4 in PR #503 (shipped in v4.2.0), item 5 in PR #505 (evals only, no runtime change, rides along with the next deploy). Closing.