rbrooks/Quest-Board

Watch

Fork 0

fix(usage): record the resolved provider, price managed ASR by host and model, and let live conformance reach the managed adapter (#502) #503

Merged

claude-bot merged 5 commits from fix/502-usage-pricing-live into main

2026-09-05 05:50:28 +00:00

claude-bot commented

2026-09-05 05:41:44 +00:00

Contributor

Refs #502 — items 1–4 (item 5, the synthetic ASR leg, follows the #433 harness merge).

What

Usage rows record the resolved adapter, not the configured string (item 1). record_llm_call already recorded provider.capabilities.name; only the two fallbacks were wrong — the ASR identity unconditionally, and the LLM identity on runs that died before their first LLM call, which is exactly the population that has already paid for ASR. New usage_service.resolve_asr_identity() / resolve_llm_identity() return (adapter name, model, host); the ASR model goes through the adapter's own default resolution so a blank model on Groq is billed as whisper-large-v3-turbo, which is what the row now says. Migration e9fa0b1c2d3e (after d8e9fa0b1c2d, one head, round-tripped): three nullable columns asr_model, asr_endpoint_host, llm_endpoint_host; nothing backfilled. Host is provider_slots.endpoint_host, the same string the concurrency limiter keys on, so there is one notion of "which box".
Managed ASR is priceable (item 2). ASR_HOURLY_USD is now {host: {model-prefix: rate}} with the same longest-prefix rule as the token table (so gpt-4o-mini-transcribe is not matched by gpt-4o-transcribe); estimate_asr_cost checks self-hosted first so a bundled server behind a vendor-looking hostname is never billed at that vendor's rate; subdomain matching mirrors the registry sniff so routing and pricing cannot disagree. Rates: api.openai.com whisper-1 / gpt-4o-transcribe $0.36/hr, gpt-4o-mini-transcribe $0.18/hr; api.groq.com whisper-large-v3-turbo $0.04/hr. Groq's 10-second per-request minimum is documented as making the estimate a floor (~1.77× on VAD-cut spans), not modelled — the row has no request count, and adding one changes a transcription signature; whisper-large-v3 is omitted because its rate could not be confirmed.
The rate table knows the Claude 5 family (item 3): Sonnet 5 $2/$10, Opus 5 $5/$25, Fable 5.1 $10/$50 (cached at 10 %), provenance stated as third-party aggregators read 2026-09-05; PRICING_AS_OF bumped. Longest-prefix keeps claude-sonnet-5 and claude-sonnet-4-5 apart — Sonnet 5 is the cheaper one, so the near-miss is expensive either way.
--live conformance can drive the managed ASR adapter (item 4): QB_CONFORMANCE_ASR_PROVIDER / QB_CONFORMANCE_ASR_MODEL routed through resolve_asr_provider; two named skips; no case's assertions changed; Makefile and DEVELOPMENT updated with a worked Groq example. Seven wiring tests run without --live so a mis-pointed harness cannot fail silently.

Rebased over #501 with no conflicts; the rebase created two Alembic heads, re-chained in its own commit.

Verification

Backend 2213 passed, 13 skipped; ruff clean; alembic heads → e9fa0b1c2d3e; version sync OK.

🤖 Generated with Claude Code

Refs #502 — items 1–4 (item 5, the synthetic ASR leg, follows the #433 harness merge). ## What - **Usage rows record the resolved adapter, not the configured string** (item 1). `record_llm_call` already recorded `provider.capabilities.name`; only the two fallbacks were wrong — the ASR identity unconditionally, and the LLM identity on runs that died before their first LLM call, which is exactly the population that has already paid for ASR. New `usage_service.resolve_asr_identity()` / `resolve_llm_identity()` return `(adapter name, model, host)`; the ASR model goes through the adapter's own default resolution so a blank model on Groq is billed as `whisper-large-v3-turbo`, which is what the row now says. **Migration `e9fa0b1c2d3e`** (after `d8e9fa0b1c2d`, one head, round-tripped): three nullable columns `asr_model`, `asr_endpoint_host`, `llm_endpoint_host`; nothing backfilled. Host is `provider_slots.endpoint_host`, the same string the concurrency limiter keys on, so there is one notion of "which box". - **Managed ASR is priceable** (item 2). `ASR_HOURLY_USD` is now `{host: {model-prefix: rate}}` with the same longest-prefix rule as the token table (so `gpt-4o-mini-transcribe` is not matched by `gpt-4o-transcribe`); `estimate_asr_cost` checks self-hosted first so a bundled server behind a vendor-looking hostname is never billed at that vendor's rate; subdomain matching mirrors the registry sniff so routing and pricing cannot disagree. Rates: `api.openai.com` whisper-1 / gpt-4o-transcribe $0.36/hr, gpt-4o-mini-transcribe $0.18/hr; `api.groq.com` whisper-large-v3-turbo $0.04/hr. **Groq's 10-second per-request minimum is documented as making the estimate a floor (~1.77× on VAD-cut spans), not modelled** — the row has no request count, and adding one changes a transcription signature; `whisper-large-v3` is omitted because its rate could not be confirmed. - **The rate table knows the Claude 5 family** (item 3): Sonnet 5 $2/$10, Opus 5 $5/$25, Fable 5.1 $10/$50 (cached at 10 %), provenance stated as third-party aggregators read 2026-09-05; `PRICING_AS_OF` bumped. Longest-prefix keeps `claude-sonnet-5` and `claude-sonnet-4-5` apart — Sonnet 5 is the cheaper one, so the near-miss is expensive either way. - **`--live` conformance can drive the managed ASR adapter** (item 4): `QB_CONFORMANCE_ASR_PROVIDER` / `QB_CONFORMANCE_ASR_MODEL` routed through `resolve_asr_provider`; two named skips; no case's assertions changed; Makefile and DEVELOPMENT updated with a worked Groq example. Seven wiring tests run without `--live` so a mis-pointed harness cannot fail silently. Rebased over #501 with no conflicts; the rebase created two Alembic heads, re-chained in its own commit. ## Verification - Backend **2213 passed, 13 skipped**; ruff clean; `alembic heads` → `e9fa0b1c2d3e`; version sync OK. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 5 commits

2026-09-05 05:41:44 +00:00

fix(usage): record the resolved provider, not the configured string (#502 ) 251c338ddd

Admin → AI Usage classified a run as self-hosted whenever `asr_provider` was
empty, and `asr_provider` held the *configured* provider string. "Detect from
the URL" is the default in Admin → Bot Settings and stores exactly that: an
empty string. So an operator who pasted an OpenAI or Groq endpoint and left the
select alone had every metered run filed as free compute — GPU seconds and no
dollars on the one page that exists to answer "what is this costing me".

The run knows better than the config does. `process_audio` already resolves the
ASR adapter for the concurrency cap, and `record_llm_call` has always recorded
`capabilities.name`; only the two fallbacks written into the usage row went on
repeating what the settings page stored. They now record what ran.

Resolution moves to `usage_service.resolve_asr_identity` /
`resolve_llm_identity` rather than into the task, for the reason the module
already gives: "does a run record the right numbers" should be a test against a
function, not a test that drives a whole Celery task with a fake Whisper.
Neither raises — a provider name that does not resolve is a misconfiguration
the run itself reports far better, and it keeps the configured string so the
typo is still named.

The adapter name alone cannot price anything, though: `openai` is a wire shape
that OpenAI, Groq and any compatible self-hosted server all wear. So the model
actually requested (after per-host defaults) and the endpoint host are recorded
alongside it, and the LLM host with them — `openai` at api.openai.com and
`openai` at somebody's own gateway are the same string and very different
bills. One additive migration, `e9fa0b1c2d3e`, three nullable columns, nothing
backfilled: the endpoint a run used six weeks ago is not recoverable, and
writing today's configuration onto it would invent provenance.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(pricing): price managed ASR by host and model, and know the Claude 5 family (#502 ) 7530b739fe

Two holes in the rate table, both of which produced no number where a number
was the point.

`ASR_HOURLY_USD` was empty and keyed on the adapter name. It could not have
been filled: `openai` names a wire shape, and OpenAI, Groq and a self-hosted
compatible server all select that one adapter at rates differing by ninefold or
not existing at all. The key is now (host, model), which is what the previous
commit put on the row — $0.36/hr for whisper-1 and gpt-4o-transcribe, $0.18 for
gpt-4o-mini-transcribe, $0.04 for Groq's whisper-large-v3-turbo, and None for
anything else. The self-hosted check runs first, so a bundled server behind a
proxy whose hostname resembles a vendor's is not billed at that vendor's rate.

Groq's ten-second per-request minimum is documented rather than modelled. The
VAD pre-pass sends one request per speech span and the median span is around
five seconds, so a real invoice runs roughly 1.77x the audio priced here — but
the row records seconds and not request counts, and a multiplier derived from
an assumed span length is a guess wearing a measurement's clothes. The figure
is named as a floor instead, in the docstring and in OPERATIONS.
`whisper-large-v3` is omitted for the same reason: its rate was not confirmed
against Groq's own page, and an unpriced row says "compute time" while a
wrongly-priced one says a number.

`TOKEN_RATES` had never heard of the Claude 5 family, so OPERATIONS.md's own
Anthropic example — `claude-opus-5` — priced at nothing. Sonnet 5, Opus 5,
Fable 5 and Sonnet 4.6 are added, and the longest-prefix rule keeps
`claude-sonnet-5` and `claude-sonnet-4-5` on their own (different) rates. Those
four figures were read from third-party aggregators on 2026-09-05 rather than
from Anthropic's page, which the module docstring says plainly: an aggregator
lags a price change and can transcribe one wrongly, and neither failure
announces itself. `PRICING_AS_OF` moves to that date, and every row already
stores the date it was priced under.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

test(conformance): --live can drive the managed ASR adapter (#502 ) d0cd8eadbf

`LiveWhisperCase.provider()` constructed `LocalWhisperProvider` outright, which
made the managed adapter the one part of the contract live mode could never
reach. That is backwards: the bundled server is the one an operator can already
poke at, while OpenAI and Groq were only ever checked against a fake built from
their documentation — and "is that documentation still accurate" is the single
question a live run exists to answer.

The case now resolves its adapter through `registry.resolve_asr_provider`, the
way the pipeline does, driven by `QB_CONFORMANCE_ASR_PROVIDER` and
`QB_CONFORMANCE_ASR_MODEL`. Unset sniffs the URL, exactly as the LLM side has
always done, so nothing changes for anyone running it against their own
Whisper. `local` is accepted alongside the canonical `local_whisper` because
this is a variable typed into a shell.

Skips stay named, which is the rule the suite is built on. A missing URL skips
naming the URL variable, and an OpenAI-compatible endpoint on a host with no
default model skips naming the model variable rather than raising a
ProviderError from inside whichever case happened to submit audio first.

No case's assertions change. What did change is the harness, and a harness
pointed at the wrong adapter fails silently, so the wiring is asserted in the
ordinary suite — those tests take neither `asr_case` nor `llm_case`, so they
run once, without `--live` and without a network.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs: what a run cost, which endpoint it cost it on, and how to drive live mode (#502 ) e81ec5f0e8

CHANGELOG gets the operator-visible bug: a paid transcription service reported
as free compute, why it happened (the provider select defaults to "Detect from
the URL", which stores nothing), what is fixed, and that older rows are left
alone because the endpoint a session used six weeks ago is not recoverable now.
The pricing and conformance changes ride along in a sentence.

OPERATIONS gains the two facts an operator will otherwise be surprised by:
transcription is priced by endpoint host *and* model rather than by the
provider name they chose — because "OpenAI-compatible" is a wire format and not
a vendor — and the Groq figure is a floor, because their ten-second per-request
minimum is not visible from the seconds Quest Board records. The usage-row
table also now lists the adapter, model and endpoint each run resolved to.

DEVELOPMENT documents the two new `QB_CONFORMANCE_ASR_*` variables with a
worked Groq example, and says which spellings the provider one takes and when
the case skips for a missing model.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

db: chain the usage-identity columns behind #501 's campaign settings (#502 )

CI / Bot/backend version sync (pull_request) Successful in 47s

Details

CI / Backend lint (ruff) (pull_request) Successful in 50s

Details

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 1m19s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 1m43s

Details

CI / Bot tests and audit (pull_request) Successful in 1m59s

Details

CI / Docker image build (pull_request) Successful in 4m24s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 8m39s

Details

1fbba3046e

e9fa0b1c2d3e was written against c7d8e9fa0b1c because #128's d8e9fa0b1c2d had
not merged yet. It has now, against the same parent, and two revisions naming
one parent is two heads — which alembic refuses to upgrade past, and which
nobody notices until a deploy stops halfway.

Nothing else changes. The two are unrelated and additive — one adds a campaign's
own AI settings, the other three nullable columns to session_usage — so the
order between them is arbitrary and this one goes second because it merged
second.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed

2026-09-05 05:41:53 +00:00

claude-bot merged commit f0cc9990de into main

2026-09-05 05:50:28 +00:00

claude-bot deleted branch fix/502-usage-pricing-live

2026-09-05 05:50:28 +00:00

claude-bot referenced this pull request from a commit

2026-09-05 05:50:29 +00:00

Merge PR #503: usage rows record the resolved provider; managed ASR priced by host and model; live conformance reaches the managed adapter (#502)

claude-bot referenced this pull request

2026-09-05 05:51:02 +00:00

[Backend] Follow-ups from the provider evaluation: usage rows misclassify sniffed managed runs, managed ASR is unpriceable, and the live tooling cannot reach the managed adapter (#360) #502