rbrooks/Quest-Board

Watch

Fork 0

feat: per-session cost and usage telemetry (#357) #494

Merged

claude-bot merged 7 commits from feat/357-usage-telemetry into main

2026-09-05 03:16:49 +00:00

claude-bot commented

2026-09-05 03:11:23 +00:00

Contributor

Closes #357. Also lands the "cache hints" half #351 deferred here: Usage.cached_tokens is now read from Anthropic (cache_read + cache_creation) and OpenAI (prompt_tokens_details.cached_tokens).

What

Migration b6c7d8e9fa0b (after a5b6c7d8e9fa, single head): table session_usage, one row per processing run — audio seconds submitted and transcribed (the VAD lever), ASR/LLM provider and model, LLM calls / prompt / completion / cached tokens, ASR and LLM phase wall time, retry count, asr_cost_usd / llm_cost_usd (nullable), cost_is_estimate, pricing_as_of, status. Up/down/up verified on PostgreSQL 16.
Usage meter without a threaded parameter — app/services/usage_meter.py: a ContextVar scope opened by process_audio around its LLM phases; llm_service._check_usage (the funnel every transport calls) records each call into the active meter and is inert otherwise. Scope widening, deliberate: the four _summarise_* prose transports did not go through _check_usage — they open-coded the truncation checks — so the session summary itself would have gone uncounted. They now route through the funnel via audio_service._prose_usage, with truncation behaviour unchanged.
Pricing — app/services/ai_pricing.py, list rates per 1M tokens for the Anthropic and OpenAI families the registry knows, longest-prefix match, PRICING_AS_OF = 2026-05-01 (the horizon the table can vouch for; it renders on the panel so staleness is visible). Self-hosted → None, never 0, which is what lets the UI show compute time instead. PROMPT_TOKENS_INCLUDE_CACHED handles the fact that OpenAI's prompt count includes the cached share and Anthropic's does not.
Admin API — GET /api/admin/ai/usage?days= (sessions, per-campaign rollups with VAD reduction, per-month line) and /usage/campaigns/{id}. This is what answers "what does a customer cost per month".
Frontend — AdminAiUsage under the capability panel: by campaign, by month, run by run; every dollar figure badged "estimated"; self-hosted rows show GPU/CPU minutes.
Verified live-ish: process_audio's body has no test in the repo, so the instrumentation was exercised with a throwaway eager run against Postgres with faked providers — success and failure paths each wrote one row with exact arithmetic (script not committed; a committed integration test for this task is worth a follow-up).

Rebased onto main after #359 (changelog conflict, both entries kept).

Verification

Backend: 1892 passed, 10 skipped, 1 xfailed (post-rebase, conformance suite included). Frontend: 484 passed, eslint clean apart from the pre-existing warning. ruff clean. alembic heads → one head. Version sync OK; no bot contract change.

🤖 Generated with Claude Code

Closes #357. Also lands the "cache hints" half #351 deferred here: `Usage.cached_tokens` is now read from Anthropic (`cache_read + cache_creation`) and OpenAI (`prompt_tokens_details.cached_tokens`). ## What - **Migration `b6c7d8e9fa0b`** (after `a5b6c7d8e9fa`, single head): table `session_usage`, one row per processing run — audio seconds submitted and transcribed (the VAD lever), ASR/LLM provider and model, LLM calls / prompt / completion / cached tokens, ASR and LLM phase wall time, retry count, `asr_cost_usd` / `llm_cost_usd` (nullable), `cost_is_estimate`, `pricing_as_of`, `status`. Up/down/up verified on PostgreSQL 16. - **Usage meter without a threaded parameter** — `app/services/usage_meter.py`: a `ContextVar` scope opened by `process_audio` around its LLM phases; `llm_service._check_usage` (the funnel every transport calls) records each call into the active meter and is inert otherwise. **Scope widening, deliberate:** the four `_summarise_*` prose transports did *not* go through `_check_usage` — they open-coded the truncation checks — so the session summary itself would have gone uncounted. They now route through the funnel via `audio_service._prose_usage`, with truncation behaviour unchanged. - **Pricing** — `app/services/ai_pricing.py`, list rates per 1M tokens for the Anthropic and OpenAI families the registry knows, longest-prefix match, `PRICING_AS_OF = 2026-05-01` (the horizon the table can vouch for; it renders on the panel so staleness is visible). Self-hosted → `None`, never `0`, which is what lets the UI show compute time instead. `PROMPT_TOKENS_INCLUDE_CACHED` handles the fact that OpenAI's prompt count includes the cached share and Anthropic's does not. - **Admin API** — `GET /api/admin/ai/usage?days=` (sessions, per-campaign rollups with VAD reduction, per-month line) and `/usage/campaigns/{id}`. This is what answers "what does a customer cost per month". - **Frontend** — `AdminAiUsage` under the capability panel: by campaign, by month, run by run; every dollar figure badged "estimated"; self-hosted rows show GPU/CPU minutes. - **Verified live-ish:** `process_audio`'s body has no test in the repo, so the instrumentation was exercised with a throwaway eager run against Postgres with faked providers — success and failure paths each wrote one row with exact arithmetic (script not committed; a committed integration test for this task is worth a follow-up). Rebased onto main after #359 (changelog conflict, both entries kept). ## Verification - Backend: **1892 passed, 10 skipped, 1 xfailed** (post-rebase, conformance suite included). Frontend: **484 passed**, eslint clean apart from the pre-existing warning. ruff clean. `alembic heads` → one head. Version sync OK; no bot contract change. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 7 commits

2026-09-05 03:11:23 +00:00

db: record what each processing run of a session cost (#357 ) a847be6446

A hosted tier cannot be priced and a self-hoster cannot size a GPU from a
projection. The managed-stack figure this milestone starts from — about
$0.95 a session, dominated by ASR — is arithmetic over public rate cards,
and there has never been a single measured number to check it against.

`session_usage` is one row per *run*: audio submitted and audio actually
transcribed (so the VAD pre-pass's saving is measurable rather than
asserted), each phase's wall time, the LLM's token counts including the
cached share, and a list-rate cost estimate stamped with the date of the
rates it came from.

Per run and not per session, because a retry, an admin reprocess and an
erasure regenerate each spend money — collapsing them would make the most
expensive sessions look like the cheapest. Failed runs are recorded for the
same reason: ASR completes long before most failures do.

The two cost columns are nullable rather than zero-defaulted. "Self-hosted,
no dollar figure applies" and "$0.00" are different claims, and only one of
them belongs in a monthly total; a run with no cost carries its seconds
instead, which is the currency a self-hoster actually pays in.

Migration b6c7d8e9fa0b, chained after a5b6c7d8e9fa. Additive, no backfill —
there is nothing to backfill from.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(llm): count what every LLM call consumed, without threading a parameter (#357 ) 962532f8f7

The tokens were already read. Every provider's `read_usage` normalises them
and `_check_usage` is the one place all of it lands — and it used them for a
truncation check and then dropped them.

A meter now accumulates there. The obvious alternative, an accumulator
passed down from `process_audio`, is the design #484 just spent a release
removing: a parameter threaded through a dozen frames means every new call
site must remember to pass it, and the ones that forget do not fail, they
just do not count. That is exactly how seventeen of eighteen sites ended up
with no declared context window (#337). A ContextVar inverts it — the task
opens a scope and everything underneath is counted, including transports
written next year.

Inert when no scope is open, so a GM generating an NPC or asking /ask is not
billed to whatever session ran last.

Two supporting changes:

- `Usage` gains `cached_tokens`, read from Anthropic's cache_read/creation
  fields and OpenAI's prompt_tokens_details. Both bill a cached prefix at a
  fraction of the input rate and this pipeline re-sends the same campaign
  context on every call of a run, so ignoring it would overstate a hosted
  bill several-fold. The two also disagree about whether prompt_tokens
  *contains* those tokens; both are passed on as reported and ai_pricing
  holds the per-provider arithmetic.

- The four `_summarise_*` transports now go through `_check_usage` like
  every other transport. They open-coded the same two checks with the
  provider spelled as a display string, which is the near-duplication #338
  and #351 each had to repair four times — and it left the largest LLM call
  in the product, the session summary itself, outside the one funnel. The
  checks themselves are unchanged: same fields, same order, parsed by the
  provider instead of by hand.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(tasks): record what each session actually cost to process (#357 ) 12af76b1a1

`process_audio` is the only place that knows all of it at once — how much
audio was submitted and how much survived VAD, which providers were
configured, how long each phase took, and whether the run succeeded — and it
computed most of it, logged some of it, and threw all of it away.

It now times the ASR phase, meters the two LLM phases onto one meter, and
writes a row on both the success and the failure path. A failed run is
recorded because it still cost money: transcription is the expensive half
and it completes long before most failures do, so recording only successes
would under-report exactly the deployments having trouble.

`record_session_usage` is a service function rather than inline task code so
that "does a run record the right numbers" is testable without standing up a
whole Celery task with a fake Whisper and a fake LLM. It writes in a
savepoint and swallows its own errors — the same inversion
`_persist_summarisation_run` uses, and for the same reason: a flush that
raises leaves the session needing a rollback, and the next commit is the one
carrying the transcript the GM asked for.

`ai_pricing` turns those counts into dollars from a dated list-rate table,
and is honest about what that is worth: every figure is an estimate,
`cost_is_estimate` and `pricing_as_of` travel with it on every row, and a
model the table does not know gets None rather than a plausible guess.
Self-hosted providers get None too — free and unpriceable are different
claims, and the seconds already on the row are the currency there.

The VAD-off path measures the tracks directly, because it returns
(segments, None, None); without that, the deployments not running VAD would
be the ones with no audio figure at all.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(admin): expose what the AI pipeline costs, per campaign and per month (#357 ) 2738a5ba28

Two admin-only endpoints over the rows the pipeline now writes.
`GET /api/admin/ai/usage?days=30` answers the question a hosted tier cannot
be priced without — what does a customer cost per month — with a campaign
rollup, a monthly line and the individual runs behind them;
`/usage/campaigns/{id}` narrows the same shape to one group.

Every dollar figure carries `cost_is_estimate` and the rate table's date,
and rows from a self-hosted stack carry no dollars at all: `est_cost_usd` is
null and `compute_seconds` is the number that means something. The two
travel together rather than one being derived from the other, so a client
cannot render "$0.00" for a deployment that has an electricity bill instead.

Empty lists rather than a 404 on a deployment that has processed nothing —
"nothing yet" is a legitimate answer and the page should render it. The
campaign endpoint 404s only on a campaign that does not exist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

feat(frontend): show what the AI pipeline costs, in the currency you pay in (#357 ) 20777b8aa7

A panel beside the capability panel in Admin → Bot Settings: the monthly
line, a per-campaign rollup, and the recent runs behind them. Adjacent
questions — what your providers can do, and what they are costing you to do
it.

Two rules the component exists to keep.

Every dollar figure carries an "est." badge and the panel names the date of
the rate card, because a number with a currency symbol is read as
authoritative unless it visibly says otherwise. Nothing here is a bill: no
bundled provider reports what it actually charged.

A self-hosted run shows the compute time it occupied, not "$0.00". The two
are different claims — one says the run was free, which it was not; the
operator paid for it in electricity and a GPU that could have been doing
something else. The API keeps `est_cost_usd: null` separate from zero
precisely so this stays renderable.

The VAD saving gets a column of its own. ASR dominates a managed session's
cost, so the silence pre-pass is the biggest lever there is, and the ~72%
figure the milestone's projection assumes is now checkable against a real
deployment.

Admin.test.jsx's users.js mock gains the new function: it is mounted inside
Bot Settings, so without it every Bot Settings test fails inside an
unrelated component.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

test: cover the meter, the row, the rate table and the rollups (#357 ) 4c1b6bf10b

Four claims, and each one is worth nothing if it is wrong quietly.

The meter is exercised through the real transports with `httpx.AsyncClient`
faked at the *constructor* — the transports build a client, enter it as a
context manager and post through it, and swapping the transport underneath
tests all of that rather than a patched method. Both hosted usage shapes,
both with cached-token blocks, because the two providers report the cached
share in different places and with opposite relationships to the prompt
count. One test covers the prose path specifically: the session summary is
the largest LLM call in the product, and it was outside the funnel until
this issue.

Inertness gets its own test. A GM generating an NPC goes through the same
transports, and those calls must be counted against nothing — not against
whatever session ran last, which is what a module-level accumulator would
do.

Pricing is tested for a known model (exact arithmetic, cached tokens
included), for OpenAI's prompt_tokens *containing* the cached ones where
Anthropic's does not, for an unknown model (None, not a guess), and for a
self-hosted run (no dollars, seconds present).

The endpoints are seeded across a month boundary — subtracting 35 days
always crosses one — because "what does a customer cost per month" is the
question they exist for, and a rollup that silently totals two months looks
exactly like one that bucketed correctly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs: what is recorded about AI usage, and what the dollar figures are worth (#357 )

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 50s

Details

CI / Bot tests and audit (pull_request) Successful in 1m25s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 1m26s

Details

CI / Docker image build (pull_request) Successful in 45s

Details

CI / Bot/backend version sync (pull_request) Successful in 50s

Details

CI / Backend lint (ruff) (pull_request) Successful in 54s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 5m23s

Details

d389248f49

An OPERATIONS section on the new panel: what each row holds and why those
are the numbers that matter (audio summed across per-speaker tracks, not the
length of the session), that costs are list-rate estimates against a stated
date rather than an invoice, where to add a model the rate table does not
know, and that a self-hoster reads compute time in place of dollars because
"$0.00" would claim the run was free.

Changelog entry under Unreleased, leading with the migration id.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed

2026-09-05 03:11:38 +00:00

claude-bot merged commit 47c31d7705 into main

2026-09-05 03:16:49 +00:00

claude-bot referenced this pull request from a commit

2026-09-05 03:16:49 +00:00

Merge PR #494: per-session cost and usage telemetry (#357)

claude-bot deleted branch feat/357-usage-telemetry

2026-09-05 03:16:49 +00:00

claude-bot referenced this pull request

2026-09-05 03:19:24 +00:00

[Backend] Per-session cost and usage telemetry #357