rbrooks/Quest-Board

Watch

Fork 0

[Backend] session_usage.asr_model is left empty while llm_model is populated (#357) #508

New issue

Closed

opened 2026-09-05 07:31:53 +00:00 by claude-bot · 3 comments

claude-bot commented

2026-09-05 07:31:53 +00:00

Collaborator

Found validating v4.2.0 on dev (2026-09-05). Two session_usage rows written by real end-to-end runs through the bundled WhisperX server:

asr_provider=local_whisper, asr_endpoint_host=10.3.0.28:8091, asr_model= (empty)
llm_provider=llamacpp, llm_model=qwen3.5

The Whisper server reports large-v3-turbo in /health, and the preflight (#354) captures it, so the value is available at the point the row is written; it is just not being copied into the usage row. Low severity, but the AI usage panel's ASR model column reads blank, and #502 item 2 keys managed ASR pricing on host + model, so an empty model will also defeat the cost estimate for a managed provider once one is configured.

Fix: populate asr_model from the resolved provider's capabilities (or the response's reported model) where the row is recorded in reminder_tasks.py, with a test that a run through the stub provider stores a non-empty model.

Found validating v4.2.0 on dev (2026-09-05). Two `session_usage` rows written by real end-to-end runs through the bundled WhisperX server: - `asr_provider=local_whisper`, `asr_endpoint_host=10.3.0.28:8091`, `asr_model=` (empty) - `llm_provider=llamacpp`, `llm_model=qwen3.5` The Whisper server reports `large-v3-turbo` in `/health`, and the preflight (#354) captures it, so the value is available at the point the row is written; it is just not being copied into the usage row. Low severity, but the AI usage panel's ASR model column reads blank, and #502 item 2 keys managed ASR pricing on host + model, so an empty model will also defeat the cost estimate for a managed provider once one is configured. Fix: populate `asr_model` from the resolved provider's capabilities (or the response's reported model) where the row is recorded in `reminder_tasks.py`, with a test that a run through the stub provider stores a non-empty model.

claude-bot referenced this issue

2026-09-05 07:34:17 +00:00

chore(release): v4.2.0 — AI Provider Abstraction & BYO-AI #504

claude-bot commented

2026-09-05 15:00:05 +00:00

Author

Collaborator

Picking this up for v4.2.1. Root cause is that resolve_asr_identity returns the configured model, and the default configuration leaves the model blank so the bundled server chooses. Fix: fall back to the model the server actually reports (capability discovery already reads it from /health) when the configured one is empty, so the usage row carries what served the run.

Picking this up for **v4.2.1**. Root cause is that `resolve_asr_identity` returns the *configured* model, and the default configuration leaves the model blank so the bundled server chooses. Fix: fall back to the model the server actually reports (capability discovery already reads it from `/health`) when the configured one is empty, so the usage row carries what served the run.

rbrooks referenced this issue from a commit

2026-09-05 15:43:38 +00:00

fix(webapp): record the model the bundled transcriber reports (#508)

claude-bot referenced this issue from a pull request that will close it,

2026-09-05 15:43:55 +00:00

fix: record the model the bundled transcriber reports, and three harness fixes from the v4.2.0 validation #510

claude-bot closed this issue

2026-09-05 16:02:15 +00:00

rbrooks referenced this issue from a commit

2026-09-05 16:03:00 +00:00

chore(release): v4.2.1

claude-bot referenced this issue

2026-09-05 16:03:15 +00:00

chore(release): v4.2.1 #511

claude-bot commented

2026-09-05 16:03:22 +00:00

Author

Collaborator

Fixed in PR #510 (merged), shipping in v4.2.1.

The cause was structural rather than a dropped assignment: the bundled adapter takes no model parameter, since the WhisperX server loads one at startup and its /transcribe form has no field to override it, so resolve_asr_identity had only the configured string to work from, and that is empty on every self-hosted deployment. llm_model was populated because the LLM side genuinely has a configured model.

The server does say what it is running, in /health, on both API v1 and v2, and capability discovery was already parsing that body and discarding the field. So AsrCapabilities now carries the reported model, and resolve_asr_identity falls back to it: configured model → the id the adapter will request (resolve_model(), unchanged for OpenAI and Groq) → the model the endpoint reported → None. Unknown stays None; an empty string was never an honest answer.

No new request on the hot path. provider.capabilities is a read of the last probe, so the only wrinkle was ordering: process_audio captured identity before anything had probed the endpoint. It now resolves a second time right after the vocabulary step (which already probes, cached per endpoint) and still before transcription, so a run that dies mid-ASR records what was serving it. The early resolution stays for runs that fail before that.

Checked that this cannot mis-price anything: estimate_asr_cost short-circuits on the self-hosted adapter names before any rate lookup, so a populated model on local_whisper still yields no dollar figure, and a test pins exactly that. Admin → AI Usage now shows the model beside the transcription time, as the LLM column always has. Older rows keep their empty column, since the model a session used six weeks ago is not recoverable.

Backend suite 2323 passed / 13 skipped, frontend 505 passed, ruff clean. Verify on dev after the v4.2.1 deploy: the next processed session's usage row should read large-v3-turbo.

Fixed in PR #510 (merged), shipping in v4.2.1. The cause was structural rather than a dropped assignment: the bundled adapter takes no model parameter, since the WhisperX server loads one at startup and its `/transcribe` form has no field to override it, so `resolve_asr_identity` had only the configured string to work from, and that is empty on every self-hosted deployment. `llm_model` was populated because the LLM side genuinely has a configured model. The server does say what it is running, in `/health`, on both API v1 and v2, and capability discovery was already parsing that body and discarding the field. So `AsrCapabilities` now carries the reported model, and `resolve_asr_identity` falls back to it: configured model → the id the adapter will request (`resolve_model()`, unchanged for OpenAI and Groq) → the model the endpoint reported → `None`. Unknown stays `None`; an empty string was never an honest answer. No new request on the hot path. `provider.capabilities` is a read of the last probe, so the only wrinkle was ordering: `process_audio` captured identity before anything had probed the endpoint. It now resolves a second time right after the vocabulary step (which already probes, cached per endpoint) and still before transcription, so a run that dies mid-ASR records what was serving it. The early resolution stays for runs that fail before that. Checked that this cannot mis-price anything: `estimate_asr_cost` short-circuits on the self-hosted adapter names before any rate lookup, so a populated model on `local_whisper` still yields no dollar figure, and a test pins exactly that. Admin → AI Usage now shows the model beside the transcription time, as the LLM column always has. Older rows keep their empty column, since the model a session used six weeks ago is not recoverable. Backend suite 2323 passed / 13 skipped, frontend 505 passed, ruff clean. Verify on dev after the v4.2.1 deploy: the next processed session's usage row should read `large-v3-turbo`.

claude-bot commented

2026-09-05 16:42:17 +00:00

Author

Collaborator

Verified live on dev (v4.2.1). The synthetic session was resubmitted through POST /api/bot/sessions/{id}/audio with force=true and processed end to end (16:31 → 16:41 UTC, status=succeeded). The new session_usage row:

asr_provider	asr_model	asr_endpoint_host	llm_provider	llm_model
`local_whisper`	`large-v3-turbo`	`10.3.0.28:8091`	`llamacpp`	`qwen3.5`

The two rows from the v4.2.0 validation this morning keep their empty column, as the changelog says they will.

**Verified live on dev (v4.2.1).** The synthetic session was resubmitted through `POST /api/bot/sessions/{id}/audio` with `force=true` and processed end to end (16:31 → 16:41 UTC, `status=succeeded`). The new `session_usage` row: | asr_provider | asr_model | asr_endpoint_host | llm_provider | llm_model | |---|---|---|---|---| | `local_whisper` | **`large-v3-turbo`** | `10.3.0.28:8091` | `llamacpp` | `qwen3.5` | The two rows from the v4.2.0 validation this morning keep their empty column, as the changelog says they will.

claude-bot referenced this issue

2026-09-05 17:30:38 +00:00

chore(release): v4.2.2 #513