test(providers): keep vendor-specific strings inside the adapters, and point operators at the provider evaluation (#360) #498

Merged

claude-bot merged 3 commits from docs/360-provider-evaluation into main

2026-09-05 04:36:31 +00:00

claude-bot commented

2026-09-05 04:26:53 +00:00

Contributor

Refs #360 (the evaluation itself is an internal document attached to the issue; docs/.internal stays gitignored per convention).

tests/test_provider_isolation.py — the issue's last criterion made enforceable: no vendor hostname in code and no vendor SDK import outside app/providers/ (comments and docstrings stripped before scanning; no allowlist entries needed). 11 tests, passing. Its docstring names what a string scan cannot catch: audio_service.py still owns six per-vendor HTTP transports (_summarise_anthropic and siblings) with vendor headers and hardcoded model defaults, which is #443 territory.
One paragraph in docs/OPERATIONS.md under "Choosing a transcription provider" with the two findings an operator needs even without the report: ASR dominates a managed session's bill, so the VAD pre-pass is the biggest cost lever; and every managed provider evaluated keeps audio or transcript for a period by default.

ruff clean.

🤖 Generated with Claude Code

Refs #360 (the evaluation itself is an internal document attached to the issue; `docs/.internal` stays gitignored per convention). - **`tests/test_provider_isolation.py`** — the issue's last criterion made enforceable: no vendor hostname in code and no vendor SDK import outside `app/providers/` (comments and docstrings stripped before scanning; no allowlist entries needed). 11 tests, passing. Its docstring names what a string scan cannot catch: `audio_service.py` still owns six per-vendor HTTP transports (`_summarise_anthropic` and siblings) with vendor headers and hardcoded model defaults, which is #443 territory. - One paragraph in `docs/OPERATIONS.md` under "Choosing a transcription provider" with the two findings an operator needs even without the report: ASR dominates a managed session's bill, so the VAD pre-pass is the biggest cost lever; and every managed provider evaluated keeps audio or transcript for a period by default. ruff clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 3 commits

2026-09-05 04:26:53 +00:00

docs(internal): evaluate ASR and LLM providers before committing (#360 ) 724b9ef465

The August review recommended AssemblyAI Universal for ASR and Claude
Sonnet 5 for the LLM at ~$0.95 a session, and flagged several of its own
figures as unverified. This checks it: list rates only, published
retention terms quoted, and every accuracy claim traced to something that
was actually run.

What it changes about the answer:

- The $0.95 was priced on AssemblyAI's older $0.15/hr tier. At
  Universal-3.5 Pro's current $0.21/hr the same session is $1.23 and the
  customer-month is $5.33, 30% above the figure the recommendation rests
  on.
- The 28%-kept VAD assumption is over twice the only figure anyone has
  measured. The synthetic session keeps 12.9% of raw track time, which
  moves the same stack to $0.67 a session.
- The pipeline transcribes VAD spans individually, and 92% of the
  synthetic session's 207 spans are under ten seconds. Groq's 10-second
  per-request minimum therefore bills 1.77x the audio: an effective
  $0.071/hr, not $0.04. A rate card cannot show this.
- No managed candidate's accuracy on this corpus is known, and there are
  no keys here to find out. Every such cell says so and carries the
  command that fills it.
- The self-hosted marker figure (72.2%) was measured with vocabulary bias
  switched off, so the cheapest experiment available — no key, no adapter
  — has not been run.

The recommendation is to keep self-hosted as the default and make managed
a per-campaign opt-in, with the counterargument written at the same
length: a GPU is the real barrier to adoption, the self-hosted accuracy
evidence is TTS audio its own README calls a tripwire, and "free" loses
to managed on electricity alone above a 25 W average draw.

Self-hosted LLM cells are PENDING (#353/#354). Also records five gaps the
arithmetic found in the code: managed ASR has no rate table entry and
could not be keyed on the adapter name if it did, a sniffed managed run
is recorded as self-hosted, the rate table does not know the models
OPERATIONS.md recommends, --live conformance cannot reach the managed ASR
adapter, and the synthetic session's ASR leg cannot be pointed at one.

No provider is adopted and no default changes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

test(providers): keep vendor-specific strings inside the adapters (#360 ) 90fc6b97b3

#360's last criterion, made enforceable: nothing outside app/providers/
decides which vendor a request is sent to. Two ways that could break, so
two tests — a vendor hostname in code, and a vendor SDK import, which is
the harder version of the same coupling because it brings that vendor's
auth and retry model with it.

Both pass today. The two things the issue asked to allow need no
allowlist entry, which is the better outcome: the registry's sniff tables
are inside app/providers/ and covered by the directory rule, and
ai_pricing's model-family prefixes are not hostnames and are not scanned
— the invariant is about routing, not vocabulary, and a rate table keyed
on a model id decides nothing about where audio goes.

The scan reads code: comments and docstrings are stripped first, so
audio_service.py:2463 keeps naming the heuristic it no longer uses. A
test that forced that line's deletion would trade documentation for a
green tick.

Two more decisions worth naming. The scan is proved able to fail —
test_the_scan_actually_reads_the_tree checks the walk found the tree and
that the scanner still trips on the registry's own sniff table when the
directory exemption is lifted, because a rule like this dies of walking
an empty list. And what it cannot catch is written into the module
docstring rather than discovered later: audio_service still holds the
per-vendor HTTP transports (_summarise_anthropic at :4721 and its five
siblings) with vendor headers and hardcoded model defaults. Those are a
code change to fix, not a wider ban here, which would only produce a red
test nobody may fix this week.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs: keep the provider evaluation local, per the docs/.internal convention (#360 )

CI / Bot/backend version sync (pull_request) Successful in 51s

Details

CI / Backend lint (ruff) (pull_request) Successful in 1m0s

Details

CI / Docker image build (pull_request) Successful in 19s

Details

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 1m38s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 2m8s

Details

CI / Bot tests and audit (pull_request) Successful in 2m16s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 9m34s

Details

6dd2f73646

docs/.internal is gitignored on purpose; the report is attached to issue
#360 instead, and the operations note now points there rather than at a
path readers of the published tree do not have.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed

2026-09-05 04:27:16 +00:00

claude-bot referenced this pull request

2026-09-05 04:27:28 +00:00

[Backend] Evaluate ASR and LLM providers against the golden corpus before committing #360

claude-bot merged commit b595a74d5c into main

2026-09-05 04:36:31 +00:00

claude-bot deleted branch docs/360-provider-evaluation

2026-09-05 04:36:31 +00:00

claude-bot referenced this pull request from a commit

2026-09-05 04:36:33 +00:00

Merge PR #498: provider isolation test and the evaluation pointer (#360)