[Backend] Provider conformance test suite #359
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found in the August 2026 session lifecycle review (#319).
Why
An abstraction with one real implementation is not an abstraction. The point of this milestone is that providers can be swapped without a rewrite, and the only way that stays true is a suite every adapter must pass.
It also protects the fallback story: keeping a self-hosted adapter deployed and CI-tested means a managed provider's price rise, rate limit, or outage is an adapter switch rather than an emergency.
Proposed fix
A conformance suite each adapter runs against: submit and retrieve,
track_idpassthrough and association-by-id, word-timestamp normalisation, vocabulary bias accepted, declared constraints honoured, schema-constrained output where claimed, prompt-token reporting, concurrency limit respected, and correct behaviour on oversized input, a truncated response, and a provider error.Run it in CI against a deterministic stub and a local self-hosted adapter. Provide a documented command to run it against a live managed provider on demand, plus a monthly canary through the fallback path so the escape hatch is known to work before it is needed.
Acceptance criteria
Landed in PR #493 (merged 2026-09-05, CI green; 1814 passed / 10 skipped / 1 xfailed).
tests/conformance/— 15 ASR + 10 LLM cases parametrised over adapter cases, run in CI against deterministic wire fakes and the local WhisperX adapter;--livemode withQB_CONFORMANCE_*(skips when unset, refuses underCI);make conformance-live.LocalWhisperProviderraised a bareRuntimeErrorfor every failure (429 and 503 indistinguishable) →app/providers/errors.pytyped errors withretry_after; declaredmax_audio_seconds/max_byteswere never enforced →enforce_input_limits()first in everysubmit; and a second adapter (StubAsrProvider, deliberately unregistered) now takes the opposite branch everywhere the contract allows.provider_canary(beat every 30 days) round-trips the configured ASR and LLM and writes{ran_at, asr_ok, llm_ok, …}to app_settings keyprovider_canary.Open, on purpose: the
_structured_*LLM transports still raise bareRuntimeError(the case carries that expectation and tightens by one line when they are typed); the #356 enforcement case is anxfailplaceholder until that lands; live word-level assertions skip on a silence probe until a speech fixture exists (#433).