test(providers): a conformance suite every adapter must pass, plus a monthly canary (#359) #493
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/359-provider-conformance"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #359.
An abstraction with one real implementation is not an abstraction. This adds the suite that defines the bar for an adapter, runs it in CI against deterministic stubs and the local WhisperX adapter, documents a live mode, and adds a monthly canary through the configured (self-hosted fallback) path.
What
tests/conformance/— parametrised over adapter cases (cases.py), not copied per provider: 15 ASR cases (submit→polltrack_idpassthrough, association by id not order, no speaker field anywhere, monotonic word spans inside their segment,word_timestampsdeclaration agrees with the words and the degraded flag, vocabulary accepted and truncated tomax_vocabulary_termsin the actual request, declared byte/duration limits enforced with zero requests made,needs_chunkingconsistent, 5xx → typed error, 429 → typed rate-limit error withRetry-Afterparsed, empty transcript is not an error, a concurrency limit is declared) and 10 LLM cases (well-formed capabilities, schema spelling matches_apply_json_schema,read_usagenormalises each provider's shape, truncated structured raises / truncated prose returns flagged, oversized prompt warned-and-still-sent pinning #351's deliberate choice, provider error surfaces, prose round trip). Adapters passing in CI:local_whisperagainst a Whisper wire fake,stub,stub_async, and all four LLM providers against their wire fakes. 88 passed / 10 skipped (all named and intentional) / 1 xfail — the xfail is the #356 enforcement case, left visible rather than absent.LocalWhisperProviderraised a bareRuntimeErrorfor every failure (429 indistinguishable from 503) → newapp/providers/errors.pywithProviderError(RuntimeError),ProviderRateLimitError(retry_after),ProviderInputTooLargeError, derived fromRuntimeErrorso existing handlers keep catching them;max_audio_seconds/max_byteswere declared and never checked →enforce_input_limits()runs first in everysubmit. A second adapter,StubAsrProvider(app/providers/stub.py), takes the opposite branch wherever the contract allows (measured word timestamps, keyterm vocabulary, declared limits, asyncpoll), so the contract is no longer a description of one implementation. Deliberately not registered as selectable.pytest tests/conformance --livewithQB_CONFORMANCE_*env vars (skips, never fails, when unset; refuses to run underCI).make conformance-live.app/tasks/canary_tasks.py::provider_canary, beat every 30 days, a minimal round trip through the configured ASR and LLM, result written to app_settings keyprovider_canary({ran_at, asr_ok, llm_ok, asr_provider, llm_provider, error};None= not configured, so an unconfigured side is not a red light). WARNING on failure. 10 tests.docs/DEVELOPMENT.md"Adding a provider adapter"; OPERATIONS notes the canary.Rebased onto main after #355 and #358 (changelog conflict, all entries kept).
Known gaps, stated
_structured_*LLM transports still raise a bareRuntimeError; the case carries that expectation with a comment and tightens by editing one line once they are typed (out of the file whitelist for this build).Verification
🤖 Generated with Claude Code
The conformance suite proves the adapters are correct against fakes. It cannot prove the box at the other end of the operator's endpoint URL is still there, still speaks the same dialect, and still accepts the key it was given six months ago — and nothing else does either, because every provider call in this product happens on a path a GM triggered. A Whisper server that died in April is found by a failed session in July, with the recording already made. provider_canary transcribes one second of synthesized silence through the configured ASR adapter and asks the configured LLM to reply with "OK", then writes {ran_at, asr_ok, llm_ok, asr_provider, llm_provider, error} to the app_settings key `provider_canary` and logs at WARNING when either side fails. Monthly, matching prune-audit-log, because the failure it catches is measured in months and because it is the one scheduled task that makes an outbound call to a possibly metered provider. It grades nothing, retries nothing, and alerts nobody: a canary that retried would hide exactly the intermittent failure worth seeing. "Not configured" records None rather than False — a deployment that never set up transcription is not broken, and a red light for it would train the operator to ignore the field. The result is written in its own session after both checks, so a failure is still recorded; the months it says "no" are the point. No UI. Exposing it on the bot-settings response is not the one-line additive change the brief allowed for — it needs a schema field and two handler changes — so the read belongs with #358's capability panel, which imports CANARY_SETTING_KEY rather than repeating the string. Tested through the conformance fakes rather than by mocking the adapters: a canary built on mocked provider calls keeps passing after the adapter it exercises stops being reachable, which is the failure it exists to detect. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>