feat: explain provider capability gaps instead of degrading silently (#358) #492
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/358-capability-gaps"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #358. Also finishes the criterion #351 deliberately left partial: a run is now marked
schema_degradedfrom the provider's declaration, before any output is parsed, not only when the tolerant parser repairs.What
app/services/capability_service.py:derive_capability_gaps(asr, llm, llm_config)→CapabilityGap(code, severity, title, consequence)in plain language, worst-first. Catalogue:llm_schema_not_enforced(high),llm_window_needs_chunking(medium, N computed with the summariser's ownprompt_budget_tokens+split_transcript_windowsagainst a documented 60k-token representative transcript: 8k→13 passes, 32k→3, 65k→2, ≥128k→1),llm_no_prompt_token_reporting(medium),llm_front_truncates(low, medium when prompt tokens are also unreported),llm_provider_guessed(low),asr_no_word_timestamps(medium),asr_no_vocabulary(low).GET /api/admin/ai/capabilities→{asr, llm, gaps}; an unconfigured side reportsconfigured: falserather than erroring. The admin Test LLM / Test Whisper responses gainprovider,provider_pinned,gaps— on the failure path too, since a wrongly-guessed provider is a plausible cause of the failure.a5b6c7d8e9fa: one nullable JSONBsummarisation_runs.capability_gaps(NULL = not recorded,[]= no gaps).process_audioseeds the run with both sides' gaps;summarise()marksschema_degradedfrom the declaration up front and never raises over provenance.Rebased onto main after #355 (one changelog conflict, both entries kept).
Verification
🤖 Generated with Claude Code
`AsrCapabilities` and `LlmCapabilities` have named word timestamps, schema enforcement, the context window and prompt-token reporting since #350/#351. Nothing read any of it. The declarations existed so the pipeline could branch; the person who picked the model was never told what the branch cost them — and the person hitting it is usually a self-hoster who chose on VRAM and price. `capability_service.derive_capability_gaps` turns the declarations into a list of `{code, severity, title, consequence}` in a GM's terms. Each entry says what someone will *see*: "the model is not held to the format Quest Board asks for, so events can be dropped with nothing failing", not "schema_enforcement is ADVISORY". Mechanism stays in the provider docstrings, where it already is. Pure by construction — capability objects in, list out, no database, no HTTP, no settings — so each gap is unit-testable, including the combinations no bundled provider declares today (ADVISORY enforcement, unreported prompt tokens). Those are exactly the shapes a future managed adapter will have. Two decisions worth recording: - The chunking gap reports "a typical session takes N passes", not "your window is small", because only the first is actionable. N comes from the summariser's own `prompt_budget_tokens` and `split_transcript_windows` run against a documented ~60k-token representative session (3.5 hours, five speakers). A second copy of that arithmetic is how the #349 harness once measured a fix as doing nothing. - Front-truncation is `low` while the provider still reports prompt tokens — the existing warning fires and an operator can act — and `medium` when it does not, because the pair leaves nothing anywhere able to notice. Gap codes are a separate namespace from the `DEGRADED_*` codes in `app.providers`: those say what happened to one result, these say what the configuration cannot do before anything has run, and only one of them has a fix. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>