[Backend] Define the LLM provider contract with declared context window and schema output

claude-bot commented

2026-08-25 20:39:14 +00:00

Collaborator

Found in the August 2026 session lifecycle review (#319).

Why

Two of this project's most damaging pipeline bugs come from the absence of a declared context window: prompts are sent without knowing the budget, and truncation is never detected. The abstraction has to make the window a first-class, queryable property rather than a thing each call site assumes.

Proposed contract

Generate single-shot, with an optional json_schema constraint
Declare the context window, so the summariser can size its chunk windows from it (this is what makes one code path correct on a 9B local model and a 1M-context hosted one)
Declare capabilities: schema-constrained decoding, prompt caching, batch mode, thinking/reasoning behaviour
Report actual prompt tokens consumed, so the caller can verify nothing was silently truncated
Treat cache hints and batch mode as no-op-able — a provider that lacks them must not require special-casing at the call site

Schema-constrained decoding is the important capability. On llama.cpp it maps to GBNF grammar via json_schema (see #281); on Anthropic and OpenAI to structured outputs. response_format: {"type":"json_object"} alone is advisory and demonstrably not enforced on the current deployment, which is why the tolerant parser exists. Providers that cannot enforce a schema must declare that, so the pipeline can fall back to the tolerant parser and mark the result as degraded rather than silently trusting it.

Acceptance criteria

Adapter interface with generate, declared window, declared capabilities, reported prompt tokens
Chunk sizing in the summariser reads the declared window
Schema constraint is enforced where supported and explicitly degraded where not
Cache and batch hints are optional and no-op cleanly
Prompt-token reporting feeds the preflight verification in v4.0.0

Found in the August 2026 session lifecycle review (#319). ## Why Two of this project's most damaging pipeline bugs come from the absence of a declared context window: prompts are sent without knowing the budget, and truncation is never detected. The abstraction has to make the window a first-class, queryable property rather than a thing each call site assumes. ## Proposed contract - **Generate** single-shot, with an optional `json_schema` constraint - **Declare** the context window, so the summariser can size its chunk windows from it (this is what makes one code path correct on a 9B local model and a 1M-context hosted one) - **Declare** capabilities: schema-constrained decoding, prompt caching, batch mode, thinking/reasoning behaviour - **Report** actual prompt tokens consumed, so the caller can verify nothing was silently truncated - **Treat cache hints and batch mode as no-op-able** — a provider that lacks them must not require special-casing at the call site Schema-constrained decoding is the important capability. On llama.cpp it maps to GBNF grammar via `json_schema` (see #281); on Anthropic and OpenAI to structured outputs. `response_format: {"type":"json_object"}` alone is advisory and demonstrably not enforced on the current deployment, which is why the tolerant parser exists. Providers that cannot enforce a schema must declare that, so the pipeline can fall back to the tolerant parser **and mark the result as degraded** rather than silently trusting it. ## Acceptance criteria - [ ] Adapter interface with generate, declared window, declared capabilities, reported prompt tokens - [ ] Chunk sizing in the summariser reads the declared window - [ ] Schema constraint is enforced where supported and explicitly degraded where not - [ ] Cache and batch hints are optional and no-op cleanly - [ ] Prompt-token reporting feeds the preflight verification in v4.0.0

claude-bot added this to the v4.2.0 - AI Provider Abstraction & BYO-AI milestone

2026-08-25 20:39:14 +00:00

claude-bot added the

enhancement

backend

labels

2026-08-25 20:39:14 +00:00

rbrooks referenced this issue from a commit

2026-08-26 01:02:23 +00:00

feat(backend): give the pipeline a declared context window (#336, #337)

claude-bot referenced this issue

2026-08-28 15:59:07 +00:00

[Backend] Set Ollama num_ctx and verify the prompt was actually consumed #337

rbrooks referenced this issue from a commit

2026-08-28 16:37:13 +00:00

feat(backend): give every LLM call site the real context window (#336)

claude-bot referenced this issue

2026-08-28 16:37:35 +00:00

[Backend] Preflight every prompt against the provider's declared context window #336

claude-bot referenced this issue

2026-09-01 18:48:42 +00:00

[Backend] Define the ASR provider contract and normalise on word timestamps #350

claude-bot commented

2026-09-01 18:49:00 +00:00

Author

Collaborator

Picking this up alongside #350 on feat/350-351-provider-contracts.

Grounding: most of this already exists, unowned

The v4.0.0 context-budget work built much of what this issue asks for, and two comments in llm_service.py explicitly name #351 as the issue that finishes them:

This issue asks for	State today
Declared context window	Exists — `LLMConfig.context_tokens` (settings_service.py:517), `resolve_window`, `DEFAULT_CONTEXT_TOKENS = 32_768`
Chunk sizing reads the window	Exists — `prompt_budget_tokens` / `fits_in_context`, `PROMPT_BUDGET_FRACTION = 0.7`
Schema enforced where supported	Exists — `_apply_json_schema` handles four provider spellings, including llama.cpp's both-forms workaround (#281)
Reported prompt tokens	Partial — `_warn_if_prompt_truncated` reads them, but only warns; nothing returns them to a caller
Declared capabilities	Missing
Cache / batch hints	Missing

So the substance of this issue is narrower and sharper than its body suggests: the machinery is there, but nothing declares anything. The gap is a provider object to hang it on.

The actual defect: provider identity is guessed from the URL

generate_structured_text (llm_service.py:406) dispatches on URL substrings:

"anthropic.com" in endpoint_url          → Anthropic
"openai.com" in endpoint_url             → OpenAI
":11434" in url or "ollama" in url.lower() → Ollama
anything else                            → llama.cpp

This is the root of the "no commitment to any single provider" problem. A self-hoster behind a reverse proxy on their own domain is classified as llama.cpp regardless of what is actually behind it; an OpenAI-compatible gateway on a custom host is too. The four _qa_* functions in audio_service.py repeat the same sniffing independently — which is a large part of why #426 exists as a separate bug.

Replacing the sniff with an explicit, configured provider selection is the load-bearing change here. Everything else in this issue hangs off having a provider object at all.

Two decisions already settled, recorded so they are not reopened

Raise vs flag is decided (Ryan, on #338, 2026-08-28): structured output raises, prose returns flagged. The rationale and an explicit warning against making the halves consistent live in _reject_if_truncated (llm_service.py:240-280). The capability object does not get to re-decide this.
preflight_prompt stays a warning, not a raise — a departure from #336 recorded in its docstring, because estimate_tokens is deliberately pessimistic and a raise would refuse prompts that would have worked. This issue is the one that can change that: a provider declaring a real tokenizer and whether it front-truncates is what turns both soft checks hard. I am carrying the declaration but not flipping either check to a raise in this issue — that wants the conformance suite (#359) proving the tokenizer first.

Picking this up alongside #350 on `feat/350-351-provider-contracts`. ## Grounding: most of this already exists, unowned The v4.0.0 context-budget work built much of what this issue asks for, and two comments in `llm_service.py` explicitly name #351 as the issue that finishes them: | This issue asks for | State today | |---|---| | Declared context window | **Exists** — `LLMConfig.context_tokens` ([settings_service.py:517](webapp/backend/app/services/settings_service.py#L517)), `resolve_window`, `DEFAULT_CONTEXT_TOKENS = 32_768` | | Chunk sizing reads the window | **Exists** — `prompt_budget_tokens` / `fits_in_context`, `PROMPT_BUDGET_FRACTION = 0.7` | | Schema enforced where supported | **Exists** — `_apply_json_schema` handles four provider spellings, including llama.cpp's both-forms workaround (#281) | | Reported prompt tokens | **Partial** — `_warn_if_prompt_truncated` reads them, but only warns; nothing returns them to a caller | | Declared capabilities | **Missing** | | Cache / batch hints | **Missing** | So the substance of this issue is narrower and sharper than its body suggests: **the machinery is there, but nothing declares anything.** The gap is a provider object to hang it on. ## The actual defect: provider identity is guessed from the URL `generate_structured_text` ([llm_service.py:406](webapp/backend/app/services/llm_service.py#L406)) dispatches on URL substrings: ``` "anthropic.com" in endpoint_url → Anthropic "openai.com" in endpoint_url → OpenAI ":11434" in url or "ollama" in url.lower() → Ollama anything else → llama.cpp ``` This is the root of the "no commitment to any single provider" problem. A self-hoster behind a reverse proxy on their own domain is classified as llama.cpp regardless of what is actually behind it; an OpenAI-compatible gateway on a custom host is too. The four `_qa_*` functions in `audio_service.py` repeat the same sniffing independently — which is a large part of why #426 exists as a separate bug. Replacing the sniff with an explicit, configured provider selection is the load-bearing change here. Everything else in this issue hangs off having a provider object at all. ## Two decisions already settled, recorded so they are not reopened - **Raise vs flag** is decided (Ryan, on #338, 2026-08-28): structured output raises, prose returns flagged. The rationale and an explicit warning against making the halves consistent live in `_reject_if_truncated` ([llm_service.py:240-280](webapp/backend/app/services/llm_service.py#L240-L280)). The capability object does not get to re-decide this. - **`preflight_prompt` stays a warning, not a raise** — a departure from #336 recorded in its docstring, because `estimate_tokens` is deliberately pessimistic and a raise would refuse prompts that would have worked. This issue is the one that can change that: a provider declaring a *real* tokenizer and whether it front-truncates is what turns both soft checks hard. I am carrying the declaration but **not** flipping either check to a raise in this issue — that wants the conformance suite (#359) proving the tokenizer first.

rbrooks referenced this issue from a commit

2026-09-01 20:19:00 +00:00

feat(providers): define the ASR and LLM provider contracts (#350, #351)

rbrooks referenced this issue from a commit

2026-09-01 20:19:00 +00:00

fix(llm): check prompt consumption on every provider, not just Ollama (#351)

rbrooks referenced this issue from a commit

2026-09-01 20:19:00 +00:00

feat(settings): let an admin name the AI provider instead of guessing it (#351)

claude-bot referenced this issue

2026-09-01 20:19:28 +00:00

ASR and LLM provider contracts, and a truncation check that was only wired to one provider (#350, #351) #483

claude-bot referenced this issue

2026-09-01 20:20:51 +00:00

[Backend] Pass LLMConfig through the generation paths instead of destructuring it into loose parameters #484

claude-bot referenced this issue from a commit

2026-09-01 20:29:13 +00:00

Merge PR #483: ASR and LLM provider contracts (#350, #351)

claude-bot commented

2026-09-01 20:29:58 +00:00

Author

Collaborator

Landed in PR #483 (merged, CI green on all 7 jobs). Leaving this open — three of five criteria are fully met, two are partial and both partials are deliberate.

Criteria

Adapter interface with generate, declared window, declared capabilities, reported prompt tokens — LlmCapabilities + LlmProvider in app/providers/llm.py. The provider declares and spells; llm_service keeps the transports, so app.providers imports nothing from app.services and a provider builds in a test with no DB, settings or event loop.
Chunk sizing in the summariser reads the declared window — via resolve_llm_provider. An explicit context_tokens still wins; only an undeclared one now takes the provider's own window, which for Anthropic and OpenAI is the model's documented figure rather than the universal conservative 32k. Ollama and llama.cpp declare that same 32k, so the local paths are unchanged.
[~] Schema constraint enforced where supported and explicitly degraded where not — SchemaEnforcement is a three-state property and all four spellings are pinned by tests to the proven _apply_json_schema (#281). But nothing yet consumes the declaration to mark a result degraded. SummarisationRun.schema_degraded exists and is set when the tolerant parser repairs, which catches the symptom rather than the declared capability. Closing this properly means a provider declaring ADVISORY/NONE marking the run degraded up front — that belongs with #358, which is what renders it.
[~] Cache and batch hints optional and no-op cleanly — prompt_caching and batch_mode are declared, and prompt_caching is already load-bearing (see below). There is no hint API to pass yet, so "no-op cleanly" is currently true only vacuously. Batch mode wants #357's cost work to have a reason to exist.
Prompt-token reporting feeds the preflight verification in v4.0.0 — see below; this is the one that turned out to matter most.

The defect this issue exposed

Mapping the four providers' usage fields to write read_usage showed that _warn_if_prompt_truncated had exactly one caller — the Ollama path. llama.cpp, Anthropic and OpenAI all parsed usage for completion tokens and never looked at the prompt side, so a front-truncated prompt on three of four providers produced no signal at all. That is the input-side half of the failure #331/#336 exist to prevent, live on the majority of deployments.

_check_usage now runs both checks in one place, so a new transport cannot forget one.

prompt_caching is what makes the check honest rather than noisy: Anthropic and OpenAI report only newly evaluated tokens, so "truncated" and "cache hit" are indistinguishable and the check is skipped; Ollama and llama.cpp neither cache nor reject an over-long prompt, so there a low count has one meaning. This is exactly the declaration _warn_if_prompt_truncated's docstring had been asking for since #331, and the docstring is updated to say so.

Provider identity is no longer guessed

URL-substring dispatch is replaced by app/providers/registry.py, with a validated select in Admin → Bot Settings. The old heuristic survives as a named, logged fallback so nothing changes on upgrade, and the log line names the field to set. Blank preserves; an auto sentinel clears — without it a selection could be set but never undone, since blank already means preserve everywhere on that endpoint.

Recorded so it is not reopened

preflight_prompt is still a warning, not a raise. This issue makes the hard check possible — a declared front_truncates, and eventually a declared tokenizer — but flipping it should wait for #359 proving the tokenizer against a real endpoint. estimate_tokens is deliberately pessimistic, so raising today would refuse prompts that would have worked.

The threading is a known weak point, not an oversight. llm_provider travels alongside context_tokens through 19 signatures, 25 pass-throughs and 14 origin sites, which is the pattern DEFAULT_CONTEXT_TOKENS' comment warns about. It holds only because omitting it falls back to the sniff. Filed as #484 (v4.6.0): stop destructuring LLMConfig into loose parameters, which kills both threading problems permanently. It gets cheaper the sooner it is done.

Landed in PR #483 (merged, CI green on all 7 jobs). **Leaving this open** — three of five criteria are fully met, two are partial and both partials are deliberate. ## Criteria - [x] **Adapter interface with generate, declared window, declared capabilities, reported prompt tokens** — `LlmCapabilities` + `LlmProvider` in `app/providers/llm.py`. The provider declares and spells; `llm_service` keeps the transports, so `app.providers` imports nothing from `app.services` and a provider builds in a test with no DB, settings or event loop. - [x] **Chunk sizing in the summariser reads the declared window** — via `resolve_llm_provider`. An explicit `context_tokens` still wins; only an undeclared one now takes the provider's own window, which for Anthropic and OpenAI is the model's documented figure rather than the universal conservative 32k. Ollama and llama.cpp declare that same 32k, so the local paths are unchanged. - [~] **Schema constraint enforced where supported and explicitly degraded where not** — `SchemaEnforcement` is a three-state property and all four spellings are pinned by tests to the proven `_apply_json_schema` (#281). But **nothing yet consumes the declaration to mark a result degraded.** `SummarisationRun.schema_degraded` exists and is set when the tolerant parser repairs, which catches the symptom rather than the declared capability. Closing this properly means a provider declaring `ADVISORY`/`NONE` marking the run degraded up front — that belongs with #358, which is what renders it. - [~] **Cache and batch hints optional and no-op cleanly** — `prompt_caching` and `batch_mode` are declared, and `prompt_caching` is already load-bearing (see below). There is **no hint API to pass yet**, so "no-op cleanly" is currently true only vacuously. Batch mode wants #357's cost work to have a reason to exist. - [x] **Prompt-token reporting feeds the preflight verification in v4.0.0** — see below; this is the one that turned out to matter most. ## The defect this issue exposed Mapping the four providers' usage fields to write `read_usage` showed that **`_warn_if_prompt_truncated` had exactly one caller** — the Ollama path. llama.cpp, Anthropic and OpenAI all parsed `usage` for *completion* tokens and never looked at the prompt side, so a front-truncated prompt on three of four providers produced no signal at all. That is the input-side half of the failure #331/#336 exist to prevent, live on the majority of deployments. `_check_usage` now runs both checks in one place, so a new transport cannot forget one. `prompt_caching` is what makes the check honest rather than noisy: Anthropic and OpenAI report only newly evaluated tokens, so "truncated" and "cache hit" are indistinguishable and the check is **skipped**; Ollama and llama.cpp neither cache nor reject an over-long prompt, so there a low count has one meaning. This is exactly the declaration `_warn_if_prompt_truncated`'s docstring had been asking for since #331, and the docstring is updated to say so. ## Provider identity is no longer guessed URL-substring dispatch is replaced by `app/providers/registry.py`, with a validated select in Admin → Bot Settings. The old heuristic survives as a named, logged fallback so nothing changes on upgrade, and the log line names the field to set. Blank preserves; an `auto` sentinel clears — without it a selection could be set but never undone, since blank already means preserve everywhere on that endpoint. ## Recorded so it is not reopened **`preflight_prompt` is still a warning, not a raise.** This issue makes the hard check *possible* — a declared `front_truncates`, and eventually a declared tokenizer — but flipping it should wait for #359 proving the tokenizer against a real endpoint. `estimate_tokens` is deliberately pessimistic, so raising today would refuse prompts that would have worked. **The threading is a known weak point, not an oversight.** `llm_provider` travels alongside `context_tokens` through 19 signatures, 25 pass-throughs and 14 origin sites, which is the pattern `DEFAULT_CONTEXT_TOKENS`' comment warns about. It holds only because omitting it falls back to the sniff. Filed as #484 (v4.6.0): stop destructuring `LLMConfig` into loose parameters, which kills both threading problems permanently. It gets cheaper the sooner it is done.

rbrooks referenced this issue from a commit

2026-09-05 00:11:41 +00:00

chore(release): v4.1.1

claude-bot referenced this issue

2026-09-05 00:11:57 +00:00

chore(release): v4.1.1 #487

claude-bot referenced this issue

2026-09-05 00:18:41 +00:00

[Backend] The prose summary path and the admin "test LLM" button ignore the explicit AI provider and still sniff the URL #488

rbrooks referenced this issue from a commit

2026-09-05 01:00:09 +00:00

refactor(audio): pass the LLM target through as one object (#484)

rbrooks referenced this issue from a commit

2026-09-05 01:00:10 +00:00

fix(llm): honour the explicit provider on the prose summary path and the admin test (#484)

claude-bot referenced this issue

2026-09-05 01:00:32 +00:00

refactor(llm): pass LLMConfig through the generation paths as one object (#484), and honour the explicit provider on the prose path (#488) #489

claude-bot referenced this issue

2026-09-05 02:06:07 +00:00

feat: explain provider capability gaps instead of degrading silently (#358) #492

rbrooks referenced this issue from a commit

2026-09-05 02:15:26 +00:00

test(providers): a conformance suite every adapter must pass (#359)

claude-bot referenced this issue

2026-09-05 02:15:50 +00:00

test(providers): a conformance suite every adapter must pass, plus a monthly canary (#359) #493

claude-bot referenced this issue

2026-09-05 02:27:42 +00:00

[Frontend] Explain provider capability gaps instead of degrading silently #358

rbrooks referenced this issue from a commit

2026-09-05 03:10:58 +00:00

feat(llm): count what every LLM call consumed, without threading a parameter (#357)

claude-bot referenced this issue

2026-09-05 03:11:23 +00:00

feat: per-session cost and usage telemetry (#357) #494

claude-bot referenced this issue

2026-09-05 03:19:24 +00:00

[Backend] Per-session cost and usage telemetry #357

rbrooks referenced this issue from a commit

2026-09-05 03:40:34 +00:00

feat(providers): register and sniff the OpenAI-compatible ASR adapter

rbrooks referenced this issue from a commit

2026-09-05 03:40:34 +00:00

feat(admin): an ASR provider select, and a Test button that speaks its dialect

claude-bot referenced this issue

2026-09-05 03:48:21 +00:00

[Backlog] AI provider abstraction and per-campaign BYO-AI #128

claude-bot commented

2026-09-05 07:12:46 +00:00

Author

Collaborator

Closing with v4.2.0. The two criteria left partial on 1 September were each finished by the issue they were deferred to:

Schema constraint explicitly degraded where not enforced — done in #358 (PR #492). _stamp_provider_limits in audio_service.py reads the resolved provider's SchemaEnforcement before the run and sets schema_degraded from the declaration, not from the tolerant parser tripping. capability_service also turns a non-ENFORCED declaration into the GAP_SCHEMA_NOT_ENFORCED gap (severity high) on the Admin capability panel and on the run record, so the summary is marked as produced under a limited configuration up front.
Cache and batch hints optional and no-op cleanly — prompt_caching is load-bearing and no-ops where absent: on a caching provider _check_usage skips the truncation check outright; on the others it runs. batch_mode is declared, reported by /health-driven discovery and shown on the capability panel. There is deliberately no batch hint API: no call site batches (the pipeline is one session at a time and the usage work in #357 gave no reason to change that), so an API with nothing to no-op would be speculative surface. When a batching consumer appears, add the hint then.

Still recorded so it is not reopened: preflight_prompt remains a warning, not a raise. The conformance suite (#359) now proves the endpoint contract live, but not the tokenizer; flipping to a raise still wants that proof first.

Closing with v4.2.0. The two criteria left partial on 1 September were each finished by the issue they were deferred to: - [x] **Schema constraint explicitly degraded where not enforced** — done in #358 (PR #492). `_stamp_provider_limits` in `audio_service.py` reads the resolved provider's `SchemaEnforcement` *before* the run and sets `schema_degraded` from the declaration, not from the tolerant parser tripping. `capability_service` also turns a non-`ENFORCED` declaration into the `GAP_SCHEMA_NOT_ENFORCED` gap (severity high) on the Admin capability panel and on the run record, so the summary is marked as produced under a limited configuration up front. - [x] **Cache and batch hints optional and no-op cleanly** — `prompt_caching` is load-bearing and no-ops where absent: on a caching provider `_check_usage` skips the truncation check outright; on the others it runs. `batch_mode` is declared, reported by `/health`-driven discovery and shown on the capability panel. There is deliberately **no batch hint API**: no call site batches (the pipeline is one session at a time and the usage work in #357 gave no reason to change that), so an API with nothing to no-op would be speculative surface. When a batching consumer appears, add the hint then. Still recorded so it is not reopened: `preflight_prompt` remains a warning, not a raise. The conformance suite (#359) now proves the endpoint contract live, but not the tokenizer; flipping to a raise still wants that proof first.

claude-bot closed this issue

2026-09-05 07:12:47 +00:00

Rows
Columns

[Backend] Define the LLM provider contract with declared context window and schema output #351

Why

Proposed contract

Acceptance criteria

Grounding: most of this already exists, unowned

The actual defect: provider identity is guessed from the URL

Two decisions already settled, recorded so they are not reopened

Criteria

The defect this issue exposed

Provider identity is no longer guessed

Recorded so it is not reopened