rbrooks/Quest-Board
Watch
1
0
Fork
You've already forked Quest-Board
 0
Code Issues 77 Pull requests 2 Projects Releases 42 Packages Wiki Activity Actions

[Backend] Define the LLM provider contract with declared context window and schema output #351

New issue
Closed
opened 2026-08-25 20:39:14 +00:00 by claude-bot · 3 comments
claude-bot commented 2026-08-25 20:39:14 +00:00
Contributor
Copy link

Found in the August 2026 session lifecycle review (#319).

Why

Two of this project's most damaging pipeline bugs come from the absence of a declared context window: prompts are sent without knowing the budget, and truncation is never detected. The abstraction has to make the window a first-class, queryable property rather than a thing each call site assumes.

Proposed contract

  • Generate single-shot, with an optional json_schema constraint
  • Declare the context window, so the summariser can size its chunk windows from it (this is what makes one code path correct on a 9B local model and a 1M-context hosted one)
  • Declare capabilities: schema-constrained decoding, prompt caching, batch mode, thinking/reasoning behaviour
  • Report actual prompt tokens consumed, so the caller can verify nothing was silently truncated
  • Treat cache hints and batch mode as no-op-able — a provider that lacks them must not require special-casing at the call site

Schema-constrained decoding is the important capability. On llama.cpp it maps to GBNF grammar via json_schema (see #281); on Anthropic and OpenAI to structured outputs. response_format: {"type":"json_object"} alone is advisory and demonstrably not enforced on the current deployment, which is why the tolerant parser exists. Providers that cannot enforce a schema must declare that, so the pipeline can fall back to the tolerant parser and mark the result as degraded rather than silently trusting it.

Acceptance criteria

  • Adapter interface with generate, declared window, declared capabilities, reported prompt tokens
  • Chunk sizing in the summariser reads the declared window
  • Schema constraint is enforced where supported and explicitly degraded where not
  • Cache and batch hints are optional and no-op cleanly
  • Prompt-token reporting feeds the preflight verification in v4.0.0
Found in the August 2026 session lifecycle review (#319). ## Why Two of this project's most damaging pipeline bugs come from the absence of a declared context window: prompts are sent without knowing the budget, and truncation is never detected. The abstraction has to make the window a first-class, queryable property rather than a thing each call site assumes. ## Proposed contract - **Generate** single-shot, with an optional `json_schema` constraint - **Declare** the context window, so the summariser can size its chunk windows from it (this is what makes one code path correct on a 9B local model and a 1M-context hosted one) - **Declare** capabilities: schema-constrained decoding, prompt caching, batch mode, thinking/reasoning behaviour - **Report** actual prompt tokens consumed, so the caller can verify nothing was silently truncated - **Treat cache hints and batch mode as no-op-able** — a provider that lacks them must not require special-casing at the call site Schema-constrained decoding is the important capability. On llama.cpp it maps to GBNF grammar via `json_schema` (see #281); on Anthropic and OpenAI to structured outputs. `response_format: {"type":"json_object"}` alone is advisory and demonstrably not enforced on the current deployment, which is why the tolerant parser exists. Providers that cannot enforce a schema must declare that, so the pipeline can fall back to the tolerant parser **and mark the result as degraded** rather than silently trusting it. ## Acceptance criteria - [ ] Adapter interface with generate, declared window, declared capabilities, reported prompt tokens - [ ] Chunk sizing in the summariser reads the declared window - [ ] Schema constraint is enforced where supported and explicitly degraded where not - [ ] Cache and batch hints are optional and no-op cleanly - [ ] Prompt-token reporting feeds the preflight verification in v4.0.0
claude-bot commented 2026-09-01 18:49:00 +00:00
Author
Contributor
Copy link

Picking this up alongside #350 on feat/350-351-provider-contracts.

Grounding: most of this already exists, unowned

The v4.0.0 context-budget work built much of what this issue asks for, and two comments in llm_service.py explicitly name #351 as the issue that finishes them:

This issue asks for State today
Declared context window Exists — LLMConfig.context_tokens (settings_service.py:517), resolve_window, DEFAULT_CONTEXT_TOKENS = 32_768
Chunk sizing reads the window Exists — prompt_budget_tokens / fits_in_context, PROMPT_BUDGET_FRACTION = 0.7
Schema enforced where supported Exists — _apply_json_schema handles four provider spellings, including llama.cpp's both-forms workaround (#281)
Reported prompt tokens Partial — _warn_if_prompt_truncated reads them, but only warns; nothing returns them to a caller
Declared capabilities Missing
Cache / batch hints Missing

So the substance of this issue is narrower and sharper than its body suggests: the machinery is there, but nothing declares anything. The gap is a provider object to hang it on.

The actual defect: provider identity is guessed from the URL

generate_structured_text (llm_service.py:406) dispatches on URL substrings:

"anthropic.com" in endpoint_url          → Anthropic
"openai.com" in endpoint_url             → OpenAI
":11434" in url or "ollama" in url.lower() → Ollama
anything else                            → llama.cpp

This is the root of the "no commitment to any single provider" problem. A self-hoster behind a reverse proxy on their own domain is classified as llama.cpp regardless of what is actually behind it; an OpenAI-compatible gateway on a custom host is too. The four _qa_* functions in audio_service.py repeat the same sniffing independently — which is a large part of why #426 exists as a separate bug.

Replacing the sniff with an explicit, configured provider selection is the load-bearing change here. Everything else in this issue hangs off having a provider object at all.

Two decisions already settled, recorded so they are not reopened

  • Raise vs flag is decided (Ryan, on #338, 2026-08-28): structured output raises, prose returns flagged. The rationale and an explicit warning against making the halves consistent live in _reject_if_truncated (llm_service.py:240-280). The capability object does not get to re-decide this.
  • preflight_prompt stays a warning, not a raise — a departure from #336 recorded in its docstring, because estimate_tokens is deliberately pessimistic and a raise would refuse prompts that would have worked. This issue is the one that can change that: a provider declaring a real tokenizer and whether it front-truncates is what turns both soft checks hard. I am carrying the declaration but not flipping either check to a raise in this issue — that wants the conformance suite (#359) proving the tokenizer first.
Picking this up alongside #350 on `feat/350-351-provider-contracts`. ## Grounding: most of this already exists, unowned The v4.0.0 context-budget work built much of what this issue asks for, and two comments in `llm_service.py` explicitly name #351 as the issue that finishes them: | This issue asks for | State today | |---|---| | Declared context window | **Exists** — `LLMConfig.context_tokens` ([settings_service.py:517](webapp/backend/app/services/settings_service.py#L517)), `resolve_window`, `DEFAULT_CONTEXT_TOKENS = 32_768` | | Chunk sizing reads the window | **Exists** — `prompt_budget_tokens` / `fits_in_context`, `PROMPT_BUDGET_FRACTION = 0.7` | | Schema enforced where supported | **Exists** — `_apply_json_schema` handles four provider spellings, including llama.cpp's both-forms workaround (#281) | | Reported prompt tokens | **Partial** — `_warn_if_prompt_truncated` reads them, but only warns; nothing returns them to a caller | | Declared capabilities | **Missing** | | Cache / batch hints | **Missing** | So the substance of this issue is narrower and sharper than its body suggests: **the machinery is there, but nothing declares anything.** The gap is a provider object to hang it on. ## The actual defect: provider identity is guessed from the URL `generate_structured_text` ([llm_service.py:406](webapp/backend/app/services/llm_service.py#L406)) dispatches on URL substrings: ``` "anthropic.com" in endpoint_url → Anthropic "openai.com" in endpoint_url → OpenAI ":11434" in url or "ollama" in url.lower() → Ollama anything else → llama.cpp ``` This is the root of the "no commitment to any single provider" problem. A self-hoster behind a reverse proxy on their own domain is classified as llama.cpp regardless of what is actually behind it; an OpenAI-compatible gateway on a custom host is too. The four `_qa_*` functions in `audio_service.py` repeat the same sniffing independently — which is a large part of why #426 exists as a separate bug. Replacing the sniff with an explicit, configured provider selection is the load-bearing change here. Everything else in this issue hangs off having a provider object at all. ## Two decisions already settled, recorded so they are not reopened - **Raise vs flag** is decided (Ryan, on #338, 2026-08-28): structured output raises, prose returns flagged. The rationale and an explicit warning against making the halves consistent live in `_reject_if_truncated` ([llm_service.py:240-280](webapp/backend/app/services/llm_service.py#L240-L280)). The capability object does not get to re-decide this. - **`preflight_prompt` stays a warning, not a raise** — a departure from #336 recorded in its docstring, because `estimate_tokens` is deliberately pessimistic and a raise would refuse prompts that would have worked. This issue is the one that can change that: a provider declaring a *real* tokenizer and whether it front-truncates is what turns both soft checks hard. I am carrying the declaration but **not** flipping either check to a raise in this issue — that wants the conformance suite (#359) proving the tokenizer first.
claude-bot commented 2026-09-01 20:29:58 +00:00
Author
Contributor
Copy link

Landed in PR #483 (merged, CI green on all 7 jobs). Leaving this open — three of five criteria are fully met, two are partial and both partials are deliberate.

Criteria

  • Adapter interface with generate, declared window, declared capabilities, reported prompt tokens — LlmCapabilities + LlmProvider in app/providers/llm.py. The provider declares and spells; llm_service keeps the transports, so app.providers imports nothing from app.services and a provider builds in a test with no DB, settings or event loop.
  • Chunk sizing in the summariser reads the declared window — via resolve_llm_provider. An explicit context_tokens still wins; only an undeclared one now takes the provider's own window, which for Anthropic and OpenAI is the model's documented figure rather than the universal conservative 32k. Ollama and llama.cpp declare that same 32k, so the local paths are unchanged.
  • [~] Schema constraint enforced where supported and explicitly degraded where not — SchemaEnforcement is a three-state property and all four spellings are pinned by tests to the proven _apply_json_schema (#281). But nothing yet consumes the declaration to mark a result degraded. SummarisationRun.schema_degraded exists and is set when the tolerant parser repairs, which catches the symptom rather than the declared capability. Closing this properly means a provider declaring ADVISORY/NONE marking the run degraded up front — that belongs with #358, which is what renders it.
  • [~] Cache and batch hints optional and no-op cleanly — prompt_caching and batch_mode are declared, and prompt_caching is already load-bearing (see below). There is no hint API to pass yet, so "no-op cleanly" is currently true only vacuously. Batch mode wants #357's cost work to have a reason to exist.
  • Prompt-token reporting feeds the preflight verification in v4.0.0 — see below; this is the one that turned out to matter most.

The defect this issue exposed

Mapping the four providers' usage fields to write read_usage showed that _warn_if_prompt_truncated had exactly one caller — the Ollama path. llama.cpp, Anthropic and OpenAI all parsed usage for completion tokens and never looked at the prompt side, so a front-truncated prompt on three of four providers produced no signal at all. That is the input-side half of the failure #331/#336 exist to prevent, live on the majority of deployments.

_check_usage now runs both checks in one place, so a new transport cannot forget one.

prompt_caching is what makes the check honest rather than noisy: Anthropic and OpenAI report only newly evaluated tokens, so "truncated" and "cache hit" are indistinguishable and the check is skipped; Ollama and llama.cpp neither cache nor reject an over-long prompt, so there a low count has one meaning. This is exactly the declaration _warn_if_prompt_truncated's docstring had been asking for since #331, and the docstring is updated to say so.

Provider identity is no longer guessed

URL-substring dispatch is replaced by app/providers/registry.py, with a validated select in Admin → Bot Settings. The old heuristic survives as a named, logged fallback so nothing changes on upgrade, and the log line names the field to set. Blank preserves; an auto sentinel clears — without it a selection could be set but never undone, since blank already means preserve everywhere on that endpoint.

Recorded so it is not reopened

preflight_prompt is still a warning, not a raise. This issue makes the hard check possible — a declared front_truncates, and eventually a declared tokenizer — but flipping it should wait for #359 proving the tokenizer against a real endpoint. estimate_tokens is deliberately pessimistic, so raising today would refuse prompts that would have worked.

The threading is a known weak point, not an oversight. llm_provider travels alongside context_tokens through 19 signatures, 25 pass-throughs and 14 origin sites, which is the pattern DEFAULT_CONTEXT_TOKENS' comment warns about. It holds only because omitting it falls back to the sniff. Filed as #484 (v4.6.0): stop destructuring LLMConfig into loose parameters, which kills both threading problems permanently. It gets cheaper the sooner it is done.

Landed in PR #483 (merged, CI green on all 7 jobs). **Leaving this open** — three of five criteria are fully met, two are partial and both partials are deliberate. ## Criteria - [x] **Adapter interface with generate, declared window, declared capabilities, reported prompt tokens** — `LlmCapabilities` + `LlmProvider` in `app/providers/llm.py`. The provider declares and spells; `llm_service` keeps the transports, so `app.providers` imports nothing from `app.services` and a provider builds in a test with no DB, settings or event loop. - [x] **Chunk sizing in the summariser reads the declared window** — via `resolve_llm_provider`. An explicit `context_tokens` still wins; only an undeclared one now takes the provider's own window, which for Anthropic and OpenAI is the model's documented figure rather than the universal conservative 32k. Ollama and llama.cpp declare that same 32k, so the local paths are unchanged. - [~] **Schema constraint enforced where supported and explicitly degraded where not** — `SchemaEnforcement` is a three-state property and all four spellings are pinned by tests to the proven `_apply_json_schema` (#281). But **nothing yet consumes the declaration to mark a result degraded.** `SummarisationRun.schema_degraded` exists and is set when the tolerant parser repairs, which catches the symptom rather than the declared capability. Closing this properly means a provider declaring `ADVISORY`/`NONE` marking the run degraded up front — that belongs with #358, which is what renders it. - [~] **Cache and batch hints optional and no-op cleanly** — `prompt_caching` and `batch_mode` are declared, and `prompt_caching` is already load-bearing (see below). There is **no hint API to pass yet**, so "no-op cleanly" is currently true only vacuously. Batch mode wants #357's cost work to have a reason to exist. - [x] **Prompt-token reporting feeds the preflight verification in v4.0.0** — see below; this is the one that turned out to matter most. ## The defect this issue exposed Mapping the four providers' usage fields to write `read_usage` showed that **`_warn_if_prompt_truncated` had exactly one caller** — the Ollama path. llama.cpp, Anthropic and OpenAI all parsed `usage` for *completion* tokens and never looked at the prompt side, so a front-truncated prompt on three of four providers produced no signal at all. That is the input-side half of the failure #331/#336 exist to prevent, live on the majority of deployments. `_check_usage` now runs both checks in one place, so a new transport cannot forget one. `prompt_caching` is what makes the check honest rather than noisy: Anthropic and OpenAI report only newly evaluated tokens, so "truncated" and "cache hit" are indistinguishable and the check is **skipped**; Ollama and llama.cpp neither cache nor reject an over-long prompt, so there a low count has one meaning. This is exactly the declaration `_warn_if_prompt_truncated`'s docstring had been asking for since #331, and the docstring is updated to say so. ## Provider identity is no longer guessed URL-substring dispatch is replaced by `app/providers/registry.py`, with a validated select in Admin → Bot Settings. The old heuristic survives as a named, logged fallback so nothing changes on upgrade, and the log line names the field to set. Blank preserves; an `auto` sentinel clears — without it a selection could be set but never undone, since blank already means preserve everywhere on that endpoint. ## Recorded so it is not reopened **`preflight_prompt` is still a warning, not a raise.** This issue makes the hard check *possible* — a declared `front_truncates`, and eventually a declared tokenizer — but flipping it should wait for #359 proving the tokenizer against a real endpoint. `estimate_tokens` is deliberately pessimistic, so raising today would refuse prompts that would have worked. **The threading is a known weak point, not an oversight.** `llm_provider` travels alongside `context_tokens` through 19 signatures, 25 pass-throughs and 14 origin sites, which is the pattern `DEFAULT_CONTEXT_TOKENS`' comment warns about. It holds only because omitting it falls back to the sniff. Filed as #484 (v4.6.0): stop destructuring `LLMConfig` into loose parameters, which kills both threading problems permanently. It gets cheaper the sooner it is done.
rbrooks referenced this issue from a commit 2026-09-05 00:11:41 +00:00
chore(release): v4.1.1
claude-bot commented 2026-09-05 07:12:46 +00:00
Author
Contributor
Copy link

Closing with v4.2.0. The two criteria left partial on 1 September were each finished by the issue they were deferred to:

  • Schema constraint explicitly degraded where not enforced — done in #358 (PR #492). _stamp_provider_limits in audio_service.py reads the resolved provider's SchemaEnforcement before the run and sets schema_degraded from the declaration, not from the tolerant parser tripping. capability_service also turns a non-ENFORCED declaration into the GAP_SCHEMA_NOT_ENFORCED gap (severity high) on the Admin capability panel and on the run record, so the summary is marked as produced under a limited configuration up front.
  • Cache and batch hints optional and no-op cleanly — prompt_caching is load-bearing and no-ops where absent: on a caching provider _check_usage skips the truncation check outright; on the others it runs. batch_mode is declared, reported by /health-driven discovery and shown on the capability panel. There is deliberately no batch hint API: no call site batches (the pipeline is one session at a time and the usage work in #357 gave no reason to change that), so an API with nothing to no-op would be speculative surface. When a batching consumer appears, add the hint then.

Still recorded so it is not reopened: preflight_prompt remains a warning, not a raise. The conformance suite (#359) now proves the endpoint contract live, but not the tokenizer; flipping to a raise still wants that proof first.

Closing with v4.2.0. The two criteria left partial on 1 September were each finished by the issue they were deferred to: - [x] **Schema constraint explicitly degraded where not enforced** — done in #358 (PR #492). `_stamp_provider_limits` in `audio_service.py` reads the resolved provider's `SchemaEnforcement` *before* the run and sets `schema_degraded` from the declaration, not from the tolerant parser tripping. `capability_service` also turns a non-`ENFORCED` declaration into the `GAP_SCHEMA_NOT_ENFORCED` gap (severity high) on the Admin capability panel and on the run record, so the summary is marked as produced under a limited configuration up front. - [x] **Cache and batch hints optional and no-op cleanly** — `prompt_caching` is load-bearing and no-ops where absent: on a caching provider `_check_usage` skips the truncation check outright; on the others it runs. `batch_mode` is declared, reported by `/health`-driven discovery and shown on the capability panel. There is deliberately **no batch hint API**: no call site batches (the pipeline is one session at a time and the usage work in #357 gave no reason to change that), so an API with nothing to no-op would be speculative surface. When a batching consumer appears, add the hint then. Still recorded so it is not reopened: `preflight_prompt` remains a warning, not a raise. The conformance suite (#359) now proves the endpoint contract live, but not the tokenizer; flipping to a raise still wants that proof first.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
ui/table-notes-label
renovate/npm-vitest-vulnerability
chore/87-align-python-3.12
v4.3.0
v4.2.3
v4.2.2
v4.2.1
v4.2.0
v4.1.2
v4.1.1
v4.1.0
v4.0.1
v4.0.0
v3.11.4
v3.11.3
v3.11.2
v3.11.1
v3.11.0
v3.10.0
v3.9.1
v3.9.0
v3.8.0
v3.7.0
v3.6.1
v3.6.0
v3.5.0
v3.4.1
v3.4.0
v3.3.0
v3.2.0
v3.1.0
v3.0.0
v2.1.0
v2.0.0
v1.1.0
v1.0.0
v0.12.0
v0.11.0
v0.10.0
v0.9.1
v0.3.3
v0.9.0
v0.8.0
v0.3.2
v0.3.1
v0.3.0
v0.7.0
v0.6.0
v0.5.0
v0.4.0
v0.2.1
v0.2.0
v0.1.1
v0.1.0
Labels
Clear labels   
backend
Issue related to the backend

  
bot
Issue related to the bot

  
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
frontend
Issue related to the frontend

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
recording
Issue related to the recording system

  
security
Security hardening / vulnerability work

  
wontfix
This won't be fixed

No labels
backend
bot
bug
duplicate
enhancement
frontend
help wanted
invalid
question
recording
security
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/Quest-Board#351
No description provided.