[Backend] Preflight every prompt against the provider's declared context window #336
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Severity: CRITICAL. Found in the August 2026 session lifecycle review (#319).
The defect
Nothing anywhere estimates prompt size before sending, and nothing checks afterwards whether the server actually consumed the whole prompt. The #293 work guards the output cap only (
llm_service.py:55-92).usage.prompt_tokens,prompt_eval_count, and llama.cpp'struncatedflag are all returned and all ignored.Per provider:
audio_service.py:2104-2135) — sends nomax_tokensand no context options. Builds with context shift silently drop the front of the prompt, which can take the system prompt and the character legend with it, and summarise the tail of the session as though it were the whole thing.finish_reasonis read but only logged.Proposed fix
Estimate tokens before sending (characters ÷ ~3.4 is adequate; a real tokenizer where available), compare against the provider's declared window, and never send a prompt above ~70% of it — route to the chunked path instead. After the call, compare the reported prompt-token count against the estimate and fail loudly on a material shortfall. That is the input-side equivalent of #293 and it is what would have caught a silent truncation.
Depends on the provider abstraction declaring a context limit per provider (v4.2.0).
Acceptance criteria
Done in
ede1ac2, withc54d9af(#337) doing the load-bearing half. One criterion amended, one moved.What was actually wrong
The body says nothing preflights and nothing checks consumption. That was true when filed and stopped being true in
ccc29d6, which builtestimate_tokens,prompt_budget_tokens,fits_in_context,LLMConfig.context_tokens,resolve_context_tokensand_warn_if_prompt_truncated. The machinery was all there.What was missing is that it went almost nowhere. One of eighteen
generate_structured_textcall sites passed a window (beat_service.py:454). Highlights, the entire lore pipeline, rephrase/promote/statblock/convert_stats, name generation, session titles, journal entries and the GM Workbench tool runner all sent prompts against a window they never declared — and becauseNonemeant "skip budgeting entirely", declaring nothing switched off thenum_ctxprotection too.Acceptance criteria
preflight_promptruns on all four provider paths in llm_service.py:355-420 and all four in_dispatch_prose. It counts the system prompt too, which shares the window; a preflight that ignores it is the accounting error that drops the speaker legend.context_tokensis now threaded through the 13 domain functions inaudio_serviceand passed fromllm_cfg.context_tokensat all 18 call sites (reminder_tasks,lore_service,generation_service,planning_tasks). The constant is the floor for anything undeclared, not the value._warn_if_prompt_truncated, on every provider response.Prompts above the threshold route to the chunked path rather than being sent and hoped for→ amended, see below.The criterion I did not meet as written
"Prompts above the threshold route to the chunked path."
The transport layer cannot route. It has no idea how to split a lore-merge or a statblock prompt — only callers that own a chunked path can, and the two that do (
summarise,extraction_windows) already route. Building a generic chunker into the transport would mean inventing a split for prompts whose structure it cannot see.So over-budget preflights warn loudly and send. It also does not raise, for a second reason:
estimate_tokensis characters ÷ 3.4 and deliberately pessimistic, so a raise would refuse prompts that would have worked. Refusing to summarise a session beats summarising a tenth of one — but it does not beat summarising all of it._warn_if_prompt_truncatedreached the same conclusion from the other direction, and both become hard checks once a provider can declare a real tokenizer and whether it front-truncates (#351 / v4.2.0).Scope moved off this issue
/ask— one of the 15 unprotected call sites, and now #426 (v4.2.0). Not fixed here._summarise_chunked.One thing worth remembering
Threading
context_tokensbroke 55 tests across ~20 files, and not visibly. Those files build a fakeLLMConfigas aSimpleNamespaceor an ad-hoctype(...), none carryingcontext_tokens. Reading it raisedAttributeErrorinside task bodies that catch exceptions — so the symptom was not a crash but a generation result recorded asfailedin tests assertingready. A hand-rolled double drifts from the dataclass it stands for, and the drift surfaces as a wrong status rather than an error.1,284 → 1,286 passing, lint clean.