rbrooks/Quest-Board
Watch
1
0
Fork
You've already forked Quest-Board
 0
Code Issues 58 Pull requests 7 Projects Releases 76 Packages Wiki Activity Actions

[Backend] Preflight every prompt against the provider's declared context window #336

New issue
Closed
opened 2026-08-25 20:38:55 +00:00 by claude-bot · 1 comment
claude-bot commented 2026-08-25 20:38:55 +00:00
Collaborator
Copy link

Severity: CRITICAL. Found in the August 2026 session lifecycle review (#319).

The defect

Nothing anywhere estimates prompt size before sending, and nothing checks afterwards whether the server actually consumed the whole prompt. The #293 work guards the output cap only (llm_service.py:55-92). usage.prompt_tokens, prompt_eval_count, and llama.cpp's truncated flag are all returned and all ignored.

Per provider:

  • llama.cpp (audio_service.py:2104-2135) — sends no max_tokens and no context options. Builds with context shift silently drop the front of the prompt, which can take the system prompt and the character legend with it, and summarise the tail of the session as though it were the whole thing. finish_reason is read but only logged.
  • OpenAI / Anthropic — input overflow is a loud 400, so these fail safely on input.

Proposed fix

Estimate tokens before sending (characters ÷ ~3.4 is adequate; a real tokenizer where available), compare against the provider's declared window, and never send a prompt above ~70% of it — route to the chunked path instead. After the call, compare the reported prompt-token count against the estimate and fail loudly on a material shortfall. That is the input-side equivalent of #293 and it is what would have caught a silent truncation.

Depends on the provider abstraction declaring a context limit per provider (v4.2.0).

Acceptance criteria

  • Every LLM call site preflights an estimate against a declared window
  • Prompts above the threshold route to the chunked path rather than being sent and hoped for
  • Reported prompt tokens are compared against the estimate; a shortfall fails loudly
  • The declared window is provider-supplied, not a constant
  • Tests cover under-budget, over-budget, and a provider that reports a truncated prompt
**Severity: CRITICAL.** Found in the August 2026 session lifecycle review (#319). ## The defect **Nothing anywhere estimates prompt size before sending, and nothing checks afterwards whether the server actually consumed the whole prompt.** The #293 work guards the *output* cap only (`llm_service.py:55-92`). `usage.prompt_tokens`, `prompt_eval_count`, and llama.cpp's `truncated` flag are all returned and all ignored. Per provider: - **llama.cpp** (`audio_service.py:2104-2135`) — sends no `max_tokens` and no context options. Builds with context shift **silently drop the front of the prompt**, which can take the system prompt and the character legend with it, and summarise the tail of the session as though it were the whole thing. `finish_reason` is read but only logged. - **OpenAI / Anthropic** — input overflow is a loud 400, so these fail safely on input. ## Proposed fix Estimate tokens before sending (characters ÷ ~3.4 is adequate; a real tokenizer where available), compare against the provider's declared window, and **never send a prompt above ~70% of it** — route to the chunked path instead. After the call, compare the reported prompt-token count against the estimate and fail loudly on a material shortfall. That is the input-side equivalent of #293 and it is what would have caught a silent truncation. Depends on the provider abstraction declaring a context limit per provider (v4.2.0). ## Acceptance criteria - [ ] Every LLM call site preflights an estimate against a declared window - [ ] Prompts above the threshold route to the chunked path rather than being sent and hoped for - [ ] Reported prompt tokens are compared against the estimate; a shortfall fails loudly - [ ] The declared window is provider-supplied, not a constant - [ ] Tests cover under-budget, over-budget, and a provider that reports a truncated prompt
claude-bot commented 2026-08-28 16:37:35 +00:00
Author
Collaborator
Copy link

Done in ede1ac2, with c54d9af (#337) doing the load-bearing half. One criterion amended, one moved.

What was actually wrong

The body says nothing preflights and nothing checks consumption. That was true when filed and stopped being true in ccc29d6, which built estimate_tokens, prompt_budget_tokens, fits_in_context, LLMConfig.context_tokens, resolve_context_tokens and _warn_if_prompt_truncated. The machinery was all there.

What was missing is that it went almost nowhere. One of eighteen generate_structured_text call sites passed a window (beat_service.py:454). Highlights, the entire lore pipeline, rephrase/promote/statblock/convert_stats, name generation, session titles, journal entries and the GM Workbench tool runner all sent prompts against a window they never declared — and because None meant "skip budgeting entirely", declaring nothing switched off the num_ctx protection too.

Acceptance criteria

  • Every LLM call site preflights an estimate against a declared window — preflight_prompt runs on all four provider paths in llm_service.py:355-420 and all four in _dispatch_prose. It counts the system prompt too, which shares the window; a preflight that ignores it is the accounting error that drops the speaker legend.
  • The declared window is provider-supplied, not a constant — context_tokens is now threaded through the 13 domain functions in audio_service and passed from llm_cfg.context_tokens at all 18 call sites (reminder_tasks, lore_service, generation_service, planning_tasks). The constant is the floor for anything undeclared, not the value.
  • Reported prompt tokens are compared against the estimate — _warn_if_prompt_truncated, on every provider response.
  • Tests cover under-budget, over-budget, and a provider that reports a truncated prompt — test_context_budget.py (including one proving the window survives into the statblock fallback call, not just the first attempt) and test_llm_transport.py:793-830.
  • Prompts above the threshold route to the chunked path rather than being sent and hoped for → amended, see below.

The criterion I did not meet as written

"Prompts above the threshold route to the chunked path."

The transport layer cannot route. It has no idea how to split a lore-merge or a statblock prompt — only callers that own a chunked path can, and the two that do (summarise, extraction_windows) already route. Building a generic chunker into the transport would mean inventing a split for prompts whose structure it cannot see.

So over-budget preflights warn loudly and send. It also does not raise, for a second reason: estimate_tokens is characters ÷ 3.4 and deliberately pessimistic, so a raise would refuse prompts that would have worked. Refusing to summarise a session beats summarising a tenth of one — but it does not beat summarising all of it. _warn_if_prompt_truncated reached the same conclusion from the other direction, and both become hard checks once a provider can declare a real tokenizer and whether it front-truncates (#351 / v4.2.0).

Scope moved off this issue

  • /ask — one of the 15 unprotected call sites, and now #426 (v4.2.0). Not fixed here.
  • Parallel map dispatch — belongs to #356, per the note in _summarise_chunked.

One thing worth remembering

Threading context_tokens broke 55 tests across ~20 files, and not visibly. Those files build a fake LLMConfig as a SimpleNamespace or an ad-hoc type(...), none carrying context_tokens. Reading it raised AttributeError inside task bodies that catch exceptions — so the symptom was not a crash but a generation result recorded as failed in tests asserting ready. A hand-rolled double drifts from the dataclass it stands for, and the drift surfaces as a wrong status rather than an error.

1,284 → 1,286 passing, lint clean.

**Done in `ede1ac2`**, with `c54d9af` (#337) doing the load-bearing half. One criterion amended, one moved. ## What was actually wrong The body says nothing preflights and nothing checks consumption. That was true when filed and stopped being true in `ccc29d6`, which built `estimate_tokens`, `prompt_budget_tokens`, `fits_in_context`, `LLMConfig.context_tokens`, `resolve_context_tokens` and `_warn_if_prompt_truncated`. The machinery was all there. What was missing is that it went almost nowhere. **One** of eighteen `generate_structured_text` call sites passed a window ([beat_service.py:454](webapp/backend/app/services/beat_service.py#L454)). Highlights, the entire lore pipeline, rephrase/promote/statblock/convert_stats, name generation, session titles, journal entries and the GM Workbench tool runner all sent prompts against a window they never declared — and because `None` meant "skip budgeting entirely", declaring nothing switched off the `num_ctx` protection too. ## Acceptance criteria - [x] **Every LLM call site preflights an estimate against a declared window** — `preflight_prompt` runs on all four provider paths in [llm_service.py:355-420](webapp/backend/app/services/llm_service.py#L355-L420) and all four in [`_dispatch_prose`](webapp/backend/app/services/audio_service.py#L1888-L1960). It counts the system prompt too, which shares the window; a preflight that ignores it is the accounting error that drops the speaker legend. - [x] **The declared window is provider-supplied, not a constant** — `context_tokens` is now threaded through the 13 domain functions in `audio_service` and passed from `llm_cfg.context_tokens` at all 18 call sites (`reminder_tasks`, `lore_service`, `generation_service`, `planning_tasks`). The constant is the floor for anything undeclared, not the value. - [x] **Reported prompt tokens are compared against the estimate** — `_warn_if_prompt_truncated`, on every provider response. - [x] **Tests cover under-budget, over-budget, and a provider that reports a truncated prompt** — [test_context_budget.py](webapp/backend/tests/test_context_budget.py) (including one proving the window survives into the statblock *fallback* call, not just the first attempt) and [test_llm_transport.py:793-830](webapp/backend/tests/test_llm_transport.py#L793-L830). - [ ] ~~Prompts above the threshold route to the chunked path rather than being sent and hoped for~~ → **amended, see below.** ## The criterion I did not meet as written *"Prompts above the threshold route to the chunked path."* The transport layer cannot route. It has no idea how to split a lore-merge or a statblock prompt — only callers that own a chunked path can, and the two that do (`summarise`, `extraction_windows`) already route. Building a generic chunker into the transport would mean inventing a split for prompts whose structure it cannot see. So over-budget preflights **warn loudly and send**. It also does not raise, for a second reason: `estimate_tokens` is characters ÷ 3.4 and deliberately pessimistic, so a raise would refuse prompts that would have worked. Refusing to summarise a session beats summarising a tenth of one — but it does not beat summarising all of it. `_warn_if_prompt_truncated` reached the same conclusion from the other direction, and both become hard checks once a provider can declare a real tokenizer and whether it front-truncates (**#351 / v4.2.0**). ## Scope moved off this issue - **`/ask`** — one of the 15 unprotected call sites, and now **#426** (v4.2.0). Not fixed here. - **Parallel map dispatch** — belongs to **#356**, per the note in `_summarise_chunked`. ## One thing worth remembering Threading `context_tokens` broke **55 tests** across ~20 files, and not visibly. Those files build a fake `LLMConfig` as a `SimpleNamespace` or an ad-hoc `type(...)`, none carrying `context_tokens`. Reading it raised `AttributeError` *inside* task bodies that catch exceptions — so the symptom was not a crash but a generation result recorded as `failed` in tests asserting `ready`. A hand-rolled double drifts from the dataclass it stands for, and the drift surfaces as a wrong status rather than an error. 1,284 → 1,286 passing, lint clean.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/1328-fill-it-in
feat/1328-fill-frontend
feat/1328-fill-backend
feat/9-foundry-push
fix/9-foundry-live
feat/9-foundry-push-frontend
renovate/celery-5.x
feat/5-foundry-connect
feat/9-foundry-push-backend
feat/9-foundry-push-module
feat/5-foundry-frontend
feat/5-foundry-backend
feat/5-foundry-module
release/v4.8.0-rc1
docs/d341-convert-draft
feat/958-update-to-version
feat/958-convert-sheet
feat/958-pathbuilder-import-frontend
fix/1360-heading-order
feat/958-pathbuilder-import-backend
feat/11-weekly-digest
feat/1341-auto-update
docs/d332-superseded
chore/1345-names-cleanup
feat/1342-mentioned-pages
feat/1328-recipes
fix/1346-pgdg-key
perf/1247-shared-recording-stream
chore/132-gitleaks-allowlist
release/v4.7.1
spike/foundry-module
release/v4.7.1-rc1
docs/decisions-d332-d337
fix/bot-certifi
chore/release-4.7.0
chore/release-4.7.0-rc4
chore/deps-4.7.0
chore/release-4.7.0-rc3
fix/1280-quote-must-say-it
fix/1257-names-row-squeeze
chore/release-4.7.0-rc2
fix/1245-pg-upgrade-count
chore/release-4.7.0-rc1
feat/1257-names-in-prep
feat/1257-be
feat/1257-fe
feat/1257-rating-fixes
design/1328-fill-it-in
feat/1257-name-splits
fix/1326-redis-503
feat/1280-fact-suggestions
feat/1280-be
feat/1280-fe
design/1280-fact-suggestions
feat/1279-infobox
feat/1279-be
feat/1279-fe
feat/1321-shelf-copies
design/1279-infobox
feat/1243-table-find
feat/1257-who-rated
feat/1243-prep-drawer
feat/1243-sheet-pickers
design/1243-browse-in-place
feat/1315-hotwords-default
feat/1242-system-filters
feat/1257-more-name-styles
feat/506-leg3-hotwords
design/1242-system-filters
fix/1302-packs-polish
feat/1241-rules-library
design/1241-rules-library
feat/1245-postgres-18
integ/4.6.2
fix/1263-map-ui-2
fix/1263-map-server
fix/1263-map-ui-1
fix/1261-editor-link-menu
fix/1229-axe-4.14-name-mismatch
renovate/axllent-mailpit-1.x
renovate/eslint-monorepo
renovate/authlib-1.x
chore/release-4.6.0
chore/release-4.6.0-rc2
fix/460-rc2-screens
fix/460-rc2-server
fix/release-prerelease-exit
chore/release-4.6.0-rc1
feat/125-campaign-templates
feat/126-calendar
feat/1181-pictures-and-files
feat/1190-event-people
integ/4.6.0
feat/1191-line-of-the-night
feat/1200-starter-packs
chore/bot-multidict-and-playwright-group
fix/1202-this-week-answered-count
fix/122-124-leftovers
fix/1178-get-routes-that-write
fix/1203-journeys-check-names
design/125-campaign-templates
feat/126-calendar-engine
design/126-in-world-calendar
fix/1189-cited-stamp-in-range
fix/1189-v4-looser-check
fix/1189-richer-beats
fix/1113-export-bind-lists
fix/1154-release-notes-by-file
chore/release-4.5.0
chore/release-4.5.0-rc12
fix/1135-wiki-markup-as-text
fix/demo-usage-cost-in-window
chore/release-4.5.0-rc11
fix/1121-sheet-consistency
fix/1121-sheet-5e-terms-and-card-hp
copy/1122-busiest-pages
fix/1121-relevant-choices
fix/1121-wiki-create-npc-save-sheet-links
fix/1029-menu-stays-on-screen
docs/prototype-transcript-bin-wording
fix/1023-editor-save-row-back-scroll
fix/1020-search-exact-first
fix/1020-table-prep
fix/1020-shell-wordmark-footer-bell
fix/1020-wiki-article-party
fix/872-admin-kept-until-phone-tables
feat/l13-milestone-level-waiting
fix/1000-1001-bell-older-setup-context
fix/999-pronouns-in-prompts
fix/l19-other-class-feats
fix/1010-transcript-window-ordinary-confirm
docs/d254-d269-decisions
fix/702-member-search
fix/680-transcript-bin
fix/993-unidentified-prompts
fix/841-redis-timeouts
fix/990-binned-not-exported
fix/830-export-prep-shelf
fix/deps-audit-2026-10-01
fix/877-title-reads
fix/798-audio-received-at
build/lane11-17-j7
build/lane11-16-levelup
build/lane11-14-seams
build/lane11-14-seed
build/lane11-15-start
build/lane11-13-old-block-out
fix/980-review-r3
build/lane11-12-block
build/lane11-8-packs
fix/demo-seed-canonical-names
build/lane11-11-hot-half
docs/l11-decisions
build/lane11-7-convert
build/lane11-10-editing
fix/npm-audit-brace-expansion
build/lane11-9-sheet-page
fix/bot-pyjwt
build/lane11-6-content
fix/lane11-rank-names
build/lane11-2b-language
build/lane11-4-js-evaluator
build/lane11-3-server
build/lane11-5a-pf2e
build/lane11-5b-5e
feat/design-admin
build/lane11-2-evaluator
build/lane11-1-sheet-pieces
build/lane11-0-drawing
fix/943-quest-objectives-fk
fix/prepsheet-shelf-flake
fix/945-leftovers
build/demo-gm-copy
fix/901-email-landings
build/lane10-8-journeys
build/demo-seed-lanes-8-10
fix/887-gm-own-confirmation
build/lane9-5-j1-admin
fix/918-audience-click
fix/useasync-overtaken
ci/939-critical-path
test/937-backend-suite-speed
chore/claude-orchestration
fix/924-xp-gm-only
fix/session-stale-poll
docs/decisions-d239-d242
fix/930-pypdf
build/lane8-journeys
build/925-quest-editor
fix/894-private-page-pictures
build/lane8-p4-drafts
build/lane8-p7-map
build/lane8-p6-handouts-rules
fix/wiki-fixture-names
build/lane10-d227-test-usage
fix/rsvp-horizon-test-tuesday
fix/media-volume-owner
build/lane8-p3-editor
fix/912-unidentified-rename
build/lane8-p5-party-quests
build/lane8-handout-takeback
build/lane10-pickers
build/lane8-p2-article
build/lane8-6-map
build/lane10-7b-email-pages
build/lane8-5-handouts
build/lane10-7a-test-the-bot-pages
build/lane8-p1-pages
build/lane8-4-quests
build/lane8-2b-feeds
build/lane8-3-parties
build/lane8-2a-links
build/lane10-5-commands
build/lane9-6-seed
build/lane10-4b-reactions
build/lane9-4d-pages
build/lane9-4c-pages
build/lane10-4a-posts
build/lane10-3-email
fix/891-web-fonts
build/lane9-4b-pages
build/lane10-6-test-the-bot
build/lane10-2-notify-out
build/lane8-1-wiki-text
build/lane9-4a-frame
design/lane10-drawings
fix/j6-budget
fix/j5-phone-flake
fix/869-test-leaks
fix/843-record-start-followups
build/lane7-all
build/lane6-all
build/lane9-b2-server
chore/changelog-fragments
docs/decisions-d218
build/lane9-a-owner-overview
fix/rule-card-level-label
fix/834-retention-semantics
fix/o3c-record-without-announcement
docs/decisions-d179
docs/decisions-d131-d170
fix/851-commit-before-response
build/lane6-j3-seed
build/lane9-b1-d67-identity
fix/841-blocking-loop
build/lane9-c-backups-audit
fix/vote-race-and-start-commit
fix/recording-announcement
fix/proposal-read-leak
fix/last-admin-guard
build/lane6-p4c-planning
build/lane6-p4b-margin
build/lane6-p4a-sheet
build/lane5-j4-seed
build/lane6-base
build/lane7-pages-talk
build/lane5-pages
build/lane7-pages-safety
build/lane7-p3-safety
build/lane7-p2-notify
build/lane6-p3-planning
build/lane6-p2-history
build/lane6-p1b-items
build/lane7-p1-threads
build/lane5-p6-faces
build/lane6-p1a-sheet
build/lane5-p5-find
build/lane5-p4a-board
build/lane5-p4b-scenes
build/lane5-p3-live
build/lane5-base
build/lane5-p2-cards
docs/lane5-decisions
build/review-r2-wide
fix/review-r1
build/lane5-p1-scenes
build/demo-seed
build/lane4-journeys-j5
build/lane4-page-review-b
build/lane4-page-review-a
build/lane4-page-read
build/lane4-journeys
build/lane4-page-sess2
build/lane4-page-sess1
build/lane4-notes
build/lane4-discord-on-approval
build/lane4-scheduling
build/lane4-versions
build/760-session-number
build/lane4-decisions
build/797-erase-no-regenerate
build/test-race-audit
build/lane3-completions
build/794-dark-only
build/796-titles-scroll
build/791-retention-anchor
build/a11y-sweep
build/shell-followups
build/retention-preview
build/753-backup-readback
docs/d91-d93
build/lane3-j1
build/ci-harness-paths-selftest
fix/useauth-test-race
build/ci-harness-paths
build/777-ai-settings-read
build/lane3-settings-c
build/lane3-settings-b
build/lane3-j6
build/lane3-sessionform
build/lane3-settings-a
build/lane3-members
build/lane3-thisweek
build/lane3-story
chore/87-align-python-3.12
v4.8.0-rc1
v4.7.1
v4.7.1-rc1
v4.7.0
v4.7.0-rc4
v4.7.0-rc3
v4.7.0-rc2
v4.7.0-rc1
v4.6.2
v4.6.2-rc4
v4.6.2-rc3
v4.6.2-rc2
v4.6.2-rc1
v4.6.1
v4.6.1-rc1
v4.6.0
v4.6.0-rc2
v4.6.0-rc1
v4.5.3
v4.5.3-rc1
v4.5.2
v4.5.2-rc1
v4.5.1
v4.5.1-rc2
v4.5.1-rc1
v4.5.0
v4.5.0-rc12
v4.5.0-rc11
v4.5.0-rc10
v4.5.0-rc9
v4.5.0-rc8
v4.5.0-rc7
v4.5.0-rc6
v4.5.0-rc5
v4.5.0-rc4
v4.5.0-rc3
v4.5.0-rc2
v4.5.0-rc1
v4.4.1
v4.4.0
v4.3.0
v4.2.3
v4.2.2
v4.2.1
v4.2.0
v4.1.2
v4.1.1
v4.1.0
v4.0.1
v4.0.0
v3.11.4
v3.11.3
v3.11.2
v3.11.1
v3.11.0
v3.10.0
v3.9.1
v3.9.0
v3.8.0
v3.7.0
v3.6.1
v3.6.0
v3.5.0
v3.4.1
v3.4.0
v3.3.0
v3.2.0
v3.1.0
v3.0.0
v2.1.0
v2.0.0
v1.1.0
v1.0.0
v0.12.0
v0.11.0
v0.10.0
v0.9.1
v0.3.3
v0.9.0
v0.8.0
v0.3.2
v0.3.1
v0.3.0
v0.7.0
v0.6.0
v0.5.0
v0.4.0
v0.2.1
v0.2.0
v0.1.1
v0.1.0
Labels
Clear labels   
backend
Issue related to the backend

  
bot
Issue related to the bot

  
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
frontend
Issue related to the frontend

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
recording
Issue related to the recording system

  
security
Security hardening / vulnerability work

  
wontfix
This won't be fixed

No labels
backend
bot
bug
duplicate
enhancement
frontend
help wanted
invalid
question
recording
security
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/Quest-Board#336
No description provided.