rbrooks/Quest-Board
Watch
1
0
Fork
You've already forked Quest-Board
 0
Code Issues 58 Pull requests 7 Projects Releases 76 Packages Wiki Activity Actions

[Backend] Set Ollama num_ctx and verify the prompt was actually consumed #337

New issue
Closed
opened 2026-08-25 20:38:56 +00:00 by claude-bot · 1 comment
claude-bot commented 2026-08-25 20:38:56 +00:00
Collaborator
Copy link

Severity: CRITICAL. Found in the August 2026 session lifecycle review (#319).

The defect

num_ctx is never set anywhere in the repository. Verified by grep across _summarise_ollama (audio_service.py:2089-2101) and _structured_ollama (llm_service.py:181-208).

Ollama's default context is 2048-4096 tokens and it silently truncates input beyond it. Any Ollama-configured install is therefore summarising roughly 2-3% of a real session while presenting the result as a summary of the whole thing — with no error, no warning, and a plausible-looking output. prompt_eval_count comes back in the response and is ignored.

This is not a hypothetical for a self-hosted-first product: Ollama is the most likely local backend a self-hoster reaches for.

Proposed fix

Set options.num_ctx explicitly on every Ollama call, sourced from the provider's declared window. Read prompt_eval_count from the response and compare it against the preflight estimate; a material shortfall means the prompt was truncated, and that must fail loudly rather than persist a confidently wrong summary.

Acceptance criteria

  • options.num_ctx set on every Ollama request
  • prompt_eval_count compared against the estimate on every response
  • A truncated prompt raises rather than returning a result
  • The configured value is surfaced in Admin → Bot Settings with an explanation
  • A regression test asserts num_ctx is present in the request body
**Severity: CRITICAL.** Found in the August 2026 session lifecycle review (#319). ## The defect **`num_ctx` is never set anywhere in the repository.** Verified by grep across `_summarise_ollama` (`audio_service.py:2089-2101`) and `_structured_ollama` (`llm_service.py:181-208`). Ollama's default context is 2048-4096 tokens and it **silently truncates** input beyond it. Any Ollama-configured install is therefore summarising roughly 2-3% of a real session while presenting the result as a summary of the whole thing — with no error, no warning, and a plausible-looking output. `prompt_eval_count` comes back in the response and is ignored. This is not a hypothetical for a self-hosted-first product: Ollama is the most likely local backend a self-hoster reaches for. ## Proposed fix Set `options.num_ctx` explicitly on every Ollama call, sourced from the provider's declared window. Read `prompt_eval_count` from the response and compare it against the preflight estimate; a material shortfall means the prompt was truncated, and that must fail loudly rather than persist a confidently wrong summary. ## Acceptance criteria - [ ] `options.num_ctx` set on every Ollama request - [ ] `prompt_eval_count` compared against the estimate on every response - [ ] A truncated prompt raises rather than returning a result - [ ] The configured value is surfaced in Admin → Bot Settings with an explanation - [ ] A regression test asserts `num_ctx` is present in the request body
claude-bot commented 2026-08-28 15:59:07 +00:00
Author
Collaborator
Copy link

Done in c54d9af. Verified against acceptance criteria below, with one criterion amended rather than met as written.

The issue's premise was half out of date

The body says "num_ctx is never set anywhere in the repository. Verified by grep." That stopped being true in ccc29d6, which added it to both _structured_ollama and _summarise_ollama — and test_ollama_greedy_decoding_does_not_clobber_num_ctx has been green ever since.

The defect was still real, just one level in: the whole protection sat behind if context_tokens:, and of the eighteen generate_structured_text call sites, exactly one passed a window (beat_service.py:454). So almost every Ollama request in the product still fell back to Ollama's 2-4k default and was silently truncated — the exact failure this issue describes, in a file that looked like it fixed it. A grep for num_ctx said "fixed"; a grep for context_tokens= said otherwise.

Deviation from the proposed fix, recorded

The issue proposes sourcing num_ctx "from the provider's declared window", i.e. threading context_tokens through the other seventeen call sites. That is now #336's job, but it is not what makes this safe, and shipping only that would have been a mistake: it fixes seventeen call sites and leaves the eighteenth — written next month — unprotected again, with no test able to notice, because "nobody passed a window" and "this call doesn't need one" are indistinguishable.

So None now resolves to a conservative default instead of switching budgeting off (llm_service.py:61-84), and num_ctx is set unconditionally (llm_service.py:474-490, audio_service.py:3849-3854). #336's plumbing becomes an accuracy improvement rather than the thing standing between a self-hoster and a summary of 3% of their session.

32,768 is not a new memory risk on Ollama. It is the value LLMConfig already defaults to, so the largest prompt in the product — summarise via get_llm_config — has been sending num_ctx: 32768 all along. Applying it to the smaller calls cannot raise the allocation ceiling above one already in production.

Acceptance criteria

  • options.num_ctx set on every Ollama request — llm_service.py:474 (structured) and audio_service.py:3849 (prose). Unconditional on both.
  • prompt_eval_count compared against the estimate on every response — _warn_if_prompt_truncated, llm_service.py:503 and audio_service.py:3862. Joined now by preflight_prompt, the input-side check that runs before the request — which matters because a backend with context shift returns an ordinary 200 for a prompt whose front it just discarded, and reports nothing at all.
  • The configured value is surfaced in Admin → Bot Settings with an explanation — already shipped: Admin.jsx:731-754, including the resolved value currently in effect.
  • A regression test asserts num_ctx is present in the request body — five of them, test_llm_transport.py:722-790, covering structured, prose, an explicit window, and five flavours of junk setting.
  • A truncated prompt raises rather than returning a result → amended. See below.

The one criterion not met as written

"A truncated prompt raises rather than returning a result."

_warn_if_prompt_truncated warns and returns, and has since ccc29d6, for a documented reason: providers that cache a prompt prefix legitimately report only newly-evaluated tokens, so a low prompt_eval_count means either "your prompt was truncated" or "most of it was cached" and from here those are identical. Raising would fail perfectly good requests on a hosted provider.

preflight_prompt makes the same call for a different reason — estimate_tokens is characters ÷ 3.4 and deliberately pessimistic, so a raise would refuse prompts that would have worked.

Both become hard checks when the provider abstraction can declare a real tokenizer and whether a backend front-truncates. That is #351 / v4.2.0, and this criterion belongs there rather than here. Consistent with the same decision on #338: flag, don't raise, where the signal is ambiguous.

Verified against the old behaviour

All ten new tests were run against the pre-change code with app/ stashed. All ten fail there, and with the right failure mode — KeyError: 'options', i.e. the key was absent from the request body entirely, not merely an assertion mismatch. The two that pass on both sides are regression guards on the explicit-window path and are labelled as such.

A latent crash turned up while testing it: if context_tokens: let a non-numeric value through to int(), so a typo'd context window in Admin raised ValueError from the transport layer rather than falling back. Covered by the parametrised test.

1,273 → 1,284 passing, lint clean.

**Done in `c54d9af`.** Verified against acceptance criteria below, with one criterion amended rather than met as written. ## The issue's premise was half out of date The body says *"`num_ctx` is never set anywhere in the repository. Verified by grep."* That stopped being true in `ccc29d6`, which added it to both `_structured_ollama` and `_summarise_ollama` — and `test_ollama_greedy_decoding_does_not_clobber_num_ctx` has been green ever since. The defect was still real, just one level in: the whole protection sat behind `if context_tokens:`, and of the eighteen `generate_structured_text` call sites, **exactly one** passed a window (`beat_service.py:454`). So almost every Ollama request in the product still fell back to Ollama's 2-4k default and was silently truncated — the exact failure this issue describes, in a file that looked like it fixed it. A grep for `num_ctx` said "fixed"; a grep for `context_tokens=` said otherwise. ## Deviation from the proposed fix, recorded The issue proposes sourcing `num_ctx` "from the provider's declared window", i.e. threading `context_tokens` through the other seventeen call sites. That is now #336's job, but it is **not** what makes this safe, and shipping only that would have been a mistake: it fixes seventeen call sites and leaves the eighteenth — written next month — unprotected again, with no test able to notice, because "nobody passed a window" and "this call doesn't need one" are indistinguishable. So `None` now resolves to a conservative default instead of switching budgeting off ([llm_service.py:61-84](webapp/backend/app/services/llm_service.py#L61-L84)), and `num_ctx` is set unconditionally ([llm_service.py:474-490](webapp/backend/app/services/llm_service.py#L474-L490), [audio_service.py:3849-3854](webapp/backend/app/services/audio_service.py#L3849-L3854)). #336's plumbing becomes an accuracy improvement rather than the thing standing between a self-hoster and a summary of 3% of their session. **32,768 is not a new memory risk on Ollama.** It is the value `LLMConfig` already defaults to, so the largest prompt in the product — `summarise` via `get_llm_config` — has been sending `num_ctx: 32768` all along. Applying it to the smaller calls cannot raise the allocation ceiling above one already in production. ## Acceptance criteria - [x] **`options.num_ctx` set on every Ollama request** — [llm_service.py:474](webapp/backend/app/services/llm_service.py#L474) (structured) and [audio_service.py:3849](webapp/backend/app/services/audio_service.py#L3849) (prose). Unconditional on both. - [x] **`prompt_eval_count` compared against the estimate on every response** — `_warn_if_prompt_truncated`, [llm_service.py:503](webapp/backend/app/services/llm_service.py#L503) and [audio_service.py:3862](webapp/backend/app/services/audio_service.py#L3862). Joined now by `preflight_prompt`, the input-side check that runs *before* the request — which matters because a backend with context shift returns an ordinary 200 for a prompt whose front it just discarded, and reports nothing at all. - [x] **The configured value is surfaced in Admin → Bot Settings with an explanation** — already shipped: [Admin.jsx:731-754](webapp/frontend/src/pages/Admin.jsx#L731-L754), including the resolved value currently in effect. - [x] **A regression test asserts `num_ctx` is present in the request body** — five of them, [test_llm_transport.py:722-790](webapp/backend/tests/test_llm_transport.py#L722-L790), covering structured, prose, an explicit window, and five flavours of junk setting. - [ ] ~~A truncated prompt raises rather than returning a result~~ → **amended.** See below. ## The one criterion not met as written *"A truncated prompt raises rather than returning a result."* `_warn_if_prompt_truncated` warns and returns, and has since `ccc29d6`, for a documented reason: providers that cache a prompt prefix legitimately report only newly-evaluated tokens, so a low `prompt_eval_count` means either "your prompt was truncated" or "most of it was cached" and from here those are identical. Raising would fail perfectly good requests on a hosted provider. `preflight_prompt` makes the same call for a different reason — `estimate_tokens` is characters ÷ 3.4 and deliberately pessimistic, so a raise would refuse prompts that would have worked. Both become hard checks when the provider abstraction can declare a real tokenizer and whether a backend front-truncates. That is **#351 / v4.2.0**, and this criterion belongs there rather than here. Consistent with the same decision on #338: flag, don't raise, where the signal is ambiguous. ## Verified against the old behaviour All ten new tests were run against the pre-change code with `app/` stashed. All ten fail there, and with the right failure mode — `KeyError: 'options'`, i.e. the key was absent from the request body entirely, not merely an assertion mismatch. The two that pass on both sides are regression guards on the explicit-window path and are labelled as such. A latent crash turned up while testing it: `if context_tokens:` let a non-numeric value through to `int()`, so a typo'd context window in Admin raised `ValueError` from the transport layer rather than falling back. Covered by the parametrised test. 1,273 → 1,284 passing, lint clean.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/1328-fill-it-in
feat/1328-fill-frontend
feat/1328-fill-backend
feat/9-foundry-push
fix/9-foundry-live
feat/9-foundry-push-frontend
renovate/celery-5.x
feat/5-foundry-connect
feat/9-foundry-push-backend
feat/9-foundry-push-module
feat/5-foundry-frontend
feat/5-foundry-backend
feat/5-foundry-module
release/v4.8.0-rc1
docs/d341-convert-draft
feat/958-update-to-version
feat/958-convert-sheet
feat/958-pathbuilder-import-frontend
fix/1360-heading-order
feat/958-pathbuilder-import-backend
feat/11-weekly-digest
feat/1341-auto-update
docs/d332-superseded
chore/1345-names-cleanup
feat/1342-mentioned-pages
feat/1328-recipes
fix/1346-pgdg-key
perf/1247-shared-recording-stream
chore/132-gitleaks-allowlist
release/v4.7.1
spike/foundry-module
release/v4.7.1-rc1
docs/decisions-d332-d337
fix/bot-certifi
chore/release-4.7.0
chore/release-4.7.0-rc4
chore/deps-4.7.0
chore/release-4.7.0-rc3
fix/1280-quote-must-say-it
fix/1257-names-row-squeeze
chore/release-4.7.0-rc2
fix/1245-pg-upgrade-count
chore/release-4.7.0-rc1
feat/1257-names-in-prep
feat/1257-be
feat/1257-fe
feat/1257-rating-fixes
design/1328-fill-it-in
feat/1257-name-splits
fix/1326-redis-503
feat/1280-fact-suggestions
feat/1280-be
feat/1280-fe
design/1280-fact-suggestions
feat/1279-infobox
feat/1279-be
feat/1279-fe
feat/1321-shelf-copies
design/1279-infobox
feat/1243-table-find
feat/1257-who-rated
feat/1243-prep-drawer
feat/1243-sheet-pickers
design/1243-browse-in-place
feat/1315-hotwords-default
feat/1242-system-filters
feat/1257-more-name-styles
feat/506-leg3-hotwords
design/1242-system-filters
fix/1302-packs-polish
feat/1241-rules-library
design/1241-rules-library
feat/1245-postgres-18
integ/4.6.2
fix/1263-map-ui-2
fix/1263-map-server
fix/1263-map-ui-1
fix/1261-editor-link-menu
fix/1229-axe-4.14-name-mismatch
renovate/axllent-mailpit-1.x
renovate/eslint-monorepo
renovate/authlib-1.x
chore/release-4.6.0
chore/release-4.6.0-rc2
fix/460-rc2-screens
fix/460-rc2-server
fix/release-prerelease-exit
chore/release-4.6.0-rc1
feat/125-campaign-templates
feat/126-calendar
feat/1181-pictures-and-files
feat/1190-event-people
integ/4.6.0
feat/1191-line-of-the-night
feat/1200-starter-packs
chore/bot-multidict-and-playwright-group
fix/1202-this-week-answered-count
fix/122-124-leftovers
fix/1178-get-routes-that-write
fix/1203-journeys-check-names
design/125-campaign-templates
feat/126-calendar-engine
design/126-in-world-calendar
fix/1189-cited-stamp-in-range
fix/1189-v4-looser-check
fix/1189-richer-beats
fix/1113-export-bind-lists
fix/1154-release-notes-by-file
chore/release-4.5.0
chore/release-4.5.0-rc12
fix/1135-wiki-markup-as-text
fix/demo-usage-cost-in-window
chore/release-4.5.0-rc11
fix/1121-sheet-consistency
fix/1121-sheet-5e-terms-and-card-hp
copy/1122-busiest-pages
fix/1121-relevant-choices
fix/1121-wiki-create-npc-save-sheet-links
fix/1029-menu-stays-on-screen
docs/prototype-transcript-bin-wording
fix/1023-editor-save-row-back-scroll
fix/1020-search-exact-first
fix/1020-table-prep
fix/1020-shell-wordmark-footer-bell
fix/1020-wiki-article-party
fix/872-admin-kept-until-phone-tables
feat/l13-milestone-level-waiting
fix/1000-1001-bell-older-setup-context
fix/999-pronouns-in-prompts
fix/l19-other-class-feats
fix/1010-transcript-window-ordinary-confirm
docs/d254-d269-decisions
fix/702-member-search
fix/680-transcript-bin
fix/993-unidentified-prompts
fix/841-redis-timeouts
fix/990-binned-not-exported
fix/830-export-prep-shelf
fix/deps-audit-2026-10-01
fix/877-title-reads
fix/798-audio-received-at
build/lane11-17-j7
build/lane11-16-levelup
build/lane11-14-seams
build/lane11-14-seed
build/lane11-15-start
build/lane11-13-old-block-out
fix/980-review-r3
build/lane11-12-block
build/lane11-8-packs
fix/demo-seed-canonical-names
build/lane11-11-hot-half
docs/l11-decisions
build/lane11-7-convert
build/lane11-10-editing
fix/npm-audit-brace-expansion
build/lane11-9-sheet-page
fix/bot-pyjwt
build/lane11-6-content
fix/lane11-rank-names
build/lane11-2b-language
build/lane11-4-js-evaluator
build/lane11-3-server
build/lane11-5a-pf2e
build/lane11-5b-5e
feat/design-admin
build/lane11-2-evaluator
build/lane11-1-sheet-pieces
build/lane11-0-drawing
fix/943-quest-objectives-fk
fix/prepsheet-shelf-flake
fix/945-leftovers
build/demo-gm-copy
fix/901-email-landings
build/lane10-8-journeys
build/demo-seed-lanes-8-10
fix/887-gm-own-confirmation
build/lane9-5-j1-admin
fix/918-audience-click
fix/useasync-overtaken
ci/939-critical-path
test/937-backend-suite-speed
chore/claude-orchestration
fix/924-xp-gm-only
fix/session-stale-poll
docs/decisions-d239-d242
fix/930-pypdf
build/lane8-journeys
build/925-quest-editor
fix/894-private-page-pictures
build/lane8-p4-drafts
build/lane8-p7-map
build/lane8-p6-handouts-rules
fix/wiki-fixture-names
build/lane10-d227-test-usage
fix/rsvp-horizon-test-tuesday
fix/media-volume-owner
build/lane8-p3-editor
fix/912-unidentified-rename
build/lane8-p5-party-quests
build/lane8-handout-takeback
build/lane10-pickers
build/lane8-p2-article
build/lane8-6-map
build/lane10-7b-email-pages
build/lane8-5-handouts
build/lane10-7a-test-the-bot-pages
build/lane8-p1-pages
build/lane8-4-quests
build/lane8-2b-feeds
build/lane8-3-parties
build/lane8-2a-links
build/lane10-5-commands
build/lane9-6-seed
build/lane10-4b-reactions
build/lane9-4d-pages
build/lane9-4c-pages
build/lane10-4a-posts
build/lane10-3-email
fix/891-web-fonts
build/lane9-4b-pages
build/lane10-6-test-the-bot
build/lane10-2-notify-out
build/lane8-1-wiki-text
build/lane9-4a-frame
design/lane10-drawings
fix/j6-budget
fix/j5-phone-flake
fix/869-test-leaks
fix/843-record-start-followups
build/lane7-all
build/lane6-all
build/lane9-b2-server
chore/changelog-fragments
docs/decisions-d218
build/lane9-a-owner-overview
fix/rule-card-level-label
fix/834-retention-semantics
fix/o3c-record-without-announcement
docs/decisions-d179
docs/decisions-d131-d170
fix/851-commit-before-response
build/lane6-j3-seed
build/lane9-b1-d67-identity
fix/841-blocking-loop
build/lane9-c-backups-audit
fix/vote-race-and-start-commit
fix/recording-announcement
fix/proposal-read-leak
fix/last-admin-guard
build/lane6-p4c-planning
build/lane6-p4b-margin
build/lane6-p4a-sheet
build/lane5-j4-seed
build/lane6-base
build/lane7-pages-talk
build/lane5-pages
build/lane7-pages-safety
build/lane7-p3-safety
build/lane7-p2-notify
build/lane6-p3-planning
build/lane6-p2-history
build/lane6-p1b-items
build/lane7-p1-threads
build/lane5-p6-faces
build/lane6-p1a-sheet
build/lane5-p5-find
build/lane5-p4a-board
build/lane5-p4b-scenes
build/lane5-p3-live
build/lane5-base
build/lane5-p2-cards
docs/lane5-decisions
build/review-r2-wide
fix/review-r1
build/lane5-p1-scenes
build/demo-seed
build/lane4-journeys-j5
build/lane4-page-review-b
build/lane4-page-review-a
build/lane4-page-read
build/lane4-journeys
build/lane4-page-sess2
build/lane4-page-sess1
build/lane4-notes
build/lane4-discord-on-approval
build/lane4-scheduling
build/lane4-versions
build/760-session-number
build/lane4-decisions
build/797-erase-no-regenerate
build/test-race-audit
build/lane3-completions
build/794-dark-only
build/796-titles-scroll
build/791-retention-anchor
build/a11y-sweep
build/shell-followups
build/retention-preview
build/753-backup-readback
docs/d91-d93
build/lane3-j1
build/ci-harness-paths-selftest
fix/useauth-test-race
build/ci-harness-paths
build/777-ai-settings-read
build/lane3-settings-c
build/lane3-settings-b
build/lane3-j6
build/lane3-sessionform
build/lane3-settings-a
build/lane3-members
build/lane3-thisweek
build/lane3-story
chore/87-align-python-3.12
v4.8.0-rc1
v4.7.1
v4.7.1-rc1
v4.7.0
v4.7.0-rc4
v4.7.0-rc3
v4.7.0-rc2
v4.7.0-rc1
v4.6.2
v4.6.2-rc4
v4.6.2-rc3
v4.6.2-rc2
v4.6.2-rc1
v4.6.1
v4.6.1-rc1
v4.6.0
v4.6.0-rc2
v4.6.0-rc1
v4.5.3
v4.5.3-rc1
v4.5.2
v4.5.2-rc1
v4.5.1
v4.5.1-rc2
v4.5.1-rc1
v4.5.0
v4.5.0-rc12
v4.5.0-rc11
v4.5.0-rc10
v4.5.0-rc9
v4.5.0-rc8
v4.5.0-rc7
v4.5.0-rc6
v4.5.0-rc5
v4.5.0-rc4
v4.5.0-rc3
v4.5.0-rc2
v4.5.0-rc1
v4.4.1
v4.4.0
v4.3.0
v4.2.3
v4.2.2
v4.2.1
v4.2.0
v4.1.2
v4.1.1
v4.1.0
v4.0.1
v4.0.0
v3.11.4
v3.11.3
v3.11.2
v3.11.1
v3.11.0
v3.10.0
v3.9.1
v3.9.0
v3.8.0
v3.7.0
v3.6.1
v3.6.0
v3.5.0
v3.4.1
v3.4.0
v3.3.0
v3.2.0
v3.1.0
v3.0.0
v2.1.0
v2.0.0
v1.1.0
v1.0.0
v0.12.0
v0.11.0
v0.10.0
v0.9.1
v0.3.3
v0.9.0
v0.8.0
v0.3.2
v0.3.1
v0.3.0
v0.7.0
v0.6.0
v0.5.0
v0.4.0
v0.2.1
v0.2.0
v0.1.1
v0.1.0
Labels
Clear labels   
backend
Issue related to the backend

  
bot
Issue related to the bot

  
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
frontend
Issue related to the frontend

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
recording
Issue related to the recording system

  
security
Security hardening / vulnerability work

  
wontfix
This won't be fixed

No labels
backend
bot
bug
duplicate
enhancement
frontend
help wanted
invalid
question
recording
security
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/Quest-Board#337
No description provided.