rbrooks/Quest-Board

Watch

Fork 0

[Backend] Prod's configured LLM context (65536) is larger than the server's real per-request window (40960), so long-prompt calls fail with 400 #613

New issue

Closed

opened 2026-09-12 01:30:01 +00:00 by claude-bot · 2 comments

claude-bot commented

2026-09-12 01:30:01 +00:00

Collaborator

What happens

The quote board got nothing from the v4.4.0 reprocess of session 5b0d6ac1:

extract_highlights: LLM call failed for window 1/1: RuntimeError
process_audio: extracted 0 highlights — keeping any existing highlights rather than clearing them

The session's stored highlights are therefore stale — 7 rows extracted from the transcript that reprocessing has since replaced.

Cause

Reproduced deterministically against the real transcript. The message the log deliberately withholds is:

llama.cpp returned 400: {"error":{"code":400,
  "message":"request (46527 tokens) exceeds the available context size (40960 tokens), try increasing it",
  "type":"exceed_context_size_error","n_prompt_tokens":46527,"n_ctx":40960}}

Quest Board is configured for a 65536-token context (llm_config.context_tokens).
The router advertises --ctx-size 81920 for gemma4, but that is the instance context shared across its slots. The window available to one request is 40960.
fits_in_context therefore passes a prompt it should have split: 70% of 65536 is ~45900 tokens, and the built request came to 46527 — comfortably inside the believed budget and well outside the real one.
max_tokens is irrelevant. The default 2048, 8192 and None all fail identically, because the prompt alone exceeds the window before generation starts.

This is not specific to highlights. Highlights is simply the first caller to send one large window. The beats pipeline survives because it chunks small. Any long-prompt call on prod — a large lore extraction window, a long stat-block conversion — is exposed to the same 400.

Why it was invisible

extract_highlights logs type(exc).__name__ and never the message, so a configuration error that names its own fix ("try increasing it") surfaced as the bare word RuntimeError. That reticence is deliberate (#279: a provider can echo the prompt into an error), but it cost a diagnosis here.

Fix

Correct the configured context for this deployment to 40960, in Admin → Bot Settings. Cheapest, and immediately unblocks every caller.
Or give the model a real 65536+ per-request window on the router (fewer slots, or a larger --ctx-size), if the GPU allows.
Stop trusting the configured number blindly. The provider contract already has a place to declare the window; a mismatch between the declared context and the server's actual n_ctx is discoverable — llama.cpp returns it in the error body (n_ctx), and a 400 of type: exceed_context_size_error could clamp the stored value and retry once, rather than failing the window silently.
Log the provider's message at WARNING for 4xx specifically. A 400 is the server describing a caller mistake, not an echo risk in the way a 200-with-garbage is; the current blanket rule made a self-describing error unreadable.

Follow-up

session_highlights for 5b0d6ac1 still holds 7 rows from the pre-reprocess transcript. Once the context is corrected they should be re-extracted so the quote board matches the transcript that now exists.

Found while reprocessing the 9/8 session on v4.4.0 (beats went 4/23 → 88/93; the summary itself is unaffected by this bug).

## What happens The quote board got nothing from the v4.4.0 reprocess of session `5b0d6ac1`: ``` extract_highlights: LLM call failed for window 1/1: RuntimeError process_audio: extracted 0 highlights — keeping any existing highlights rather than clearing them ``` The session's stored highlights are therefore **stale** — 7 rows extracted from the transcript that reprocessing has since replaced. ## Cause Reproduced deterministically against the real transcript. The message the log deliberately withholds is: ``` llama.cpp returned 400: {"error":{"code":400, "message":"request (46527 tokens) exceeds the available context size (40960 tokens), try increasing it", "type":"exceed_context_size_error","n_prompt_tokens":46527,"n_ctx":40960}} ``` - **Quest Board is configured for a 65536-token context** (`llm_config.context_tokens`). - **The router advertises `--ctx-size 81920`** for `gemma4`, but that is the *instance* context shared across its slots. The window available to one request is **40960**. - `fits_in_context` therefore passes a prompt it should have split: 70% of 65536 is ~45900 tokens, and the built request came to 46527 — comfortably inside the believed budget and well outside the real one. - `max_tokens` is irrelevant. The default 2048, 8192 and `None` all fail identically, because the prompt alone exceeds the window before generation starts. **This is not specific to highlights.** Highlights is simply the first caller to send one large window. The beats pipeline survives because it chunks small. Any long-prompt call on prod — a large lore extraction window, a long stat-block conversion — is exposed to the same 400. ## Why it was invisible `extract_highlights` logs `type(exc).__name__` and never the message, so a configuration error that names its own fix ("try increasing it") surfaced as the bare word `RuntimeError`. That reticence is deliberate (#279: a provider can echo the prompt into an error), but it cost a diagnosis here. ## Fix 1. **Correct the configured context for this deployment** to 40960, in Admin → Bot Settings. Cheapest, and immediately unblocks every caller. 2. **Or give the model a real 65536+ per-request window** on the router (fewer slots, or a larger `--ctx-size`), if the GPU allows. 3. **Stop trusting the configured number blindly.** The provider contract already has a place to declare the window; a mismatch between the declared context and the server's actual `n_ctx` is discoverable — llama.cpp returns it in the error body (`n_ctx`), and a 400 of `type: exceed_context_size_error` could clamp the stored value and retry once, rather than failing the window silently. 4. **Log the provider's message at WARNING for 4xx specifically.** A 400 is the server describing a caller mistake, not an echo risk in the way a 200-with-garbage is; the current blanket rule made a self-describing error unreadable. ## Follow-up `session_highlights` for `5b0d6ac1` still holds 7 rows from the pre-reprocess transcript. Once the context is corrected they should be re-extracted so the quote board matches the transcript that now exists. Found while reprocessing the 9/8 session on v4.4.0 (beats went 4/23 → 88/93; the summary itself is unaffected by this bug).

claude-bot added the

backend

bug

labels

2026-09-12 01:30:05 +00:00

claude-bot commented

2026-09-12 01:45:48 +00:00

Author

Collaborator

Correction: the context setting was not the root cause

The owner set the context to 40960 — the server's true per-request window — and the highlights call still failed, now on window 1 of 2. Measured against the real transcript:

	tokens
`estimate_tokens` for window 1's prompt + system prompt	27,174
what llama.cpp actually counted (`n_prompt_tokens`)	43,382
the window it had (`n_ctx`)	40,960

So the budget arithmetic is fine and the configured context is now correct. The estimator is what is wrong: _CHARS_PER_TOKEN = 3.4 under-counts this text by about 60%.

That constant is documented as "deliberately on the low side so the estimate errs toward 'too big'" — and for English prose it does. A merged transcript is not prose: every line carries [01:12:47] Viquilonto (Viq): and the body is short fragments, disfluencies and names. Punctuation and rare tokens push the real ratio close to 2 characters per token, so the estimate lands far under and fits_in_context cheerfully passes a prompt the server must reject.

This invalidates fix options 1 and 2 as written above. No configured context value fixes it in general, because the error scales with the text: a bigger window just moves the point at which a long session overruns. Lowering the setting to ~25000 would absorb this session (40960 × 27174/43382), but that is a magic number hiding a broken estimate.

Chosen fix: make the 400 self-correcting

Agreed with the owner:

Raise a typed error, not a string. llm_service currently converts the 400 into a bare RuntimeError whose message is the only record of n_ctx / n_prompt_tokens. A dedicated exception carrying both is what makes recovery possible — and it fixes the diagnosability complaint above for free.
Learn the real ratio from the rejection. The server tells us exactly what it counted. n_prompt_tokens / estimate_tokens(prompt) is a measured calibration for this endpoint and this kind of text; persist it and budget against it thereafter.
Retry the window once, re-split. Windowing belongs to the caller (highlights, beat extraction, lore chunking), so the recovery has to happen there: on the typed error, re-split with the corrected budget and try again. One retry, not a loop.

Worth noting the blast radius this explains: the beats pipeline has been silently sending over-large prompts all along and only survived because its windows are small enough to stay under the real limit by luck. #575's "it will be sent anyway" warning is the same underestimate viewed from the other side.

Also: a caller that ignored a failed window

Re-extracting the quote board for 5b0d6ac1 with window 1 failing and window 2 succeeding produced a board built only from the session's last twenty minutes, and it was written to the database. extract_highlights returns whatever it collected without telling the caller that a window was lost, so "partial" is indistinguishable from "complete". Whatever shape the retry takes, a caller that persists highlights needs to know how many windows failed — and ideally refuse to replace a good set with a partial one.

## Correction: the context setting was not the root cause The owner set the context to 40960 — the server's true per-request window — and the highlights call **still failed**, now on window 1 of 2. Measured against the real transcript: | | tokens | |---|---| | `estimate_tokens` for window 1's prompt + system prompt | **27,174** | | what llama.cpp actually counted (`n_prompt_tokens`) | **43,382** | | the window it had (`n_ctx`) | 40,960 | So the budget arithmetic is fine and the configured context is now correct. **The estimator is what is wrong**: `_CHARS_PER_TOKEN = 3.4` under-counts this text by about 60%. That constant is documented as "deliberately on the low side so the estimate errs toward 'too big'" — and for English prose it does. A merged transcript is not prose: every line carries `[01:12:47] Viquilonto (Viq): ` and the body is short fragments, disfluencies and names. Punctuation and rare tokens push the real ratio close to 2 characters per token, so the estimate lands far under and `fits_in_context` cheerfully passes a prompt the server must reject. **This invalidates fix options 1 and 2 as written above.** No configured context value fixes it in general, because the error scales with the text: a bigger window just moves the point at which a long session overruns. Lowering the setting to ~25000 would absorb *this* session (40960 × 27174/43382), but that is a magic number hiding a broken estimate. ## Chosen fix: make the 400 self-correcting Agreed with the owner: 1. **Raise a typed error, not a string.** `llm_service` currently converts the 400 into a bare `RuntimeError` whose message is the only record of `n_ctx` / `n_prompt_tokens`. A dedicated exception carrying both is what makes recovery possible — and it fixes the diagnosability complaint above for free. 2. **Learn the real ratio from the rejection.** The server tells us exactly what it counted. `n_prompt_tokens / estimate_tokens(prompt)` is a measured calibration for this endpoint and this kind of text; persist it and budget against it thereafter. 3. **Retry the window once, re-split.** Windowing belongs to the caller (highlights, beat extraction, lore chunking), so the recovery has to happen there: on the typed error, re-split with the corrected budget and try again. One retry, not a loop. Worth noting the blast radius this explains: **the beats pipeline has been silently sending over-large prompts all along** and only survived because its windows are small enough to stay under the real limit by luck. #575's "it will be sent anyway" warning is the same underestimate viewed from the other side. ## Also: a caller that ignored a failed window Re-extracting the quote board for `5b0d6ac1` with window 1 failing and window 2 succeeding produced a board built **only from the session's last twenty minutes**, and it was written to the database. `extract_highlights` returns whatever it collected without telling the caller that a window was lost, so "partial" is indistinguishable from "complete". Whatever shape the retry takes, a caller that persists highlights needs to know how many windows failed — and ideally refuse to replace a good set with a partial one.

claude-bot referenced this issue

2026-09-12 01:51:00 +00:00

[Backend] Recover from a context-window rejection: learn the real token ratio and re-split the window once #614

rbrooks referenced this issue from a commit

2026-09-12 02:41:30 +00:00

fix(webapp): recover from a context-window rejection instead of losing the window (#614, #613)

claude-bot referenced this issue

2026-09-12 02:43:35 +00:00

fix(webapp): recover from a context-window rejection instead of losing the window (#614, #613) #615

claude-bot referenced this issue from a commit

2026-09-12 03:03:53 +00:00

Merge pull request 'fix(webapp): recover from a context-window rejection instead of losing the window (#614, #613)' (#615) from fix/614-context-window-retry into main

rbrooks referenced this issue from a commit

2026-09-12 03:05:41 +00:00

chore(release): v4.4.1

claude-bot referenced this issue

2026-09-12 03:08:57 +00:00

chore(release): v4.4.1 #616

claude-bot commented

2026-09-12 03:35:11 +00:00

Author

Collaborator

Resolved by #614, released as v4.4.1 and verified on dev and production — see #614 for the measured before/after.

What this issue got right, and what it got wrong

The first diagnosis here — "the configured context (65536) is larger than the server's real window (40960)" — was only half the story, and the fix it proposed did not work. The owner corrected the setting to 40,960 and the call still failed, because the estimator under-counts a transcript by ~1.6×: at a 40,960 window the budget is 28,672 estimated tokens, and a window sized to that measured 41,934 at the server. No configured value fixes that, since the error scales with the text.

Recorded because the wrong half was plausible and self-consistent, and acting on it cost a round trip through a production deploy.

What actually fixed it

A rejection is now recoverable rather than fatal: the endpoint's own count teaches the app how far its estimate was off, the window is re-split at the corrected budget and retried once, and — the half that mattered most — a caller that persists results refuses to replace a stored set with a partial one.

The diagnosability complaint is also addressed

This issue noted that extract_highlights logged only type(exc).__name__, so an error whose message literally said "try increasing it" surfaced as the bare word RuntimeError. The typed ContextWindowExceededError now carries n_ctx and n_prompt_tokens as fields, and the callers log both counts, so the next occurrence of this class of failure names its own numbers.

One loose end, unrelated to the fix

Every LLM call on production logs:

No LLM provider configured; guessed 'llamacpp' from the endpoint URL. Set the provider
explicitly in Admin → Bot Settings — the guess is a URL substring match and is wrong for
any proxy or gateway on a custom domain.

Harmless on this deployment, because the guess happens to be right, but it is one field in Admin → Bot Settings and it would stop the repeated warning. Flagging rather than filing; open an issue if it should be enforced instead of guessed.

Closing.

Resolved by #614, released as **v4.4.1** and verified on dev and production — see #614 for the measured before/after. ## What this issue got right, and what it got wrong The first diagnosis here — "the configured context (65536) is larger than the server's real window (40960)" — was **only half the story, and the fix it proposed did not work**. The owner corrected the setting to 40,960 and the call still failed, because the estimator under-counts a transcript by ~1.6×: at a 40,960 window the budget is 28,672 estimated tokens, and a window sized to that measured **41,934** at the server. No configured value fixes that, since the error scales with the text. Recorded because the wrong half was plausible and self-consistent, and acting on it cost a round trip through a production deploy. ## What actually fixed it A rejection is now recoverable rather than fatal: the endpoint's own count teaches the app how far its estimate was off, the window is re-split at the corrected budget and retried once, and — the half that mattered most — a caller that persists results refuses to replace a stored set with a partial one. ## The diagnosability complaint is also addressed This issue noted that `extract_highlights` logged only `type(exc).__name__`, so an error whose message literally said *"try increasing it"* surfaced as the bare word `RuntimeError`. The typed `ContextWindowExceededError` now carries `n_ctx` and `n_prompt_tokens` as fields, and the callers log both counts, so the next occurrence of this class of failure names its own numbers. ## One loose end, unrelated to the fix Every LLM call on production logs: ``` No LLM provider configured; guessed 'llamacpp' from the endpoint URL. Set the provider explicitly in Admin → Bot Settings — the guess is a URL substring match and is wrong for any proxy or gateway on a custom domain. ``` Harmless on this deployment, because the guess happens to be right, but it is one field in Admin → Bot Settings and it would stop the repeated warning. Flagging rather than filing; open an issue if it should be enforced instead of guessed. Closing.

claude-bot closed this issue

2026-09-12 03:35:11 +00:00