rbrooks/Quest-Board

Watch

Fork 0

[Backend] Recover from a context-window rejection: learn the real token ratio and re-split the window once #614

New issue

Closed

opened 2026-09-12 01:51:00 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-09-12 01:51:00 +00:00

Collaborator

The fix for the failure diagnosed in #613. That issue explains why a prompt the estimator sized at 27,174 tokens was counted by the server as 43,382; this one is the recovery.

Scope decided with the owner: add a recovery path, do not touch the estimator. Calls that succeed today must behave identically — the beats pipeline currently produces good summaries and is not being re-tuned here. What changes is that a rejection becomes recoverable instead of losing a window silently.

1. A typed exception

llm_service turns every non-200 into a bare string:

llama.cpp returned {status}: {text[:200]} (the llama.cpp path)
Ollama returned {status}: {text[:200]} (the Ollama path)

Add ContextWindowExceededError(RuntimeError) carrying n_ctx, n_prompt_tokens, estimated_tokens and provider, raised from both sites when the response body's error.type is exceed_context_size_error (llama.cpp's name; match defensively — fall back to the generic RuntimeError when the body is not that shape).

It must subclass RuntimeError. Several callers already catch broad exceptions and must keep behaving exactly as they do now if they choose not to handle it.

Note resp.text[:200] is only just long enough to hold the error body — parse before truncating, not after.

2. Learn the ratio from the rejection

The server reports what it actually counted. n_prompt_tokens / estimate_tokens(prompt) is a measured calibration for that endpoint and that kind of text (this session: ~1.6).

Cache it per endpoint URL, process-level, in the shape providers/asr.py already uses for its capability probe — one cache keyed by URL, not one per adapter instance.
Never write it back into llm_config. That blob is encrypted, LLMConfig is a frozen dataclass, and context_tokens is the operator's own setting — one they may have just set by hand. Overwriting it would be a silent config edit.
Keep the highest observed ratio, and clamp it to something sane (a ratio below 1 means the estimate was already pessimistic; ignore it).
Budgeting divides the available window by the cached ratio when one exists.

3. Retry once, at the caller

Windowing belongs to the caller, so recovery does too. Three call sites do their own splitting:

audio_service.extract_highlights — split_transcript_windows(transcript, budget)
audio_service.split_windows_for_extraction — the beats path
lore_service.chunk_transcript_for_extraction — the lore pipeline

On ContextWindowExceededError: re-split that window's text with the corrected budget and try again, once. Not a loop. If the retry also fails, treat it as a failed window exactly as today.

4. A caller must not persist a partial result

The bug that motivated this, and the more damaging half. Re-extracting the quote board for session 5b0d6ac1 with window 1 failing and window 2 succeeding wrote a board built only from the last twenty minutes of the session — including a slur — and deleted the previous good rows to do it. extract_highlights returns what it collected with no indication that a window was lost, so "partial" and "complete" are indistinguishable to its caller.

Return the failure count alongside the results (or raise when every window failed).
process_audio's highlights block must not run its delete-then-reinsert when any window failed. It already declines to wipe on an empty result for exactly this reason (#397 protects approved and hand-edited rows) — extend that judgement to partial results.

Tests

A 400 whose body carries exceed_context_size_error raises ContextWindowExceededError with both counts populated, from each of the two transport paths.
A 400 of any other shape still raises plain RuntimeError.
The ratio is cached per endpoint, is not persisted to settings, and a ratio below 1 is ignored.
A window rejected once is re-split and retried exactly once; a second rejection does not retry again.
extract_highlights reports failed windows to its caller.
process_audio leaves existing highlights untouched when a window failed, and replaces them only when every window succeeded.
A calibrated endpoint sizes its next windows smaller without any settings change.

Verify after merge

Re-extract the quote board for 5b0d6ac1 on prod and confirm it covers the whole session rather than its tail. Its 8 current rows are from the partial run and are to be replaced wholesale.

The fix for the failure diagnosed in #613. That issue explains *why* a prompt the estimator sized at 27,174 tokens was counted by the server as 43,382; this one is the recovery. **Scope decided with the owner: add a recovery path, do not touch the estimator.** Calls that succeed today must behave identically — the beats pipeline currently produces good summaries and is not being re-tuned here. What changes is that a rejection becomes recoverable instead of losing a window silently. ## 1. A typed exception `llm_service` turns every non-200 into a bare string: - `llama.cpp returned {status}: {text[:200]}` (the llama.cpp path) - `Ollama returned {status}: {text[:200]}` (the Ollama path) Add `ContextWindowExceededError(RuntimeError)` carrying `n_ctx`, `n_prompt_tokens`, `estimated_tokens` and `provider`, raised from **both** sites when the response body's `error.type` is `exceed_context_size_error` (llama.cpp's name; match defensively — fall back to the generic `RuntimeError` when the body is not that shape). **It must subclass `RuntimeError`.** Several callers already catch broad exceptions and must keep behaving exactly as they do now if they choose not to handle it. Note `resp.text[:200]` is only just long enough to hold the error body — parse before truncating, not after. ## 2. Learn the ratio from the rejection The server reports what it actually counted. `n_prompt_tokens / estimate_tokens(prompt)` is a measured calibration for that endpoint and that kind of text (this session: ~1.6). - Cache it **per endpoint URL**, process-level, in the shape `providers/asr.py` already uses for its capability probe — one cache keyed by URL, not one per adapter instance. - **Never write it back into `llm_config`.** That blob is encrypted, `LLMConfig` is a frozen dataclass, and `context_tokens` is the operator's own setting — one they may have just set by hand. Overwriting it would be a silent config edit. - Keep the highest observed ratio, and clamp it to something sane (a ratio below 1 means the estimate was already pessimistic; ignore it). - Budgeting divides the available window by the cached ratio when one exists. ## 3. Retry once, at the caller Windowing belongs to the caller, so recovery does too. Three call sites do their own splitting: - `audio_service.extract_highlights` — `split_transcript_windows(transcript, budget)` - `audio_service.split_windows_for_extraction` — the beats path - `lore_service.chunk_transcript_for_extraction` — the lore pipeline On `ContextWindowExceededError`: re-split that window's text with the corrected budget and try again, **once**. Not a loop. If the retry also fails, treat it as a failed window exactly as today. ## 4. A caller must not persist a partial result The bug that motivated this, and the more damaging half. Re-extracting the quote board for session `5b0d6ac1` with window 1 failing and window 2 succeeding wrote a board built **only from the last twenty minutes of the session** — including a slur — and deleted the previous good rows to do it. `extract_highlights` returns what it collected with no indication that a window was lost, so "partial" and "complete" are indistinguishable to its caller. - Return the failure count alongside the results (or raise when every window failed). - `process_audio`'s highlights block must **not** run its delete-then-reinsert when any window failed. It already declines to wipe on an empty result for exactly this reason (#397 protects approved and hand-edited rows) — extend that judgement to partial results. ## Tests - A 400 whose body carries `exceed_context_size_error` raises `ContextWindowExceededError` with both counts populated, from each of the two transport paths. - A 400 of any other shape still raises plain `RuntimeError`. - The ratio is cached per endpoint, is not persisted to settings, and a ratio below 1 is ignored. - A window rejected once is re-split and retried exactly once; a second rejection does not retry again. - `extract_highlights` reports failed windows to its caller. - `process_audio` leaves existing highlights untouched when a window failed, and replaces them only when every window succeeded. - A calibrated endpoint sizes its next windows smaller without any settings change. ## Verify after merge Re-extract the quote board for `5b0d6ac1` on prod and confirm it covers the whole session rather than its tail. Its 8 current rows are from the partial run and are to be replaced wholesale.

claude-bot added the

backend

bug

labels

2026-09-12 01:51:08 +00:00

rbrooks referenced this issue from a commit

2026-09-12 02:41:30 +00:00

fix(webapp): recover from a context-window rejection instead of losing the window (#614, #613)

claude-bot referenced this issue from a pull request that will close it,

2026-09-12 02:43:34 +00:00

fix(webapp): recover from a context-window rejection instead of losing the window (#614, #613) #615

claude-bot closed this issue

2026-09-12 03:03:52 +00:00

claude-bot referenced this issue from a commit

2026-09-12 03:03:53 +00:00

Merge pull request 'fix(webapp): recover from a context-window rejection instead of losing the window (#614, #613)' (#615) from fix/614-context-window-retry into main

rbrooks referenced this issue from a commit

2026-09-12 03:05:41 +00:00

chore(release): v4.4.1

claude-bot referenced this issue

2026-09-12 03:08:57 +00:00

chore(release): v4.4.1 #616

claude-bot commented

2026-09-12 03:34:59 +00:00

Author

Collaborator

Shipped in PR #615, released as v4.4.1 (a6abef1), and verified on both dev and production.

Verified on dev (read-only, nothing persisted)

The real session, the real endpoint, and a genuine rejection rather than a rehearsal:

Extracting highlights from 2 window(s) of a 40960-token context
POST /v1/chat/completions "HTTP/1.1 400 Bad Request"
extract_highlights: window 1/2 rejected as too large for the endpoint's context window
  (estimated 26396 tokens; the endpoint counted 41878 against a 40960-token window)
  — recording the ratio and retrying once with a smaller split
POST /v1/chat/completions "HTTP/1.1 200 OK"
POST /v1/chat/completions "HTTP/1.1 200 OK"


ratio learned	1.587
prompt budget before → after	28,672 → 18,072
`failed_windows`	0
quotes	spanning 00:03 → 01:12

Verified on production

Same shape — estimated 26,367, endpoint counted 41,934, ratio 1.59, budget 28,672 → 18,028, failed_windows=0.

With a clean result the guarded write was allowed, and the board that had been left holding only the session's last twenty minutes was replaced: 8 stale rows → 10 fresh ones, spanning 00:01 to 01:12. Anchoring corrected 1 speaker and 2 timestamps, dropped nothing.

The guard earned its place immediately

The re-extraction script carried the same rule the fix adds to process_audio — refuse to write unless failed_windows == 0. That is precisely what was missing when this issue was created: a partial board overwrote a complete one, and the delete had already destroyed the good version by the time anyone could tell.

No settings changed on either host. The correction is measured from the rejection and held in memory, so the operator's hand-set context window is untouched — confirmed by the ratio cache being empty at the start of each run and populated only after the 400.

Closing.

Shipped in PR #615, released as **v4.4.1** (`a6abef1`), and verified on both dev and production. ## Verified on dev (read-only, nothing persisted) The real session, the real endpoint, and a genuine rejection rather than a rehearsal: ``` Extracting highlights from 2 window(s) of a 40960-token context POST /v1/chat/completions "HTTP/1.1 400 Bad Request" extract_highlights: window 1/2 rejected as too large for the endpoint's context window (estimated 26396 tokens; the endpoint counted 41878 against a 40960-token window) — recording the ratio and retrying once with a smaller split POST /v1/chat/completions "HTTP/1.1 200 OK" POST /v1/chat/completions "HTTP/1.1 200 OK" ``` | | | |---|---| | ratio learned | **1.587** | | prompt budget before → after | 28,672 → **18,072** | | `failed_windows` | **0** | | quotes | spanning 00:03 → 01:12 | ## Verified on production Same shape — estimated 26,367, endpoint counted 41,934, ratio **1.59**, budget 28,672 → **18,028**, `failed_windows=0`. With a clean result the guarded write was allowed, and the board that had been left holding only the session's last twenty minutes was replaced: **8 stale rows → 10 fresh ones, spanning 00:01 to 01:12**. Anchoring corrected 1 speaker and 2 timestamps, dropped nothing. ## The guard earned its place immediately The re-extraction script carried the same rule the fix adds to `process_audio` — refuse to write unless `failed_windows == 0`. That is precisely what was missing when this issue was created: a partial board overwrote a complete one, and the delete had already destroyed the good version by the time anyone could tell. No settings changed on either host. The correction is measured from the rejection and held in memory, so the operator's hand-set context window is untouched — confirmed by the ratio cache being empty at the start of each run and populated only after the 400. Closing.

claude-bot referenced this issue

2026-09-12 03:35:11 +00:00

[Backend] Prod's configured LLM context (65536) is larger than the server's real per-request window (40960), so long-prompt calls fail with 400 #613