rbrooks/Quest-Board

Watch

Fork 0

fix(webapp): recover from a context-window rejection instead of losing the window (#614, #613) #615

Merged

claude-bot merged 1 commit from fix/614-context-window-retry into main

2026-09-12 03:03:52 +00:00

claude-bot commented

2026-09-12 02:43:34 +00:00

Collaborator

Fixes #614, for the failure diagnosed in #613.

What went wrong

A real prompt that estimate_tokens sized at 27,174 tokens was counted by llama.cpp at 43,382 against a 40,960-token window, and rejected:

{"error":{"code":400,"message":"request (43382 tokens) exceeds the available context size (40960 tokens)…",
  "type":"exceed_context_size_error","n_prompt_tokens":43382,"n_ctx":40960}}

_CHARS_PER_TOKEN = 3.4 is tuned for prose. A merged transcript is not prose — every line carries [01:12:47] Name (label): plus short disfluent fragments — so the estimate ran ~1.6× under and fits_in_context passed a prompt the server had to refuse.

The damaging half was not the rejection but what a caller did with it. Re-extracting a real session's quote board with window 1 rejected and window 2 succeeding wrote a board built only from the session's last twenty minutes — and the delete-then-reinsert had already destroyed the previous good one. A nonempty result was indistinguishable from a complete one.

What this changes

Scope: a recovery path only. estimate_tokens, _CHARS_PER_TOKEN and PROMPT_BUDGET_FRACTION are untouched, and no call that succeeds today reaches any new code — the cache is only ever populated by a rejection. The beats pipeline that currently produces good summaries is deliberately not re-tuned.

ContextWindowExceededError(RuntimeError) carrying n_ctx, n_prompt_tokens, estimated_tokens and provider, raised from a shared helper at both the llama.cpp and Ollama non-200 sites. It parses the untruncated body — the error JSON only just fits inside the existing resp.text[:200] — and any other 400 shape still raises the original message character-for-character, which a test pins.
A per-endpoint calibration learned from the rejection. n_prompt_tokens / estimated_tokens, cached process-wide by endpoint URL, highest ratio wins, ratios ≤ 1 ignored. Modelled on the ASR capability cache, and for the same reasons — including not keying on the API key.
Never persisted. context_tokens is the operator's own hand-set value and llm_config is encrypted at rest, so a learned correction stays in memory; a fresh process re-learns it rather than trusting a stale number, and no operator-visible setting moves without the operator moving it.
One retry, at the three windowing callers — highlights, beat extraction, lore chunking — re-splitting that window's own text at the corrected budget. One retry, not a loop, matching #572's reasoning.
extract_highlights now returns HighlightsExtraction(highlights, failed_windows), and process_audio skips the replace entirely when any window failed, keeping whatever was already stored. The #397 approved/hand-edited protection on the delete is unchanged.

The beats retry is opt-in (context_tokens/endpoint_url default to None), so the #349 eval harness keeps its existing behaviour rather than silently changing what the evals measure.

Notes from review

A re-split window contributes several entries to quotes_by_window, so _interleave gives it proportionally more of the final quote slots than an un-split window. Defensible (more text, more windows) and left as-is, but worth knowing.
The delegate found two names in my brief were wrong: the beats windowing function is extraction_windows, not split_windows_for_extraction, and chunk_transcript_for_extraction's caller is generate_lore_proposals — the retry went into lore_chunk_extract, where the LLM call and therefore the rejection actually happen. Both correct as built.

Verification

Full backend suite: 3429 passed, 13 skipped (skip count unchanged from baseline).
ruff format --check and ruff check clean under CI's pinned 0.4.4.
No migration; transport and service layers only.

An autouse conftest fixture clears the new cache between tests, mirroring the existing ASR one, because endpoint URLs are reused across test files.

After merge

v4.4.1, then dev, then prod — and re-extract the affected session's quote board, which currently holds 8 rows from the partial run.

🤖 Generated with Claude Code

Fixes #614, for the failure diagnosed in #613. ## What went wrong A real prompt that `estimate_tokens` sized at **27,174** tokens was counted by llama.cpp at **43,382** against a 40,960-token window, and rejected: ``` {"error":{"code":400,"message":"request (43382 tokens) exceeds the available context size (40960 tokens)…", "type":"exceed_context_size_error","n_prompt_tokens":43382,"n_ctx":40960}} ``` `_CHARS_PER_TOKEN = 3.4` is tuned for prose. A merged transcript is not prose — every line carries `[01:12:47] Name (label): ` plus short disfluent fragments — so the estimate ran ~1.6× under and `fits_in_context` passed a prompt the server had to refuse. **The damaging half was not the rejection but what a caller did with it.** Re-extracting a real session's quote board with window 1 rejected and window 2 succeeding wrote a board built only from the session's last twenty minutes — and the delete-then-reinsert had already destroyed the previous good one. A nonempty result was indistinguishable from a complete one. ## What this changes **Scope: a recovery path only.** `estimate_tokens`, `_CHARS_PER_TOKEN` and `PROMPT_BUDGET_FRACTION` are untouched, and no call that succeeds today reaches any new code — the cache is only ever populated by a rejection. The beats pipeline that currently produces good summaries is deliberately not re-tuned. - **`ContextWindowExceededError(RuntimeError)`** carrying `n_ctx`, `n_prompt_tokens`, `estimated_tokens` and `provider`, raised from a shared helper at both the llama.cpp and Ollama non-200 sites. It parses the **untruncated** body — the error JSON only just fits inside the existing `resp.text[:200]` — and any other 400 shape still raises the original message character-for-character, which a test pins. - **A per-endpoint calibration learned from the rejection.** `n_prompt_tokens / estimated_tokens`, cached process-wide by endpoint URL, highest ratio wins, ratios ≤ 1 ignored. Modelled on the ASR capability cache, and for the same reasons — including not keying on the API key. - **Never persisted.** `context_tokens` is the operator's own hand-set value and `llm_config` is encrypted at rest, so a learned correction stays in memory; a fresh process re-learns it rather than trusting a stale number, and no operator-visible setting moves without the operator moving it. - **One retry, at the three windowing callers** — highlights, beat extraction, lore chunking — re-splitting that window's own text at the corrected budget. One retry, not a loop, matching #572's reasoning. - **`extract_highlights` now returns `HighlightsExtraction(highlights, failed_windows)`**, and `process_audio` skips the replace entirely when any window failed, keeping whatever was already stored. The #397 approved/hand-edited protection on the delete is unchanged. The beats retry is opt-in (`context_tokens`/`endpoint_url` default to `None`), so the #349 eval harness keeps its existing behaviour rather than silently changing what the evals measure. ## Notes from review - A re-split window contributes several entries to `quotes_by_window`, so `_interleave` gives it proportionally more of the final quote slots than an un-split window. Defensible (more text, more windows) and left as-is, but worth knowing. - The delegate found two names in my brief were wrong: the beats windowing function is `extraction_windows`, not `split_windows_for_extraction`, and `chunk_transcript_for_extraction`'s caller is `generate_lore_proposals` — the retry went into `lore_chunk_extract`, where the LLM call and therefore the rejection actually happen. Both correct as built. ## Verification - Full backend suite: **3429 passed, 13 skipped** (skip count unchanged from baseline). - `ruff format --check` and `ruff check` clean under CI's pinned 0.4.4. - No migration; transport and service layers only. An autouse conftest fixture clears the new cache between tests, mirroring the existing ASR one, because endpoint URLs are reused across test files. ## After merge v4.4.1, then dev, then prod — and re-extract the affected session's quote board, which currently holds 8 rows from the partial run. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit

2026-09-12 02:43:35 +00:00

fix(webapp): recover from a context-window rejection instead of losing the window (#614 , #613 )

CI / Backend lint (ruff) (pull_request) Successful in 27s

Details

CI / Docker image build (pull_request) Successful in 27s

Details

CI / Bot/backend version sync (pull_request) Successful in 38s

Details

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 55s

Details

CI / Bot tests and audit (pull_request) Successful in 1m46s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 2m30s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 7m38s

Details

CI / Synthetic session harness (no GPU, no LLM) (pull_request) Successful in 20m9s

Details

f6594dccb0

llama.cpp's own token count sometimes runs well ahead of
llm_service.estimate_tokens on a merged transcript — every line carries a
`[HH:MM:SS] Speaker: ` stamp plus short disfluent fragments the char/token
ratio was never tuned against. A real prompt sized at 27174 tokens was
counted by llama.cpp at 43382 against a 40960-token window and rejected
outright, with no recovery: the window (or chunk) was simply lost.

Per the scope agreed on #613, estimate_tokens/_CHARS_PER_TOKEN/
PROMPT_BUDGET_FRACTION and the arithmetic behind a call that already
succeeds are unchanged. This adds a recovery path on top:

- llm_service.ContextWindowExceededError(RuntimeError) is raised from the
  llama.cpp and Ollama transports when a non-200 body parses as llama.cpp's
  `exceed_context_size_error` shape (parsed from the untruncated body, since
  the existing 200-char truncation only just fits the error JSON). Any other
  400 shape still raises the same plain RuntimeError, message unchanged.

- A process-wide cache keyed by endpoint URL (llm_service._context_ratio_cache,
  mirroring app.providers.asr's capability cache) records the highest observed
  ratio of the endpoint's own count to our estimate, ignoring ratios <= 1.
  effective_prompt_budget_tokens() applies it to narrow the prompt budget --
  never written to LLMConfig or settings, which stay exactly what the operator
  set.

- Three windowing callers each get one retry, not a loop: on
  ContextWindowExceededError, record the ratio, re-split the rejected window's
  own text at the corrected budget, and try each piece once.
  - audio_service.extract_highlights (per-window, now returns
    HighlightsExtraction(highlights, failed_windows) instead of a bare list)
  - audio_service._extract_one_pass / gather_validated_beats (the beats path
    behind extraction_windows), which now take optional context_tokens/
    endpoint_url so the #349 eval harness is unaffected
  - tasks.reminder_tasks.lore_chunk_extract, re-chunking its own chunk_text
    via lore_service.chunk_text at a corrected char budget (llm_service.
    chars_for_token_budget converts the corrected token budget back to chars)

- tasks.reminder_tasks.process_audio's highlights block now skips its
  delete-then-reinsert whenever any window failed, not just when the result
  is empty -- on the session that motivated this, a nonempty-but-partial
  result (one of two windows lost) was being written as if it were the whole
  quote board, after the delete had already destroyed the previous good one.
  The #397 approved/edited guard on the delete is untouched.

Tests: extended test_llm_transport.py (typed exception, generic 400s
unaffected), test_context_budget.py (cache semantics, no settings write),
test_highlights.py (retry/resplit/calibration, HighlightsExtraction),
test_failed_windows.py (beats-path retry, eval-harness fallback), and a new
test_lore_context_retry.py for the lore_chunk_extract path. Full suite (3429
passed, 13 skipped) and ruff format/check both clean.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed

2026-09-12 02:44:00 +00:00

claude-bot merged commit 3e643963f3 into main

2026-09-12 03:03:52 +00:00

claude-bot deleted branch fix/614-context-window-retry

2026-09-12 03:03:52 +00:00

claude-bot referenced this pull request from a commit

2026-09-12 03:03:53 +00:00

Merge pull request 'fix(webapp): recover from a context-window rejection instead of losing the window (#614, #613)' (#615) from fix/614-context-window-retry into main

claude-bot referenced this pull request

2026-09-12 03:34:59 +00:00

[Backend] Recover from a context-window rejection: learn the real token ratio and re-split the window once #614