A summarisation-stage prompt of ~31k tokens was sent against a 22.9k budget; find the call and chunk it #575

New issue

Closed

opened 2026-09-09 06:43:53 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-09-09 06:43:53 +00:00

Collaborator

Seen on the dev replay of 2026-09-09 (llamacpp/qwen3.5, 32,768-token window):

llamacpp prompt is ~31348 tokens against a 22937-token budget (32768-token window at 70%) — it will be sent anyway

The beats path windows its extraction and composes from a short event list, so the oversized prompt is almost certainly a call that still sends the whole compacted transcript (about 26k tokens for this 80-minute session) in one go: the highlights extraction (extract_highlights, which is handed transcript directly in process_audio) or one of the fan-out tasks. On a front-truncating server that silently drops the system prompt and the speaker legend, which is the worst possible thing to lose; on llama.cpp the behaviour depends on the server's context handling.

Fix. Identify the call from the log ordering (the warning sits between the compose call and the highlights persistence), then give it the same windowing the beats path has, or sample the transcript with the existing sample_evenly / truncate_keeping_ends helpers when the budget is exceeded, and make the "sent anyway" path an explicit decision per call site rather than the default.

Related: #341 (compaction), #356/#357 (the budget and telemetry that produced the warning).

**Seen on the dev replay of 2026-09-09** (llamacpp/qwen3.5, 32,768-token window): ``` llamacpp prompt is ~31348 tokens against a 22937-token budget (32768-token window at 70%) — it will be sent anyway ``` The beats path windows its extraction and composes from a short event list, so the oversized prompt is almost certainly a call that still sends the **whole compacted transcript** (about 26k tokens for this 80-minute session) in one go: the highlights extraction (`extract_highlights`, which is handed `transcript` directly in `process_audio`) or one of the fan-out tasks. On a front-truncating server that silently drops the system prompt and the speaker legend, which is the worst possible thing to lose; on llama.cpp the behaviour depends on the server's context handling. **Fix.** Identify the call from the log ordering (the warning sits between the compose call and the highlights persistence), then give it the same windowing the beats path has, or sample the transcript with the existing `sample_evenly` / `truncate_keeping_ends` helpers when the budget is exceeded, and make the "sent anyway" path an explicit decision per call site rather than the default. Related: #341 (compaction), #356/#357 (the budget and telemetry that produced the warning).

rbrooks referenced this issue from a commit

2026-09-09 17:35:02 +00:00

fix(backend): window the highlights call instead of sending the whole transcript (#575)

rbrooks referenced this issue from a commit

2026-09-09 17:35:37 +00:00

fix(backend): window the highlights call instead of sending the whole transcript (#575)

claude-bot referenced this issue

2026-09-09 17:35:50 +00:00

fix(backend): four summarisation-pipeline defects found by replaying a real session (#566, #567, #572, #575) #583

claude-bot referenced this issue from a commit

2026-09-09 17:52:35 +00:00

Merge PR #583: fix(backend): four summarisation-pipeline defects found by replaying a real session (#566, #567, #572, #575)

claude-bot referenced this issue

2026-09-09 18:38:32 +00:00

Beat extraction silently loses whole windows on the local Qwen router, and logs the failure with an empty message #572

claude-bot referenced this issue

2026-09-12 01:45:48 +00:00

[Backend] Prod's configured LLM context (65536) is larger than the server's real per-request window (40960), so long-prompt calls fail with 400 #613

claude-bot commented

2026-09-12 19:41:48 +00:00

Author

Collaborator

Closing — fixed by #583 (commit 5817504, "window the highlights call instead of sending the whole transcript") and never closed afterwards.

The issue's guess was right: the oversized prompt was extract_highlights, which process_audio hands the whole transcript. It is now windowed like every other call in the module that reads a whole transcript, with the session-wide quote cap applied after the merge rather than trusted to a single prompt. The reasoning is recorded in its docstring, including why "send it anyway" is the wrong default for this particular caller:

"Send it anyway" is a reasonable default for a warning — the token count is an estimate and refusing to summarise beats summarising a tenth of a session — but it is not a reasonable design for the one caller that knows how to split its own input.

That is also the right answer to the issue's "make the 'sent anyway' path an explicit decision per call site": it stays the global default, and the call site that can do better does better.

Worth noting the second half of this problem was fixed separately and has since shipped: #613/#614 (v4.4.1) made a context rejection recalibrate the endpoint's token estimate and re-split rather than dropping the window, so even an under-estimate that slips past the windowing now self-corrects instead of losing content. The token estimator being ~1.6x under on transcripts was the reason a windowed call could still overshoot.

Verified on main today: tests/test_highlights.py passes as part of the 91 tests run with test_failed_windows.py and test_empty_hook.py. The dev replay of 2026-09-09 (comment on #572) recorded the highlights call running windowed with no over-budget warning.

Closing — fixed by **#583** (commit `5817504`, "window the highlights call instead of sending the whole transcript") and never closed afterwards. The issue's guess was right: the oversized prompt was `extract_highlights`, which `process_audio` hands the whole `transcript`. It is now windowed like every other call in the module that reads a whole transcript, with the session-wide quote cap applied after the merge rather than trusted to a single prompt. The reasoning is recorded in its docstring, including why "send it anyway" is the wrong default *for this particular caller*: > "Send it anyway" is a reasonable default for a *warning* — the token count is an estimate and refusing to summarise beats summarising a tenth of a session — but it is not a reasonable design for the one caller that knows how to split its own input. That is also the right answer to the issue's "make the 'sent anyway' path an explicit decision per call site": it stays the global default, and the call site that can do better does better. Worth noting the second half of this problem was fixed separately and has since shipped: #613/#614 (v4.4.1) made a context rejection recalibrate the endpoint's token estimate and re-split rather than dropping the window, so even an under-estimate that slips past the windowing now self-corrects instead of losing content. The token estimator being ~1.6x under on transcripts was the reason a windowed call could still overshoot. Verified on `main` today: `tests/test_highlights.py` passes as part of the 91 tests run with `test_failed_windows.py` and `test_empty_hook.py`. The dev replay of 2026-09-09 (comment on #572) recorded the highlights call running windowed with no over-budget warning.

claude-bot closed this issue

2026-09-12 19:41:51 +00:00