A summarisation-stage prompt of ~31k tokens was sent against a 22.9k budget; find the call and chunk it #575

New issue

Open

opened 2026-09-09 06:43:53 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-09-09 06:43:53 +00:00

Contributor

Seen on the dev replay of 2026-09-09 (llamacpp/qwen3.5, 32,768-token window):

llamacpp prompt is ~31348 tokens against a 22937-token budget (32768-token window at 70%) — it will be sent anyway

The beats path windows its extraction and composes from a short event list, so the oversized prompt is almost certainly a call that still sends the whole compacted transcript (about 26k tokens for this 80-minute session) in one go: the highlights extraction (extract_highlights, which is handed transcript directly in process_audio) or one of the fan-out tasks. On a front-truncating server that silently drops the system prompt and the speaker legend, which is the worst possible thing to lose; on llama.cpp the behaviour depends on the server's context handling.

Fix. Identify the call from the log ordering (the warning sits between the compose call and the highlights persistence), then give it the same windowing the beats path has, or sample the transcript with the existing sample_evenly / truncate_keeping_ends helpers when the budget is exceeded, and make the "sent anyway" path an explicit decision per call site rather than the default.

Related: #341 (compaction), #356/#357 (the budget and telemetry that produced the warning).

**Seen on the dev replay of 2026-09-09** (llamacpp/qwen3.5, 32,768-token window): ``` llamacpp prompt is ~31348 tokens against a 22937-token budget (32768-token window at 70%) — it will be sent anyway ``` The beats path windows its extraction and composes from a short event list, so the oversized prompt is almost certainly a call that still sends the **whole compacted transcript** (about 26k tokens for this 80-minute session) in one go: the highlights extraction (`extract_highlights`, which is handed `transcript` directly in `process_audio`) or one of the fan-out tasks. On a front-truncating server that silently drops the system prompt and the speaker legend, which is the worst possible thing to lose; on llama.cpp the behaviour depends on the server's context handling. **Fix.** Identify the call from the log ordering (the warning sits between the compose call and the highlights persistence), then give it the same windowing the beats path has, or sample the transcript with the existing `sample_evenly` / `truncate_keeping_ends` helpers when the budget is exceeded, and make the "sent anyway" path an explicit decision per call site rather than the default. Related: #341 (compaction), #356/#357 (the budget and telemetry that produced the warning).