[Backend] Chunk the transcript by time window, sized from the provider's declared context #331
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Severity: CRITICAL. Found in the August 2026 session lifecycle review (#319). First milestone of the summarisation re-architecture — roughly 70% of the benefit for 40% of the work.
The defect
summarise()sends the entire transcript in one prompt, with no chunking and no map-reduce anywhere in the path (audio_service.py:865-910). The budget does not fit the target workload:--ctx-size 131072 --parallel 2gives half the nominal size per slotIt sometimes fits, barely, with no headroom, and a chatty table or a 4-hour session pushes it over. The design is out of budget at exactly the workload it was built for.
Real values are one grep away:
audio_service.py:884logs the character count, and anything at or above ~190 KB is at or over the slot budget.Proposed fix
Split the transcript on timestamp boundaries into windows, with 1-2 lines of overlap, then map each window to a fixed mini-form (
time_range, 4-8 bullet events with actors, NPCs introduced, decisions) and reduce once over the ordered mini-summaries with an explicit instruction to preserve the given order and time ranges.Size the window from the provider's declared context, not a constant. This is the load-bearing design decision: it makes one code path correct on a 9B local model with a 65k slot and on a 1M-context hosted model, where the whole transcript collapses into a single window naturally. It ties directly to the provider abstraction in v4.2.0, which must declare a context limit per provider.
Chunking here is a context-management device, not an ordering device — the ordering guarantee comes from the beat validator in this milestone, not from the chunk boundaries. That distinction matters: naive chunking (summarise pieces to prose, ask a model to stitch them) genuinely does scramble chronology.
Acceptance criteria
Re-scoping one criterion, from the #336/#337/#338 pass.
Map calls run in parallel where the provider allows it→ belongs to #356 (v4.2.0), not here._summarise_chunkedsummarises windows sequentially, deliberately, and says why in the code: a local llama.cpp server has a small fixed slot count, so firing every window at once queues behind it anyway — or trips a hosted provider's rate limit. Doing this correctly needs to know how much concurrency a given provider allows, which is exactly what the provider abstraction is for. Building a fixed concurrency here would be a guess that is wrong on both self-hosted and hosted, in opposite directions.The rest of this issue's criteria are unaffected. Noting it rather than silently leaving the box unticked.
Also relevant to "window size is derived from the provider's declared context window, never hardcoded": since
ede1ac2(#336) every call site passes the real window fromllm_cfg.context_tokens, and sincec54d9af(#337) a caller that declares nothing gets a conservative default rather than no budgeting at all. Note that extraction windows are separately capped by_EXTRACTION_CONTEXT_CAP— that is #423's measured result (a 131k model handed a whole session finds fewer events, not more), and it is a deliberate ceiling on top of the derived size, not a hardcoding of it.Verified; three gaps fixed in
ed441c8anddd8f305. One of them was a worker-killer.Criteria
process_audio, false on the other call site. Fixed.Map calls run in parallel where the provider allows it— re-scoped to #356, per the comment above.The fold recursion was unbounded
When the reduce prompt still exceeds the budget,
_summarise_chunkedfolds its own notes down by calling itself. That terminates only if each pass actually shrinks the text, and a model is under no obligation to shorten what it is given. A model answering a condense request with something the same length recursed until the OOM killer took the worker — leaving the session in "processing" forever with nothing logged.Found by writing the first test for that branch, which killed the test runner. The branch has existed since this issue shipped and had no coverage: the map-reduce test asserts
>= 2windows, which a two-window transcript satisfies. Now bounded at three folds, then the notes are sampled to fit —sample_evenlyrather than truncation, because head-truncation drops the end of the session, which is the part a GM most wants (#340).I was wrong about the window reaching every call site
My earlier comment here said "since
ede1ac2(#336) every call site passes the real window fromllm_cfg.context_tokens". That was wrong, and the miss was the one that mattered.select_canonical_name(routers/sessions.py) calledsummarise()with nocontext_tokens. Withwindow = 0, thefits_in_contextbranch is skipped entirely and the whole transcript goes in one prompt — precisely the defect this issue exists to prevent. #336 threaded the window through everygenerate_structured_textcaller and missed this one, because it reaches the provider throughsummariseinstead. Fixed indd8f305.Worth noting what made it invisible: three test doubles built the LLM config as a
SimpleNamespacewithoutcontext_tokens, so adding the read surfaced as a 500 from the router, not a quietfailedstatus — unlike the ~20 doubles inede1ac2, where a task-levelexceptswallowed the sameAttributeErrorand recorded a wrong status instead of an error.Overlap
_WINDOW_OVERLAP_LINESwas a module constant no caller, env var or setting could reach — so "configurable overlap" was not met. Now a parameter with aQB_WINDOW_OVERLAP_LINESoverride, the same shape asQB_EXTRACTION_CONTEXT_CAPand for the same reason: the useful value moves with the model and should be settled with the #349 harness rather than by argument.Tests
"…and one that needs twelve" is now end-to-end rather than splitter-level, asserting every window's notes reach reduce in order — losing one loses that stretch of the session silently. Plus the fold recursion, and overlap at 0, default and 4.
Closing.