Summary length is set by the verified-beat count: a strong extractor makes compose write a 10,000-character recap #568

New issue

Closed

opened 2026-09-09 04:47:45 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-09-09 04:47:45 +00:00

Contributor

Observation, not a bug report yet. COMPOSE_SYSTEM_PROMPT asks for 3-6 paragraphs and says every verified event must be represented (#423). Those two instructions only agree while extraction returns a few dozen beats. Replaying the 2026-09-09 session (80 minutes, six speakers) through the real pipeline with different models standing in for the LLM:

model	verified beats	summary length
qwen3.5 (prod)	4	1,950 chars
Haiku	21	2,500 chars
Fable	44	8,800 chars
Opus	84	10,200 chars
Sonnet	93	8,800 chars

The more capable the extractor, the more beats survive validation (including small ones: a dice roll, a joke the GM narrated, someone pointing at a tree), and the composer is forbidden to leave any out, so the "summary" becomes a full recap. The local model has the opposite problem: it cites so few timestamps correctly that only 4 of 23 beats verified, and the summary is built on scraps and gets the geography wrong.

Things to decide.

Whether compose should get every beat, or the top N by kind (reveal, decision, combat, travel first; social and other last) with the rest kept on the run record and in the anchors.
Whether extraction should be told to prefer fewer, larger beats (the prompt currently rewards granularity, since nothing penalises it).
Whether the recovery pass (#423) should stay as is: on Opus it recovered one dropped event and introduced duplicated sentences, because the reminder makes the model restate events it had already covered.

Related: the unsorted-beats bug filed alongside this makes the long lists worse, since an out-of-order 90-beat list is what produced Sonnet's out-of-order fifth paragraph.

**Observation, not a bug report yet.** `COMPOSE_SYSTEM_PROMPT` asks for 3-6 paragraphs *and* says every verified event must be represented (#423). Those two instructions only agree while extraction returns a few dozen beats. Replaying the 2026-09-09 session (80 minutes, six speakers) through the real pipeline with different models standing in for the LLM: | model | verified beats | summary length | |---|---|---| | qwen3.5 (prod) | 4 | 1,950 chars | | Haiku | 21 | 2,500 chars | | Fable | 44 | 8,800 chars | | Opus | 84 | 10,200 chars | | Sonnet | 93 | 8,800 chars | The more capable the extractor, the more beats survive validation (including small ones: a dice roll, a joke the GM narrated, someone pointing at a tree), and the composer is forbidden to leave any out, so the "summary" becomes a full recap. The local model has the opposite problem: it cites so few timestamps correctly that only 4 of 23 beats verified, and the summary is built on scraps and gets the geography wrong. **Things to decide.** - Whether compose should get every beat, or the top N by kind (reveal, decision, combat, travel first; social and other last) with the rest kept on the run record and in the anchors. - Whether extraction should be told to prefer fewer, larger beats (the prompt currently rewards granularity, since nothing penalises it). - Whether the recovery pass (#423) should stay as is: on Opus it recovered one dropped event and introduced duplicated sentences, because the reminder makes the model restate events it had already covered. Related: the unsorted-beats bug filed alongside this makes the long lists worse, since an out-of-order 90-beat list is what produced Sonnet's out-of-order fifth paragraph.

claude-bot referenced this issue

2026-09-09 06:29:43 +00:00

Beat extraction silently loses whole windows on the local Qwen router, and logs the failure with an empty message #572

claude-bot referenced this issue

2026-09-10 04:47:33 +00:00

Beat extraction reads a GM's answer to a table question as present-tense narration ("The party is currently located in Ironroot Hold") #584

claude-bot commented

2026-09-10 04:55:13 +00:00

Author

Contributor

Decided (2026-09-10, owner): a short summary and a full event log, rather than choosing between them.

Compose keeps writing 3 to 6 paragraphs, but from the most significant verified events rather than all of them (reveals, decisions, combat and travel ahead of social and other), so the paragraph target and the "represent everything you are given" rule stop contradicting each other.
Every verified event is still shown, as a timestamped list rendered in code from the beats. No extra LLM call, nothing selected away. beat_service.render_summary_anchors already produces exactly this shape and is currently called by nothing, so the backend work is small.
The log is collapsed by default under the summary and never shows unverified beats; those stay on the GM-only run detail as today.

Sized against the 2026-09-09 replay: Gemma 4 26B verified 59 events, compose dropped 12 and the retry recovered 2. Under this design nothing is lost regardless of how many events a session yields.

**Decided (2026-09-10, owner):** a short summary *and* a full event log, rather than choosing between them. - Compose keeps writing 3 to 6 paragraphs, but from the most significant verified events rather than all of them (reveals, decisions, combat and travel ahead of social and other), so the paragraph target and the "represent everything you are given" rule stop contradicting each other. - Every verified event is still shown, as a timestamped list rendered in code from the beats. No extra LLM call, nothing selected away. `beat_service.render_summary_anchors` already produces exactly this shape and is currently called by nothing, so the backend work is small. - The log is collapsed by default under the summary and never shows unverified beats; those stay on the GM-only run detail as today. Sized against the 2026-09-09 replay: Gemma 4 26B verified 59 events, compose dropped 12 and the retry recovered 2. Under this design nothing is lost regardless of how many events a session yields.