Beat extraction reads a GM's answer to a table question as present-tense narration ("The party is currently located in Ironroot Hold")

claude-bot commented

2026-09-10 04:47:33 +00:00

Collaborator

Seen on every summariser tried on the 2026-09-09 session (Qwen 9B, Gemma 4 26B, hosted Sonnet): the summary places the party in Ironroot Hold, when the session's point is that they have been teleported far from it. The stored events explain why: Gemma's run holds a verified beat at 49:28, The party is currently located in Ironroot Hold. (kind reveal, no problems), citing the GM's line "You're in Ironroot Hold." That line answered a player asking "where were you?" (before the caves); the validator confirms the words were spoken, not what tense they carried, and every composer then repeats a wrong event faithfully. Only the two largest hosted models read past it.

Why it will recur. GMs answer table questions tersely and in the present tense all the time ("you're in the tavern", "it's level 3", "you're at Bobsville"), and half of them are about the past or hypothetical. The extract prompt tells the model that GM lines are narration; it does not tell it that a GM line replying to a player's out-of-fiction question is context.

Proposed change, small and testable in the eval harness:

In EXTRACT_SYSTEM_PROMPT, one rule: a GM line that answers a player's question about earlier events, rules, or the map describes context, not something that happened now; record it only if the surrounding lines show the fiction changing.
In the harness, add a fixture exchange of that shape (player: "where were we before?"; GM: "you're in X") with a golden that contains no "party is in X" beat, so the rule is measured rather than hoped.

Related: #423 (extraction passes), #349 (eval harness), #568 (what compose is asked to keep).

**Seen on every summariser tried on the 2026-09-09 session** (Qwen 9B, Gemma 4 26B, hosted Sonnet): the summary places the party *in* Ironroot Hold, when the session's point is that they have been teleported far from it. The stored events explain why: Gemma's run holds a **verified** beat at 49:28, `The party is currently located in Ironroot Hold.` (kind reveal, no problems), citing the GM's line "You're in Ironroot Hold." That line answered a player asking "where were you?" (before the caves); the validator confirms the words were spoken, not what tense they carried, and every composer then repeats a wrong event faithfully. Only the two largest hosted models read past it. **Why it will recur.** GMs answer table questions tersely and in the present tense all the time ("you're in the tavern", "it's level 3", "you're at Bobsville"), and half of them are about the past or hypothetical. The extract prompt tells the model that GM lines are narration; it does not tell it that a GM line replying to a player's out-of-fiction question is context. **Proposed change**, small and testable in the eval harness: - In `EXTRACT_SYSTEM_PROMPT`, one rule: a GM line that answers a player's question about earlier events, rules, or the map describes context, not something that happened now; record it only if the surrounding lines show the fiction changing. - In the harness, add a fixture exchange of that shape (player: "where were we before?"; GM: "you're in X") with a golden that contains no "party is in X" beat, so the rule is measured rather than hoped. Related: #423 (extraction passes), #349 (eval harness), #568 (what compose is asked to keep).

rbrooks referenced this issue from a commit

2026-09-10 05:26:15 +00:00

fix(backend): stop the summariser reading a GM's answer as an event (#584)

claude-bot referenced this issue from a pull request that will close it,

2026-09-10 05:26:29 +00:00

fix(backend): stop the summariser reading a GM's answer as an event (#584) #585

claude-bot referenced this issue

2026-09-10 05:26:37 +00:00

The eval harness cannot penalise a summary for containing an event that never happened #586

claude-bot closed this issue

2026-09-10 05:47:57 +00:00

claude-bot referenced this issue from a commit

2026-09-10 05:47:58 +00:00

Merge PR #585: fix(backend): stop the summariser reading a GM's answer as an event (#584)

claude-bot commented

2026-09-10 06:33:14 +00:00

Author

Collaborator

Reopening: the fix did not take on a real run. Re-summarised the 2026-09-09 session on dev at 06:31 UTC with #585 deployed (Gemma 4 26B, 97 beats extracted, 81 verified). Two verified beats now assert the location, where there was one before:

[49:28] reveal  The GM reveals the party's current location is Ironroot Hold.
[49:31] reveal  The GM reveals the party is in Ironroot Hold and notes the distance between the marked points.

and the summary says "the GM clarified that the party's current location is Ironroot Hold". A LIKE '%in Ironroot Hold%' check passes, which is a good reminder that the phrase was never the thing to test for.

Why the detector did not catch it, which is the useful part. answer_shaped_beats logged nothing at all, so it found no candidates. The reason is the one its own author flagged as a footnote: the pipeline detects against the compacted transcript (#341), where consecutive same-speaker lines merge into one line carrying the run's first timestamp. The exchange looks like this raw:

Harrowhark: You're in Bobsville, right?
GM: No, you were not in Bobsville.
Harrowhark: With all the Bobs?
GM: No.
Viquilonto (Viq): New Town.
GM: You're in Ironroot Hold.

After compaction the GM's replies merge into one long line, so the "≤ 8 words" test that gives the predicate its precision can never fire, and the player question that precedes the answer is no longer adjacent to it. The detector is structurally blind to exactly the shape it was written for.

Suggested next step. Detect against the raw transcript rather than the compacted one, resolving a beat's cited stamp to the span of raw lines that the merged line covers (from the cited stamp up to the next merged line's stamp), and applying the word-count and preceding-question tests to the raw lines inside it. The stored sessions.transcript is the uncompacted text and is already in hand where summarise is called. Then re-measure the 1-of-71 precision figure on raw lines, since that number was also taken on the compacted form and is likely optimistic.

And the prompt rule alone is not enough, at least for a 26B local model: it is in the extract prompt for this run and the beat was produced anyway, twice.

**Reopening: the fix did not take on a real run.** Re-summarised the 2026-09-09 session on dev at 06:31 UTC with #585 deployed (Gemma 4 26B, 97 beats extracted, 81 verified). Two verified beats now assert the location, where there was one before: ``` [49:28] reveal The GM reveals the party's current location is Ironroot Hold. [49:31] reveal The GM reveals the party is in Ironroot Hold and notes the distance between the marked points. ``` and the summary says "the GM clarified that the party's current location is Ironroot Hold". A `LIKE '%in Ironroot Hold%'` check passes, which is a good reminder that the phrase was never the thing to test for. **Why the detector did not catch it, which is the useful part.** `answer_shaped_beats` logged nothing at all, so it found no candidates. The reason is the one its own author flagged as a footnote: the pipeline detects against the **compacted** transcript (#341), where consecutive same-speaker lines merge into one line carrying the run's first timestamp. The exchange looks like this raw: ``` Harrowhark: You're in Bobsville, right? GM: No, you were not in Bobsville. Harrowhark: With all the Bobs? GM: No. Viquilonto (Viq): New Town. GM: You're in Ironroot Hold. ``` After compaction the GM's replies merge into one long line, so the "≤ 8 words" test that gives the predicate its precision can never fire, and the player question that precedes the answer is no longer adjacent to it. The detector is structurally blind to exactly the shape it was written for. **Suggested next step.** Detect against the raw transcript rather than the compacted one, resolving a beat's cited stamp to the *span* of raw lines that the merged line covers (from the cited stamp up to the next merged line's stamp), and applying the word-count and preceding-question tests to the raw lines inside it. The stored `sessions.transcript` is the uncompacted text and is already in hand where `summarise` is called. Then re-measure the 1-of-71 precision figure on raw lines, since that number was also taken on the compacted form and is likely optimistic. **And the prompt rule alone is not enough**, at least for a 26B local model: it is in the extract prompt for this run and the beat was produced anyway, twice.

rbrooks referenced this issue from a commit

2026-09-10 07:09:27 +00:00

fix(backend): check for an answer-shaped beat against the lines that were spoken (#584)

claude-bot referenced this issue

2026-09-10 07:09:43 +00:00

fix(backend): pick the summary's events by time, and check for an answer-shaped beat against what was spoken (#589, #584) #591

claude-bot commented

2026-09-10 07:10:00 +00:00

Author

Collaborator

Correcting two things I wrote in the comment above, both found by measuring rather than reasoning:

The "1 of 71 GM lines" figure was already a raw-transcript measurement. The golden file has 218 lines and 71 GM lines; its compacted form has 209 and 66. So the precision claim was never optimistic in the way I said, and it is unchanged by the fix. The docstring now names which text the number refers to.
My second mechanism was wrong. Merging GM lines cannot separate a question from its answer, because a player's line always breaks a GM run and merging can only ever bring the two closer together. The real second mechanism is that a player's question merges with that player's own next remark, so the merged line no longer ends in a question mark and the lookback finds nothing. Both mechanisms now have tests, each paired with an assertion that the compacted path reports nothing, so neither can rot into passing for the wrong reason.

Also worth recording: the mapping from a merged line back to its raw lines is built by walking the two line lists in step and matching on second and speaker, not by the stamp arithmetic I proposed. On the golden transcript that arithmetic produces an empty window for four lines that collide on the same second, which would have put a fresh blind spot inside the fix for a blind spot.

And the honest limitation: the golden fixture cannot demonstrate this bug at all. It is authored as strict turn-taking, so only 9 of its 209 lines merge two raw lines and none merge three, and the raw and compacted measurements agree for that reason rather than because the change is a no-op. On real speech, where compaction removes a quarter of the lines, the predicate now matches shapes it structurally could not — and its false-positive rate there is still unmeasured. That is why it continues to report rather than reject.

Fix is in the PR opened alongside this comment. Leaving this issue open: the prompt rule from #585 demonstrably did not stop a 26B local model producing the beat twice, so the underlying problem is not closed.

**Correcting two things I wrote in the comment above**, both found by measuring rather than reasoning: 1. **The "1 of 71 GM lines" figure was already a raw-transcript measurement.** The golden file has 218 lines and 71 GM lines; its compacted form has 209 and 66. So the precision claim was never optimistic in the way I said, and it is unchanged by the fix. The docstring now names which text the number refers to. 2. **My second mechanism was wrong.** Merging *GM* lines cannot separate a question from its answer, because a player's line always breaks a GM run and merging can only ever bring the two closer together. The real second mechanism is that a **player's question merges with that player's own next remark**, so the merged line no longer ends in a question mark and the lookback finds nothing. Both mechanisms now have tests, each paired with an assertion that the compacted path reports nothing, so neither can rot into passing for the wrong reason. Also worth recording: the mapping from a merged line back to its raw lines is built by walking the two line lists in step and matching on second and speaker, **not** by the stamp arithmetic I proposed. On the golden transcript that arithmetic produces an empty window for four lines that collide on the same second, which would have put a fresh blind spot inside the fix for a blind spot. And the honest limitation: **the golden fixture cannot demonstrate this bug at all.** It is authored as strict turn-taking, so only 9 of its 209 lines merge two raw lines and none merge three, and the raw and compacted measurements agree for that reason rather than because the change is a no-op. On real speech, where compaction removes a quarter of the lines, the predicate now matches shapes it structurally could not — and its false-positive rate there is still unmeasured. That is why it continues to report rather than reject. Fix is in the PR opened alongside this comment. Leaving this issue open: the prompt rule from #585 demonstrably did not stop a 26B local model producing the beat twice, so the underlying problem is not closed.

claude-bot referenced this issue from a commit

2026-09-10 07:30:59 +00:00

Merge PR #591: fix(backend): pick the summary's events by time, and check for an answer-shaped beat against what was spoken (#589, #584)

claude-bot referenced this issue

2026-09-11 03:32:01 +00:00

Beat extraction records the GM's table admin as story events #596

claude-bot referenced this issue

2026-09-11 03:32:18 +00:00

[Frontend] Show a GM the verified events behind a summary #424

rbrooks referenced this issue from a commit

2026-09-11 05:13:19 +00:00

feat(webapp): label the GM running the table, and never narrate it (#596)