fix(backend): stop the summariser reading a GM's answer as an event (#584) #585

Merged

claude-bot merged 1 commit from fix/584-gm-answer-context into main

2026-09-10 05:47:56 +00:00

claude-bot commented

2026-09-10 05:26:28 +00:00

Contributor

Closes #584.

The defect. Every summariser tried on the 2026-09-09 session, local and hosted, put the party in Ironroot Hold, the one place the session establishes they are far away from. The run holds a verified beat, "The party is currently located in Ironroot Hold", citing the GM's line "You're in Ironroot Hold" — which answered a player asking where they had been before the caves. The validator confirms the words were spoken and who spoke them; it cannot read tense, so a false event passes every check and every composer repeats it.

The fix is the prompt. One rule beside the existing table-talk rule: a GM line answering a player's question, about earlier events, the rules, the map or a name, states context rather than something happening now, and is not an event unless the surrounding lines show the fiction changing. The narration and narrated rules are untouched; this carves an exception out of them.

The code-side check reports, and never rejects. answer_shaped_beats flags a beat whose cited lines are all short GM lines shortly after a player's question, logged as a count. It is not a problems entry, on evidence rather than taste: a problems entry removes the beat from compose and from the anchors, so one false positive silently deletes a real event. Measured on the golden transcript, the tight predicate flags 1 of 71 GM lines; loosening the lookback to six lines with no length bound flags 40 of 71, including the session's pivotal reveal. A terse GM would lose their most important beat, and there is no real-speech corpus in the repo to fix a threshold against. So it is an instrument, not a gate, and its docstring is honest about what it misses.

Eval note. The scores are byte-identical before and after (coverage 0.950, attribution 0.950, chronology 1.000, beat validation 0.950), and that identity is itself the finding: all four scorers measure recall or correctness, so none can penalise a summary for containing an extra false event. Filed as a follow-up.

Tests: 12 new in test_beat_service.py including a case pinning that the validator still accepts the trap beat, plus end-to-end coverage that the count is logged and the beat text is not. Full suite 3,097 passed; formatted with CI's ruff.

🤖 Generated with Claude Code

Closes #584. **The defect.** Every summariser tried on the 2026-09-09 session, local and hosted, put the party *in* Ironroot Hold, the one place the session establishes they are far away from. The run holds a *verified* beat, "The party is currently located in Ironroot Hold", citing the GM's line "You're in Ironroot Hold" — which answered a player asking where they had been *before* the caves. The validator confirms the words were spoken and who spoke them; it cannot read tense, so a false event passes every check and every composer repeats it. **The fix is the prompt.** One rule beside the existing table-talk rule: a GM line answering a player's question, about earlier events, the rules, the map or a name, states context rather than something happening now, and is not an event unless the surrounding lines show the fiction changing. The narration and `narrated` rules are untouched; this carves an exception out of them. **The code-side check reports, and never rejects.** `answer_shaped_beats` flags a beat whose cited lines are all short GM lines shortly after a player's question, logged as a count. It is not a `problems` entry, on evidence rather than taste: a problems entry removes the beat from compose *and* from the anchors, so one false positive silently deletes a real event. Measured on the golden transcript, the tight predicate flags 1 of 71 GM lines; loosening the lookback to six lines with no length bound flags 40 of 71, including the session's pivotal reveal. A terse GM would lose their most important beat, and there is no real-speech corpus in the repo to fix a threshold against. So it is an instrument, not a gate, and its docstring is honest about what it misses. **Eval note.** The scores are byte-identical before and after (coverage 0.950, attribution 0.950, chronology 1.000, beat validation 0.950), and that identity is itself the finding: all four scorers measure recall or correctness, so none can penalise a summary for containing an *extra* false event. Filed as a follow-up. Tests: 12 new in `test_beat_service.py` including a case pinning that the validator still accepts the trap beat, plus end-to-end coverage that the count is logged and the beat text is not. Full suite 3,097 passed; formatted with CI's ruff. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit

2026-09-10 05:26:29 +00:00

fix(backend): stop the summariser reading a GM's answer as an event (#584 )

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 1m4s

Details

CI / Backend lint (ruff) (pull_request) Successful in 55s

Details

CI / Bot/backend version sync (pull_request) Successful in 30s

Details

CI / Docker image build (pull_request) Successful in 30s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 3m7s

Details

CI / Bot tests and audit (pull_request) Successful in 2m44s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 12m43s

Details

CI / Synthetic session harness (no GPU, no LLM) (pull_request) Successful in 17m4s

Details

0e23eea91b

On one real session every summariser tried — a 9B local model, a 26B one and a
frontier hosted model — put the party in the place the session was about them
being teleported away from. All three were faithfully repeating one *verified*
beat: the GM had said "you're in <place>", a few turns into the players asking
where they had been *before* the caves. The words were spoken and the speaker
spoke them, which is everything `validate_beats` can decide from the text; the
tense was answering a question about the past, which it cannot.

So the fix is in extraction, not compose or validation. One rule is added to
`EXTRACT_SYSTEM_PROMPT`: a GM line answering a player's question — about earlier
events, the rules, the map or a name — states context rather than something
happening now, and may only be recorded as an event when the surrounding lines
show the fiction changing. The existing rule that GM lines are narration and NPC
dialogue is untouched; this carves an exception out of it rather than replacing
it.

`beat_service.answer_shaped_beats` reports the shape but never rejects it, and
that is a deliberate decision rather than caution. A `problems` entry keeps a
beat out of compose and out of the anchors, so a false positive would delete a
real event from a summary — the trade #333 exists to refuse — and the word-count
bound is the only thing between the heuristic and a laconic GM's most
consequential line. Measured before choosing: on the synthetic session's golden
transcript the shape matches 1 of 71 GM lines (that one table talk), and none of
the 25 canned beats in the eval fixtures; but that is one authored transcript,
and the real session cannot be committed, so the false-positive rate on real
speech is unmeasured. The pipeline logs the count only, which is what a later
decision to reject would be made on.

Tested at the unit level rather than as an eval fixture: the four scorers in
`evals/scoring.py` are all recall or correctness measures, so none of them can
penalise a summary for containing an *extra* false beat, and the audio harness's
`FORBIDDEN` probes — which can — need new authored lines and therefore new TTS
clips, a manifest rebuild and an ASR recalibration.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

claude-bot scheduled this pull request to auto merge when all checks succeed

2026-09-10 05:26:46 +00:00

claude-bot referenced this pull request

2026-09-10 05:32:53 +00:00

feat(webapp): a bounded summary and a complete event log (#568) #587

claude-bot merged commit 26d68235df into main

2026-09-10 05:47:56 +00:00

claude-bot referenced this pull request from a commit

2026-09-10 05:47:58 +00:00

Merge PR #585: fix(backend): stop the summariser reading a GM's answer as an event (#584)

claude-bot deleted branch fix/584-gm-answer-context

2026-09-10 05:47:58 +00:00