rbrooks/Quest-Board

Watch

Fork 0

Summary ends with an empty "What's next?" heading on Gemma 4 #595

New issue

Closed

opened 2026-09-11 03:31:47 +00:00 by claude-bot · 2 comments

claude-bot commented

2026-09-11 03:31:47 +00:00

Collaborator

What happens

Both summary-only reruns with Gemma 4 26B (the router's qwen3.5 alias) on the dev replay session ended like this:

…Wyatt was able to identify a mid-sized city located a few miles from an exit.

What's next?

The runs were on 2026-09-10 at 06:31 UTC (at #587) and 2026-09-11 at 03:28 UTC (at 6533fa1, which includes #591). In both, the heading is there with nothing after it. Prod is also on Gemma, because the router is shared, so a real session's summary will end the same way.

Why

Both compose prompts ask the model to "end with a short 'What's next?' hook" (beat_service.py ~828, audio_service.py ~139 and ~2879). Gemma reads the quoted phrase as a heading to print, prints it, and stops. The material for a hook was there. The last events are the party learning of a mid-sized city a few miles from an exit, and the circus rumour of a faster way between towns.

_WHATS_NEXT_RE already finds the hook for note_unsummarised_windows, so the code can tell where it is.

Fix

Say what the hook is. Reword both prompts to ask for content, not a heading: e.g. "End with a final short paragraph that begins 'What's next?' and says, in one or two sentences, what the party is left facing or has decided to do."
Guard the output. If the summary ends with the hook and nothing after it (only whitespace or punctuation following _WHATS_NEXT_RE), drop the empty heading rather than publish it. An empty heading reads as broken; a summary without a hook doesn't. Count it on the run record, the way truncated is, so it's visible rather than silent.
Test it at the request level, per feedback_llm_test_seam: assert the reworded instruction is in the system prompt, and run the guard on a stubbed reply that ends with a bare "What's next?".

Found while comparing summaries for #591.

## What happens Both summary-only reruns with Gemma 4 26B (the router's `qwen3.5` alias) on the dev replay session ended like this: ``` …Wyatt was able to identify a mid-sized city located a few miles from an exit. What's next? ``` The runs were on 2026-09-10 at 06:31 UTC (at #587) and 2026-09-11 at 03:28 UTC (at 6533fa1, which includes #591). In both, the heading is there with nothing after it. **Prod is also on Gemma**, because the router is shared, so a real session's summary will end the same way. ## Why Both compose prompts ask the model to *"end with a short 'What's next?' hook"* (`beat_service.py` ~828, `audio_service.py` ~139 and ~2879). Gemma reads the quoted phrase as a heading to print, prints it, and stops. The material for a hook was there. The last events are the party learning of a mid-sized city a few miles from an exit, and the circus rumour of a faster way between towns. `_WHATS_NEXT_RE` already finds the hook for `note_unsummarised_windows`, so the code can tell where it is. ## Fix 1. **Say what the hook is.** Reword both prompts to ask for content, not a heading: e.g. "End with a final short paragraph that begins 'What's next?' and says, in one or two sentences, what the party is left facing or has decided to do." 2. **Guard the output.** If the summary ends with the hook and nothing after it (only whitespace or punctuation following `_WHATS_NEXT_RE`), drop the empty heading rather than publish it. An empty heading reads as broken; a summary without a hook doesn't. Count it on the run record, the way `truncated` is, so it's visible rather than silent. 3. **Test it** at the request level, per [[feedback_llm_test_seam]]: assert the reworded instruction is in the system prompt, and run the guard on a stubbed reply that ends with a bare "What's next?". Found while comparing summaries for #591.

claude-bot added the

backend

bug

labels

2026-09-11 03:32:24 +00:00

claude-bot commented

2026-09-11 04:46:05 +00:00

Author

Collaborator

Picking this up (2026-09-11), in one branch with #588's exact-duplicate case. Both changes are deterministic, so one dev rerun will show their effect.

The plan follows the body above:

Prompts: reword every prompt that asks for the hook so it asks for content, not a heading.
Guard: drop an empty trailing hook, on both the compose path and the prose path. The guard runs before note_unsummarised_windows places its notes.
Record: a hook_dropped column on summarisation_runs, so it's visible and not silent.

#596 (the table-admin kind) is being built in parallel. It changes what the model produces, so it will be measured separately, after this one.

**Picking this up** (2026-09-11), in one branch with #588's exact-duplicate case. Both changes are deterministic, so one dev rerun will show their effect. The plan follows the body above: - **Prompts:** reword every prompt that asks for the hook so it asks for content, not a heading. - **Guard:** drop an empty trailing hook, on both the compose path and the prose path. The guard runs before `note_unsummarised_windows` places its notes. - **Record:** a `hook_dropped` column on `summarisation_runs`, so it's visible and not silent. #596 (the table-admin kind) is being built in parallel. It changes what the model produces, so it will be measured separately, after this one.

claude-bot referenced this issue

2026-09-11 04:46:09 +00:00

Beat dedupe misses same-moment duplicates and split moments, inflating the event log #588

claude-bot referenced this issue

2026-09-11 04:46:14 +00:00

Beat extraction records the GM's table admin as story events #596

rbrooks referenced this issue from a commit

2026-09-11 05:10:36 +00:00

fix(backend): drop an empty "What's next?" hook, collapse exact duplicate beats (#595, #588)

claude-bot referenced this issue

2026-09-11 05:12:02 +00:00

fix(backend): no empty "What's next?", and collapse exact-duplicate beats (#595, #588) #598

claude-bot referenced this issue from a commit

2026-09-11 05:33:35 +00:00

Merge pull request 'fix(backend): no empty "What's next?", and collapse exact-duplicate beats (#595, #588)' (#598) from fix/595-588-hook-and-dupes into main

claude-bot commented

2026-09-11 05:46:48 +00:00

Author

Collaborator

Fixed and measured. Merged in PR #598 (main 0935b7a) and deployed to dev, then checked with a summary-only rerun on the replay session (Gemma 4 26B, run 6d3e6f54, 2026-09-11 05:45 UTC).

The summary now ends with a real hook:

What's next? The party now knows they are near a mid-sized city and must decide how to proceed toward it.

hook_dropped is false: the reworded instruction (HOOK_INSTRUCTION) was enough on its own, and the guard didn't need to fire. The guard stays for models that still print only the heading, and when it fires it's recorded on the run rather than hidden.
Run 3, before the fix, ended on a bare "What's next?", as did the run before that.

Closing.

**Fixed and measured.** Merged in PR #598 (main `0935b7a`) and deployed to dev, then checked with a summary-only rerun on the replay session (Gemma 4 26B, run `6d3e6f54`, 2026-09-11 05:45 UTC). The summary now ends with a real hook: > What's next? The party now knows they are near a mid-sized city and must decide how to proceed toward it. - **`hook_dropped` is false:** the reworded instruction (`HOOK_INSTRUCTION`) was enough on its own, and the guard didn't need to fire. The guard stays for models that still print only the heading, and when it fires it's recorded on the run rather than hidden. - **Run 3, before the fix**, ended on a bare "What's next?", as did the run before that. Closing.

claude-bot closed this issue

2026-09-11 05:47:03 +00:00

claude-bot referenced this issue

2026-09-11 06:10:33 +00:00

feat(backend): label the GM's table admin instead of narrating it (#596) #599