The unsupported-sentence critic counts sentence-opening words as unknown names #567

New issue

Open

opened 2026-09-09 04:47:35 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-09-09 04:47:35 +00:00

Contributor

Symptom. beat_service.unsupported_sentences (#334) treats any capitalised word not in the closed set of speaker labels, character names and verified-beat actors as a name the summary invented. Sentence-initial words are capitalised, so ordinary prose openers are flagged: replaying the 2026-09-09 session through the real pipeline, the reasons recorded were 'Early', 'Fortunately', 'Undeterred', 'Continuing' (Haiku), 'Ahead', 'Getting', 'Whatever' (Opus) and 'Among', 'Rounding' (Sonnet). None is a name. The Qwen run's unsupported_sentences = 3 on the same session is likely the same thing.

Why it matters. summarisation_runs.unsupported_sentences is the number a GM or an eval reads as "sentences that name someone the transcript does not account for". Inflated by capitalisation, it cannot distinguish a run with a real hallucinated name from one with a few adverbs, which is the one distinction #334 exists to make.

Fix. In _capitalised_names, skip the first token of each sentence unless it is also capitalised mid-sentence elsewhere in the summary or matches a known name, and skip a small stoplist of common sentence openers. Add a test with a summary that opens sentences with "Fortunately" and "Meanwhile" and names nobody unknown, asserting zero findings, alongside the existing positive case.

**Symptom.** `beat_service.unsupported_sentences` (#334) treats any capitalised word not in the closed set of speaker labels, character names and verified-beat actors as a name the summary invented. Sentence-initial words are capitalised, so ordinary prose openers are flagged: replaying the 2026-09-09 session through the real pipeline, the reasons recorded were `'Early'`, `'Fortunately'`, `'Undeterred'`, `'Continuing'` (Haiku), `'Ahead'`, `'Getting'`, `'Whatever'` (Opus) and `'Among'`, `'Rounding'` (Sonnet). None is a name. The Qwen run's `unsupported_sentences = 3` on the same session is likely the same thing. **Why it matters.** `summarisation_runs.unsupported_sentences` is the number a GM or an eval reads as "sentences that name someone the transcript does not account for". Inflated by capitalisation, it cannot distinguish a run with a real hallucinated name from one with a few adverbs, which is the one distinction #334 exists to make. **Fix.** In `_capitalised_names`, skip the first token of each sentence unless it is also capitalised mid-sentence elsewhere in the summary or matches a known name, and skip a small stoplist of common sentence openers. Add a test with a summary that opens sentences with "Fortunately" and "Meanwhile" and names nobody unknown, asserting zero findings, alongside the existing positive case.