The unsupported-sentence critic counts sentence-opening words as unknown names #567

New issue

Closed

opened 2026-09-09 04:47:35 +00:00 by claude-bot · 1 comment

claude-bot commented

2026-09-09 04:47:35 +00:00

Collaborator

Symptom. beat_service.unsupported_sentences (#334) treats any capitalised word not in the closed set of speaker labels, character names and verified-beat actors as a name the summary invented. Sentence-initial words are capitalised, so ordinary prose openers are flagged: replaying the 2026-09-09 session through the real pipeline, the reasons recorded were 'Early', 'Fortunately', 'Undeterred', 'Continuing' (Haiku), 'Ahead', 'Getting', 'Whatever' (Opus) and 'Among', 'Rounding' (Sonnet). None is a name. The Qwen run's unsupported_sentences = 3 on the same session is likely the same thing.

Why it matters. summarisation_runs.unsupported_sentences is the number a GM or an eval reads as "sentences that name someone the transcript does not account for". Inflated by capitalisation, it cannot distinguish a run with a real hallucinated name from one with a few adverbs, which is the one distinction #334 exists to make.

Fix. In _capitalised_names, skip the first token of each sentence unless it is also capitalised mid-sentence elsewhere in the summary or matches a known name, and skip a small stoplist of common sentence openers. Add a test with a summary that opens sentences with "Fortunately" and "Meanwhile" and names nobody unknown, asserting zero findings, alongside the existing positive case.

**Symptom.** `beat_service.unsupported_sentences` (#334) treats any capitalised word not in the closed set of speaker labels, character names and verified-beat actors as a name the summary invented. Sentence-initial words are capitalised, so ordinary prose openers are flagged: replaying the 2026-09-09 session through the real pipeline, the reasons recorded were `'Early'`, `'Fortunately'`, `'Undeterred'`, `'Continuing'` (Haiku), `'Ahead'`, `'Getting'`, `'Whatever'` (Opus) and `'Among'`, `'Rounding'` (Sonnet). None is a name. The Qwen run's `unsupported_sentences = 3` on the same session is likely the same thing. **Why it matters.** `summarisation_runs.unsupported_sentences` is the number a GM or an eval reads as "sentences that name someone the transcript does not account for". Inflated by capitalisation, it cannot distinguish a run with a real hallucinated name from one with a few adverbs, which is the one distinction #334 exists to make. **Fix.** In `_capitalised_names`, skip the first token of each sentence unless it is also capitalised mid-sentence elsewhere in the summary or matches a known name, and skip a small stoplist of common sentence openers. Add a test with a summary that opens sentences with "Fortunately" and "Meanwhile" and names nobody unknown, asserting zero findings, alongside the existing positive case.

rbrooks referenced this issue from a commit

2026-09-09 17:35:02 +00:00

fix(backend): stop the unsupported-sentence critic reading openers as names (#567)

rbrooks referenced this issue from a commit

2026-09-09 17:35:02 +00:00

fix(backend): window the highlights call instead of sending the whole transcript (#575)

rbrooks referenced this issue from a commit

2026-09-09 17:35:37 +00:00

fix(backend): stop the unsupported-sentence critic reading openers as names (#567)

rbrooks referenced this issue from a commit

2026-09-09 17:35:37 +00:00

fix(backend): window the highlights call instead of sending the whole transcript (#575)

claude-bot referenced this issue

2026-09-09 17:35:50 +00:00

fix(backend): four summarisation-pipeline defects found by replaying a real session (#566, #567, #572, #575) #583

claude-bot referenced this issue from a commit

2026-09-09 17:52:35 +00:00

Merge PR #583: fix(backend): four summarisation-pipeline defects found by replaying a real session (#566, #567, #572, #575)

claude-bot referenced this issue

2026-09-09 18:38:32 +00:00

Beat extraction silently loses whole windows on the local Qwen router, and logs the failure with an empty message #572

claude-bot commented

2026-09-12 20:08:55 +00:00

Author

Collaborator

Closing — fixed by #583 (commit 6e197c9, "stop the unsupported-sentence critic reading openers as names"), the third of that PR's four fixes to go unclosed. Verified on main today.

The fix is structural rather than a stoplist, which is the right call and is argued in the code:

_capitalised_names is replaced by _vouched_for_words: a word capitalised at the start of a sentence tells you nothing, because English capitalises every sentence's first word; the same word capitalised anywhere else in the same summary does, because nothing but a name or an acronym is capitalised mid-sentence in composed prose. So the summary vouches for its own names.
The stoplist of real offenders — "Early", "Fortunately", "Undeterred", "Continuing", "Ahead", "Getting", "Whatever", "Among", "Rounding" — is kept only as belt-and-braces for a word that turns up capitalised mid-sentence by coincidence. The comment is explicit that it "is deliberately not an attempt to enumerate the openers a scribe can reach for; that set has no end, which is why the rule and not the list is the fix."

Covered by tests in test_beat_service.py, test_summarisation_runs.py and test_summarise_beats.py. The dev replay that motivated it reported critic count 0 afterwards.

Closing — fixed by **#583** (commit `6e197c9`, "stop the unsupported-sentence critic reading openers as names"), the third of that PR's four fixes to go unclosed. Verified on `main` today. The fix is structural rather than a stoplist, which is the right call and is argued in the code: - `_capitalised_names` is replaced by `_vouched_for_words`: a word capitalised at the *start* of a sentence tells you nothing, because English capitalises every sentence's first word; the same word capitalised **anywhere else in the same summary** does, because nothing but a name or an acronym is capitalised mid-sentence in composed prose. So the summary vouches for its own names. - The stoplist of real offenders — "Early", "Fortunately", "Undeterred", "Continuing", "Ahead", "Getting", "Whatever", "Among", "Rounding" — is kept only as belt-and-braces for a word that turns up capitalised mid-sentence by coincidence. The comment is explicit that it "is deliberately not an attempt to enumerate the openers a scribe can reach for; that set has no end, which is why the rule and not the list is the fix." Covered by tests in `test_beat_service.py`, `test_summarisation_runs.py` and `test_summarise_beats.py`. The dev replay that motivated it reported critic count 0 afterwards.

claude-bot closed this issue

2026-09-12 20:08:58 +00:00