The eval harness cannot penalise a summary for containing an event that never happened #586

New issue

Open

opened 2026-09-10 05:26:37 +00:00 by claude-bot · 0 comments

claude-bot commented

2026-09-10 05:26:37 +00:00

Contributor

Found while fixing #584. All four scorers in evals/scoring.py — coverage, attribution, chronology, beat validation — measure recall or correctness of what the summary does contain. None of them can mark a summary down for containing an extra claim that is false. So the defect in #584, a verified-but-wrong beat that put the party in the wrong town and survived every model we tried, scores 1.000 across the board, and the fix for it moves no number.

That is why #584 was found by a person reading a summary rather than by the harness, and it will be true of the next defect of the same shape.

Proposed. A precision metric: per-fixture forbidden phrase groups in the fixture JSON, scored as "none of these appear in the summary", reported alongside coverage. The synthetic-session harness already has the only precision-shaped scorer in the codebase (synthetic_session/scorers.py::score_table_talk), so the vocabulary exists; this extends it to the text fixtures.

Two things to get right:

A forbidden group has to be robust to paraphrase, or it measures wording. Groups of alternatives ("is in Ironroot Hold", "located in Ironroot Hold", "arrived at Ironroot Hold") rather than one string.
The fixture that carries #584's case needs authored lines in the scene, not a canned stub beat, because the stub path cannot exercise a prompt change. The synthetic-session scene would have to be re-authored to add the exchange, which the README's re-authoring order makes a deliberate job (piper has no seed; the clips are the artifact).

Until then, note on any prompt-level change that the harness cannot confirm it.

Found while fixing #584. All four scorers in `evals/scoring.py` — coverage, attribution, chronology, beat validation — measure recall or correctness of what the summary *does* contain. None of them can mark a summary down for containing an **extra** claim that is false. So the defect in #584, a verified-but-wrong beat that put the party in the wrong town and survived every model we tried, scores 1.000 across the board, and the fix for it moves no number. That is why #584 was found by a person reading a summary rather than by the harness, and it will be true of the next defect of the same shape. **Proposed.** A precision metric: per-fixture `forbidden` phrase groups in the fixture JSON, scored as "none of these appear in the summary", reported alongside coverage. The synthetic-session harness already has the only precision-shaped scorer in the codebase (`synthetic_session/scorers.py::score_table_talk`), so the vocabulary exists; this extends it to the text fixtures. Two things to get right: - A forbidden group has to be robust to paraphrase, or it measures wording. Groups of alternatives ("is in Ironroot Hold", "located in Ironroot Hold", "arrived at Ironroot Hold") rather than one string. - The fixture that carries #584's case needs authored lines in the scene, not a canned stub beat, because the stub path cannot exercise a prompt change. The synthetic-session scene would have to be re-authored to add the exchange, which the README's re-authoring order makes a deliberate job (piper has no seed; the clips are the artifact). Until then, note on any prompt-level change that the harness cannot confirm it.

claude-bot referenced this issue

2026-09-11 03:32:01 +00:00

Beat extraction records the GM's table admin as story events #596

claude-bot referenced this issue

2026-09-11 03:32:18 +00:00

[Frontend] Show a GM the verified events behind a summary #424

claude-bot referenced this issue

2026-09-11 13:40:31 +00:00

[Frontend] Show a GM the verified events behind a summary #424