rbrooks/Quest-Board

Watch

Fork 0

feat(evals): score what a summary should not say (#586) #633

Merged

claude-bot merged 1 commit from feat/586-precision-metric into main

2026-09-12 21:57:13 +00:00

claude-bot commented

2026-09-12 21:15:24 +00:00

Collaborator

The harness could not fail a summary for containing something false — only for missing something true. This adds the metric that can.

The gap

All four scorers measure recall or correctness over what the summary does contain. #584 was a verified beat that put the party in a town they had only discussed from the road; it survived every model tried, scored 1.000 across the board, and was found by a person reading the prose. The fix for it moved no number.

What this adds

forbidden[] on a fixture — claims a summary of this session must not make. Same mentions shape as an event, and matched by exactly the same rule (every group needs one alternative present), so there is one matching rule in scoring.py and not two. Groups are alternatives so a claim measures meaning rather than wording, and a second group expresses "only false in combination" — "Ironroot Hold" alone is fine, the session discusses it.
score_precision → the share of claims the summary did not make, with the fired claim ids as evidence.
Carried through the runner into the report table, the aggregates, the history record (including which claims fired — a precision drop is only actionable if you know which one) and a --fail-under-precision gate.

None, not 1.0, when a fixture pins nothing. "Not measured" is not a clean pass, and unpinned fixtures are excluded from the mean rather than averaged in — otherwise the aggregate would rise as fixtures without the check were added.

The fixture that proves it works

fixtures/synthetic/fabricated_location.json makes #584 reproducible. The party discuss Ironroot Hold from the road and deliberately take the other fork; one beat says they went in, citing lines that really do mention it and really were spoken by the named player — so every grounding check passes.

fixture                      provider model         coverage  attrib.  chron.(tau)  chron.(0-1)  beat val.  precision  cov.in beats
clean-run                    stub     stub             1.000    1.000        1.000        1.000      1.000      1.000         1.000
fabricated-location          stub     stub             1.000    1.000        1.000        1.000      1.000      0.500         1.000
hallucinated-citation        stub     stub             0.800    1.000        1.000        1.000      0.800        n/a         0.800

That row is the whole argument: perfect on everything the harness could previously see, and wrong.

Two things worth knowing about authoring it — both cost me a cycle and are now documented:

A beat whose evidence falls outside its own t_start/t_end is dropped before validation rather than failing it, so the false beat had to be internally consistent (which is closer to #584's real shape anyway).
The false beat originally said "…before they went on to the caravan", which pulled another event's first-mention offset into its sentence and dropped attribution to 0.75. A fixture meant to fail exactly one metric has to be written not to disturb the others.

Scope I did not take

The issue also notes the synthetic-session (audio) scene would need re-authoring to carry this case there. I left that alone deliberately — per CLAUDE.md and the harness README, the committed .opus clips are the artifact (piper has no seed), so re-authoring is a separate, deliberate job rather than something to slip into this PR. The text-fixture half needs no audio and is complete here.

Tests

Seven unit tests for the scorer (paraphrase groups, every-group-required, case-insensitivity, not-measured None, an unusable claim that cannot fire) plus the fixture's pinned expected.precision, newly asserted in the existing parametrised test. That assertion also fails a fixture carrying forbidden[] with no pinned precision — a forbidden claim nothing asserts a number about is a check that cannot fail.

Backend suite: 3457 passed, 13 skipped. ruff check / ruff format --check clean under CI's pinned 0.4.4.

Closes #586

🤖 Generated with Claude Code

The harness could not fail a summary for containing something false — only for missing something true. This adds the metric that can. ## The gap All four scorers measure recall or correctness over what the summary *does* contain. #584 was a verified beat that put the party in a town they had only discussed from the road; it survived every model tried, scored **1.000 across the board**, and was found by a person reading the prose. The fix for it moved no number. ## What this adds - **`forbidden[]` on a fixture** — claims a summary of this session must not make. Same `mentions` shape as an event, and matched by exactly the same rule (every group needs one alternative present), so there is one matching rule in `scoring.py` and not two. Groups are *alternatives* so a claim measures meaning rather than wording, and a second group expresses "only false in combination" — "Ironroot Hold" alone is fine, the session discusses it. - **`score_precision`** → the share of claims the summary did *not* make, with the fired claim ids as evidence. - Carried through the runner into the **report table, the aggregates, the history record** (including which claims fired — a precision drop is only actionable if you know which one) and a **`--fail-under-precision`** gate. `None`, not `1.0`, when a fixture pins nothing. "Not measured" is not a clean pass, and unpinned fixtures are excluded from the mean rather than averaged in — otherwise the aggregate would *rise* as fixtures without the check were added. ## The fixture that proves it works `fixtures/synthetic/fabricated_location.json` makes #584 reproducible. The party discuss Ironroot Hold from the road and deliberately take the other fork; one beat says they went in, citing lines that really do mention it and really were spoken by the named player — so every grounding check passes. ``` fixture provider model coverage attrib. chron.(tau) chron.(0-1) beat val. precision cov.in beats clean-run stub stub 1.000 1.000 1.000 1.000 1.000 1.000 1.000 fabricated-location stub stub 1.000 1.000 1.000 1.000 1.000 0.500 1.000 hallucinated-citation stub stub 0.800 1.000 1.000 1.000 0.800 n/a 0.800 ``` That row is the whole argument: perfect on everything the harness could previously see, and wrong. Two things worth knowing about authoring it — both cost me a cycle and are now documented: - A beat whose `evidence` falls outside its own `t_start`/`t_end` is dropped before validation rather than failing it, so the false beat had to be internally consistent (which is closer to #584's real shape anyway). - The false beat originally said "…before they went on to the caravan", which pulled another event's first-mention offset into its sentence and dropped **attribution** to 0.75. A fixture meant to fail exactly one metric has to be written not to disturb the others. ## Scope I did not take The issue also notes the synthetic-session (audio) scene would need re-authoring to carry this case there. I left that alone deliberately — per `CLAUDE.md` and the harness README, the committed `.opus` clips *are* the artifact (piper has no seed), so re-authoring is a separate, deliberate job rather than something to slip into this PR. The text-fixture half needs no audio and is complete here. ## Tests Seven unit tests for the scorer (paraphrase groups, every-group-required, case-insensitivity, not-measured `None`, an unusable claim that cannot fire) plus the fixture's pinned `expected.precision`, newly asserted in the existing parametrised test. That assertion also fails a fixture carrying `forbidden[]` with no pinned precision — a forbidden claim nothing asserts a number about is a check that cannot fail. Backend suite: `3457 passed, 13 skipped`. `ruff check` / `ruff format --check` clean under CI's pinned 0.4.4. Closes #586 🤖 Generated with [Claude Code](https://claude.com/claude-code)

claude-bot added 1 commit

2026-09-12 21:15:25 +00:00

feat(evals): score what a summary should not say (#586 )

CI / Bot/backend version sync (pull_request) Successful in 24s

Details

CI / Backend lint (ruff) (pull_request) Successful in 26s

Details

CI / Summarisation accuracy eval harness (stub provider) (pull_request) Successful in 2m14s

Details

CI / Bot tests and audit (pull_request) Successful in 3m25s

Details

CI / Frontend tests, audit, and build (pull_request) Successful in 4m4s

Details

CI / Docker image build (pull_request) Successful in 4m29s

Details

CI / Synthetic session harness (no GPU, no LLM) (pull_request) Successful in 14m56s

Details

CI / Backend migration, tests, and audit (pull_request) Successful in 16m37s

Details

6c31411bf9

All four scorers in `evals/scoring.py` measure recall or correctness of what
the summary *does* contain, so none could penalise it for an extra claim that
is false. #584 was exactly that shape — a verified-but-wrong beat that put the
party in a town they had only discussed from the road — and it scored 1.000
across the board while a person reading the prose spotted it immediately.

A fixture can now carry `forbidden[]`: claims a summary of this session must
not make, each a group of alternatives so it measures meaning rather than
wording. `score_precision` reports the share the summary did *not* make, and
the runner carries it into the report table, the aggregates, the history
record and a `--fail-under-precision` gate.

A claim fires on exactly the same rule as coverage — every group must have one
alternative present — so there is one matching rule in this module and not
two. `None` (not 1.0) when a fixture pins nothing: "not measured" is not a
clean pass, and such fixtures are excluded from the mean rather than averaged
in, or the aggregate would rise as fixtures *without* the check were added.

`fixtures/synthetic/fabricated_location.json` makes #584 reproducible: the
party discuss Ironroot Hold from the road and take the other fork, and a beat
says they went in, citing lines that really do mention it and really were
spoken by the named player. It scores 1.000 on coverage, attribution,
chronology and beat validation, and 0.500 on precision. Before this there was
no scorer that could fail it, so its expected block would have been
indistinguishable from clean_run.json's.

The existing `expected.absent[]` hook (#596) stays for phrases asserted
directly by the test; `forbidden[]` is for claims about the session, because
it produces a reported, thresholdable number instead of only failing a test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

claude-bot merged commit 7a1afb0c5a into main

2026-09-12 21:57:13 +00:00

claude-bot deleted branch feat/586-precision-metric

2026-09-12 21:57:14 +00:00

claude-bot referenced this pull request from a commit

2026-09-12 21:57:15 +00:00

Merge pull request 'feat(evals): score what a summary should not say (#586)' (#633) from feat/586-precision-metric into main