[Backend] Add a golden-transcript accuracy eval harness

claude-bot commented

2026-08-25 20:39:05 +00:00

Collaborator

Severity: MEDIUM. Found in the August 2026 session lifecycle review (#319).

Why

The capture bug survived for months because nothing measured whether the pipeline was any good. Every check in the system verified that a step completed, not that its output was correct. With a provider abstraction landing in v4.2.0 and models being swapped in and out, that gap becomes permanent unless something closes it.

Proposed fix

Build a small fixture corpus: a handful of hand-checked transcripts with known ground truth — who said what, in what order, and what actually happened. Synthetic is fine and avoids using real player data; one real session with permission would be better for realism.

Then a harness that runs the full pipeline against them and scores:

Chronology — are events in the summary in the true order?
Attribution — is each action credited to the right actor?
Coverage — what fraction of significant events survive into the summary?
Beat validation rate — what fraction of emitted beats pass the code validator? This is the cheapest ongoing model-quality signal, and it comes free once the validator exists.

Run it in CI against a stub provider for regressions, and on demand against a real provider when swapping models. Record the numbers over time so "we changed the prompt and it got better" becomes a measurable claim.

Acceptance criteria

A fixture corpus with ground truth exists, containing no real player data unless explicitly consented
The harness scores chronology, attribution, coverage, and beat validation rate
CI runs it against a deterministic stub provider
A documented command runs it against a configured live provider
Results are recorded over time, not just pass/fail

**Severity: MEDIUM.** Found in the August 2026 session lifecycle review (#319). ## Why The capture bug survived for months because **nothing measured whether the pipeline was any good**. Every check in the system verified that a step completed, not that its output was correct. With a provider abstraction landing in v4.2.0 and models being swapped in and out, that gap becomes permanent unless something closes it. ## Proposed fix Build a small fixture corpus: a handful of hand-checked transcripts with known ground truth — who said what, in what order, and what actually happened. Synthetic is fine and avoids using real player data; one real session with permission would be better for realism. Then a harness that runs the full pipeline against them and scores: - **Chronology** — are events in the summary in the true order? - **Attribution** — is each action credited to the right actor? - **Coverage** — what fraction of significant events survive into the summary? - **Beat validation rate** — what fraction of emitted beats pass the code validator? This is the cheapest ongoing model-quality signal, and it comes free once the validator exists. Run it in CI against a stub provider for regressions, and on demand against a real provider when swapping models. Record the numbers over time so "we changed the prompt and it got better" becomes a measurable claim. ## Acceptance criteria - [ ] A fixture corpus with ground truth exists, containing no real player data unless explicitly consented - [ ] The harness scores chronology, attribution, coverage, and beat validation rate - [ ] CI runs it against a deterministic stub provider - [ ] A documented command runs it against a configured live provider - [ ] Results are recorded over time, not just pass/fail

claude-bot added this to the v4.0.0 - Session Pipeline Accuracy milestone

2026-08-25 20:39:05 +00:00

claude-bot added the

enhancement

backend

labels

2026-08-25 20:39:05 +00:00

claude-bot referenced this issue

2026-08-26 14:53:01 +00:00

Spike: does the LLM endpoint honour constrained decoding (json_schema / grammar)? #281

rbrooks referenced this issue from a commit

2026-08-26 18:28:03 +00:00

fix(backend): beat extraction was capped at 2048 output tokens (#349)

rbrooks referenced this issue from a commit

2026-08-26 18:28:03 +00:00

feat(backend): a golden-transcript eval harness, and what it found (#349)

claude-bot referenced this issue

2026-08-26 18:28:22 +00:00

[Backend] Summary quality varies 8x run-to-run on identical input #423

claude-bot commented

2026-08-26 18:28:51 +00:00

Author

Collaborator

Done in 65e929b, with a production fix it found in 3bf85b3.

Acceptance

A fixture corpus with ground truth exists, containing no real player data unless explicitly consented
The harness scores chronology, attribution, coverage, and beat validation rate
CI runs it against a deterministic stub provider
A documented command runs it against a configured live provider
Results are recorded over time, not just pass/fail

1,182 passing before → 1,202.

On the real-data question

The GM consented to their own campaign being used, so the corpus has one real fixture: the first session recorded after the #320 clock fix, 1,529 lines over 77 minutes, eleven events hand-verified against the transcript line that supports each.

It is not in git. evals/fixtures/private/ is a self-ignoring directory (* plus !.gitignore), because consent to be recorded is not consent to have five other people's verbatim speech committed into a repository that underpins a hosted product and runs through CI. The committed corpus is synthetic; history.jsonl records scores only, never content. git add --dry-run over the tree confirms only .gitignore is ever stageable from there.

Two ground-truth claims needed checking rather than taking from the approved summary. The summary credits Viquilonto with tripping the mimic and Viq never says "trip" — he strikes at 00:18:10 and asks to "prone this fool" at 00:18:54, which is a Trip. And it bundles Harrowhark's demoralize with the Chilling Darkness kill though a demoralize also happened at 00:05:31; there were several attempts, so that is compression, not error. Both verified; the summary is sound.

The synthetic fixtures are adversarial

Three of four are deliberately wrong — scrambled order, a misattributed actor, a hallucinated citation — and CI asserts the scorers catch each. A scorer that silently returned 1.0 would pass a corpus of correct fixtures, and scrambled chronology and misattribution are the two failure modes that opened this milestone. Scores are 1.000/1.000/1.000 for the clean run, tau −1.000 for the scrambled one, attribution 0.800 for the misattributed one.

Chronology returns None, not 1.0, when fewer than two events are covered. A harness reporting a perfect score for having measured nothing is worse than one that does not run.

What it found

A production bug on its first real run (3bf85b3). extract_beats never passed max_tokens, so it inherited generate_structured_text's 2048 default while nearly every other structured call passes None. It is the one call whose response length scales with its input. Window size comes from the provider's context, so a 131k-context model puts a whole 77-minute session in one window and asks for every beat in it — which does not fit in 2048 tokens, so _reject_if_truncated threw the entire extraction away and the pipeline fell back to prose, the path where chronology and attribution stop being checked. Beat extraction failed hardest on the largest context windows. No existing test could have caught it: they all either stub the LLM or use a transcript short enough to fit.

Then the real finding. Three consecutive runs, same session, same model (qwen3.5-9B at 131k), same prompt:

run	coverage	attribution	chronology (tau)	beat validation
1	0.091	0.000	not measurable	0.240
2	0.727	0.375	0.857	0.824
3	0.545	0.667	1.000	0.765

Coverage varies 8× across identical inputs — 1 of 11 events to 8 of 11. Filed as #423 with what I ruled out (not the cap, not reasoning tokens, not windowing, not the validator) and where I would look first (sampling temperature; structured extraction against a fixed schema gains nothing from sampling diversity).

I nearly reported the first run's numbers on their own. They would have been a real-looking measurement of something that isn't true, from n=1 — which is the same class of error as the bug this milestone exists to fix, so it is worth naming.

Notes

The stub scores the validated beat rendering rather than a composed summary, since compose needs a live model. Stated plainly in the README rather than papered over — CI's honest job is regression-guarding the harness and its scorers, not measuring model quality.
QB_EVAL_ENDPOINT is a base URL; the client appends /v1/chat/completions. Passing the full path gives a 404, which cost a run.
QB_EVAL_GIT_SHA overrides SHA detection. The normal way to run this is a container with the repo bind-mounted and no git binary, which recorded every run as unknown — and a score you cannot attach to a commit cannot answer the question the history file exists to answer.

Closing.

Done in `65e929b`, with a production fix it found in `3bf85b3`. ## Acceptance - [x] A fixture corpus with ground truth exists, containing no real player data unless explicitly consented - [x] The harness scores chronology, attribution, coverage, and beat validation rate - [x] CI runs it against a deterministic stub provider - [x] A documented command runs it against a configured live provider - [x] Results are recorded over time, not just pass/fail 1,182 passing before → **1,202**. ## On the real-data question The GM consented to their own campaign being used, so the corpus has one real fixture: the first session recorded after the #320 clock fix, 1,529 lines over 77 minutes, eleven events hand-verified against the transcript line that supports each. It is **not in git**. `evals/fixtures/private/` is a self-ignoring directory (`*` plus `!.gitignore`), because consent to be *recorded* is not consent to have five other people's verbatim speech committed into a repository that underpins a hosted product and runs through CI. The committed corpus is synthetic; `history.jsonl` records scores only, never content. `git add --dry-run` over the tree confirms only `.gitignore` is ever stageable from there. Two ground-truth claims needed checking rather than taking from the approved summary. The summary credits Viquilonto with tripping the mimic and Viq never says "trip" — he strikes at 00:18:10 and asks to "prone this fool" at 00:18:54, which is a Trip. And it bundles Harrowhark's demoralize with the Chilling Darkness kill though a demoralize also happened at 00:05:31; there were several attempts, so that is compression, not error. Both verified; the summary is sound. ## The synthetic fixtures are adversarial Three of four are deliberately **wrong** — scrambled order, a misattributed actor, a hallucinated citation — and CI asserts the scorers catch each. A scorer that silently returned 1.0 would pass a corpus of correct fixtures, and scrambled chronology and misattribution are the two failure modes that opened this milestone. Scores are 1.000/1.000/1.000 for the clean run, tau −1.000 for the scrambled one, attribution 0.800 for the misattributed one. Chronology returns `None`, not 1.0, when fewer than two events are covered. A harness reporting a perfect score for having measured nothing is worse than one that does not run. ## What it found **A production bug on its first real run** (`3bf85b3`). `extract_beats` never passed `max_tokens`, so it inherited `generate_structured_text`'s 2048 default while nearly every other structured call passes `None`. It is the one call whose response length scales with its *input*. Window size comes from the provider's context, so a 131k-context model puts a whole 77-minute session in one window and asks for every beat in it — which does not fit in 2048 tokens, so `_reject_if_truncated` threw the entire extraction away and the pipeline fell back to prose, the path where chronology and attribution stop being checked. **Beat extraction failed hardest on the largest context windows.** No existing test could have caught it: they all either stub the LLM or use a transcript short enough to fit. **Then the real finding.** Three consecutive runs, same session, same model (qwen3.5-9B at 131k), same prompt: | run | coverage | attribution | chronology (tau) | beat validation | |---|---|---|---|---| | 1 | 0.091 | 0.000 | not measurable | 0.240 | | 2 | 0.727 | 0.375 | 0.857 | 0.824 | | 3 | 0.545 | 0.667 | 1.000 | 0.765 | Coverage varies **8×** across identical inputs — 1 of 11 events to 8 of 11. Filed as #423 with what I ruled out (not the cap, not reasoning tokens, not windowing, not the validator) and where I would look first (sampling temperature; structured extraction against a fixed schema gains nothing from sampling diversity). I nearly reported the first run's numbers on their own. They would have been a real-looking measurement of something that isn't true, from n=1 — which is the same class of error as the bug this milestone exists to fix, so it is worth naming. ## Notes - The stub scores the validated beat rendering rather than a composed summary, since compose needs a live model. Stated plainly in the README rather than papered over — CI's honest job is regression-guarding the harness and its scorers, not measuring model quality. - `QB_EVAL_ENDPOINT` is a **base URL**; the client appends `/v1/chat/completions`. Passing the full path gives a 404, which cost a run. - `QB_EVAL_GIT_SHA` overrides SHA detection. The normal way to run this is a container with the repo bind-mounted and no `git` binary, which recorded every run as `unknown` — and a score you cannot attach to a commit cannot answer the question the history file exists to answer. Closing.

claude-bot closed this issue

2026-08-26 18:28:54 +00:00

rbrooks referenced this issue from a commit

2026-08-26 18:51:18 +00:00

fix(backend): stop sampling at random during structured extraction (#423)

rbrooks referenced this issue from a commit

2026-08-26 20:15:32 +00:00

fix(backend): stop compose silently discarding verified events (#423)

rbrooks referenced this issue from a commit

2026-08-27 18:30:06 +00:00

perf(backend): set the extraction window from a sweep, not a guess (#423)

claude-bot referenced this issue