[Backend] Add a golden-transcript accuracy eval harness #349
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Severity: MEDIUM. Found in the August 2026 session lifecycle review (#319).
Why
The capture bug survived for months because nothing measured whether the pipeline was any good. Every check in the system verified that a step completed, not that its output was correct. With a provider abstraction landing in v4.2.0 and models being swapped in and out, that gap becomes permanent unless something closes it.
Proposed fix
Build a small fixture corpus: a handful of hand-checked transcripts with known ground truth — who said what, in what order, and what actually happened. Synthetic is fine and avoids using real player data; one real session with permission would be better for realism.
Then a harness that runs the full pipeline against them and scores:
Run it in CI against a stub provider for regressions, and on demand against a real provider when swapping models. Record the numbers over time so "we changed the prompt and it got better" becomes a measurable claim.
Acceptance criteria
Done in
65e929b, with a production fix it found in3bf85b3.Acceptance
1,182 passing before → 1,202.
On the real-data question
The GM consented to their own campaign being used, so the corpus has one real fixture: the first session recorded after the #320 clock fix, 1,529 lines over 77 minutes, eleven events hand-verified against the transcript line that supports each.
It is not in git.
evals/fixtures/private/is a self-ignoring directory (*plus!.gitignore), because consent to be recorded is not consent to have five other people's verbatim speech committed into a repository that underpins a hosted product and runs through CI. The committed corpus is synthetic;history.jsonlrecords scores only, never content.git add --dry-runover the tree confirms only.gitignoreis ever stageable from there.Two ground-truth claims needed checking rather than taking from the approved summary. The summary credits Viquilonto with tripping the mimic and Viq never says "trip" — he strikes at 00:18:10 and asks to "prone this fool" at 00:18:54, which is a Trip. And it bundles Harrowhark's demoralize with the Chilling Darkness kill though a demoralize also happened at 00:05:31; there were several attempts, so that is compression, not error. Both verified; the summary is sound.
The synthetic fixtures are adversarial
Three of four are deliberately wrong — scrambled order, a misattributed actor, a hallucinated citation — and CI asserts the scorers catch each. A scorer that silently returned 1.0 would pass a corpus of correct fixtures, and scrambled chronology and misattribution are the two failure modes that opened this milestone. Scores are 1.000/1.000/1.000 for the clean run, tau −1.000 for the scrambled one, attribution 0.800 for the misattributed one.
Chronology returns
None, not 1.0, when fewer than two events are covered. A harness reporting a perfect score for having measured nothing is worse than one that does not run.What it found
A production bug on its first real run (
3bf85b3).extract_beatsnever passedmax_tokens, so it inheritedgenerate_structured_text's 2048 default while nearly every other structured call passesNone. It is the one call whose response length scales with its input. Window size comes from the provider's context, so a 131k-context model puts a whole 77-minute session in one window and asks for every beat in it — which does not fit in 2048 tokens, so_reject_if_truncatedthrew the entire extraction away and the pipeline fell back to prose, the path where chronology and attribution stop being checked. Beat extraction failed hardest on the largest context windows. No existing test could have caught it: they all either stub the LLM or use a transcript short enough to fit.Then the real finding. Three consecutive runs, same session, same model (qwen3.5-9B at 131k), same prompt:
Coverage varies 8× across identical inputs — 1 of 11 events to 8 of 11. Filed as #423 with what I ruled out (not the cap, not reasoning tokens, not windowing, not the validator) and where I would look first (sampling temperature; structured extraction against a fixed schema gains nothing from sampling diversity).
I nearly reported the first run's numbers on their own. They would have been a real-looking measurement of something that isn't true, from n=1 — which is the same class of error as the bug this milestone exists to fix, so it is worth naming.
Notes
QB_EVAL_ENDPOINTis a base URL; the client appends/v1/chat/completions. Passing the full path gives a 404, which cost a run.QB_EVAL_GIT_SHAoverrides SHA detection. The normal way to run this is a container with the repo bind-mounted and nogitbinary, which recorded every run asunknown— and a score you cannot attach to a commit cannot answer the question the history file exists to answer.Closing.