[Backend] Validate beats in code before any prose is written #333
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Severity: CRITICAL. Found in the August 2026 session lifecycle review (#319). This is the issue that makes accuracy independent of model size.
Why
The two reported failure modes — scrambled chronology and misattributed actions — are both currently trusted to the model. This makes them checked.
Crucially, the check is pure code and free. It runs the same way on a 9B local model and on a frontier hosted one. Given that self-hosted is a first-class target, this is the only design whose correctness floor does not ride on model size.
Proposed fix
After the map phase, before any prose is generated:
evidencetimestamp must correspond to a real transcript line.narratedand the actor must appear in the text of a cited GM line.t_start <= t_end, both inside the session, and consistent with the cited evidence.sorted()guarantee rather than a model behaviour. This is the entire point.Beats that fail validation are flagged, not silently dropped — surface them to the GM as "unverified" so a real event with a bad citation is recoverable rather than invisible. Silent dropping would trade one invisible failure for another.
Acceptance criteria
sorted(), not by the modelVerified against acceptance criteria, and five defects fixed in
152048e. This is the issue whose stated purpose is that the accuracy floor "does not ride on model size" because the check is pure code — so a check that does not run matters more here than anywhere else in the pipeline.Criteria
validate_beatsis synchronous and makes no request.t_start <= t_endwas repaired in_coerce_beat, never recorded._coerce_beatclampedt_starttotranscript_seconds, thenvalidate_beatstestedbeat.t_start > transcript_seconds."starts after the session ends"could not be appended under any input.sorted()— and the sort key is now derived from verified data rather than a field nothing checked.used_beatswas computed every run and never persisted, so the runs that skipped validation were the ones leaving no trace. Now every run gets a row.The test that could not fail
test_a_beat_starting_after_the_session_ends_is_flaggedasserted onlyresult.beat.t_start <= max(index), which the clamp guarantees unconditionally. It never touchedresult.okorresult.problems. Its own comment hedged — "the range problem may resolve" — which reads like the author noticing and writing a weaker assertion rather than chasing it down.Why the evidence check re-anchors instead of rejecting
My first implementation flagged a beat whose range contained none of its citations. The
scrambled_chronologyeval fixture immediately showed that was wrong: all five of its beats failed, so five real, correctly-attributed events dropped out of the summary. That fixture exists precisely to demonstrate this gap — its notes say so, and its expectedchronology_tau: -1.0encoded the defect.The evidence has already been checked against the transcript; the declared range has been checked by nothing. When they disagree, the verified value wins. So the range is re-anchored onto the citations, which makes the chronology guarantee stronger than it was — the sort key now comes from transcript-verified timestamps rather than a model-supplied field.
That fixture now scores
tau = +1.0with coverage, attribution and validation all 1.0. Its expectations and notes are rewritten to record the fix rather than the gap; the five reversed timestamps still make it hand-checkable in both directions.A hallucinated citation is never used as an anchor — that would trade a wrong time the validator reported for a wrong time it invented.
BeatValidation.repairscarries corrections, separate fromproblemsand not affectingok: a beat the code could fix did not fail. Persisted tosession_beats.repairsand exposed through #424's API, because a silent correction is indistinguishable from a correct answer.dedupe_beats preferred hallucinations
It kept whichever duplicate had more citations, with no reference to the verdict — and
gather_validated_beatsvalidates before merging, so both sides already carried one. A second-pass beat citing the same real line plus one invented stamp beat a clean single-citation beat.render_beats_for_composeskips failing beats, so the event then vanished from the summary having been both correctly extracted and correctly checked. Verified beats now win outright.The loop's docstring described a filter that did not exist
"'Did this pass find anything new' has to mean anything new that survives checking" sat three lines above
gained = len(merged) - len(validations), which counted flagged beats. A pass of fresh hallucinations looked like progress and ran to the cap — a full re-extraction over every window, paid for nothing.test_a_pass_of_pure_hallucination_does_not_keep_the_loop_runningpassed for the wrong reason: its invented beat was identical every pass, sodedupe_beatscollapsed it. It would have gone on passing if the.okfilter it is named for were deleted — which it was, because it never existed. It now varies actor and stamp per pass, and fails against the old code.Verified by mutation, not by reading
Reverting
gainedand the dedupe rule fails exactly the three tests named above and nothing else._coerce_beat's silent discards are now counted and logged.1,306 → 1,324 passing. Migration
f7a8b0c1d2e3, applied and rolled back against a real database. Closing.