Richer session events, and three checks that stop dropping real ones (#1189) #1192
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/1189-v4-prompt"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #1189. This is the owner's choice after the private Session 14 comparison: the v4 prompt with the looser checks.
Prompt
EXTRACT_SYSTEM_PROMPTnow asks for one or two sentences: who did what, to whom or with what, and how it turned out. Every name and detail must come from the cited lines, the example has a single actor, people the action was done to stay out ofactors, and the outcome line is cited as well. The reasons for each clause are in the code comment.Validation (
beat_service.validate_beats)[start-end]line, or at its end, resolves to that line and is recorded as a repair. Before, it was rejected as "not in the transcript".CREDIT_ACTOR_INSIDE_CITED_SPAN, now on). An actor is credited from their own line when it sits inside a two-ended cited exchange of at most 120 s with nobody but the GM and the beat's own actors in it.GM_NEIGHBOUR_SECONDS = 5, on). An event crediting the GM that cites only someone else's line is kept when the GM's line is within 5 s before or after that line, with nobody in between. This applies to the GM only, never a player.Measured
On the synthetic fixtures, compared with the current prompt: beat validation 0.840 → 0.928 and coverage 0.751 → 0.895 with v4 and the looser check, and attribution 0.515 → 0.798.
On the real Session 14 (read-only), v4 had about 60 story events against 36. (a) and (b) recovered 3 GM events and gave none to the wrong person. The look-forward adds two more of the remaining shapes. The eval history rows are in
evals/results/history.jsonl.The eval harness also gains
--detail, which reports how specific the events are and where their names come from.Tests
On docker-test, from a git archive:
test_beat_service,test_summarise_*,test_evals_*, and others).changelog.py checkand the version sync are OK.New tests cover the look-forward in both directions: credited, someone in between, outside the window, and never for a player. They also cover the GM line that starts inside a cited player range.
No migration and no bot contract change.
🤖 Generated with Claude Code
The richer-beat prompt v4 cites the GM's set-up and outcome lines and not the player's own line between them ("Astrid split the ooze" citing the GM's 00:01:05 and 00:01:58, Astrid's swing at 00:01:22 uncited), so the beat is rejected for "did not speak any cited line" although actor and exchange are right. beat_service.CREDIT_ACTOR_INSIDE_CITED_SPAN (default False, so nothing changes in production) credits such an actor when the beat cites at least two real lines no more than CITED_SPAN_MAX_SECONDS (120) apart, the actor spoke between them, and nobody but the GM and the beat's own actors spoke between them. The actor's lines are added to the evidence and the credit is recorded as a repair. No window around the citations: the line after a cited GM outcome is where a bystander's reaction sits (Fenwick's "Nice work!"). Live, Gemma 4 26B (alias qwen3.5), 6 synthetic fixtures + Halloway Reach, v4 prompt applied for the runs only (not committed), switch on, three runs, recorded as 9b10a86+1189-v4-loose; current prompt re-run once as9b10a86: v4-loose: coverage 0.895 [0.857-0.914], attribution 0.798 [0.774-0.838], chronology 0.849 [0.786-0.881], beat validation 0.928 [0.923-0.940], precision 1.000, words/beat 15.9, invented names 0, verified 59.7 [56-67] current recheck: 0.736 / 0.476 / 0.810 / 0.840 / 1.000, 11.1 words, 74 Rescued 8 beats a run (the ooze kill and the silver split, on the five clean_run-transcript fixtures), every one attributed correctly; still rejected 8.3 a run, all citing a single line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>