[Frontend] Show a GM the verified events behind a summary #424
Closed
opened 2026-08-27 21:45:04 +00:00 by claude-bot
·
6 comments
No Branch/Tag specified
main
feat/1328-fill-it-in
feat/1328-fill-frontend
feat/1328-fill-backend
feat/9-foundry-push
fix/9-foundry-live
feat/9-foundry-push-frontend
renovate/celery-5.x
feat/5-foundry-connect
feat/9-foundry-push-backend
feat/9-foundry-push-module
feat/5-foundry-frontend
feat/5-foundry-backend
feat/5-foundry-module
release/v4.8.0-rc1
docs/d341-convert-draft
feat/958-update-to-version
feat/958-convert-sheet
feat/958-pathbuilder-import-frontend
fix/1360-heading-order
feat/958-pathbuilder-import-backend
feat/11-weekly-digest
feat/1341-auto-update
docs/d332-superseded
chore/1345-names-cleanup
feat/1342-mentioned-pages
feat/1328-recipes
fix/1346-pgdg-key
perf/1247-shared-recording-stream
chore/132-gitleaks-allowlist
release/v4.7.1
spike/foundry-module
release/v4.7.1-rc1
docs/decisions-d332-d337
fix/bot-certifi
chore/release-4.7.0
chore/release-4.7.0-rc4
chore/deps-4.7.0
chore/release-4.7.0-rc3
fix/1280-quote-must-say-it
fix/1257-names-row-squeeze
chore/release-4.7.0-rc2
fix/1245-pg-upgrade-count
chore/release-4.7.0-rc1
feat/1257-names-in-prep
feat/1257-be
feat/1257-fe
feat/1257-rating-fixes
design/1328-fill-it-in
feat/1257-name-splits
fix/1326-redis-503
feat/1280-fact-suggestions
feat/1280-be
feat/1280-fe
design/1280-fact-suggestions
feat/1279-infobox
feat/1279-be
feat/1279-fe
feat/1321-shelf-copies
design/1279-infobox
feat/1243-table-find
feat/1257-who-rated
feat/1243-prep-drawer
feat/1243-sheet-pickers
design/1243-browse-in-place
feat/1315-hotwords-default
feat/1242-system-filters
feat/1257-more-name-styles
feat/506-leg3-hotwords
design/1242-system-filters
fix/1302-packs-polish
feat/1241-rules-library
design/1241-rules-library
feat/1245-postgres-18
integ/4.6.2
fix/1263-map-ui-2
fix/1263-map-server
fix/1263-map-ui-1
fix/1261-editor-link-menu
fix/1229-axe-4.14-name-mismatch
renovate/axllent-mailpit-1.x
renovate/eslint-monorepo
renovate/authlib-1.x
chore/release-4.6.0
chore/release-4.6.0-rc2
fix/460-rc2-screens
fix/460-rc2-server
fix/release-prerelease-exit
chore/release-4.6.0-rc1
feat/125-campaign-templates
feat/126-calendar
feat/1181-pictures-and-files
feat/1190-event-people
integ/4.6.0
feat/1191-line-of-the-night
feat/1200-starter-packs
chore/bot-multidict-and-playwright-group
fix/1202-this-week-answered-count
fix/122-124-leftovers
fix/1178-get-routes-that-write
fix/1203-journeys-check-names
design/125-campaign-templates
feat/126-calendar-engine
design/126-in-world-calendar
fix/1189-cited-stamp-in-range
fix/1189-v4-looser-check
fix/1189-richer-beats
fix/1113-export-bind-lists
fix/1154-release-notes-by-file
chore/release-4.5.0
chore/release-4.5.0-rc12
fix/1135-wiki-markup-as-text
fix/demo-usage-cost-in-window
chore/release-4.5.0-rc11
fix/1121-sheet-consistency
fix/1121-sheet-5e-terms-and-card-hp
copy/1122-busiest-pages
fix/1121-relevant-choices
fix/1121-wiki-create-npc-save-sheet-links
fix/1029-menu-stays-on-screen
docs/prototype-transcript-bin-wording
fix/1023-editor-save-row-back-scroll
fix/1020-search-exact-first
fix/1020-table-prep
fix/1020-shell-wordmark-footer-bell
fix/1020-wiki-article-party
fix/872-admin-kept-until-phone-tables
feat/l13-milestone-level-waiting
fix/1000-1001-bell-older-setup-context
fix/999-pronouns-in-prompts
fix/l19-other-class-feats
fix/1010-transcript-window-ordinary-confirm
docs/d254-d269-decisions
fix/702-member-search
fix/680-transcript-bin
fix/993-unidentified-prompts
fix/841-redis-timeouts
fix/990-binned-not-exported
fix/830-export-prep-shelf
fix/deps-audit-2026-10-01
fix/877-title-reads
fix/798-audio-received-at
build/lane11-17-j7
build/lane11-16-levelup
build/lane11-14-seams
build/lane11-14-seed
build/lane11-15-start
build/lane11-13-old-block-out
fix/980-review-r3
build/lane11-12-block
build/lane11-8-packs
fix/demo-seed-canonical-names
build/lane11-11-hot-half
docs/l11-decisions
build/lane11-7-convert
build/lane11-10-editing
fix/npm-audit-brace-expansion
build/lane11-9-sheet-page
fix/bot-pyjwt
build/lane11-6-content
fix/lane11-rank-names
build/lane11-2b-language
build/lane11-4-js-evaluator
build/lane11-3-server
build/lane11-5a-pf2e
build/lane11-5b-5e
feat/design-admin
build/lane11-2-evaluator
build/lane11-1-sheet-pieces
build/lane11-0-drawing
fix/943-quest-objectives-fk
fix/prepsheet-shelf-flake
fix/945-leftovers
build/demo-gm-copy
fix/901-email-landings
build/lane10-8-journeys
build/demo-seed-lanes-8-10
fix/887-gm-own-confirmation
build/lane9-5-j1-admin
fix/918-audience-click
fix/useasync-overtaken
ci/939-critical-path
test/937-backend-suite-speed
chore/claude-orchestration
fix/924-xp-gm-only
fix/session-stale-poll
docs/decisions-d239-d242
fix/930-pypdf
build/lane8-journeys
build/925-quest-editor
fix/894-private-page-pictures
build/lane8-p4-drafts
build/lane8-p7-map
build/lane8-p6-handouts-rules
fix/wiki-fixture-names
build/lane10-d227-test-usage
fix/rsvp-horizon-test-tuesday
fix/media-volume-owner
build/lane8-p3-editor
fix/912-unidentified-rename
build/lane8-p5-party-quests
build/lane8-handout-takeback
build/lane10-pickers
build/lane8-p2-article
build/lane8-6-map
build/lane10-7b-email-pages
build/lane8-5-handouts
build/lane10-7a-test-the-bot-pages
build/lane8-p1-pages
build/lane8-4-quests
build/lane8-2b-feeds
build/lane8-3-parties
build/lane8-2a-links
build/lane10-5-commands
build/lane9-6-seed
build/lane10-4b-reactions
build/lane9-4d-pages
build/lane9-4c-pages
build/lane10-4a-posts
build/lane10-3-email
fix/891-web-fonts
build/lane9-4b-pages
build/lane10-6-test-the-bot
build/lane10-2-notify-out
build/lane8-1-wiki-text
build/lane9-4a-frame
design/lane10-drawings
fix/j6-budget
fix/j5-phone-flake
fix/869-test-leaks
fix/843-record-start-followups
build/lane7-all
build/lane6-all
build/lane9-b2-server
chore/changelog-fragments
docs/decisions-d218
build/lane9-a-owner-overview
fix/rule-card-level-label
fix/834-retention-semantics
fix/o3c-record-without-announcement
docs/decisions-d179
docs/decisions-d131-d170
fix/851-commit-before-response
build/lane6-j3-seed
build/lane9-b1-d67-identity
fix/841-blocking-loop
build/lane9-c-backups-audit
fix/vote-race-and-start-commit
fix/recording-announcement
fix/proposal-read-leak
fix/last-admin-guard
build/lane6-p4c-planning
build/lane6-p4b-margin
build/lane6-p4a-sheet
build/lane5-j4-seed
build/lane6-base
build/lane7-pages-talk
build/lane5-pages
build/lane7-pages-safety
build/lane7-p3-safety
build/lane7-p2-notify
build/lane6-p3-planning
build/lane6-p2-history
build/lane6-p1b-items
build/lane7-p1-threads
build/lane5-p6-faces
build/lane6-p1a-sheet
build/lane5-p5-find
build/lane5-p4a-board
build/lane5-p4b-scenes
build/lane5-p3-live
build/lane5-base
build/lane5-p2-cards
docs/lane5-decisions
build/review-r2-wide
fix/review-r1
build/lane5-p1-scenes
build/demo-seed
build/lane4-journeys-j5
build/lane4-page-review-b
build/lane4-page-review-a
build/lane4-page-read
build/lane4-journeys
build/lane4-page-sess2
build/lane4-page-sess1
build/lane4-notes
build/lane4-discord-on-approval
build/lane4-scheduling
build/lane4-versions
build/760-session-number
build/lane4-decisions
build/797-erase-no-regenerate
build/test-race-audit
build/lane3-completions
build/794-dark-only
build/796-titles-scroll
build/791-retention-anchor
build/a11y-sweep
build/shell-followups
build/retention-preview
build/753-backup-readback
docs/d91-d93
build/lane3-j1
build/ci-harness-paths-selftest
fix/useauth-test-race
build/ci-harness-paths
build/777-ai-settings-read
build/lane3-settings-c
build/lane3-settings-b
build/lane3-j6
build/lane3-sessionform
build/lane3-settings-a
build/lane3-members
build/lane3-thisweek
build/lane3-story
chore/87-align-python-3.12
v4.8.0-rc1
v4.7.1
v4.7.1-rc1
v4.7.0
v4.7.0-rc4
v4.7.0-rc3
v4.7.0-rc2
v4.7.0-rc1
v4.6.2
v4.6.2-rc4
v4.6.2-rc3
v4.6.2-rc2
v4.6.2-rc1
v4.6.1
v4.6.1-rc1
v4.6.0
v4.6.0-rc2
v4.6.0-rc1
v4.5.3
v4.5.3-rc1
v4.5.2
v4.5.2-rc1
v4.5.1
v4.5.1-rc2
v4.5.1-rc1
v4.5.0
v4.5.0-rc12
v4.5.0-rc11
v4.5.0-rc10
v4.5.0-rc9
v4.5.0-rc8
v4.5.0-rc7
v4.5.0-rc6
v4.5.0-rc5
v4.5.0-rc4
v4.5.0-rc3
v4.5.0-rc2
v4.5.0-rc1
v4.4.1
v4.4.0
v4.3.0
v4.2.3
v4.2.2
v4.2.1
v4.2.0
v4.1.2
v4.1.1
v4.1.0
v4.0.1
v4.0.0
v3.11.4
v3.11.3
v3.11.2
v3.11.1
v3.11.0
v3.10.0
v3.9.1
v3.9.0
v3.8.0
v3.7.0
v3.6.1
v3.6.0
v3.5.0
v3.4.1
v3.4.0
v3.3.0
v3.2.0
v3.1.0
v3.0.0
v2.1.0
v2.0.0
v1.1.0
v1.0.0
v0.12.0
v0.11.0
v0.10.0
v0.9.1
v0.3.3
v0.9.0
v0.8.0
v0.3.2
v0.3.1
v0.3.0
v0.7.0
v0.6.0
v0.5.0
v0.4.0
v0.2.1
v0.2.0
v0.1.1
v0.1.0
Labels
Clear labels
backend
Issue related to the backend
bot
Issue related to the bot
bug
Something is not working
duplicate
This issue or pull request already exists
enhancement
New feature
frontend
Issue related to the frontend
help wanted
Need some help
invalid
Something is wrong
question
More information is needed
recording
Issue related to the recording system
security
Security hardening / vulnerability work
wontfix
This won't be fixed
No labels
backend
bot
bug
duplicate
enhancement
frontend
help wanted
invalid
question
recording
security
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set
Reference
rbrooks/Quest-Board#424
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The last open criterion from #423, reframed. That issue closed with the coverage floor raised from 0.091 to 0.727, but one criterion unmet: "a run that covers drastically fewer events than expected is visible to the GM rather than silent."
Why this is a UI issue and not a detector
I tried twice in #423 to detect a lossy run in code, comparing the composed summary against the beats it was built from. Both failed on real data, in opposite directions:
Across six live runs the detector flagged nothing while the eval harness measured 1–2 verified events per run missing from the prose. The harness only manages it because its fixtures carry hand-authored per-event mention groups; production has only the beat's own summary text, which is a much weaker signal.
beat_service.unrepresented_beatsstill exists and is correct in clear-cut cases, but it is not a foundation to build a warning on.The reframing: the GM does not need a machine to tell them something is missing. They were at the session. Show them the verified events beside the summary and they will see a gap in seconds — and, unlike a heuristic, they will also spot the summary that covers everything but characterises it wrongly.
What already exists
beat_service.validate_beatsproduces aBeatValidationper beat, carrying.okand, when not ok,.problemsnaming why ('Kira' did not speak any cited line,cites 00:41:12, which is not in the transcript).t_start/t_end,actors,kind, andevidence— cited transcript timestamps, already validated against the transcript.render_beats_for_composerenders them in time order.What does not exist, and is the real work
Beats are never persisted. They live in
_summarise_from_beatsfor the duration of one Celery task and are gone. There is no model, no table, no API. Showing them means storing them, which is a migration plus a router, and is most of this issue.Worth deciding at design time whether they are stored per summarisation run (so a reprocess produces a second set and the GM can compare) or last-write-wins. #423 established that two runs over the same audio genuinely differ, which makes the comparison view more interesting than it sounds.
Acceptance criteria
Notes
unrepresented_beats. It is retained for the retry path, where being occasionally wrong is cheap; it is not accurate enough to drive anything a GM is shown.webapp/backend/evals/results/history.jsonland reproducible withpython -m evals --fixtures evals/fixtures/private --provider live --repeat 4.Picking up the backend half now, in parallel with the v4.4.0 design phases (#535 phase 4); the screen that shows the events beside the summary comes in the session lane of the rebuild, once the design direction is chosen.
Decisions, so the screen does not decide them by accident:
GET /api/sessions/{id}/beats(latest run, or?run_id=) andGET /api/sessions/{id}/beats/runs.unrepresented_beats, as the body says.One migration, chained on the v4.3.0 head.
Correction: the backend half already shipped in v4.0.0. The body's premise ("beats are never persisted") was true when it was filed and is not true now. Two commits landed it:
2791c24(the rows:summarisation_runs+session_beats, migrationd5e6f7a8b0c1, extended byf7a8b0c1d2e3for provenance) andaa4bff9("expose the events behind a summary over the API (#333, #424)"), whose message already says this issue becomes a frontend-only job.What exists on main today, for whoever builds the screen:
GET /api/sessions/{id}/summarisation-runs— every run, newest first, each with its beats inline (so an older run is selected from this list; there is no?run_id=).GET /api/sessions/{id}/summarisation-runs/latest— 404 when the session has no run; the list returns[]instead, deliberately, because sessions summarised before v4.0.0 have no record and none can be reconstructed.get_session_for_gm. Documented indocs/API.md.t_start/t_end, actors, kind, evidence,ok,problemsandrepairs(a repaired beat passed, and a silent correction would otherwise be indistinguishable from a correct one).extraction_passes > 1.Verified on a fresh database: the full chain applies, one head (
0b1c2d3e4f5a), both tables with cascading FKs and the partial index on problems; 45 tests pass acrosstest_summarisation_runs.pyandtest_capability_gaps.py.Two notes for the remaining work: the service tests drive
session_beat_servicedirectly and there is no HTTP-level 403 test for these endpoints (a codebase-wide gap, not this issue's); and the first acceptance criterion is met, so the four that remain are all screen work, which lands in the session lane of the v4.4.0 rebuild.A case for letting the GM correct an event and regenerate the summary, not just see the events.
On the dev replay (Gemma 4 26B, run
a0784433, 2026-09-11), the summary says the journal "revealed that the party was actually inside Ironroot Hold". That's wrong, and no model, prompt or detector could be expected to catch it, because it's what the GM said:Thirty seconds later: "the point that you think you entered at next to Ironroot Hold and this point that you are at now, they are a considerable distance apart."
The GM misspoke, saying "you're in" for "you were in". The extracted event, "The GM reveals the party is in Ironroot Hold" (00:49:28), verifies correctly against the transcript. #584's check doesn't flag it either: it isn't a short reply to a player's question.
It has survived every model and pipeline change since the first Qwen run: Qwen 9B, Gemma before #587, Gemma after #587, and now Gemma after #591. The robust fix is the GM: when the verified events sit beside the summary, let the GM strike or rewrite one event ("The map marks where they entered the caves, just outside Ironroot Hold, a long way from here") and recompose the summary from the corrected list.
That costs one compose call, not a re-extraction, and it's exactly the correction the event log makes possible. It also gives the eval work in #586 a source of labelled false events. Worth designing into the Review screen that v4.4.0 is mocking up now.
Picking up the correct-and-recompose half (2026-09-11, owner's choice), ahead of the v4.4.0 rebuild. The API is the part that lasts. The UI goes into today's event log on the session page, and the rebuild's Review screen will reuse the API.
Design decisions, so the screen doesn't make them by accident:
recomposed_from_run_idand who corrected it. The new run runs only the compose step, one model call and no re-extraction, using the source run's own speaker legend.The motivating case is Ironroot Hold: rewrite that one event, regenerate, and the error is gone.
Correct and regenerate is merged (PR #602, main
8f26886) and deployed to dev (migration9e0f1a2b3c4d). All three new routes answer 401 unauthenticated, and the foursummary_recompose_*columns are present.The design as built (details in the PR):
compose_from_validationsprocess_audio uses.For the owner to try on dev: rewrite the Ironroot Hold event (00:49:28) on the replay session and regenerate.
One known limitation: restore doesn't keep hand edits made after a regenerate. The confirmation says so.
Still open: the rest of this issue's original criteria (showing the verified events beside the summary on the Review screen) belongs to the v4.4.0 rebuild, which will reuse these endpoints. Leaving the issue open for that.
Checked against main (
2b077778), 2 October: built in lane 4, criterion by criterion.summarisation_runsandsession_beats(v4.0.0, per the earlier comment here).webapp/frontend/src/pages/session/ReviewEvents.jsx, "The events it was built from" on the Review page; each row is time, who ("Unclear" when none), what. Data fromGET /api/sessions/{id}/event-logandsummarisation-runs/latest.ReviewEvents.jsx:227), and the header counts them.onCite, D104).Covered by
ReviewView.test.jsx"lists each event — time, who, what — with the one that did not check out, and folds the table notes" and "cites the transcript by its time, and opens it at that line (D104)"; backendtests/test_event_log.py.Beyond the issue: strike, reword, relabel and emphasise. Not built, and not in the acceptance list: comparing two runs side by side. Closing; that would be its own issue if wanted.