rbrooks/Quest-Board
Watch
1
0
Fork
You've already forked Quest-Board
 0
Code Issues 77 Pull requests 2 Projects Releases 42 Packages Wiki Activity Actions

[Frontend] Show a GM the verified events behind a summary #424

New issue
Open
opened 2026-08-27 21:45:04 +00:00 by claude-bot · 5 comments
claude-bot commented 2026-08-27 21:45:04 +00:00
Contributor
Copy link

The last open criterion from #423, reframed. That issue closed with the coverage floor raised from 0.091 to 0.727, but one criterion unmet: "a run that covers drastically fewer events than expected is visible to the GM rather than silent."

Why this is a UI issue and not a detector

I tried twice in #423 to detect a lossy run in code, comparing the composed summary against the beats it was built from. Both failed on real data, in opposite directions:

  • Any two content words in common — too weak. A session's beats share heavily specific vocabulary (mimic, amphora, amulet), so a dropped event's words appear anyway because the beat next to it got written up.
  • Words unique within the session — too strict. Most beats have no word of their own, so they are skipped unchecked.

Across six live runs the detector flagged nothing while the eval harness measured 1–2 verified events per run missing from the prose. The harness only manages it because its fixtures carry hand-authored per-event mention groups; production has only the beat's own summary text, which is a much weaker signal. beat_service.unrepresented_beats still exists and is correct in clear-cut cases, but it is not a foundation to build a warning on.

The reframing: the GM does not need a machine to tell them something is missing. They were at the session. Show them the verified events beside the summary and they will see a gap in seconds — and, unlike a heuristic, they will also spot the summary that covers everything but characterises it wrongly.

What already exists

  • beat_service.validate_beats produces a BeatValidation per beat, carrying .ok and, when not ok, .problems naming why ('Kira' did not speak any cited line, cites 00:41:12, which is not in the transcript).
  • Every beat has t_start/t_end, actors, kind, and evidence — cited transcript timestamps, already validated against the transcript.
  • render_beats_for_compose renders them in time order.
  • Beats that fail validation are deliberately flagged, not dropped (#333), so the failures are available too and are often the interesting ones.

What does not exist, and is the real work

Beats are never persisted. They live in _summarise_from_beats for the duration of one Celery task and are gone. There is no model, no table, no API. Showing them means storing them, which is a migration plus a router, and is most of this issue.

Worth deciding at design time whether they are stored per summarisation run (so a reprocess produces a second set and the GM can compare) or last-write-wins. #423 established that two runs over the same audio genuinely differ, which makes the comparison view more interesting than it sounds.

Acceptance criteria

  • Verified events are persisted alongside the summary they produced
  • A GM can see them next to the summary, in time order, with who each is attributed to
  • Events that failed validation are shown too, with the reason, and visibly distinguished from verified ones
  • Each event can be traced to the transcript lines it cites
  • Nothing here blocks or delays the summary a GM sees today

Notes

  • Do not build this on unrepresented_beats. It is retained for the retry path, where being occasionally wrong is cheap; it is not accurate enough to drive anything a GM is shown.
  • Sits with #422 in this milestone: both are "the pipeline knows something the GM cannot see".
  • The numbers that motivate it are in webapp/backend/evals/results/history.jsonl and reproducible with python -m evals --fixtures evals/fixtures/private --provider live --repeat 4.
The last open criterion from #423, reframed. That issue closed with the coverage floor raised from 0.091 to 0.727, but one criterion unmet: *"a run that covers drastically fewer events than expected is visible to the GM rather than silent."* ## Why this is a UI issue and not a detector I tried twice in #423 to detect a lossy run in code, comparing the composed summary against the beats it was built from. Both failed on real data, in opposite directions: - **Any two content words in common** — too weak. A session's beats share heavily specific vocabulary (mimic, amphora, amulet), so a dropped event's words appear anyway because the beat next to it got written up. - **Words unique within the session** — too strict. Most beats have no word of their own, so they are skipped unchecked. Across six live runs the detector flagged nothing while the eval harness measured 1–2 verified events per run missing from the prose. The harness only manages it because its fixtures carry *hand-authored per-event mention groups*; production has only the beat's own summary text, which is a much weaker signal. `beat_service.unrepresented_beats` still exists and is correct in clear-cut cases, but it is not a foundation to build a warning on. **The reframing:** the GM does not need a machine to tell them something is missing. They were at the session. Show them the verified events beside the summary and they will see a gap in seconds — and, unlike a heuristic, they will also spot the summary that covers everything but characterises it wrongly. ## What already exists - `beat_service.validate_beats` produces a `BeatValidation` per beat, carrying `.ok` and, when not ok, `.problems` naming why (`'Kira' did not speak any cited line`, `cites 00:41:12, which is not in the transcript`). - Every beat has `t_start`/`t_end`, `actors`, `kind`, and `evidence` — cited transcript timestamps, already validated against the transcript. - `render_beats_for_compose` renders them in time order. - Beats that fail validation are deliberately **flagged, not dropped** (#333), so the failures are available too and are often the interesting ones. ## What does not exist, and is the real work **Beats are never persisted.** They live in `_summarise_from_beats` for the duration of one Celery task and are gone. There is no model, no table, no API. Showing them means storing them, which is a migration plus a router, and is most of this issue. Worth deciding at design time whether they are stored per summarisation *run* (so a reprocess produces a second set and the GM can compare) or last-write-wins. #423 established that two runs over the same audio genuinely differ, which makes the comparison view more interesting than it sounds. ## Acceptance criteria - [ ] Verified events are persisted alongside the summary they produced - [ ] A GM can see them next to the summary, in time order, with who each is attributed to - [ ] Events that failed validation are shown too, with the reason, and visibly distinguished from verified ones - [ ] Each event can be traced to the transcript lines it cites - [ ] Nothing here blocks or delays the summary a GM sees today ## Notes - Do **not** build this on `unrepresented_beats`. It is retained for the retry path, where being occasionally wrong is cheap; it is not accurate enough to drive anything a GM is shown. - Sits with #422 in this milestone: both are "the pipeline knows something the GM cannot see". - The numbers that motivate it are in `webapp/backend/evals/results/history.jsonl` and reproducible with `python -m evals --fixtures evals/fixtures/private --provider live --repeat 4`.
claude-bot commented 2026-09-06 19:48:11 +00:00
Author
Contributor
Copy link

Picking up the backend half now, in parallel with the v4.4.0 design phases (#535 phase 4); the screen that shows the events beside the summary comes in the session lane of the rebuild, once the design direction is chosen.

Decisions, so the screen does not decide them by accident:

  • Per summarisation run, not last-write-wins. #423 showed two runs over the same audio genuinely differ, and the comparison is the interesting view.
  • Never in the GM's way. The run is written after the summary is saved, inside the same task; a failure to persist is logged and the summary is unaffected.
  • GM-only read, on the same authorisation as transcript approval: GET /api/sessions/{id}/beats (latest run, or ?run_id=) and GET /api/sessions/{id}/beats/runs.
  • Nothing is built on unrepresented_beats, as the body says.

One migration, chained on the v4.3.0 head.

Picking up the **backend half** now, in parallel with the v4.4.0 design phases (#535 phase 4); the screen that shows the events beside the summary comes in the session lane of the rebuild, once the design direction is chosen. Decisions, so the screen does not decide them by accident: - **Per summarisation run**, not last-write-wins. #423 showed two runs over the same audio genuinely differ, and the comparison is the interesting view. - **Never in the GM's way.** The run is written after the summary is saved, inside the same task; a failure to persist is logged and the summary is unaffected. - **GM-only read**, on the same authorisation as transcript approval: `GET /api/sessions/{id}/beats` (latest run, or `?run_id=`) and `GET /api/sessions/{id}/beats/runs`. - Nothing is built on `unrepresented_beats`, as the body says. One migration, chained on the v4.3.0 head.
claude-bot commented 2026-09-06 19:53:16 +00:00
Author
Contributor
Copy link

Correction: the backend half already shipped in v4.0.0. The body's premise ("beats are never persisted") was true when it was filed and is not true now. Two commits landed it: 2791c24 (the rows: summarisation_runs + session_beats, migration d5e6f7a8b0c1, extended by f7a8b0c1d2e3 for provenance) and aa4bff9 ("expose the events behind a summary over the API (#333, #424)"), whose message already says this issue becomes a frontend-only job.

What exists on main today, for whoever builds the screen:

  • GET /api/sessions/{id}/summarisation-runs — every run, newest first, each with its beats inline (so an older run is selected from this list; there is no ?run_id=).
  • GET /api/sessions/{id}/summarisation-runs/latest — 404 when the session has no run; the list returns [] instead, deliberately, because sessions summarised before v4.0.0 have no record and none can be reconstructed.
  • Both GM-only via get_session_for_gm. Documented in docs/API.md.
  • Each run records the speaker legend as used, extraction passes, used beats, unsupported sentences, schema degradation, capability gaps and truncation; each beat carries t_start/t_end, actors, kind, evidence, ok, problems and repairs (a repaired beat passed, and a silent correction would otherwise be indistinguishable from a correct one).
  • The write path uses a savepoint, not a bare try/except: a failed flush would otherwise poison the session and make the later commit of the transcript and summary raise, which is the inversion this issue exists to prevent.
  • Runs are appended, never replaced; an adaptive multi-pass retry (#423) yields one run with extraction_passes > 1.

Verified on a fresh database: the full chain applies, one head (0b1c2d3e4f5a), both tables with cascading FKs and the partial index on problems; 45 tests pass across test_summarisation_runs.py and test_capability_gaps.py.

Two notes for the remaining work: the service tests drive session_beat_service directly and there is no HTTP-level 403 test for these endpoints (a codebase-wide gap, not this issue's); and the first acceptance criterion is met, so the four that remain are all screen work, which lands in the session lane of the v4.4.0 rebuild.

**Correction: the backend half already shipped in v4.0.0.** The body's premise ("beats are never persisted") was true when it was filed and is not true now. Two commits landed it: `2791c24` (the rows: `summarisation_runs` + `session_beats`, migration `d5e6f7a8b0c1`, extended by `f7a8b0c1d2e3` for provenance) and `aa4bff9` ("expose the events behind a summary over the API (#333, #424)"), whose message already says this issue becomes a frontend-only job. What exists on main today, for whoever builds the screen: - `GET /api/sessions/{id}/summarisation-runs` — every run, newest first, each with its beats inline (so an older run is selected from this list; there is no `?run_id=`). - `GET /api/sessions/{id}/summarisation-runs/latest` — 404 when the session has no run; the list returns `[]` instead, deliberately, because sessions summarised before v4.0.0 have no record and none can be reconstructed. - Both GM-only via `get_session_for_gm`. Documented in `docs/API.md`. - Each run records the speaker legend as used, extraction passes, used beats, unsupported sentences, schema degradation, capability gaps and truncation; each beat carries `t_start`/`t_end`, actors, kind, evidence, `ok`, `problems` **and `repairs`** (a repaired beat passed, and a silent correction would otherwise be indistinguishable from a correct one). - The write path uses a savepoint, not a bare try/except: a failed flush would otherwise poison the session and make the *later* commit of the transcript and summary raise, which is the inversion this issue exists to prevent. - Runs are appended, never replaced; an adaptive multi-pass retry (#423) yields one run with `extraction_passes > 1`. Verified on a fresh database: the full chain applies, one head (`0b1c2d3e4f5a`), both tables with cascading FKs and the partial index on problems; 45 tests pass across `test_summarisation_runs.py` and `test_capability_gaps.py`. Two notes for the remaining work: the service tests drive `session_beat_service` directly and there is no HTTP-level 403 test for these endpoints (a codebase-wide gap, not this issue's); and the first acceptance criterion is met, so the four that remain are all screen work, which lands in the session lane of the v4.4.0 rebuild.
claude-bot commented 2026-09-11 03:32:18 +00:00
Author
Contributor
Copy link

A case for letting the GM correct an event and regenerate the summary, not just see the events.

On the dev replay (Gemma 4 26B, run a0784433, 2026-09-11), the summary says the journal "revealed that the party was actually inside Ironroot Hold". That's wrong, and no model, prompt or detector could be expected to catch it, because it's what the GM said:

GM: Where were you? · Harrowhark: You're in Bobsville, right? · GM: No, you were not in Bobsville · Viq: New Town. · GM: You're in Ironroot Hold. … so there's like a point that's marked a little bit outside of it … that is probably the point that you entered the cave system.

Thirty seconds later: "the point that you think you entered at next to Ironroot Hold and this point that you are at now, they are a considerable distance apart."

The GM misspoke, saying "you're in" for "you were in". The extracted event, "The GM reveals the party is in Ironroot Hold" (00:49:28), verifies correctly against the transcript. #584's check doesn't flag it either: it isn't a short reply to a player's question.

It has survived every model and pipeline change since the first Qwen run: Qwen 9B, Gemma before #587, Gemma after #587, and now Gemma after #591. The robust fix is the GM: when the verified events sit beside the summary, let the GM strike or rewrite one event ("The map marks where they entered the caves, just outside Ironroot Hold, a long way from here") and recompose the summary from the corrected list.

That costs one compose call, not a re-extraction, and it's exactly the correction the event log makes possible. It also gives the eval work in #586 a source of labelled false events. Worth designing into the Review screen that v4.4.0 is mocking up now.

**A case for letting the GM correct an event and regenerate the summary, not just see the events.** On the dev replay (Gemma 4 26B, run `a0784433`, 2026-09-11), the summary says the journal "revealed that the party was actually inside Ironroot Hold". That's wrong, and **no model, prompt or detector could be expected to catch it**, because it's what the GM said: > GM: Where were you? · Harrowhark: You're in Bobsville, right? · GM: No, you were not in Bobsville · Viq: New Town. · **GM: You're in Ironroot Hold.** … so there's like a point that's marked a little bit outside of it … that is probably the point that you entered the cave system. Thirty seconds later: *"the point that you think you entered at next to Ironroot Hold and this point that you are at now, they are a considerable distance apart."* The GM misspoke, saying "you're in" for "you were in". The extracted event, "The GM reveals the party is in Ironroot Hold" (00:49:28), verifies correctly against the transcript. #584's check doesn't flag it either: it isn't a short reply to a player's question. It has survived every model and pipeline change since the first Qwen run: Qwen 9B, Gemma before #587, Gemma after #587, and now Gemma after #591. **The robust fix is the GM:** when the verified events sit beside the summary, let the GM strike or rewrite one event ("The map marks where they entered the caves, just outside Ironroot Hold, a long way from here") and **recompose the summary from the corrected list**. That costs one compose call, not a re-extraction, and it's exactly the correction the event log makes possible. It also gives the eval work in #586 a source of labelled false events. Worth designing into the Review screen that v4.4.0 is mocking up now.
claude-bot commented 2026-09-11 13:40:31 +00:00
Author
Contributor
Copy link

Picking up the correct-and-recompose half (2026-09-11, owner's choice), ahead of the v4.4.0 rebuild. The API is the part that lasts. The UI goes into today's event log on the session page, and the rebuild's Review screen will reuse the API.

Design decisions, so the screen doesn't make them by accident:

  • Corrections never rewrite an existing run. Runs are append-only.
    • The GM strikes an event (keep it out of the summary) or rewrites its text.
    • Regenerate summary creates a new summarisation run: a copy of the source run's events with the corrections applied, each correction keeping the original text, plus recomposed_from_run_id and who corrected it. The new run runs only the compose step, one model call and no re-extraction, using the source run's own speaker legend.
    • The source run stays exactly as the model produced it.
  • The summary it replaces is kept on the new run, and a GM can restore it. #401 is the lesson here: nothing a GM wrote disappears without a way back.
  • Regenerating is an explicit GM action, with a confirmation that says it replaces the current summary, including hand edits.
    • It follows the existing approval rule: an approved summary is withdrawn first (#416's path), then corrected and re-approved.
    • It's GM-only, on the same authorisation as the runs endpoints.
  • Asynchronous, following the closest existing precedent for a GM-triggered LLM job: 202 plus a status the page can poll, with the failure shown and not swallowed.
  • Struck events drop out of the prose and show struck through to the GM. Anything players see shows the corrected text, never the struck event.
  • Every correction is recorded with its original text. That's also the labelled false-event data #586 has been missing.

The motivating case is Ironroot Hold: rewrite that one event, regenerate, and the error is gone.

**Picking up the correct-and-recompose half** (2026-09-11, owner's choice), ahead of the v4.4.0 rebuild. The API is the part that lasts. The UI goes into today's event log on the session page, and the rebuild's Review screen will reuse the API. Design decisions, so the screen doesn't make them by accident: - **Corrections never rewrite an existing run.** Runs are append-only. - The GM **strikes** an event (keep it out of the summary) or **rewrites** its text. - *Regenerate summary* creates a **new** summarisation run: a copy of the source run's events with the corrections applied, each correction keeping the original text, plus `recomposed_from_run_id` and who corrected it. The new run runs **only the compose step**, one model call and no re-extraction, using the source run's own speaker legend. - The source run stays exactly as the model produced it. - **The summary it replaces is kept on the new run,** and a GM can restore it. #401 is the lesson here: nothing a GM wrote disappears without a way back. - **Regenerating is an explicit GM action,** with a confirmation that says it replaces the current summary, including hand edits. - It follows the existing approval rule: an approved summary is withdrawn first (#416's path), then corrected and re-approved. - It's GM-only, on the same authorisation as the runs endpoints. - **Asynchronous**, following the closest existing precedent for a GM-triggered LLM job: 202 plus a status the page can poll, with the failure shown and not swallowed. - **Struck events** drop out of the prose and show struck through to the GM. Anything players see shows the corrected text, never the struck event. - **Every correction is recorded with its original text.** That's also the labelled false-event data #586 has been missing. The motivating case is Ironroot Hold: rewrite that one event, regenerate, and the error is gone.
claude-bot commented 2026-09-11 15:20:36 +00:00
Author
Contributor
Copy link

Correct and regenerate is merged (PR #602, main 8f26886) and deployed to dev (migration 9e0f1a2b3c4d). All three new routes answer 401 unauthenticated, and the four summary_recompose_* columns are present.

The design as built (details in the PR):

  • Corrections: the GM strikes or rewrites events in the session page's event log. Regenerate summary then writes a new run, a copy of the source run's beats with the corrections applied, each correction keeping its original text. Only the compose step runs, through the same compose_from_validations process_audio uses.
  • Restore: the replaced summary is kept on the new run, and "Restore the summary from before" puts it back.
  • Visibility: players see corrected text and never a struck event. The GM sees both, with the original wording.
  • Approval: this follows the existing rule. Editing an approved summary is allowed; the confirmation says the Discord copy won't change and points to Withdraw approval.
  • Async: it follows the lore-run pattern, with status on the session, a GM-only status endpoint, a 30-minute stale limit, and superseded results discarded.

For the owner to try on dev: rewrite the Ironroot Hold event (00:49:28) on the replay session and regenerate.

One known limitation: restore doesn't keep hand edits made after a regenerate. The confirmation says so.

Still open: the rest of this issue's original criteria (showing the verified events beside the summary on the Review screen) belongs to the v4.4.0 rebuild, which will reuse these endpoints. Leaving the issue open for that.

**Correct and regenerate is merged (PR #602, main `8f26886`) and deployed to dev** (migration `9e0f1a2b3c4d`). All three new routes answer 401 unauthenticated, and the four `summary_recompose_*` columns are present. **The design as built** (details in the PR): - **Corrections:** the GM strikes or rewrites events in the session page's event log. *Regenerate summary* then writes a **new** run, a copy of the source run's beats with the corrections applied, each correction keeping its original text. Only the compose step runs, through the same `compose_from_validations` process_audio uses. - **Restore:** the replaced summary is kept on the new run, and "Restore the summary from before" puts it back. - **Visibility:** players see corrected text and never a struck event. The GM sees both, with the original wording. - **Approval:** this follows the existing rule. Editing an approved summary is allowed; the confirmation says the Discord copy won't change and points to Withdraw approval. - **Async:** it follows the lore-run pattern, with status on the session, a GM-only status endpoint, a 30-minute stale limit, and superseded results discarded. **For the owner to try on dev:** rewrite the Ironroot Hold event (00:49:28) on the replay session and regenerate. **One known limitation:** restore doesn't keep hand edits made *after* a regenerate. The confirmation says so. **Still open:** the rest of this issue's original criteria (showing the verified events beside the summary on the Review screen) belongs to the v4.4.0 rebuild, which will reuse these endpoints. Leaving the issue open for that.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
ui/table-notes-label
renovate/npm-vitest-vulnerability
chore/87-align-python-3.12
v4.3.0
v4.2.3
v4.2.2
v4.2.1
v4.2.0
v4.1.2
v4.1.1
v4.1.0
v4.0.1
v4.0.0
v3.11.4
v3.11.3
v3.11.2
v3.11.1
v3.11.0
v3.10.0
v3.9.1
v3.9.0
v3.8.0
v3.7.0
v3.6.1
v3.6.0
v3.5.0
v3.4.1
v3.4.0
v3.3.0
v3.2.0
v3.1.0
v3.0.0
v2.1.0
v2.0.0
v1.1.0
v1.0.0
v0.12.0
v0.11.0
v0.10.0
v0.9.1
v0.3.3
v0.9.0
v0.8.0
v0.3.2
v0.3.1
v0.3.0
v0.7.0
v0.6.0
v0.5.0
v0.4.0
v0.2.1
v0.2.0
v0.1.1
v0.1.0
Labels
Clear labels   
backend
Issue related to the backend

  
bot
Issue related to the bot

  
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
frontend
Issue related to the frontend

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
recording
Issue related to the recording system

  
security
Security hardening / vulnerability work

  
wontfix
This won't be fixed

No labels
backend
bot
bug
duplicate
enhancement
frontend
help wanted
invalid
question
recording
security
wontfix
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rbrooks/Quest-Board#424
No description provided.